Parallel Processing

Parallel Processing is a way to improve performance on high-end machines, by running multiple actions at once as a set of separate processes. A system architecture of multiple processors with multiple cores is useful because each process can be assigned to a different core. With multiple processes running on multiple cores, the whole translation can run several times quicker than it used to.

Parallel Processing and FME

Parallel processing in FME covers not an entire workspace, but just a specific subsection of workspace. This is defined using a custom transformer, which is a natural method to subdivide a large translation.

Parallel processing is achieved by forming groups of features on the basis of attribute values, in a similar way to a Group-Based transformer. Each group is assigned to a separate process.

Activating Parallel Processing

Each custom transformer has a set of parameters - located in the Navigator window - that relates explicitly to parallel processing. Here you can determine the level of parallel processing, and an attribute that defines groups of data that will be transformed as a separate process:

Parallel Processing Levels

The parameter that controls parallel processing allows different levels of effort to be applied. Each level specifies the number of processes to run simultaneously, in relation to the number of cores available:

As in the above example, on a quad-core machine, minimal parallelism results in two simultaneous FME processes. Extreme parallelism would result in eight (assuming there are eight tasks that can be processed simultaneously).

There is also a hard cap for each FME license level:

So, if you have a Base Edition license, you are never going to get more than four processes at one time, regardless of machine type and the parallelism parameter. The quad-core machine in the above example can never have more than eight processes since that is the maximum that "extreme" parallel processing allows.

Parallel Processing Groups

The Parallel Process By parameter requires the author to select an attribute, and - as already covered - selecting attributes for a custom transformer parameter requires some consideration.

In particular, this parameter is designed so that you cannot simply pick an attribute to use:

Instead, this parameter (and one for Group By Mode) is published to make it available as one of the custom transformer parameters:

Parallel Processing and Custom Transformer Design

The keys to parallel processing are designing a custom transformer to be a standalone subsection, and the definition of groups to be processed.

In essence, features are split into groups on reaching the custom transformer, processed as a separate action, then combined back into a single group on exiting the custom transformer.

It's important to be aware that because each group is processed independently, there can be no relationship between features in different groups. If features are related, and their results depend on each other, then they must be in the same group.

The easiest way to think of this is that the Custom Transformer itself is a Group-Based transformer. Because it is a wrapper for all the individual FME transformers in the definition, it allows those transformers to operate in groups, even if they are Feature-Based.

Creating Artificial Groups

Sometimes none of the incoming data is related to each other, and can be split into arbitrary groups for processing. In these cases, where there is no identifier to define groups, one can be created manually by generating attributes with the ModuloCounter or RandomNumberGenerator transformers.

For example, here the workspace author has a large number of address features from which to create labels. To speed the process they activate parallel processing:

They could group by PostalCode, but choose not to because all addresses fall inside only one of two postal codes. Therefore they create an artificial group using the ModuloCounter. Note the Group By parameter in the custom transformer is set to the _modulo_count attribute.

In such a scenario, it's best to create only one group per intended FME process. For example, on a quad-core (four core) computer, with an FME Professional license and Extreme parallel processing, a maximum of eight processes at a time are permitted. Therefore - as shown here - the optimum number of groups to use is also eight.