Description:

The Run ETL Job step enables a FlowWright workflow to execute a predefined ETL definition as part of an automated business process.

This step is commonly used when workflows need to initiate data import, transformation, migration, synchronization, or export operations without requiring manual intervention. By embedding ETL execution within a workflow, organizations can automate end-to-end business processes that combine workflow orchestration with enterprise data processing.

The step allows selected workflow variables and global variables to be passed into the ETL definition, enabling ETL jobs to receive runtime information such as file locations, customer identifiers, processing dates, or configuration settings.

For monitoring and auditing purposes, the ETL execution instance identifier can optionally be stored in a workflow variable or global variable. This identifier can later be used with steps such as Get ETL Job Status to monitor execution progress.

Typical uses include:

• Importing customer data
• Migrating business records
• Synchronizing ERP systems
• Processing scheduled batch files
• Exporting reports
• Data warehouse loading
• Automated integration workflows

The Run ETL Job step starts the ETL process and returns control to the workflow, allowing subsequent workflow logic to continue immediately or monitor the ETL execution separately.

Usage:

The Run ETL Job step is typically used whenever workflow execution needs to initiate an ETL process.

During execution:

1. Select the ETL definition to execute.
2. Pass any required workflow variables or global variables.
3. Start the ETL job.
4. Store the ETL instance ID if configured.
5. Continue workflow execution through either the True or False return path.

A typical workflow might look like this:

Typical workflow scenarios include:

• Processing uploaded CSV files
• Executing nightly data synchronization
• Importing customer information
• Loading financial transactions
• Synchronizing inventory data
• Executing scheduled integration jobs
• Launching data migration processes

Example:

Let’s build and execute the “clsExecuteETLJobDef” example.          

• Create a new process definition named “clsExecuteETLJobDef” and open it in designer mode.
• Drag "UpdateVariable, UpdateGlobal, and clsExecuteETLJob” steps to the canvas.
• Connect the dots between the “Start” and other steps, as shown above.
• Define a variable or a global to store the result.
• Click the "updateVariable" step to configure its "Required" properties. Provide a name for the step. Click the Save button. 
• Click the "updateVariable" step to configure its "Optional" properties. Click the button to initialize multiple variables required for the process, or provide the reference and value for a single variable. Select the Expression as “No” if the variable value does not contain an expression to evaluate. Click the Save button to confirm the configuration. In the example below, the variable value stores the input file path required for the ETL definition. Click the Save button. 

• Click the "updateGlobal" step to configure its "Required" properties. Enter a name for the step. Then click the Save button. 
• Click the "updateGlobal" step to configure its "Optional" properties. Click the button to configure multiple global references. Click the Add Row(+) button to insert an empty row. Enter the global name and value. Click the Save button. You may insert multiple global references by adding more rows. Select “Has Expression” as “No” if the global value does not contain a C# expression to evaluate. Click the Save button. 

• Click the "clsExecuteETLJob" step to configure its "Required" properties. Provide a name for the step. Select the ETL definition to execute from the drop-down list. Then click the Save button.

• Click the "clsExecuteETLJob" step to configure its "Optional" properties. Click the button to configure multiple variables to pass to the ETL Definition during execution. Click the Add Row (+) button to insert an empty row. Select a pre-configured variable from the drop-down list. Click the Save button. You may insert multiple variable references by adding new rows. Provide a variable or a global reference to store the ETL Instance ID value. Click the Save button. 

• Click the button to configure multiple globals to pass to the ETL Definition during execution. Click the Add Row (+) button to insert an empty row. Select a pre-configured global from the drop-down list. Click the Save button. You may add multiple global references by adding new rows. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Navigate to the ETL - Definition page and select the definition referenced above from the table. Click the Actions - Design menu option. The design definition is displayed on the new page for modification. Select the CSV Input File step. On the Required tab, enter the variable reference passed from the process definition for the input file path. Click the Save button.

• Select the CSV Output File step. On the Required tab, enter the global reference from the process definition for the output file path. Click the Save button. 

• Navigate to the Process Definition page. Save the process definition, create a new instance, and execute it. Render the process instance. The process step has executed the ETL job, as shown below. 

• Navigate to the ETL - Instances page. Render the instance shown above. The ETL process step has been executed using the variable and global references passed from the Process instance, as shown below.  

Tips:

• Design ETL definitions to accept workflow variables whenever dynamic runtime values are required.
• Pass only the variables and globals needed by the ETL definition.
• Store the ETL instance ID whenever subsequent workflow steps need to monitor execution progress.
• Combine this step with Get ETL Job Status when workflows depend on ETL completion.
• Route the False path to logging, notification, or retry logic.
• Use descriptive ETL definition names to simplify workflow maintenance.
• Combine this step with Upload File, CSV Input File, Get ETL Job Status, Send Email, Create Process Instance, or Decision steps to build complete automated integration workflows.

Notes:

• A valid ETL definition must exist before the workflow can execute this step.
• Passing workflow variables and global variables is optional and depends on the ETL definition requirements.
• The ETL instance ID can be stored for monitoring, reporting, auditing, or troubleshooting purposes.
• The True return path indicates that the ETL job was successfully initiated, not necessarily that processing has completed.
• To determine whether an ETL job has completed successfully, use the Get ETL Job Status step with the stored ETL instance ID.
• Execution details and ETL startup information are recorded in the FlowWright workflow execution log.

Run ETL Job vs. Run ETL Validation Job:

Both workflow steps execute FlowWright ETL definitions, but they are designed for different purposes.

As a general guideline:

• Use Run ETL Job when workflows need to execute ETL processes that import, transform, migrate, synchronize, or export data.
• Use Run ETL Validation Job when workflows need to validate incoming data before continuing with business processing.

Definition Sample:

You may download the sample workflow definition from the link provided and import it into your FlowWright environment.

Note: Verify and complete any missing configuration after importing the sample, including:

• ETL definition
• Workflow variables
• Global variables
• ETL instance ID storage variable
• Success and error handling
• Optional ETL monitoring using Get ETL Job Status

After verifying the configuration, save and publish the workflow before execution.

Click here to download the Process sample file.

Click here to download the ETL Process sample file.