Description:
The Multicast Data step creates multiple identical output streams from a single incoming ETL dataset. Rather than transforming or modifying the data, this utility step duplicates the entire dataset and forwards it to multiple downstream components for independent processing.
This transform is particularly useful when the same source data must be processed in different ways. For example, one branch may prepare data for loading into an SQL database, while another branch exports the same records to a CSV file or performs additional validation and reporting.
Unlike conditional branching, every downstream connection receives the complete dataset. No records are filtered or removed, and no schema changes are made during the multicast operation.
The Multicast Data step is commonly used immediately after data extraction or after initial cleansing and validation, enabling multiple ETL paths to execute in parallel while maintaining a single source of data.
The step supports:
- Dataset duplication
- Parallel ETL processing
- Multiple downstream branches
- Shared data pipelines
- Reusable data streams
- Enterprise ETL workflows
- Integration with FlowWright ETL pipelines
This step can be used for:
- Sending data to multiple destinations
- Simultaneous reporting and database loading
- Running multiple transformation pipelines
- Creating audit and archive copies
- Parallel business processing
- Data synchronization
- Enterprise integration workflows
Inputs
- This step does not require any configuration properties.
Note: The incoming dataset is automatically duplicated and forwarded to every connected output path.
Returns
- The Multicast Data step does not define explicit return paths.
Note: The duplicated datasets are automatically delivered to all connected downstream components. Any execution or connectivity issues are recorded in the ETL execution log.
Usage:
The Multicast Data step is typically inserted after importing or validating data when multiple independent processing paths require the same dataset.
During execution, the transformation creates identical copies of every incoming record and sends them to each connected downstream branch. Each branch processes the data independently without affecting the others.
A typical ETL pipeline might look like this:
Typical multicast scenarios include:
- Exporting identical datasets to multiple file formats
- Loading the same data into multiple databases
- Running independent reporting and analytics pipelines
- Creating backup or archive copies during processing
- Sending data to auditing and operational systems simultaneously
- Supporting multiple business processes from a single import
- Distributing master data across enterprise applications
Prerequisite:
The ETL process engine service should be running to execute the ETL definitions.
Example:
Let’s build and execute the “clsMulticastDef” example.
- Create a new ETL definition named “clsMulticastDef” and open it in designer mode.
- Drag the “clsInputCSV, clsMulticast, clsFilterTransform, clsTransformSubstring, clsOutputExcel, and clsOutputCSV” steps to the canvas.
- Connect the dots between the “clsInputCSV” step and other steps, as shown above.
- Define a variable or a global to hold the virtual path.
- Click the "clsInputCSV" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.
- Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as shown below. This function helps to understand the schema at a glance.
- Click the "clsMulticast" step to configure its "Required" properties. Provide a name for the step, then click the Save button.
- Click the "clsFilterTransform" step to configure its "Required" properties. Provide a name for the step and a filter condition. Click the Save button.
- Click to configure the "clsTransformSubstring" step's "Required" properties. Provide a name for the step. Click the button to configure the substring. Click the Save button.
- Click the "clsTransformSubstring" step to configure its "Required" properties. Click the button to set the substring criteria. The pop-up window opens for configuration. Click the Add Row (+) button to insert an empty row. Select the text data type column. Enter the start position and the number of characters to create a substring. Select the output text column from the drop-down list. You may also type a new Out column name and press Enter to add it. Click the Save button. You may add multiple columns for transformation using the Add Row button.
- Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button.
- Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step and specify the variable or global reference to hold the virtual output path. Click the Save button.
- Click the "clsOutputExcel" step to configure its "Required" properties. Provide a name for the step and the path to the output XLSX file on the application server. Then click the Save button.
- Click the "clsOutputExcel" step to configure its "Optional" properties. Provide a name for the step. Provide the variable or global reference for the virtual output path. Click the Save button.
- The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.
- Save the ETL definition, create a new ETL instance, and execute. Render the ETL instance. Click the ETL step to view its properties. The “clsMulticast” step should support multiple transformations as configured for the output file. During execution, every record received by the Multicast Data step is duplicated and sent to each downstream branch. Each branch processes the complete dataset independently, allowing different transformations or destinations to execute simultaneously without requiring additional source reads. The variable or global holds the output file’s virtual path as shown below.
Tips:
- Place the Multicast Data step after common validation and cleansing operations to avoid repeating the same processing in multiple branches.
- Use multicast rather than multiple input sources when the same dataset is required by multiple ETL paths.
- Keep downstream branches independent to simplify maintenance and troubleshooting.
- Ensure each branch performs only the transformations required for its specific business purpose.
- Monitor downstream processing times when distributing very large datasets to multiple destinations.
- Document the purpose of each multicast branch to improve ETL maintainability.
- Combine this transform with Validate Data, Rename Column, Set Field Value, Pivot Transform, Aggregate Transform, CSV Output File, JSON Output File, and SQL Output components to build scalable ETL solutions.
Notes:
- The Multicast Data step does not modify the dataset or schema.
- Every connected downstream branch receives the complete input dataset.
- Processing performed in one branch does not affect the data processed by other branches.
- Additional downstream branches can be added as required by the ETL design.
- Large datasets distributed across multiple branches may increase memory usage and overall execution time, depending on the complexity of downstream processing.
- This transform does not perform filtering, aggregation, or validation; it simply duplicates the incoming data stream.
- The Multicast Data step is commonly used in reporting, data warehousing, auditing, backup, synchronization, migration, and enterprise integration scenarios.
- Review downstream branch configurations to ensure each destination processes the duplicated data appropriately.
Definition Sample:
You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.
Note: Verify and complete any missing configuration after importing the sample, including:
- Input source configuration
- Downstream branch connections
- Destination component configuration
- Transformation logic for each branch
- ETL schema validation
- Environment-specific settings
After verifying the configuration, save the ETL Definition before execution.