Description:

The Filter with Conditions step evaluates each incoming record against a configurable list of filter conditions. Records that meet the configured conditions remain in the ETL pipeline, while those that do not are excluded from further processing.

Unlike manually written filter expressions, this step provides a structured, configuration-driven approach for defining filtering rules. ETL designers configure filter operations using the Configure Filter List property, allowing business rules to be defined through the designer interface rather than writing expression syntax.

The Filter with Conditions step is commonly used immediately after data extraction or validation to reduce the dataset prior to additional transformations, such as grouping, aggregation, sorting, or exporting.

The step supports:

• Rule-based filtering
• Configurable filter operations
• Multi-condition filtering
• Column-based comparisons
• Business rule enforcement
• Enterprise ETL workflows
• Integration with FlowWright ETL pipelines

This step can be used for:

• Filtering active customer records
• Selecting transactions within a value range
• Processing specific departments or regions
• Removing incomplete records
• Preparing regulatory reporting datasets
• Business intelligence data preparation
• Enterprise integration workflows

Usage:

The Filter with Conditions step is typically placed immediately after importing and validating data so that downstream ETL components process only the records that satisfy the configured business rules.

During execution, every incoming record is evaluated against the configured filter list. Records that satisfy all configured conditions remain in the dataset, while non-matching records are excluded.

A typical ETL pipeline might look like this:

Typical filtering scenarios include:

• Selecting customers whose account status is Active
• Processing only approved purchase orders
• Filtering employees by department
• Selecting transactions above a specified amount
• Processing records within a date range
• Removing archived or inactive records
• Preparing data for reporting or analytics

Filtering records early in the ETL pipeline improves downstream processing efficiency by reducing the volume of data that subsequent transformations process.

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

Example:

Let’s build and execute the “clsFilterTransformConditionsDef” example.          

• Create a new ETL definition named “clsFilterTransformConditionsDef” and open it in designer mode. 
• Drag “clsInputCSV, clsFilterTransformConditions, and clsOutputCSV” steps to the canvas.
• Connect the dots between the “clsInputCSV” step and other steps, as shown above.
• Define a variable or a global to store the file path. 
• Click the "clsInputCSV" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.

• Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as shown below. This function helps to understand the schema at a glance.  

• Click the "clsFilterTransformConditions" step to configure its "Required" properties. Provide a name for the step. Click the button to configure the columns to transform. Then click the Save button. 

• Click the "clsFilterTransformConditions" step to configure its "Required" properties. Click the button to configure the transform list. A pop-up window appears for configuration. Click the Add Row (+) button to insert an empty row. Select the column from the drop-down list. Select the filter condition operator from the drop-down list. Enter the filter value in the text box. Select the “and/or” condition. You may add multiple columns for transformation using the Add Row button. 

• Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button. 

• Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step. Provide a variable or global reference to store the virtual output path. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Save the ETL definition, create a new ETL instance, and execute. Render the ETL instance. Click the ETL step to view its properties. The “clsFilterTransformConditions” step should filter the column values according to the output file configuration. Only records satisfying all configured filter conditions continue to downstream ETL components. The variable or global holds the output file’s virtual path as shown below. 

Tips:

• Place this transform near the beginning of the ETL pipeline to reduce the volume of data processed by later transformations.
• Use descriptive and well-organized filter conditions to simplify maintenance.
• Validate source data before applying business filters.
• Review configured conditions whenever source schemas or business rules change.
• Test filtering logic using representative datasets before deploying to production.
• Document complex filtering requirements within the ETL definition.
• Combine this transform with Validate Data, Sorting Transform, Group By Transform, Pivot Transform, Aggregate Transform, Output Column Transform, JSON Output File, or SQL Output components to build complete ETL solutions.

Notes:

• Every incoming record is evaluated independently against the configured filter operations.
• Only matching records remain in the dataset for downstream processing.
• Multiple filter conditions can be configured within a single transformation.
• The referenced columns must exist within the ETL schema.
• Invalid filter configurations or references to missing columns may result in ETL execution errors.
• This transformation performs filtering only and does not modify the remaining records.
• The Filter with Conditions step is commonly used for business rule enforcement, operational reporting, customer segmentation, compliance reporting, data preparation, and enterprise integration workflows.
• Review filter configurations periodically to ensure they remain aligned with current business requirements.

Filter with Conditions vs. Filter with Expressions:

Although both steps filter records from an ETL dataset, they are designed for different configuration styles:

In general:

• Use Filter with Conditions when business rules can be expressed using standard comparison operators configured through the designer.
• Use Filter with Expressions when filtering requires calculated values, nested logical conditions, custom functions, or advanced expression syntax.

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

• Filter condition configuration
• ETL schema validation
• Source column verification
• Downstream transformation mappings
• Destination component configuration
• Environment-specific settings

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.