clsRemoveDuplicatesTransform Step

Use this feature to identify and remove duplicate records from an ETL dataset based on one or more selected columns before passing the resulting data to downstream transformations or destination components.

Last published at: July 20th, 2026

Description:

The Remove Duplicates step eliminates duplicate records from an ETL dataset by comparing one or more selected columns. This transformation helps ensure that only unique records continue through the ETL pipeline, improving data quality and preventing redundant information from being loaded into destination systems.

The duplicate comparison criteria are configured through the Select Columns property. The selected columns collectively determine whether two records are considered duplicates. Records with identical values across the configured columns are treated as duplicates, while the first occurrence is retained for downstream processing.

The Remove Duplicates step is typically used after importing and standardizing data, but before aggregation, reporting, or loading records into databases, data warehouses, or external applications.

The step supports:

  • Duplicate record detection
  • Multi-column comparison
  • Configurable duplicate criteria
  • Data quality improvement
  • Dataset normalization
  • Enterprise ETL workflows
  • Integration with FlowWright ETL pipelines

This step can be used for:

  • Customer master data cleansing
  • Removing duplicate contacts
  • Data migration projects
  • Preparing reporting datasets
  • Preventing duplicate database records
  • Master data management
  • Enterprise data integration

 

Inputs

  • Select Columns – Configure one or more columns that are used to determine duplicate records. Records containing identical values across all selected columns are considered duplicates and are removed from the processed dataset.
 

 

Returns

  • The Remove Duplicates step does not define explicit return paths.

Note: After duplicate removal has completed successfully, ETL processing automatically continues to the next connected component. Any configuration or execution errors are recorded in the ETL execution log.

 

 

 

Usage:

The Remove Duplicates step is typically placed after importing and optionally transforming data, but before sorting, aggregation, reporting, or exporting the final dataset.

During execution, the transformation evaluates every record using the configured comparison columns. When duplicate records are detected, only the first matching record is retained while subsequent duplicates are removed from the ETL stream.

A typical ETL pipeline might look like this:

 

Typical duplicate removal scenarios include:

  • Eliminating duplicate customer records
  • Removing repeated invoice entries
  • Cleaning imported contact lists
  • Consolidating employee records
  • Preparing unique reporting datasets
  • Removing duplicate product catalog entries
  • Standardizing migration datasets before loading

The resulting dataset contains only unique records based on the configured comparison columns.

 

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

 

 

Example:

Let’s build and execute the “clsRemoveDuplicatesTransformDef” example.          

  • Create a new ETL definition named “clsRemoveDuplicatesTransformDef” and open it in designer mode. 
  • Drag the “clsInputCSV, clsRemoveDuplicatesTransform, and clsOutputCSV” steps to the canvas. 
  • Connect the dots between the “clsInputCSV” and other steps, as shown above.
  • Define a variable or a global variable to hold the virtual path.
  • Click the "clsInputCSV" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input CSV file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.

 

  • Click the "clsInputCSV" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as follows. This function helps to understand the schema at a glance.  

 

  • Click the "clsRemoveDuplicatesTransform" step to configure its "Required" properties. Provide a name for the step. Click the button to configure the columns to transform. Then click the Save button. 

 

  • Click the "clsRemoveDuplicatesTransform" step to configure its "Required" properties. Click the column button to open the transform list. A pop-up window will appear for configuration. Click the Add Row (+) button to insert an empty row. Select the column to be removed from the drop-down list. Click the Save button. You may add multiple columns for transformation by using the Add Row button.  

 

  • Click the "clsOutputCSV" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output CSV file on the application server. Click the Save button. 

 

  • Click the "clsOutputCSV" step to configure its "Optional" properties. Provide a name for the step and specify the variable or global reference to hold the virtual output path. Click the Save button. 

 

  • The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

 

  • Save the ETL Definition. Execute the ETL process. During execution, the transformation compares the configured columns for each record. Duplicate records are removed before the dataset is passed to downstream ETL components. The variable or global holds the output file’s virtual path as shown below. 

 

Tips:

  • Select only the columns that uniquely identify a record for duplicate detection.
  • Normalize or standardize data before removing duplicates to improve matching accuracy.
  • Perform duplicate removal before aggregation or loading records into destination systems.
  • Verify duplicate criteria carefully to avoid removing legitimate records.
  • Combine duplicate removal with validation and data-cleansing transforms to achieve higher data quality.
  • Test duplicate detection rules using representative sample datasets.
  • Combine this transform with Rename Column, Set Field Value, Validate Data, Sorting Transform, Aggregate Transform, or SQL Output components to create comprehensive ETL workflows.

 

Notes:

  • Duplicate detection is based solely on the configured comparison columns.
  • Multiple columns can be selected to define record uniqueness.
  • The transformation removes duplicate records but does not modify the remaining data values.
  • Source columns referenced for comparison must exist in the ETL schema.
  • Large datasets containing many duplicate comparisons may require additional processing time.
  • Invalid schema mappings or missing comparison columns may result in ETL execution errors.
  • This transform is commonly used during customer data consolidation, reporting, data migration, master data management, and enterprise integration projects.
  • Review the output dataset before loading it into production systems to ensure the duplicate removal criteria produce the expected results.

 

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

  • Duplicate comparison column selection
  • ETL schema mappings
  • Source data configuration
  • Downstream transformation connections
  • Destination mappings
  • Environment-specific settings

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.