Description:

The JSON Input File step serves as a source component in a FlowWright ETL definition by importing structured data from a JSON file.

This step reads the specified JSON document, interprets the data using a selected column schema, and exposes the resulting dataset to downstream ETL transformations. It provides a simple and efficient way to import JSON data generated by REST APIs, business applications, IoT devices, cloud services, exported reports, or custom applications.

The Show Schema option allows designers to preview the imported dataset and verify that the JSON structure correctly maps to the selected ETL schema before executing the ETL process.

Because this is an ETL source component, it is typically the first step in an ETL pipeline and provides the dataset that subsequent transforms and destination components consume.

The step supports:

• Reading JSON files
• ETL schema mapping
• Column schema selection
• Dataset preview
• Schema validation
• Integration with ETL transformations
• Import of structured JSON data

This step can be used for:

• Importing API responses
• Data migration
• System integration
• Cloud data ingestion
• Business reporting
• Data warehouse loading
• Enterprise ETL automation

Usage:

The JSON Input File step is typically used as the starting point of an ETL pipeline whenever the source data is stored in JSON format.

During execution, the step opens the specified JSON file, applies the selected column schema, validates the imported structure, and exposes the resulting dataset for downstream ETL processing.

A typical ETL pipeline might look like this:

After the dataset has been imported, subsequent ETL components can:

• Filter records
• Calculate derived values
• Aggregate data
• Perform lookups
• Sort records
• Merge datasets
• Load data into databases, files, or external systems

The JSON file should exist and conform to the expected structure before the ETL process begins.

Prerequisite:

The ETL process engine service should be running to execute the ETL definitions.

Example:

Let’s build and execute the “clsInputJSONDef” example.          

• Create a new ETL definition named “clsInputJSONDef” and open it in designer mode. 
• Drag “clsInputJSON, clsTransformFormatNumberToString, and clsOutputJSON” steps to the canvas. 
• Connect the dots between the “clsInputJSON” and other steps, as shown above.
• Define a variable or a global to hold the virtual path. 
• Click the "clsInputJSON" step to configure its "Required" properties. Provide a name for the step. Specify the path to the input JSON file on the application server. Select the ETL data schema from the drop-down list. Note: The schema's column names should match those in the input file. Click the Save button. Click here to learn about the ETL Data Schema Designer.

• Click the "clsInputJSON" step to view its "Optional" properties. Select the Show Schema button. The pop-up window displays the schema columns as shown below. This function helps to understand the schema at a glance.  

• Click the "clsTransformFormatNumberToString" step to configure its "Required" properties. Provide a name for the step. Click the button to configure the transform list, then click the Save button. 

• Click the "clsTransformFormatNumberToString" step to configure the transform list. The configuration pop-up window appears. Click the Add Row (+) button to insert an empty row. Select “integer or value” as the input source column and string as the output target column. Click the Save button. You may add multiple columns for transformation by using the Add Row button. 

• Click the "clsOutputJSON" step to configure its "Required" properties. Enter a name for the step. Enter the path to the output JSON file on the application server. Click the Save button. 

• Click the "clsOutputJSON" step to configure its "Optional" properties. Provide a name for the step. Provide a variable or global reference to store the virtual output path. Click the Save button. 

• The “Logging” configuration is necessary for documentation and also measures workflow progress and percent complete. This is achieved by configuring the step state and percent fields individually, as shown below. Configure the “Logging” using the following properties.

• Save the ETL Definition. Execute the ETL process. During execution, the component reads the JSON file, maps the imported data using the selected column schema, and makes the dataset available for downstream ETL processing. Subsequent ETL transforms can manipulate, enrich, validate, aggregate, or load the imported records into destination systems.

Tips:

• Verify that the JSON file exists and is accessible to the FlowWright server before executing the ETL process.
• Use a column schema that accurately reflects the structure of the source JSON document.
• Use the Show Schema feature to validate the imported structure before connecting downstream transforms.
• Keep JSON property names consistent to simplify schema mapping.
• Validate JSON syntax before importing large datasets.
• Store source files in a controlled location with appropriate access permissions.
• Combine this source with Filter Transform, Lookup Transform, Derived Column, Aggregate Transform, Sort Transform, or SQL Output components to build complete ETL pipelines.

Notes:

• This component functions as an ETL source and typically appears at the beginning of an ETL definition.
• The specified JSON file must be valid JSON.
• The selected column schema determines how imported properties are mapped into the ETL dataset.
• The Show Schema option is intended for design-time validation and schema inspection.
• Invalid file paths, malformed JSON documents, or incompatible schema mappings may result in ETL execution errors.
• This component imports data only and does not modify the source JSON file.
• Proper schema validation should be performed before deploying ETL definitions to production.
• JSON input files are commonly used for REST API exports, application integrations, cloud services, reporting systems, IoT data collection, and enterprise data migration.

Definition Sample:

You may download the sample ETL definition(s) from the link provided and later import them into your FlowWright ETL Designer.

Note: Verify and complete any missing configuration after importing the sample, including:

• JSON file path
• Column schema selection
• Schema validation
• Destination mappings
• ETL connection settings
• Environment-specific configuration

After verifying the configuration, save the ETL Definition before execution.

Click here to download the sample file.