From Data Prep , you can create and run sequences—or pipelines—of transformations to apply to tabular outputs in chains. For example, you can set up pipelines of common transformations to prepare data for or from your various systems of record.
Tip: If your chains use a common sequence of Tabular Transformation and File Utilities commands to update data from your systems of record, create pipelines to perform those transformations with a single Run pipeline command.
A pipeline can apply transformations to:
- Modify the layout of data, such as to add or remove columns or adjust their values
- Apply filters to remove rows from the data based on specific criteria
- Sort the data or apply summations based on specific columns
- Map relationships between data models based on defined rules
Step 1. Set up a Data Prep connection
To apply a pipeline's transformations to data in a chain, you include the Data Prep connector's Run pipeline command. If you haven't already, set up a Data Prep connection. With the Data Prep connection set up, you can the open Data Prep from Wdata Chains .
Step 2. Upload sample files
From Sample files upload sample files that represent the tabular data to transform to help ease pipeline creation.in Data Prep,
Once uploaded, you can use a sample file to:
- Quickly define the columns and types of data a pipeline interacts with
- Preview how a pipeline or mapping transformation impacts the data
Step 3. Create groups for mapping transformations
To map relationships between data models within a pipeline, you can include Mapping transformations. From Mapping groups create mapping groups to define the relationships between values and how to transform values from one system to another within a mapping transformation.in Data Prep,
Tip: To set values for a mapping transformation when the pipeline runs, set up runtime variables for the mapping group.
When you create a mapping group, you can define its rules to transform values based on an exact match, a simple pattern, or regular expression.
Step 4. Set up pipelines
To define the sequence of transformations to apply to tabular data, create pipelines from Pipelines in Data Prep.
When you create a pipeline, you:
- Define the columns and types of data it interacts with, either manually or based on a sample file or uploaded delimited file
- Set up the transformations to apply—in order—when the pipeline runs
Tip: To set values for a transformation when the pipeline runs, set up runtime variables for the pipeline.
Step 5. Run pipelines in chains
To apply the transformations to tabular data from an output earlier in a chain, use the Data Prep connector's Run pipeline command. When you set up the command, you:
- Select the pipeline to run and the tabular output to transform
- Map the tabular file's columns to the pipeline's column definition
- Set any runtime variable values for the pipeline