From Data Prep , you can create and run sequences (pipelines), as well as, create functional transformations(mapping groups).
Pipeline are a series of technical transformations to apply to tabular outputs in chains. For example, you can set up pipelines of common transformations to prepare data for or from your various systems of record.
Tip: If your chains use a common sequence of Tabular Transformation and File Utilities commands to update data from your systems of record, create pipelines to perform those transformations with a single Run pipeline command.
A pipeline can apply transformations to:
- Modify the layout of data, such as to add or remove columns or adjust their values
- Apply filters to remove rows from the data based on specific criteria
- Sort the data or apply summations based on specific columns
- Map relationships between data models based on defined rules
Mapping Groups build the relationships between data models of different enterprise systems by defining how to transform the codes/value from one system to another. They are applied within a Pipeline as a transformation step.
Mapping Groups support a range of mapping techniques that are powerful, yet simple to use, and address common or complex requirements. They are designed for users to define and update to transform and harmonize data. Mapping Groups can be shared across multiple pipelines. The interface is very Excel-like, providing a familiar and welcoming feel.
Step 1. Set up a Data Prep connection
To apply a pipeline's transformations to data in a chain, you include the Data Prep connector's Run pipeline command. If you haven't already, set up a Data Prep connection. With the Data Prep connection set up, you can the open Data Prep from Wdata Chains .
Step 2. Upload sample files
From Sample files upload sample files that represent the tabular data to transform to help ease pipeline creation.in Data Prep,
Once uploaded, you can use a sample file to:
- Quickly define the columns and types of data a pipeline interacts with
- Preview how a pipeline or mapping transformation impacts the data
Step 3. Create groups for mapping transformations
To map relationships between data models within a pipeline, you can include Mapping transformations. From Mapping groups create mapping groups to define the relationships between values and how to transform values from one system to another within a mapping transformation.in Data Prep,
Tip: To set values for a mapping transformation when the pipeline runs, set up runtime variables for the mapping group.
When you create a mapping group, you can define its rules to transform values based on an exact match, a simple pattern, or regular expression.
Step 4. Set up pipelines
A Pipeline is the collection of technical and functional transformations that are applied to data processed by Data Prep.
The technical transformations defined in a Pipeline are used to modify the data layout. Activities such as adding or removing columns, reordering columns, or inserting new columns are all examples of technical transformations.
The functional transformation is the process of building a relationship between the data models of the systems being integrated. Functional transformation is often referred to as mapping and is managed by Data Prep Mapping Groups. Mapping Groups are applied within a Pipeline as a transformation step.
To define the sequence of transformations to apply to tabular data, create pipelines from Pipelines in Data Prep.
When you create a pipeline, you:
- Define the columns and types of data it interacts with, either manually or based on a sample file or uploaded delimited file
- Set up the transformations to apply—in order—when the pipeline runs
Tip: To set values for a transformation when the pipeline runs, set up runtime variables for the pipeline.
Step 5. Run pipelines in chains
To apply the transformations to tabular data from an output earlier in a chain, use the Data Prep connector's Run pipeline command. When you set up the command, you:
- Select the pipeline to run and the tabular output to transform
- Map the tabular file's columns to the pipeline's column definition
- Set any runtime variable values for the pipeline