Manage transformations for pipelines

Pipelines specify the sequence of transformations to apply to tabular data when a chain executes a Run pipeline command. A pipeline can include different types of transformations, based on how you want to impact the data.

From a pipeline, you can add and manage its transformations as a vertical list or horizontal flowchart:

• For a vertical list, click List list.
• For a horizontal flow, click Visual swap_horiz.

Tip: To preview the transformations' impact, pin a sample file indicative of the columns and data from the Files of the pipeline.

Add transformations to a pipeline

To add a transformation to a pipeline:

1. From Wdata Chains, click Data Prep.
2. From Pipelines linear_scale, open the pipeline.
3. Add the transformation:
   • For the first transformation, click Create transformation.
     
   • For an additional transformation in a horizontal flow, click Add transformation before or after the preceding or following transformation.
     
   • For an additional transformation in a vertical list, click Add transformation after for the preceding transformation.
4. Select the transformation to add, click Next, and set up its values and rules.

   Tip: To enable flexibility, some of the transformation's values may support variables code. To prompt for these values when the pipeline runs, you can create runtime variables.

5. Click Save.

Copy a transformation

To add another instance of a transformation to a pipeline, such as to apply multiple mapping groups:

1. In the pipeline, select the existing transformation, and click Copy file_copy.
2. Update the settings of the duplicate as necessary.
3. Click Save.

Reorder transformations

To reposition a transformation, select it in the pipeline, and then click Move forward or Move back to shift it forward or backward one transformation, respectively.

Note: When you reposition a transformation, adjust the order of any transformations that depend on its result as necessary.

Delete a transformation

To remove a transformation from a pipeline:

1. In the pipeline, select the transformation, and click Delete delete.
2. In Confirm, enter delete.
3. Click Delete.

Note: If you delete a transformation that adds a column, any subsequent transformations that use that column are no longer valid.