Pipeline transformations

A pipeline can apply different types of transformations, based on how you want to impact the tabular data. Every Pipeline requires at least one (1) transformation and may be assigned a virtually unlimited number of transformations.

Note: When you set up a transformation that uses a comparison operator, such as contains, greater than, equals (=), or does not equal (!=), press ENTER after its value.

These transformations are grouped into four (4) categories:

• Transform (Technical Transformations) - modify the data layout of the payload.
• Filter - Remove rows from the data payload based on a criterion/criteria.
• Data - Sort data by specific fields and apply summation to derive data points.
• Mapping (Functional Transformations) - build relationships between data fields.

Adding transformations to a Pipeline is a very guided, but not prescriptive process. This ensures the ultimate flexibility to address your transformation needs

Note: Prior to adding transformations to your Pipeline, be sure to pin a sample file.

Requirements

Data Prep is controlled entirely at the org level and does not recognize individual workspaces or their permissions.

This means:

• Data Prep is shared among all authorized users in your org.
• Any user with access to Chain Builder also has access to Data Prep.
• All users who can create or edit chains will have the ability to manage pipelines in Data Prep.
• A single Data Prep pipeline can be used across multiple chains and workspaces within an organization.
  
  

Transform

To modify the layout of the tabular data, such as to add or remove columns or adjust their values, add a Transform transformation.

Note: When you add a column, you specify its name, the type and format of its values, and where it should appear in the tabular data.

Tip: To remove rows from a dataset, use a keyword—such as Ignore—to indicate the rows to remove. Then, use the keyword as a To value in a Mapping transformation, and set up a Smart filter transformation with an OR condition to remove any mapped field with that keyword.

Filter

To remove rows from the data based on specific criteria, add a Filter transformation.

Data

To sort tabular data or apply summations based on specific columns, add a Data transformation.

Mapping

To transform a column's values based on rules defined by a mapping group, add a Mapping transformation:

1. Select Mapping, and click Next.
2. Select the mapping group with the rules to apply.
3. Under Match columns, select which of the pipeline's columns map to the mapping group's.
4. Click Save.

If you apply multiple Mapping transformations to a column, ensure they're in the correct order within the pipeline.

Tip: When a pipeline applies a Mapping transformation, it transforms the column's values inline. To view a column's values both before and after the transformation at the same time, apply a Copy column transformation to add a duplicate of the column before the Mapping transformation. To indicate the "before" status of the copy's values, append a prefix to its name, such as Src_ or UM_, for Source or Unmapped, respectively. To later remove this column from the dataset, apply a Group by or Remove columns transformation.

Additional Transformation Functions

Add a Transformation

Additional transformations can be added to a Pipeline by clicking the green plus signs to the right or left of an existing transformation. Clicking the plus-sign to the left or right of an existing transformation adds a new transformation before or after it, respectively.

Copy a Transformation
Any existing transformation in the Pipeline can be duplicated by clicking the copy icon that is under the transformation. A copy of the transformation is created and the transformation form is opened. Update the transformation inputs as needed and save the copied transformation.

Delete a Transformation
Any transformation in a Pipeline can be deleted by clicking the trash can icon under the transformation. When prompted to confirm the delete, type the required text and then click the Delete button. Click the Back button on the confirmation form to cancel the deletion.

When deleting a transformation that creates a new column in the Pipeline, any subsequent transformation that utilizes the column that was created by the now-deleted transformation will be marked as invalid and the transformation icon (blue circle) will turn red to indicate that the transformation is no longer valid.

Note: Deleted transformations cannot be restored.

Reorder Transformations
The order of the transformations can be changed by clicking the left and right arrows under a specific transformation. By clicking the left arrow, the transformation is shifted to occur immediately before the transformation that previously proceeded it. The right arrow shifts the transformation to occur after the transformation that previously followed.

Note: A transformation can be shifted as many places to the left or right as needed. However, as with deleting a transformation, any subsequent transformations that depend on the result of the transformation being shifted must continue to follow.