Classic file types are no longer available for use as of January 2021. You can transition your classic files or download a PDF. Learn More

Creating pipelines

  • Last updated on

To transform tabular data with the Data Prep connector, you first set up the sequence—or pipeline—of transformations to apply. A pipeline provides a graphical representation of its defined transformations and enables a preview of each transformation's impact. 

Tip: If your chains use a common sequence of Tabular Transformation and File Utilities commands to update data from your systems of record, create pipelines to perform those transformations with a single Run pipeline command.


Requirements

Data Prep is controlled entirely at the org level and does not recognize individual workspaces or their permissions.

This means:

  • Data Prep is shared among all authorized users in your org.
  • Any user with access to Chain Builder also has access to Data Prep.
  • All users who can create or edit chains will have the ability to manage pipelines in Data Prep.
  • A single Data Prep pipeline can be used across multiple chains and workspaces within an organization.

Step 1. Create the pipeline

Tip: Before you create the pipeline, from Sample files, upload a sample file that represents the columns and data it will transform to easily define the pipeline's columns and enable a preview of the transformations applied. 
  1. In Wdata, click Chains and Data Prep.
    Note: To access Data Prep from Wdata Chains, first set up a Data Prep connector.
  2. From Pipelines linear_scale, under Active pipelines, create the pipeline:
    • For the first pipeline, click Create a pipeline.
    • Otherwise, click New pipeline (+) next to the search bar.
      create-pipelines_01.png
  3. Enter a name and description to help identify the pipeline.
  4. Click Create.

Step 2. Define the columns

To specify the fields the pipeline will interact with, define the columns of the data it transforms. When you define a column, you specify its name and the type and format of its data. For example, for a column with a Number data type, specify its decimal places and the characters used for its decimal and thousands separators.

Note: The column names defined for the pipeline can differ from the columns in the data it transforms.

To define the pipeline's columns, you can use the column definition from an uploaded sample file or a delimited file saved locally or on your network. You can also manually define columns.

To ease pipeline creation, we recommend you use a sample file to define its columns:

Note: To use a sample file, first upload it to Sample files.
  1. Under Define columns, click Pick from list.
    create-pipelines_02.png
  2. Select the sample file with the column definition to use, and click OK.
    Note: The sample file's column definition will replace any columns defined for the pipeline.
  3. Review the column definition, and edit the columns' names as necessary.
  4. Click Save.

To define the pipeline's columns, you can upload a file with the same column definition.

Note: The file must be delimited and contain a header row.
  1. Under Define columns, click Create from file.
    create-pipelines_03.png
  2. Browse to and select the file with the column definition to use, and click OK.
    Note: The file's column definition will replace any columns defined for the pipeline.
  3. Review the column definition, and edit the columns' names and data types as necessary.
    Note: Be sure to review and update the column definition. The pipeline uses columns names from the file's header row and guesses data types based on the data.
  4. Click Save.

To manually define a column:

  1. Under Define columns, click Add columns.
    create-pipelines_04.png
  2. Select the column's data type.
  3. Enter a name and description to help identify the column.
  4. Specify the format of the column's data, based on its type:
    • For a String column, select any special format, such as for universally unique identifiers (UUIDs), binary strings, email addresses, or uniform resource identifier (URI) web addresses.
    • For an Integer column, select the thousands separator.
    • For a Number column, enter the number of decimals places, and select the decimal and thousands separators.
    • For a Date, Time, or DateTime column, select its string-from-time (strftime) format.
    Note: A Binary column contains values such as True or False, or 1 or 0.
  5. After you define all columns, click Save.

Step 3. Set up the transformations

  1. To preview the transformations' impact, pin a sample file indicative of the columns and data to be transformed by the pipeline.
  2. Click Create transformation.
    create-pipelines_06.png
  3. Select the transformation to apply, and click Next. 
  4. Set up the transformation, and click Save.
  5. To set up any additional transformations, click Add transformation before or after the existing transformation, based on when it should occur.
    create-pipelines_07.png
    Tip: To add another instance of a transformation already in the pipeline, click its Copy, and set up the new instance as necessary.
  6. Adjust the transformations as necessary:
    • To reposition a transformation within the pipeline, click its Move forward or Move back.
    • To remove a transformation from the pipeline, click its Delete.
    Note: If you move or delete a transformation, adjust any transformations that depend on its result as necessary.

Step 4. Publish the pipeline

When the pipeline is ready for use, click Publish. 

After you publish the pipeline, you can use it with the Run pipeline command of the Data Prep connector to apply its transformations to tabular data within a chain.

Was this article helpful?
0 out of 2 found this helpful
To transform tabular data with the Data Prep connector, you first set up the sequence—or pipeline—of transformations ...