Manage variables for pipeline transformations

To enable flexible pipelines and mapping groups, you can use variable values in transformations and mapping rules. 

Requirements

Data Prep is controlled entirely at the org level and does not recognize individual workspaces or their permissions.

This means:

• Data Prep is shared among all authorized users in your org.
• Any user with access to Chain Builder also has access to Data Prep.
• All users who can create or edit chains will have the ability to manage pipelines in Data Prep.
• A single Data Prep pipeline can be used across multiple chains and workspaces within an organization.
  
  

Variable types

You can use different types of variables with a pipeline, based on the value to return.

Input a variable value

You can select a variable for some values within a transformation or mapping rule:

• To select a variable for a transformation, click Variable code in the field.
  
• To select a variable for a mapping rule, click the From or To field.
  

You can include multiple variable values in a field, or combine variables with static values.

Create runtime variables for a pipeline or mapping group

You can create runtime variables to enable values to be set for transformations or mapping groups when the pipeline runs.

Note: You can use a runtime variable only with columns of the same data type. To use a variable with multiple columns of different data types, create a separate variable for each type.

1. In the pipeline or mapping group, select the Variables code tab.
2. Add the variable:
   • If the first variable, click Create variables.
     
   • Otherwise, under Edit variables, click Add variable.
     
     
     
3. Select the data type of the columns to use the variable, such as String or Number.
4. If the variable's value is needed for the pipeline to run, check Required.

   Note: If Required is checked and the value is not entered at runtime, the pipeline won't run, even if the variable has a default value.

5. In Display name, enter a name to help identify the variable.
6. Enter the variable's default value, if applicable.
7. Click Save.

Tip: If you create a runtime variable for a mapping group, create a corresponding variable for any pipeline that uses the mapping group in a Mapping transformation. To help map the variables between the pipeline and mapping group, use the same name for both. To distinguish the mapping group's variable, append a prefix to its name, such as MG_.

Set preview values for a pipeline's runtime variables

If a pipeline uses runtime variables, set the values to use to preview its transformations.

Note: If a runtime variable is required, the pipeline requires its value to preview transformations. If a runtime variable is not required, null appears in the preview until its value is set.

1. In the pipeline, select the Variables code tab, and click Edit variables.
2. Under Set preview variables, enter the values for the pipeline's runtime variables.
3. Click Set values.

Note: The message, There was an error processing your transformation.  Exception previewing pipeline #### on sample file ####.  Variables [] are marked as required but do not have a non-empty value set. will be displayed if a runtime variable is required but a preview is not populated.

Edit runtime variables

You can edit runtime variables at any time.

1. In the pipeline or mapping group, select the Variables code tab, and click Edit variables.
2. Update the variables' data type, Required status, name, or default value as necessary.
3. Click Save.
4. If you edit the data type, also update any transformation that uses the variable value.

Delete runtime variables

If you no longer need a runtime variable, you can delete it from the pipeline or mapping group.

1. In the pipeline or mapping group, select the Variables code tab, and click Edit variables.
2. For the variable to remove, click Delete delete.
3. Click Save.
4. Update any transformation that used the variable with a new value.