Customizing mappings in Pipe17

Overview

Managing order operations in Pipe17 requires configuring connections with external systems. Pipe17 provides multiple ways to control how data is sent and received, enabling users to customize mappings to align with their business needs.

Mappings define how Pipe17’s entity schema corresponds to external system schemas. For example, you can map the "name" field on Shopify orders to the "Order ID" field in Pipe17.

Example of mapping the "name" field on Shopify orders to the "Order ID" field in Pipe17

 

Mapping Architecture

Baseline Mappings

Baseline mappings serve as factory-defined templates maintained by Pipe17. These mappings are shared across all integrations and ensure consistent data flow between systems.

• Any modifications to baseline mappings affect all integrations of that connector. 
• Changes are versioned, allowing users to compare different mapping versions. 

Comparing the current baseline mapping version to v42.

Custom Mappings

Custom mappings allow users to override baseline mappings for specific needs. If a custom mapping modifies a field already mapped in the baseline, the custom version takes precedence.

Example:

• Baseline Mapping: Maps the "status" field from Shopify to Pipe17's status field.
• Custom Mapping Override: Maps "active" instead of "status" to Pipe17’s status field.

This customization ensures that all Shopify products are treated as "active" in Pipe17, regardless of their original status in Shopify.

When a baseline mapping is customized:

• The system creates a copy of the baseline mapping and applies the changes.
• Users can view only custom mappings in the "Custom only" view.

Note: Override mappings have the same target as the baseline mapping. If the target is different, then it is a new custom mapping.

Custom Mapping Use Cases

Mappings can be customized to:

• Modify the default behavior of a baseline mapping.
• Disable a baseline mapping if it’s unnecessary.
• Map additional fields that are not included in the baseline.

Advanced: How Schema Converter evaluates mappings

Mapping conversions are handled by the Pipe17 schema converter library. Entity mappings are processed following the steps below:

1. If there are custom mappings, merge with the baseline mappings. This is also the default state of the UI when you view all mappings. The merge process is as follows:
   1. Iterate through each custom mapping.
   2. If the target matches with a baseline mapping, override that mapping field.
   3. If there is no matching target, add the mapping rule to the bottom as a new custom mapping.
2. Pass the merged mapping rules along with the document to map to the schema converter. The schema converter will then evaluate all enabled rules from top-down. Schema converter supports special patterns as well, as described in Advanced Use Cases.
3. Each mapping row is evaluated by the following logic:
   1. The source mapping is evaluated.
   2. If the result of this source mapping is null, set to the default value, if defined.
   3. If the result of this source mapping is undefined, set to the missing value, if defined.
   4. If there is a lookup table, then find the key associated to the value and use that as the final result.
   5. If there is no match, then the value is undefined. Set to the missing value if defined.
   6. At the end,
      1. If the value is undefined, the mapping target will be dropped from the resulting document.
      2. If the value is null, empty string, or any value other than undefined, the mapping target will be set to that value.

Customizing Mappings

Since mappings control data flow between external systems and Pipe17, changes should be made carefully. Follow these guidelines when developing, testing, and monitoring mapping changes.

Creating Mappings

1. Click "View documentation" to open the reference panel.
2. Check the external schema details to determine how the mapping should be structured.
3. Add a new mapping if the required field is not mapped by default.

Advanced Use Cases

Pipe17 mappings follow the schema converter format, which processes JavaScript-based transformations with special syntax rules.

Example: Using `[$0]`

line_items[$0].name → result.lineItems[$0].name

• line_items is an array.
• Each item in `line_items` maps its `name` field to the `result.lineItems` array.
• Important: The `[$0]`  syntax must be present in both the source and target for proper conversion.

Additional advanced options include:

• Lookup tables: Map source data based on predefined values.
• Fallback values: Define default and missing values if the source mapping logic evaluates to undefined or null. This is typically used to guarantee a value for required fields.

If you are familiar with JavaScript, view our Advanced Mapping Scenarios guide for some examples of common mapping scenarios that require code.

Example: Baseline mapping of Shopify order status to Pipe17 order status

In this example, the `active` field of a Shopify order is mapped to the `status` field of a Pipe17 order. 

1. The lookup table maps `true` to `active` and `false` to `inactive`. If the field is not `true` or `false`, then the lookup table will not find a match, which will update the value to `undefined`. 
2. We specify that if the value is undefined, we set it to `active`. 
3. In this way, even if the Shopify order's `active` field is an unexpected value, we guarantee that it will be mapped to a legitimate order status value in Pipe17.

 

Testing Mappings

Before saving changes, test mappings to ensure expected behavior:

1. Paste a source document on the left-hand side of the mapping tool.
2. If Pipe17 has already ingested entities, load an existing document.
3. Otherwise, retrieve a document from your external integration for testing.

Monitoring Mapping Changes

Once mappings are modified and saved, monitor their impact on data flows:

• Use the Events page to track how mappings affect incoming data.
• For Shopify order mappings, observe events related to Shopify order creation.
• To minimize disruptions, specify integration settings to pull only test orders into Pipe17.

 

Basic Structure

A mapping file consists of a rules object with a mappings array:

{
  "mappings": [
    {
      "name": "Field description",
      "enabled": true,
      "source": "sourceFieldPath",
      "target": "targetFieldPath"
    }
  ]
}

Each mapping item must have:

• name: A descriptive name for the mapping
• enabled: Boolean to enable/disable the mapping
• source: Path or expression to extract data from source
• target: Path where to place the data in target

Simple Field Mappings

Direct Field Copy

Copy a field directly from source to target:

{
  "name": "Copy product title",
  "enabled": true,
  "source": "displayName",
  "target": "title"
}

Nested Field Access

Access nested fields using dot notation:

{
  "name": "Copy customer email",
  "enabled": true,
  "source": "customer.contact.email",
  "target": "email"
}

JavaScript Expressions

Use JavaScript expressions for dynamic values:

{
  "name": "Calculate full name",
  "enabled": true,
  "source": "customer.firstName + ' ' + customer.lastName",
  "target": "fullName"
}

Working with Arrays

Array Iteration with $0-$9

Use $0 through $9 variables to iterate through arrays:

{
  "name": "Map line items",
  "enabled": true,
  "source": "orderItems[$0].sku",
  "target": "lineItems[$0].productSku"
}

Nested Array Iteration

Use multiple iterator variables for nested arrays:

{
  "name": "Map nested arrays",
  "enabled": true,
  "source": "orders[$0].items[$1].price",
  "target": "allOrders[$0].products[$1].cost"
}

Array Length and Counting

Get array length or perform calculations:

{
  "name": "Count items",
  "enabled": true,
  "source": "(items || []).length",
  "target": "totalItems"
}

Advanced Transformations

Complex JavaScript Functions

Use immediately invoked function expressions (IIFE) for complex logic:

{
  "name": "Calculate tax amount",
  "enabled": true,
  "source": "((order) => { return order.subtotal * 0.08; })(order)",
  "target": "taxAmount"
}

Conditional Logic

Use ternary operators or logical operators:

{
  "name": "Set status",
  "enabled": true,
  "source": "order.paid ? 'completed' : 'pending'",
  "target": "status"
}

Array Reduction

Aggregate array values:

{
  "name": "Total discount",
  "enabled": true,
  "source": "((items) => items.reduce((sum, item) => sum + item.discount, 0))(lineItems)",
  "target": "totalDiscount"
}

Lookup Tables

Map values using lookup tables:

{
  "name": "Convert weight unit",
  "enabled": true,
  "source": "weight_unit",
  "target": "weightUnit",
  "lookup": {
    "_lb": "pounds",
    "_kg": "kilograms",
    "_oz": "ounces",
    "_g": "grams"
  }
}

The lookup key is prefixed with underscore. If the source value matches a key, it's replaced with the corresponding value.

Default and Missing Values

Default Values

Set a default when source field doesn't exist:

{
  "name": "Product type with default",
  "enabled": true,
  "source": "productType",
  "target": "type",
  "default": "general"
}

Missing Values

Set a value when source field is null/undefined:

{
  "name": "Handle missing price",
  "enabled": true,
  "source": "price",
  "target": "cost",
  "missing": 0
}

Type Conversion

Convert values to specific types:

{
  "name": "Convert to number",
  "enabled": true,
  "source": "quantity",
  "target": "qty",
  "type": "number"
}

{
  "name": "Convert to string",
  "enabled": true,
  "source": "customerId",
  "target": "customerIdString",
  "type": "string"
}

Required Fields

Mark fields as required (will log error if missing):

{
  "name": "Required email",
  "enabled": true,
  "source": "customer.email",
  "target": "email",
  "require": true
}

Pre and Post Processing Scripts

Pre-processing Script

Run code before mappings:

{
  "name": "__PIPE17_PRE_RULE__",
  "enabled": true,
  "source": "(() => { sourcedoc.processedAt = new Date().toISOString(); })()",
  "target": "__PIPE17_PRE_RULE__"
}

Post-processing Script

Run code after mappings:

{
  "name": "__PIPE17_POST_RULE__",
  "enabled": true,
  "source": "(() => { targetdoc.mappedAt = new Date().toISOString(); })()",
  "target": "__PIPE17_POST_RULE__"
}

Complex Examples

E-commerce Order Mapping

{
  "mappings": [
    {
      "name": "Order ID",
      "enabled": true,
      "source": "order_number",
      "target": "orderId",
      "type": "string"
    },
    {
      "name": "Customer Email",
      "enabled": true,
      "source": "customer.email",
      "target": "customer.email",
      "require": true
    },
    {
      "name": "Line Items",
      "enabled": true,
      "source": "line_items[$0].sku",
      "target": "items[$0].sku"
    },
    {
      "name": "Line Item Quantity",
      "enabled": true,
      "source": "line_items[$0].quantity",
      "target": "items[$0].qty",
      "type": "number"
    },
    {
      "name": "Line Item Price",
      "enabled": true,
      "source": "line_items[$0].price * line_items[$0].quantity",
      "target": "items[$0].totalPrice"
    },
    {
      "name": "Order Total",
      "enabled": true,
      "source": "((items) => items.reduce((sum, item) => sum + (item.price * item.quantity), 0))(line_items)",
      "target": "total"
    }
  ]
}

Product Catalog Mapping

{
  "mappings": [
    {
      "name": "Product Title",
      "enabled": true,
      "source": "title || name",
      "target": "displayName",
      "default": "Untitled Product"
    },
    {
      "name": "Product Images",
      "enabled": true,
      "source": "images[$0].src",
      "target": "media[$0].url"
    },
    {
      "name": "Image Count",
      "enabled": true,
      "source": "(images || []).length",
      "target": "mediaCount"
    },
    {
      "name": "Variant SKUs",
      "enabled": true,
      "source": "variants[$0].sku",
      "target": "variations[$0].itemNumber"
    },
    {
      "name": "Inventory Status",
      "enabled": true,
      "source": "variants[$0].inventory_quantity > 0 ? 'in-stock' : 'out-of-stock'",
      "target": "variations[$0].status"
    },
    {
      "name": "Weight Conversion",
      "enabled": true,
      "source": "weight",
      "target": "shippingWeight",
      "lookup": {
        "_lb": "16",
        "_kg": "35.274"
      }
    }
  ]
}

Advanced Data Processing

{
  "mappings": [
    {
      "name": "Pre-process data",
      "enabled": true,
      "source": "(() => { sourcedoc._temp = {}; sourcedoc._temp.startTime = Date.now(); })()",
      "target": "__PIPE17_PRE_RULE__"
    },
    {
      "name": "Complex calculation",
      "enabled": true,
      "source": "((data) => { const items = data.items || []; const validItems = items.filter(i => i.status === 'active'); return validItems.map(i => ({ id: i.id, value: i.price * i.qty })); })(sourcedoc)",
      "target": "processedItems"
    },
    {
      "name": "Conditional mapping",
      "enabled": true,
      "source": "(() => { if (sourcedoc.type === 'wholesale') { return sourcedoc.price * 0.7; } else if (sourcedoc.type === 'retail') { return sourcedoc.price; } else { return sourcedoc.price * 1.2; } })()",
      "target": "finalPrice"
    },
    {
      "name": "Post-process cleanup",
      "enabled": true,
      "source": "(() => { targetdoc.processingTime = Date.now() - sourcedoc._temp.startTime; delete sourcedoc._temp; })()",
      "target": "__PIPE17_POST_RULE__"
    }
  ]
}

Available Variables in Expressions

When writing expressions, you have access to:

• sourcedoc - The entire source document
• targetdoc - The target document being built (in pre/post scripts)
• Direct field references (e.g., displayName instead of sourcedoc.displayName)
• Iterator variables $0 through $9 for array processing

Best Practices

1. Use descriptive names - Make mapping names clear and descriptive
2. Enable/disable mappings - Use the enabled flag to temporarily disable mappings
3. Handle missing data - Use default, missing, or conditional logic
4. Validate data - Use require for critical fields
5. Type safety - Use type conversion when needed
6. Test thoroughly - Test mappings with various data scenarios including edge cases

Error Handling

The converter will log errors for:

• Missing required fields
• Invalid expressions
• Type conversion failures
• Array access out of bounds

These errors are passed to the logging callback but don't stop the conversion process.

 

Summary

Customizing mappings in Pipe17 allows precise control over data synchronization. By following best practices in development, testing, and monitoring, users can optimize their order operations while maintaining accuracy and consistency across systems.