Chapter 3: Architecture of your Integration

Webhooks and the scheduler both feed the same flow workers; every worker reads and writes the state store, and failures go to the retry queue.

Webhooks and the scheduler both feed the same flow workers; every worker reads and writes the state store, and failures go to the retry queue.

You can host these as one process (a small Node.js service with a cron library), or split them across serverless functions with a managed queue and database. The design below works the same either way.

One connector, many integrations

Think of the connector as a class definition and each integration as an object instance with its own state. You define the connector once. Every customer organization that uses it gets its own integration, identified by the pair integrationId + orgKey. Your code must handle everything in those pairs.

In practice:

• Keep a registry of integrations. List your connector's integrations with the Connector API Key (GET /integrations); each comes back with its integrationId, orgKey, status and settings. Refresh the registry on a schedule and when the integrations webhook fires.
• Store one Integration API Key per pair in your secrets manager, for example under pipe17/<orgKey>/<integrationId>. The customer's admin finds it on the integration's API Key tab.
• Run every job once per integration. The scheduler loops over active integrations and passes the pair and its key to each job. Never let one customer's job read another's settings, cursors or key.
• Scope all state by the pair. Every table in Chapter 4 carries org_key and integration_id.
• Route webhooks by orgKey. Each integration registers its own webhook with its own key. Every delivery carries orgKey, which tells you whose integration it belongs to.

// scheduler.js – run each job for every active integration
import { refreshSettings, isFlowEnabled } from './settings.js';
import { pollShippingRequests } from './flows/shippingRequests.js';

export async function runAll() {
  const integrations = await refreshSettings();          // GET /integrations with the Connector API Key
  for (const integ of integrations.filter((i) => i.status === 'connected')) {
    const ctx = {
      orgKey: integ.orgKey,
      integrationId: integ.integrationId,
      key: await secrets.get(`pipe17/${integ.orgKey}/${integ.integrationId}`),   // your secrets manager
      settings: integ.settings,
    };
    if (isFlowEnabled(ctx, 'shipments.out')) await pollShippingRequests(ctx);
    // ...the other flows, each with the same ctx
  }
}

Every job and handler in the rest of this guide takes this ctx as its first argument. To keep the examples short, they leave out two things your connector must add: key: ctx.key on every API call (p17(method, path, { key: ctx.key, ... })) and ctx.orgKey + ctx.integrationId on every state lookup.

Placeholders. The examples call a few things you write yourself: mySystem.* (your system's API), state.* (your data-access module, Chapter 4), secrets (your secrets manager), contextFor() (looks up an integration in your registry), chunk() (splits a list into batches) and reportException() (Chapter 8).

Honor the flow settings

An admin turns flows on and off on the integration's Settings tab. Your connector reads them from settings.fields, a list of { path, value } entries where path is the flow name (shipments.out, fulfillments.in, inventory.in, products.out, locations.in). A flow that is off or missing must not run.

// settings.js – load every integration of your connector and read its settings
import { p17 } from './p17.js';

export async function refreshSettings() {
  const res = await p17('GET', '/integrations', {
    key: process.env.PIPE17_CONNECTOR_KEY,
    query: { keys: 'integrationId,orgKey,status,settings,connection' },
  });
  return res.integrations ?? [];
}

// ctx = { orgKey, integrationId, key, settings } for one integration (see above)
export function isFlowEnabled(ctx, path) {
  return ctx.settings?.fields?.find((f) => f.path === path)?.value === true;
}

Call refreshSettings() at startup and on a schedule (every 5 minutes is enough). Subscribe to the integrations webhook topic to pick up changes sooner. Check isFlowEnabled() at the start of every job and before handling every webhook.

Also check the integration's status. If it is disabled or error, pause all flows until it returns to connected.

Add your own settings

Use custom settings for anything that differs from one customer to the next: a SKU filter, a default shipping method, a warehouse code. Each customer then sets their own values on the integration's Settings tab, and one connector serves them all. The example below adds a productSKUs list that the products flow (7.2) uses as a filter. A setting has two halves:

Step 1 – Read the current settings definition. The list already holds your flow toggles (products.out, shipments.out…), and you must send them back unchanged.

curl -s -G "$PIPE17_BASE_URL/connectors/<connectorId>" \
  -H "X-Pipe17-Key: $PIPE17_CONNECTOR_KEY" \
  --data-urlencode "keys=settings"

Step 2 – Add the new field and send the complete list back. Treat settings.fields like entities: the PUT replaces the whole list, so always send every field, existing toggles included.

curl -s -X PUT "$PIPE17_BASE_URL/connectors/<connectorId>" \
  -H "Content-Type: application/json" \
  -H "X-Pipe17-Key: $PIPE17_CONNECTOR_KEY" \
  -d '{
    "settings": {
      "fields": [
        {
          "path": "products.out",
          "type": "boolean",
          "options": { "label": "Send products to the warehouse", "component": "switch", "defaultValue": false }
        },
        "… the other existing fields from Step 1, unchanged …",
        {
          "path": "productSKUs",
          "type": "array",
          "options": {
            "label": "Process only these SKUs",
            "helperText": "Optional. Exact SKUs to send to the warehouse. Empty means all active products.",
            "component": "text-array",
            "defaultValue": []
          }
        }
      ]
    }
  }'

Replace the quoted placeholder line with the real field objects from Step 1 before sending.

Step 3 – Each customer fills in their value. The new field appears on the Settings tab of every integration of your connector. An existing integration may need an upgrade before the field shows (see Versions and upgrades in Chapter 2). Once saved, the customer's value is on their integration record:

curl -s -G "$PIPE17_BASE_URL/integrations" \
  -H "X-Pipe17-Key: $PIPE17_CONNECTOR_KEY" \
  --data-urlencode "keys=integrationId,settings"

{
  "integrationId": "<integrationId>",
  "settings": {
    "fields": [
      { "path": "products.out", "value": true },
      { "path": "productSKUs", "value": ["TSHIRT-BLK-M", "TSHIRT-BLK-L"] }
    ]
  }
}

Step 4 – Your connector applies it. Read the value with getSetting(ctx, 'productSKUs') (defined below) and pass it to the products call as sku. Because the value belongs to each integration, one connector can serve many customers, each with its own list. The same pattern works for any per-customer option, for example an inventorySKUs filter for the inventory flow or a default shipping method.

// settings.js – add next to isFlowEnabled()
export function getSetting(ctx, path) {
  return ctx.settings?.fields?.find((f) => f.path === path)?.value;
}

Troubleshooting

• The integration runs but no data syncs
  The settings for each entity flow are off by default. On the integration's Settings tab, turn on the flows your integration supports, and confirm your code reads the latest settings.
• A workflow keeps running after an admin turns it off
  Your code may be using cached settings. Poll for integration updates with the updatedSince filter, and refresh your stored settings before each job.
• API calls fail with rate-limit errors
  Reduce how often your jobs run, or add retries with backoff.

Need Help?

If you need additional assistance:

• Use Ask Pippen, our AI agent, located at the top of the app page.
• Submit a support request with as much relevant detail as possible. Learn how to submit a request.

We're here to help you succeed with your operations.