Chapter 3: Implementing Connection Code

Coding the Custom Connection

The code for this custom connection runs in your environment, outside of Pipe17. This code will use Pipe17 APIs to transact with the system, thus establishing a connection between the two systems.

Now that we have the custom connection definition setup, we will go into actual implementation of the workflows defined in the custom connection settings. We can go through the workflows supported and deal with each entity involved.

For the sample ShipBob Fulfillment Connection, the workflows supported are:

Locations:
- Pull Locations (Warehouses) from ShipBob (optionally filter the list)
- Push Locations to Pipe17 (create or update Locations)
Order Fulfillment:
- Route Shipping Requests to ShipBob connection - can be done via Pipe17 configuration
- Push Shipping Requests to ShipBob
- Pull Fulfillment from ShipBob
- Push Fulfillment to Pipe17
- Pipe17 will push Fulfillment to Sales Channel (Order Source) - Pipe17 platform will do this automatically once a Fulfillment is created in Pipe17
Product Catalog:
- Get Product catalog information into Pipe17 - done by connecting a Product source channel or by manually uploading Products into Pipe17
- Push Products into ShipBob
Inventory Updates:
- Pull Inventory from ShipBob into Pipe17
- Push Inventory updates to other channels - Pipe17 platform will do this automatically once a destination connection to push Inventory is created in Pipe17

We will be referring to:

Pipe17 API Docs: https://apidoc.pipe17.com/
ShipBob API Docs: https://developer.shipbob.com/api-docs/
Pipe17 reference implementation, consisting of:
- Pipe17 API Client node module available in this repository: gitlab:pipe17-group/connectors/api-client
- Pipe17 3PL Reference Implementation available in this repository: gitlab.com:pipe17-group/3pl-reference-implementation.git
  - The code has README.md describing how to create a local installation as we will be referring to the code here.

Please reach out to support@pipe17.com to get access to the API Client module 
and the reference implementation code.

Background Jobs Implementation

In an ideal connection implementation you would be implementing all the workflows as background jobs that would run periodically. However, for the sake of illustrating the end to end workflows we have created a sample App and added manual triggers to showcase how data related to an entity would go to/from Pipe17 & ShipBob. The Locations workflow is implemented in both the sample App and as background jobs.

Typically you want your implementation code to run a job periodically, say every 10 min. The code will have to keep state on what it already processed. The frequency of run is also subject to API rate limits set by vendors so please check relevant limits before implementation. Pipe17 will also rate limit to prevent abuse and overuse. Pipe17 can tune rate limit by customer scenario. Please reach out to support@pipe17.com to understand your specific rate limits.

API Connection Setup

ShipBob Client

ShipBobClient creates a class that makes it easy to instantiate the client once with the required credentials and use it across the app for making calls to the ShipBob API. Please note that along with the Personal Access Token, you also need a Channel Id in the header to make calls to their API. The code here illustrates how it is done. You can learn more about channel_id at ShipBob docs here.

Note that the required credentials are being loaded from .env file.

This part of the code will also be a good place to implement safeguards around any API rate limits, data privacy etc.

Pipe17 API Client

Pipe17APIClient similarly instantiates a client once to access Pipe17 API across the app. The config to instantiate the client is once again being read in from .env file. Since we have a ready node module that is providing a ready interface we can leverage this instead of making direct REST API calls.

Local Data Storage

A local Mysql DB is created to keep track of the entities that have been processed, their status etc. The entities in this local DB follow Pipe17 schema as the canonical model - along with a few fields for tracking sync status with ShipBob. Keeping Pipe17 schema as canonical model means we will need to map fields coming from / going to ShipBob.

You do not strictly need a local DB that stores all entities to work with the Pipe17 API although you might need to store a few details such as last pulled timestamp, status, and other relevant filters for processing. Individual entities in Pipe17 allow you to store external ids that are specific to channels. You can leverage the external ids along with filters on the API such as updatedSince, and status to process only the required entities instead of the entire list available in Pipe17.

Location: has the field externalSystem to store locationValue (id or primary key) of the external channel
Shipping Request: has the field extReferenceId to store id reference to destination channel (3PL) & extShipmentId to store id reference to the source channel (Sales or ERP).
Fulfillment: has the field extFulfillmentId to store id reference to the 3PL channel
Product: has the field extProductId to store id reference to the published channel

These external ids along with status can be used to poll for status changes and moving through the workflow process without need for a separate db.

Polling / Background Jobs

A typical connection will poll for data required either from Pipe17 or the Vendor channel. Once updated data is available; further API calls are made to continue the workflows.

In this implementation we are using node-cron to run simple background jobs once every 5 mins. The code for background jobs can be seen in pollWorker.js file.

Workflows have to be processed based on the Connection settings the Org has enabled, so the first thing to poll for is the connection settings. If you look at the server.js file it begins by fetching the connection and storing the settings in local db. It then triggers a background job to poll connection that have changed since last update so that the connection workflows respect the settings made on the Org in Pipe17. For simplicity we will assume all the workflows have been enabled in this connection.