Testing Automations

• Reduce order-flow disruptions by validating logic up front

• Improve accuracy by confirming the right entities trigger the right actions

• Save time by using built-in tools—no sandbox or separate environment required

There is no single “test automation” button. Effective testing combines three capabilities:

• Test filter and/or code to confirm rule conditions

• Test entities (orders, products, etc.) to simulate real scenarios

• Automation runs to monitor execution, timing, and errors

Before You Begin

• Keep new or edited rules disabled until testing is complete via dry runs.

• When validating a rule in production, create the rule normally and add a test identifier to your filter, such as checking for an automation-test tag to isolate test orders.

• Start with narrow filters to limit scope, then broaden gradually.

• Know rule execution behavior: rules run top to bottom. For NEW_ORDER and EXCEPTION_MODIFIED triggers, processing stops after the first matching rule.

• Know where to monitor: in the app, open Automation → History to review every execution.\

How Testing Works

• Test filter: Validates that your rule would match the intended entities before you enable it.

• Test entities: Real entities in your account (for example, a test order) that meet your filter conditions.

• Automation runs: A log of each time an automation processes an entity, including matched rules, steps, status, timing, and errors. To see runs, navigate to Events → Event Log and in the Entity Type filter choose Automation runs.

Testing Automations

There are several ways to test your automation rules:

1. Built-in Testing Tools

The automation rule editor includes testing capabilities:

• Test Filter: Paste raw entity JSON (order, shipment, etc.) to see if your filter matches (returns true/false). This validates your logic before enabling a rule.

  • You can load an existing entity by its ID or paste/modify entity JSON.

  • Use this to confirm your filter conditions work as expected.

 

• Test Code Actions: For JavaScript-based actions, paste order JSON and run your action code to see what updates would be made.

  • Shows the computed changes before applying them to real data.

  • Helps debug complex JavaScript logic.

 

• Convert Filter to Code: Turn a Simple filter into JavaScript, then customize it for more complex scenarios.

 

2. Test with Real Orders

You can test automations using actual orders in a controlled way:

1. Create a test order in your sales channel with a specific tag (e.g.,pipe17_test)

2. Set up ingestion filters on your selling channel integration:

   • Go to Integrations → your selling channel → Settings

   • Use Order ID filter or Order tag filter to limit ingestion to just your test order

   • Adjust Order ingestion date if testing with older orders

3. Pull the test order:

   • Turn on Pull orders in your integration settings, or

   • Use Integration actions → Pull orders by ID/date for immediate ingestion

4. Monitor the automation:

   • Go to Automations → History to see the automation run

   • Check the Status, Steps executed, and any Error messages

   • Use View JSON to inspect full details and payloads

3. Manual Triggering

You can manually trigger automations through:

• The UI for bulk actions

• Direct API calls for specific reprocessing tasks

4. Review Automation History

The Automations → History page is essential for testing and troubleshooting:

• Filter by entity type, status, and time period

• View which rules matched and what actions were taken

• Inspect error messages if something went wrong

• Use Explain filters with Pippen to understand filter behavior

Steps

1. Create Your Test

• Define the scenario (for example, “Route orders over $500 with expedited shipping”).

• Set acceptance criteria (for example, “Only orders tagged automation-test and subtotal > $500 trigger Rule A”).

• Decide which fields you will verify after the run (destination, tags, fulfillment routing, step outputs).

2. Create Safe Test Entities

• Create one or more test orders/products that match your intended filters.

• Add a distinct tag (for example, automation-test) or a unique value (for example, a SKU suffix) to isolate test data.

3. Validate the Rule with Test Filter

• Open your rule (keep it Disabled).

• Select Test filter and confirm the tool returns exactly the entities you expect.

• If the set is incorrect, refine the filter until the preview is accurate.

4. (Optional) Learn from Automation History

Use existing Automation runs to confirm how similar scenarios behaved:

• Compare automationRuleIds in past runs to see which rules actually triggered.

• Review step order, results, and timing to anticipate outcomes for your new rule.

5. Enable the Rule for a Controlled Test

• Enable the rule.

• Create a new test entity that matches the filter.

• Keep volumes low (one or two entities) to limit impact.

6. Monitor the Execution in Automation History

Open the latest run for your test entity and verify:

• Matched rules: Confirm the expected rule triggered (check automationRuleIds).

• Step sequence: Confirm actions executed in the intended order.

• Status per step: Each step shows PENDING, COMPLETED, or FAILED.

• Timing: Compare createdAt and completedAt to gauge duration.

• Retries: Check execCount for retry attempts (the system retries up to 3 times).

• Errors: Review errorMessage and failedAt if a step fails.

7. Validate the Outcome

• Confirm the entity’s final state (for example, tags applied, routing decisions, or updates to external systems).

• If results differ from expectations, adjust the rule filter or step configuration and retest.

8. Expand Scope Safely

• Remove the test-only tag from the filter, or add additional conditions.

• Re-run Test filter to confirm the broader scope.

• Monitor results closely as volume increases.

9. Validate the Code

• Validate that the code works in all situations when using code filters or actions.

• For example, when adding a tag to an order using code, ensure it works if “The order has no tags parameter,” “The order has an empty tags parameter,” or “The order has tags.”

Best Practices

• Start disabled: New rules start disabled by default; test first to avoid impacting production flow.

• Use specific test filters: Target your automation-test tag or other unique identifiers.

• Test rule interactions: Because rules run top to bottom and stop after the first match for key triggers, verify ordering carefully.

• Monitor resilience: Review failed runs and retries. Investigate repeated execCount > 1 and recurring errorMessage patterns.

• Test edge cases: Boundaries (for example, $499.99 vs. $500), missing data, uncommon carriers, split shipments, or partial fulfillments.

Troubleshooting

The rule did not trigger

• Confirm the test entity actually matches the filter using Test filter.

• Check rule ordering; a rule above may have matched first.

• Verify the trigger type is correct for your scenario (for example, NEW_ORDER vs. EXCEPTION_MODIFIED).

The run shows FAILED on a step

• Open the step details and read errorMessage for the exact cause.

• Check failedAt and surrounding timestamps for race conditions or external delays.

• Investigate dependencies (for example, missing credentials, mapping errors, or unavailable services).

• Review execCount to see if retries succeeded; if all attempts failed, fix the root cause and retest.

Performance is slower than expected

• Compare createdAt and completedAt to isolate slow steps.

• Look for patterns across runs to identify bottlenecks in multi-step processes.

• Reduce scope temporarily (for example, limit to automation-test tag) and re-measure.

The filter matches too many or too few entities

• Simplify complex conditions; verify each condition incrementally with Test filter.

• Use explicit values (for example, exact tags, SKUs, or order amounts) before introducing ranges or wildcards.

Example Testing Approach

1. Create a test order with tag automation-test.

2. Build a rule that filters for orders where Tag = automation-test. Keep the rule Disabled.

3. Use Test filter to confirm only your test order is returned.

4. Enable the rule.

5. Create a second test order with the same tag.

6. Open Automation → Runs and review the new run: matched rule, step statuses, timing, and any errors.

7. Confirm the order was updated as expected. Adjust and repeat as needed.

Need Help?

If you need additional assistance:

• Use Ask Pippen, our AI agent, located at the top of the platform page.

• Submit a support request with as much relevant detail as possible. Learn how to submit a request.

• For urgent issues, email us directly at support@pipe17.com.

We're here to help you succeed with your operations.