The automation failures nobody scopes for.

Automations rarely fail on the day they ship. They fail in week three, when a real person does something the demo never did, and by then the client has stopped watching. The gap between a workflow that demonstrates well and one that survives contact with production is almost entirely made of cases nobody scoped.

None of these are exotic. They are the same handful of failures, in the same order, across most integration projects.

The recurring failures.

What happens	What the client sees	What prevents it
A webhook is delivered twice	Two identical records, two notifications	An idempotency key checked before any write
The provider returns 429	Records silently missing from a batch	Backoff with retry, and a queue for the overflow
A refresh token expires	The workflow stops, quietly, on a Tuesday	Token refresh handled in code, with an alert on failure
System A is written, system B fails	A CRM record with no matching invoice	Ordering writes, then reconciling what did not complete
An upstream field is renamed	Empty values written over good data	Validating the shape before writing, and refusing to write nothing
A run errors with no error path	Nothing. That is the problem	An explicit failure branch that alerts a human
A date crosses a timezone boundary	Follow-ups sent at three in the morning	Storing UTC, converting only for display
Somebody submits an apostrophe	A broken record or a failed step	Treating all input as untrusted text

The rest of this post is the detail behind the rows that cause the most damage.

Duplicates are the default, not the exception.

Every webhook provider worth using guarantees at-least-once delivery. That is not a weakness, it is the only guarantee a network can honestly offer. If the receiver is slow to acknowledge, the sender retries, and the same payload arrives again.

A workflow that creates a record on receipt will create two. If it also sends a notification, charges a card or writes to an accounting system, the second one is a real problem rather than an untidy one.

The fix is to derive a stable key from the payload and check it before doing anything:

key = hash(form_id + ":" + lowercase(email) + ":" + submitted_at)

if seen.contains(key):
    stop
seen.add(key, ttl = 24 hours)
proceed with the write

Two details decide whether this works. The key must come from the content, not from a timestamp generated at receipt, or a retry produces a different key. And the store has to be shared across runs, not a variable inside one execution, or every retry starts with an empty memory.

Where the destination system supports it, an upsert on a natural key is better still, because the deduplication happens in the place that owns the data.

Rate limits are a scheduling problem.

Limits are commonly enforced per minute, per hour and per day at the same time, and the daily one is often undocumented until you cross it. A workflow that handles one lead at a time will never notice. The same workflow pointed at a migration of existing records hits all three within a minute.

Handling this properly means treating 429 as an expected response rather than an error. Read the retry-after header when the provider sends one, back off exponentially when it does not, and cap the number of attempts so a persistent failure does not retry forever.

The part that gets skipped is what happens after the cap. A record that could not be written after five attempts has to go somewhere a human will look. If it does not, the workflow has quietly lost data and reported success.

Partial writes leave the client’s systems disagreeing.

Most useful automations write to more than one system. There is no transaction spanning a CRM and an invoicing tool. If the first write succeeds and the second fails, the client now has two systems telling different stories, and the workflow has already reported that it finished.

Three practices contain this.

Order the writes so the most important one happens last, or so the first one is harmless on its own. Record what completed, so a failed run can be resumed rather than restarted. And when a step cannot be undone, make the operation safe to repeat instead, so re-running the whole thing does no additional damage.

Nobody is watching the activity log.

The most common failure mode in production automation is not a crash. It is a workflow that stopped running and told nobody.

Two things are needed and they are different. Error alerting fires when a run fails, and should reach a channel a person actually reads, with enough context to act on. A heartbeat check fires when the workflow has not run at all within the period it should have. Error alerting cannot detect a trigger that has silently disconnected, which is exactly the failure that goes unnoticed longest.

There should also be a way to answer “is this working right now” without opening the builder. If the only person who can determine that is the person who built it, the agency has not been handed a supportable system.

Input is never the shape you were shown.

The test data has a first name, a last name, a company and a valid email. Production has a name field containing an entire sentence, an email with a plus sign in it, a phone number written with spaces and a country code, a message field with an apostrophe, and a submission where every optional field is empty.

Validate at the boundary. Decide explicitly what happens to a submission that fails validation, because dropping it silently is a decision even when nobody made it deliberately. Never let an empty upstream value overwrite good data that is already in the destination, which is the specific failure that turns a schema change into data loss.

What tested should mean before handover.

A demonstration proves the happy path works once. That is worth very little. Before a system is handed over, it should have been run against the cases that break it.

• The same payload delivered twice, and three times
• A required field empty, and a field containing unexpected characters
• The destination API returning 429, 500 and a timeout
• An expired or revoked credential
• Volume well above the expected peak
• A record that already exists in the destination
• A run that fails halfway between two writes

Then confirm the failure paths themselves. Cause a failure deliberately and check that the alert arrives, that it reaches somebody, and that it says something useful.

Reliability is the part that takes the time.

Connecting two systems is usually a day. Making the connection survive duplicates, rate limits, expired tokens, bad input and silent stoppages is the rest of the project, and it is the part that decides whether the client still trusts the system in six months.

If you have a client automation that breaks in ways nobody can explain, or a build coming up where this is the part you would rather not own, send us the requirement and what you know about how it fails. We will tell you how we would approach it and what handling it properly would involve. Describe it here in as much or as little detail as you have.