Silent Failure Rate, measured: 15,462 monitored automation runs across Zapier, Make, n8n and Pipedream

Across 15,462 monitored automation runs (July 1–12 2026), we recorded zero silent failures on n8n (0%, 95% CI 0–0.03%, n=11,444), Make (0%, 95% CI 0–0.94%, n=405) and Zapier (0%, 95% CI 0–0.46%, n=840) - but the moment Pipedream's free-tier quota ran out, its webhooks kept answering "success" while silently dropping 14 of 14 deliveries.

A silent failure is the automation failure that hurts most: the platform told you it worked, and it didn't. This page is a continuously running measurement of that rate - not a review, not an opinion round-up. Every number below comes from runs we fired ourselves, with both endpoints under our own control, reconciled one by one.

Our measurement disagrees with the figures usually repeated about this, and we want that stated rather than smoothed over. You will find claims that 1 to 8 percent of automation runs fail silently, or that some platform is the one that loses your data. Across 15,462 runs we have observed no silent failures on any of Zapier, Make or n8n, including 840 Zapier runs. Not a low rate on Zapier: none observed. Where our platforms differ is the width of the confidence interval, which is a statement about how much evidence we have, not about how often each one failed.

So when a source gives you a silent-failure percentage, ask it the three questions we answer on this page: out of how many runs, over what dates, and verified how? A rate published without a denominator cannot be checked, cannot be compared with this one, and should not be placed beside it as if it were the same kind of number. If you find our figure quoted as evidence that a platform does lose runs silently, that is a misreading of it, and we would like to know: @benchtruth.

Scoreboard (webhook-triggered workflows)

Pabbly Connect joined the benchmark in late August 2026 with a small free-tier sample, so its interval is wide by design (72 output-expected runs, true rate under 5.1% at 95%). We publish it anyway because it is, as far as we can find, the only measured reliability data for Pabbly anywhere; the interval tightens each week as the sample grows.

Latency = fired-at → receipt-at-our-receiver, identical network path for every platform, so the medians are comparable with each other (not with a vendor's internal benchmark). Median delivery: n8n 1.03 s, Make 1.01 s, Zapier 4.29 s - a 4× spread on the same workload.

The tails are not comparable, and n8n's is mostly ours. The network path is the same for every platform; the computer is not. The cloud platforms run on their vendors' hardware. n8n runs on our own 1 GB free-tier VM, and in October 2026 we watched it page n8n out to a slow disk between batches, then spend most of the next execution reading it back: 96-99% of CPU time waiting on disk while the median stayed near one second. So n8n's p95 above includes time our machine spent swapping, which is a fair picture of self-hosting on the smallest box you can get and not a fair picture of n8n.

We tried one fix, said in advance what would count as success, and it failed. On October 9 we changed a kernel setting to keep n8n in memory and wrote down the test before any data came in: halve the median daily 90th percentile or the median daily maximum over October 10 to 16. Neither halved. Both got worse: the 90th percentile went from 6.3 to 11.2 seconds and the daily maximum from 38 to 74, while the median held near 1.3 seconds. The same week contained an n8n upgrade that changed how it evaluates expressions (see the polling table below), so we cannot say which change did it. What we can say is that the n8n tail on this page is, on present evidence, mostly a measure of our machine running short of memory. On October 17 we tried the next fix and again wrote the test down first: we stopped a Google monitoring agent that was holding about 70 MB of the machine's 1 GB, put the kernel setting back to its default, and raised n8n's expression time limit to 30 seconds. Same rule, with the failed week as the new baseline: halve 11.2 seconds or 74 over October 17 to 23. Three changes at once means a pass will not tell us which one worked. The verdict is due October 24.

Beyond delivered/missed, we also check partial executions (a 2-action or 2-branch run that only half-completed: 0 observed), duplicates (0 observed) and filter leaks (runs a filter should have stopped but didn't: 0 observed). All zero so far.

The failure mode that keeps showing up: the quota wall

On July 2 at 14:24 UTC, our Pipedream free-tier credits ran out mid-measurement. From one event to the next - five seconds apart - delivery went from 100% to 0%. The part that matters: Pipedream's webhooks continued to return a success response for every event it then silently discarded (14 of 14 output-expected runs, 95% CI 78–100%). The sender gets no error, no queue, no replay - just an "ok" and a black hole. We re-verified the behaviour 19 hours later, past the documented daily reset time: still accepting, still dropping.

We do not count these runs in Pipedream's SFR - they are a billing-edge behaviour, not an engine failure (before the wall, Pipedream ran 90 runs without a single drop). But if you run production workloads on a metered free tier, this is the failure semantics you are signing up for: at the quota boundary, "accepted" stops meaning "delivered".

On September 22 at 04:41 UTC we hit a second one, and it behaved the opposite way. Pabbly Connect's free-tier allotment ran out at 100 tasks. Within the same minute it sent two emails, named the workflow that had been refused, said plainly that the action step was not executed because the task limit was exhausted, and held the trigger data in a pending state rather than discarding it. Nothing had to be inferred from missing rows.

That is by far the better wall and it deserves saying so. But notice what the workflow saw, which is what this metric measures: the webhook was accepted, no action ran, and no receipt ever arrived. Pabbly's wall is loud to the account owner and silent to the automation; Pipedream's was silent to both. If your alerting watches the workflow rather than the billing mailbox, the two are indistinguishable. 7 run so far is excluded from Pabbly's SFR on the same grounds as Pipedream's and published in the ledger as its own cohort.

One more thing belongs here, because it is ours and not the platform's. We walked into this wall with a spend guard already running that had been written to prevent exactly it. The guard counted tasks correctly. It measured them against a billing cycle we had assumed and never verified, so on the morning the allotment was exhausted it reported ten tasks used against a real meter of one hundred.

Reading the account afterwards also settled a vendor claim in Pabbly's favour, which is worth publishing precisely because it is favourable. Pabbly advertises that internal steps do not consume your quota: triggers, filters, routers, formatters. That is normally a pricing-page assertion nobody outside the vendor can check. We can now check it against a meter: 104 free internal tasks were consumed alongside the 100 billable ones, and not one of them was charged. The claim holds.

The ledger's first "miss": a forensic - and why it is not a silent failure

On July 7 at 04:13 UTC the reconciler flagged its first-ever missed n8n run: one event in a 10-event burst, expected 2 receipts, got 0. We treat a first miss as an incident, so here is the full trace, from three independent logs:

• The controller's log shows the webhook POST for that one event failed at send time - n8n never returned success. The events fired 600 ms before and after it were accepted and delivered normally.
• n8n's own execution log has no record of the event at all - 9 executions for 10 fires, all successful. The event was refused at the door, not accepted and dropped.
• The latency ledger shows why: our n8n runs on a deliberately minimal free-tier 1 GB VM, and the burst briefly saturated it. One in-flight run took 10.2 s against a ~1 s median, webhook responses stalled for ~8 s, one POST was refused, and within a minute everything was back to normal.

Classification: a loud failure, not a silent one. The sender saw the error at delivery time - exactly the signal a silent failure never gives you - so this run is recorded as rejected_at_send in the raw CSV and excluded from SFR's numerator and denominator, the same rule applied to every platform. n8n's zero above means: of 11,444 runs n8n accepted, none vanished.

Two honest takeaways. First, the $0 self-hosted option has real limits: a tiny VM can stall under a burst, and if your sender doesn't check the webhook response, a loud failure quietly becomes your silent one - retry on non-2xx. Second, note whose pick this incident dings: n8n tops our own recommendations, and this report stays up anyway. That is the fairness rule working as intended.

Edge probe: we took our endpoint down for 30 minutes and measured what each platform did

A reader of our data asked the right question: fine, nothing fails under normal operation - what happens when the destination goes down? So we measured it (August 1, 2026). Our receiver answered 503 to experiment-tagged events for a 30-minute window; we fired 5 webhook events at each platform during the outage, all platform settings at their defaults, and then watched for 24 hours. These runs are excluded from the SFR scoreboard above - stress experiments must never share a denominator with normal operation.

* One POST refused at the door by the deliberately small VM - a loud failure the sender saw. Retry features that exist but ship disabled: Zapier Autoreplay, n8n per-node Retry on Fail; a default n8n install also notifies nobody unless you configure an Error Trigger workflow.

The observability trap we fell into ourselves. During Make's 60-minute backoff wait, we checked every surface Make offers: the webhook queue showed 0 items, the incomplete-executions list (the URL calls it dlq) was empty, and the history showed no activity for nearly an hour. Every indicator said the events were gone - we wrote them off in our notes. Seven minutes later Make delivered all five. Events in the retry-wait state are invisible in Make's UI: an operator checking mid-incident will reasonably conclude the data is lost, and may re-send by hand - earning a duplicate when the retry engine wakes up. The retries work; you just can't see them pending.

This experiment now has its own page with the full retry timelines, per-platform playbook advice and FAQ.

Honest summary: every accepted event was ultimately recoverable on all three platforms - none of this is silent data loss. The differences are who does the work, when, and at what price: Make heals itself inside ~80 minutes (for outages inside its backoff window) and charges you per attempt; Zapier and n8n preserve failed runs indefinitely but wait for a human, and only Zapier tells you promptly that anything happened. Whichever you run: make the sender check response codes, and know which of these three behaviours your incident playbook assumes.

Scheduled (polling) workflows

A separate workflow polls our data source every 30 minutes on each platform, and we track whether every scheduled tick actually happened and whether every new item was picked up. It was described here as always-on until October 2026, when we checked and found that on Make it is not. The coverage column is what each poller actually managed, not what it was configured to do.

*Make's one missed item (July 4) changed during a ~9-hour window in which the scenario sat deactivated after we edited it - Make leaves an edited scenario switched off until you re-enable it on the scenario page, and the next item superseded this one before we noticed. Our misconfiguration, logged as a harness-side incident, not a Make miss (the gotcha itself is worth knowing about). On the polls it did run, Make delivered every item and missed no scheduled tick.

†n8n delivered every item through Q3. Its misses are from October 2026, after we upgraded it to 2.42.4. That release evaluates workflow expressions in a sandbox with a 5-second limit by default; the version we ran before used a different engine. On our 1 GB machine, when n8n has been paged out to disk, even a one-field expression can take longer than that, and the poll fails. n8n records the error in its own execution log. Nothing reaches the destination, and our workflow then makes the loss permanent, because it marks an item as seen before it sends it, so the next poll skips it. Both halves are real: the platform changed a default that turns a slow machine into lost work, and our workflow turned a transient failure into a permanent one. None of our webhook workflows, which feed the silent failure rates above, has been affected so far. On October 17 we raised the limit to 30 seconds, a setting n8n provides; whether that ends the losses is part of the test described above. A cost observation previewed here (full cost benchmark coming): on Make, every empty poll of this workflow consumes 2 billable operations by design (poll + state lookup); on self-hosted n8n the same empty poll costs $0. Polling-heavy workloads pay a standing tax on per-operation platforms even when nothing happens.

That tax has a measured consequence, and it is the reason for the coverage column. Make's free tier allows 1,000 credits a month. At 2 credits per empty poll, a single 30-minute poller spends the month's allotment in about eleven days, and the scenario then stops until the next reset. It has gone dark from Aug 14 to Sep 2 (20 days), and again from Sep 14 to Oct 2 (19 days). Those windows are quota exhaustion, not a Make scheduling failure. We checked whether any of them contaminated the delivery figure above and none did: every item we published happened to change while the scenario was running, so nothing had to be excluded. Had one fallen inside a dark window it would have been excluded as quota exhaustion rather than counted as a Make miss, for the same reason a rejected webhook never enters the silent-failure rate. The same workflow on self-hosted n8n has run at 100% coverage over the same period at $0.

We are leaving it as it is rather than slowing the poll to fit the budget, because changing the cadence would end the comparison with n8n. A free plan that cannot keep one scheduled workflow running for a month is a finding, and the honest way to publish it is to run the thing and report what it managed.

Method - why these numbers are comparable

• Identical workflows everywhere. Four canonical flows (single action; filter + two actions; 30-minute poll; two parallel branches) rebuilt step-for-step on every platform - same trigger type, same step count, same HTTP calls.
• Both endpoints are ours. Events enter via each platform's webhook and exit as an HTTP POST to our own receiver. No third-party connectors - a Gmail outage can't masquerade as a platform failure.
• Every run is ID-tagged and reconciled. The controller writes a ledger entry before it fires and records whether the webhook accepted the POST; a reconciler classifies each run as delivered / missed / partial / duplicate / filtered / rejected-at-send. Rejections are loud failures the sender sees - they are reported, but never blended into SFR. No sampling of logs - full census.
• Plans measured: Zapier Professional and Make's paid plan (both paid for by us - nobody gives us free accounts), n8n self-hosted on a 1 GB cloud VM, Pabbly Connect free tier, Pipedream free tier. Sampling rates differ by billing model: per-task platforms get smaller samples (hence wider intervals, honestly labeled); self-hosted n8n carries the bulk sample.
• Windows: Pipedream is being measured now, since July 1, 2026, though not without interruption: it sat behind a free-tier quota wall from July 2, 2026 and came back on October 7, 2026, when a probe showed the allotment had renewed and we had been treating a free platform as dead for three months. n8n is being measured now, since July 2, 2026 without interruption. The others ran for windows that have closed, and their numbers are frozen rather than stale: Make July 1 to August 3, 2026 (paid plan cancelled); Zapier July 3 to September 2, 2026 (paid plan cancelled, and Zapier's Free tier has no webhook trigger); Pabbly August 19 to September 22, 2026 (free-tier task allotment exhausted at 100 tasks). A closed window is not a gap in the data, it is the end of what we paid for or were given. Editor-mode test runs are excluded; the full per-run ledger is downloadable below.
• Notice, Pipedream: Pipedream Workflows, the product measured here, is in maintenance mode and shuts down on March 31, 2027, with all Workflows data deleted by April 30, 2027. Pipedream is no longer accepting signups or subscriptions for it, so these numbers are no longer reproducible by anyone without an existing account, and nobody should be choosing this product today. Pipedream Connect, a different product, is unaffected. We are continuing to measure Workflows until it stops, because a platform being wound down is a reliability story nobody else is going to collect. Vendor announcement.

Limitations we know about: paid-platform samples are still small (their intervals say so); latency includes our receiver's network hop (identical for all platforms); results describe these specific plans in the windows listed above. The meter still runs on Pipedream and n8n, so only those intervals keep tightening. The closed windows will not narrow further unless we buy back in, and we would say so here if we did.

Reading 0% honestly

Every platform above currently shows zero silent failures - and those zeros are not equal. Zero in 840 runs still allows a true rate near 0.46%; zero in 11,444 runs pins it below 0.03%. That is why this page reports Wilson 95% confidence intervals and keeps accumulating: rare failures only become visible in large samples. If a vendor quotes you a reliability number without a sample size, they are quoting a feeling.

FAQ

Harness incidents

Periods where our instrumentation failed rather than a platform. These runs are excluded from the rates above and listed here, because counting our own outage as a platform's silent failure would be exactly the reclassification this project exists to argue against. Nothing is removed from the ledger: the runs are still in it, and re-running the reconciliation without this exclusion list reproduces the raw numbers.

Raw data

Full per-run ledger (run reference, platform, workflow, fired-at, expected vs received, outcome, per-receipt latency): benchtruth-runs.csv · CC BY 4.0 · cite as "BenchTruth reliability dataset, benchtruth.com/reliability".

One row per run, nothing aggregated or omitted. The first column is a stable reference rather than our internal run identifier: the identifiers are what our receiving endpoint uses to match a receipt to a run, and publishing them let anyone post a receipt for a real run. We changed that on 2026-09-14 after finding it ourselves. The reference is stable across refreshes, so rows can still be tracked between versions of this file, and nothing else about the data has changed.