How this benchmark has changed, month by month

Every other page here tells you what the numbers are now. This one tells you how they got there: how much the sample grew each month, how much tighter the confidence intervals became as a result, which platforms joined and which lapsed, and what each experiment found when it was published.

It matters for a boring but important reason. A reliability figure without a history is a claim; a reliability figure you can watch tighten over months is a measurement. The counts below are computed from the same ledger that produces every other number on this site, so this page cannot drift away from the data even if we forget to update it.

October 2026

+1,553 monitored runs this month · 16,146 cumulative.

• Pipedream announced that Workflows, the product we measure, shuts down on March 31, 2027, with all Workflows data deleted by April 30. It is in maintenance mode and no longer accepting signups, which means our Pipedream measurement is no longer reproducible by anyone who does not already have an account. The numbers stay up because they were true, with the status printed beside them; nobody should be choosing this product on the strength of them. We are measuring it until it stops, because how a platform behaves while being wound down is a reliability story nobody else will collect.
• Pipedream had in fact been measurable again since at least October 1 and we did not know. We paused it in July when its free credits ran out, with a note saying to resume once the renewal period was confirmed, and nobody confirmed it, so the pause lasted 96 days. One probe and 45 seconds settled it. The reset-detection probe written in September to stop us guessing renewal dates had never actually fired: two later checks dropped it, and the log announced a probe that was never sent. A safeguard that reports success while doing nothing is worse than no safeguard. The probe now runs, and a separate check fails loudly if it stops.
• The polling workflow described on this site as always-on had not been on for 39 of 96 days on Make. Make's free plan allows 1,000 credits a month, one empty 30-minute poll costs 2 of them by design, so the month's allotment is gone in about eleven days and the scenario stops until the reset. It had been going dark for roughly nineteen days of every month since August and nothing noticed, because a poller that stops raises no error, it is simply absent. Measured coverage is now published next to the poll counts, computed from the log rather than written down, and the cost page carries the consequence: a free tier that cannot keep one scheduled workflow running for a month.
• Self-hosted n8n upgraded from 2.28.6 to 2.42.4 on October 9 (down 10:13:56 to 10:15:57 UTC; no measurement fell inside the window). The instance was fourteen minor versions behind a batch of security advisories published on September 30. Our n8n figures now span two versions, and the cutover time is recorded so they can be split if the versions ever behave differently. We pinned a release that had been out for two days rather than the one published that morning.
• Our n8n latency tail turned out to be mostly our own machine. n8n's median has held near one second for weeks while its 90th percentile ran several seconds and its daily maximum routinely reached 30 to 60. On October 9 we watched why: on the 1 GB VM it runs on, n8n's memory is paged out to a slow disk between batches, and the next execution spends most of its time waiting for it to come back. The network path is identical for every platform; the computer is not, so medians are comparable and tails are not. We changed one kernel setting (swappiness, 60 to 10) at 11:03 UTC and wrote down beforehand what would count as fixed: over October 10 to 16, the median daily 90th percentile below 3.2 seconds or the median daily maximum below 19.2 seconds, against a baseline of 6.3 and 38.5. The verdict is due October 17, and the same week also contains the version upgrade, so a pass would not be credited to the setting alone.
• We cut our own n8n server off the network for 14.5 hours, and our monitoring said everything was fine. While changing a memory setting on October 9 we reloaded the machine's kernel settings to check the change would survive a reboot, and the reload re-applied a cloud default that switches off IP forwarding, which the container runtime depends on. From 11:03 UTC to 01:33 the next morning n8n could neither receive events nor reach anything: 60 test events refused, 29 scheduled polls missed. The check that runs after every cycle stayed green, because it only asked whether any platform had delivered recently, and Pipedream had. Those runs are excluded and listed under harness incidents, scoped to n8n alone. The check now looks at each platform separately: a cycle where half or more of one platform's events are refused fails the run, and a platform that accepts events but returns nothing for twelve hours blocks publication. Both were tested against this outage's own data and fire on it.
• n8n's webhooks now require a key. Until October 10 they had no authentication and their paths were literally WF1, WF2 and WF4, so anyone could trigger our workflows. That could not forge a measurement, because we only count receipts for events we fired ourselves, but it could have knocked over the small machine n8n runs on. A request without the key is now refused before the workflow runs. Applying the change took n8n down for 13 minutes, logged as maintenance. One thing we did not expect: for about a minute after the restart n8n kept running the previous, unauthenticated version of two workflows, and an unauthenticated test request we sent in that minute was executed. We found it by checking which version each execution ran rather than trusting the status codes, which is now how every change of this kind is verified.
• The memory fix did not work, by the test we wrote before it ran. We had said it would count as fixed if n8n's median daily 90th percentile or median daily maximum halved over October 10 to 16. Neither did. Both got worse: 6.3 to 11.2 seconds, and 38 to 74. The same week included an n8n upgrade that switched its default expression engine to a sandboxed one with a 5-second limit, so we cannot separate the two causes; we can only report that the result failed the test we set.
• n8n lost two scheduled-poll items in October, its first since we started measuring. After the upgrade, evaluating even a one-field expression sometimes took longer than the new 5-second limit on our memory-starved machine, so the poll failed. n8n logged the error; the destination got nothing; and our workflow made the loss permanent by marking each item as seen before sending it. The polling table on /reliability now explains both halves. None of the webhook workflows behind the silent failure rates has been affected.
• Second try at n8n's memory, with the test written down before it ran. On October 17 between 00:01 and 00:04 UTC we stopped a Google monitoring agent on our n8n machine (about 70 MB of its 1 GB), put the swappiness setting back to its default of 60, and raised n8n's expression time limit from 5 to 30 seconds so a slow machine stops turning into lost poll items. The test is the same rule with the failed week as baseline: the median daily 90th percentile below 5.6 seconds or the median daily maximum below 37, over October 17 to 23, verdict October 24. With three changes at once, a pass will not say which one worked. The time limit should not move webhook latency: no webhook workflow has hit it since the upgrade, and the poll is not in the latency figures.
• A week of correcting our own published claims. Six statements across three files had drifted from the data, and five of the six made us look better than we were: a rejection count that said one when the ledger held eight, a platform roster that omitted Pabbly in four separate places, a sample described as still growing that had stopped, and measurement described as continuous on four platforms where it had ended. Each figure is now computed at build time or fails a check, because a number typed into a sentence stops being recomputed.

September 2026

+3,503 monitored runs this month · 14,593 cumulative.

• Zapier's plan lapsed to Free on September 3 and the platform paused all four of our Zaps, for exactly the reason we had published in July: webhook triggers are a premium feature, so a free plan cannot run a webhook workflow at any volume. Zapier's measurement window closes at 0 silent failures in 840 runs.
• The harness now runs at $0/month (self-hosted n8n on a free-tier VM, Pabbly's free tier, free hosting). Total spend across the whole paid measurement period was about A$65.
• Latency percentiles published for the first time (p50 through p99).
• We broke our own receiver and did not notice for 23 hours. A config file added for type checking conflicted with how the endpoint is built, and from September 17 every receipt arrived at a function that would not start. The platforms did their job: 121 runs were accepted and delivered normally, and 101 of them found nothing at our end. We found out because Pabbly emailed to say one of its steps was failing, which is a poor way to learn that a measurement system has stopped measuring. Those runs are excluded from the rates and listed as a harness incident, with the window and the cause published, because counting our own outage as the platforms' silent failures would be the exact reclassification our specification exists to forbid. Nothing was deleted from the ledger.
• We audited our own endpoint before publishing the specification and found that anyone could post a receipt to it. On its own that was minor, because our reconciliation only counts receipts matching a run it fired. What made it real is that we were publishing the full run ledger with our internal run identifiers in it, so anyone could take a real identifier, post a second receipt for it, and make that run count as a duplicate. The published data now carries a stable reference instead of the identifier, and the endpoint rejects receipts for runs fired more than three days earlier. No measurement was affected; we checked.
• We audited our own claims and found one that was false. The homepage said the test harness was published; it was not. Raw per-run data is published and the method is documented, but the harness source has never been released. The sentence now says what is true, and it will change again when the source goes out. A benchmark that will not correct itself is worth nothing.

August 2026

+4,049 monitored runs this month · 11,090 cumulative.

• Outage experiment (EP2): we took our own receiver down for 30 minutes and measured what each platform does. Make retried for 79 minutes and self-healed; Zapier and n8n waited for a human.
• Malformed-response experiment (EP3): platforms record a 200 OK carrying an error as a full success, and catching it costs nothing standing.
• Make's paid window ended; Pabbly Connect joined as the fifth platform.
• Reference pages published for the vocabulary this project introduced (silent failure vs rejected webhook, uptime vs failure rate, the 200 OK error).

July 2026

+7,041 monitored runs this month · 7,041 cumulative.

• Harness went live (July 1) and the first measurements began accumulating.
• Zapier and Make paid windows opened; billing models verified against both meters (Zapier's task meter matched our executed-action count exactly, 45 = 45).
• Pipedream's free-tier quota ran out mid-measurement and its webhooks kept returning success while discarding every event - our first measured failure mode.
• First public data post; the benchmark began appearing in AI assistant answers five days later.

Reading this table

• The last column is the point. It is the upper bound of the 95% confidence interval on the true silent-failure rate, given everything measured up to the end of that month. Every platform has observed zero silent failures throughout; what changes is how confidently we can say the true rate is small. Watching that bound fall month over month is what accumulating evidence actually looks like.
• "Paused" is not a gap in the data. Platforms stop when a paid window ends or a free tier is exhausted; their accumulated totals stay valid for the period they ran, and we say when that period was rather than quietly blending it into a current figure.
• Sample sizes are deliberately unequal. The self-hosted platform costs nothing per run so it carries the bulk sample; metered platforms get small ones. That is why their intervals stay wider, and why we publish intervals instead of bare percentages.

Current figures live on the reliability scoreboard, cost on the cost page, and delivery speed on the latency benchmark. Raw per-run data: benchtruth-runs.csv (CC BY 4.0).