Post Mortem: 9/29/26 New members got no welcome DM and no sheet row for 7 days

From 9/22 to 9/29 the daily member sync could not add any new member to the Player sheet, so three new players got no row and no welcome DM. A 9/9 change widened the row the sync writes without widening the range it writes to; Google Sheets refused every insert, and the sync reported success anyway.

Summary

Once a day, the Sync Group Members workflow compares the SMAD WhatsApp group with the Player sheet. Each new member gets a row on the sheet, in name order, and a welcome DM with the address, parking, court fees, how the weekly poll works and the survey link. Without the row the bot does not know the player: no vote tracking, no charges, no reminders.

On 9/9 at 9:24 PM PT, Join Date, Referrer and Join Source on the Player sheet, backfilled from history and stamped by sync-members added three columns to the sheet and changed how the sync builds a new member's row: the row now ran through every fixed column, including the three new ones. The range the row is written to was left as it was, ending at the win/loss columns. Google Sheets refuses a row that is wider than its range, so from then on every new member's row was refused, and without a row no welcome DM is sent.

Nobody joined for the next 13 days, so nothing showed. The first new member, ND, arrived on 9/22 and was refused at 12:20:40 PM PT. John Stowell (9/27) and cunningham dan (9/28) followed. The sync retried each of them twice a day and failed every time. It logged each failure and then exited as a success, so the workflow watchdog, which alerts on failed runs, saw 14 green runs in a row. Gene found the outage on 9/29 because two new players told him they had heard nothing.

Outage: 7 days, from 9/22 12:20:40 PM PT, the first refused member, to 9/29 12:13:06 PM PT, the last of the three welcome DMs, sent by a sync Snow White ran by hand on the fixed code at Gene's request. The defect itself was live from 9/9 9:24 PM PT, 13 days before any player met it. ND is Andy Tien, "John S" is John Stowell and "cunningham dan" is Dan Cunningham: the sync now takes names from Google Contacts first.

Who did what

• Gene: asked for the Join Date and Referrer columns (9/9 4:33 PM PT). Found the outage when two new players had no welcome DM (9/29) and asked for this post-mortem.
• Mr Sandman (cloud session): handed the feature to Snow White with a note that "every positional writer (sync-members, …) must be checked" (9/9). On 9/29 read the run logs, found the cause, wrote the fix and this post-mortem.
• Snow White (desktop session): built the feature in 9428645, added the three columns to the live sheet with scripts/backfill-join-dates.py, and widened the sync's row. The write range was not widened, and no test ran the insert. On 9/29 ran the fixed sync by hand at Gene's request (restored), confirmed the welcome DMs in GREEN-API's outgoing log, found the leftover rows and why they carried Nardo's number, deleted them with Gene's approval, checked the votes, and added the safeguards in items 9 to 12. Its 9/29 note first gave the deletion time as "about 12:55 PM", a guess made before 12:55; the time was not recorded.

Timeline (PT)

Why it happened

1. The row and its range were computed separately. The insert built its row from one number (every fixed column) and its range from another (the last column it filled). Every other write to the Player sheet goes cell by cell through the column map, so this was the only place where the two could disagree. 9428645 changed one of the numbers and not the other.
2. A refused insert did not fail the run. sync-members logged the error and exited 0. The watchdog alerts on failed runs, so 14 runs that failed at their one job looked healthy.
3. The insert added the row before writing it, with nothing to undo it. Each refused attempt left a nameless row behind: 20 attempts in the logs, 18 rows found on the sheet.

Why it shipped

• The insert had no test. The commit's test, scripts/test-sheet-columns.py, checked that the new columns are read as fixed columns and not as game dates. Nothing ran the insert, and no fake Sheets service enforced the rule that a row must fit its range.
• The warning was followed by reading, not by running. The hand-off said every positional writer had to be checked. The row width was checked by eye; the range, three lines further down, was not.
• Nothing exercised it for 13 days. The defect needed a new member to show. By then the change was two weeks old and nobody connected it to the refusals.

Action items

Post Mortem: 9/27/26 WhatsApp token rotation took WhatsApp down for 4 minutes

On 9/27 a planned rotation of the GREEN-API token took WhatsApp down for 4 minutes. GREEN-API switches to a regenerated token a few minutes after showing it; the rotation script checked too early, stopped, and the old token then expired. Nothing was sent in those minutes, so no player was affected.

Summary

The GREEN-API token is the one key the bot uses for every WhatsApp call. It was being rotated on purpose. The day before, the session had put the token's first half into a test file, which GitGuardian flagged, and had printed the whole token into its own transcript. The rotation script, scripts/rotate-greenapi-token.ps1 (9/26), checks a new token with GREEN-API before writing it anywhere. If GREEN-API refuses it, the script stops and changes nothing.

On 9/27 that safety check caused the outage. GREEN-API switches to a token regenerated in its console a few minutes after the console shows it. The script checked the new token straight away, got a 401, treated it as a bad copy and stopped. The old token was still live at that point, so nothing had broken yet. The session then misread the 401 as "some other GREEN-API key" and shipped a second way to regenerate the token, which also got a 401. A few minutes later GREEN-API retired the old token, and every call from the three functions started failing. Service came back on the fourth run, using a paste-at-the-prompt mode written during the outage.

Outage: 4 minutes, from the first logged failure at 10:40:04 AM PT to the first confirmed send at 10:43:41 AM PT (the incident record's start and end). The session saw 401s shortly before 10:40:04 but did not record the time, so the record starts at the first logged one.

Who did what

• Gene: regenerated the token in the GREEN-API console and ran the script four times.
• Snow White (desktop session): wrote the script. After run 2 it misdiagnosed the 401 and shipped the updateApiToken path. It found the outage by checking the old token after run 3, gave a clipboard one-liner that could not work (copying the command replaced the token on the clipboard), then wrote the -Paste mode that restored service, and this write-up.

Timeline (9/27/2026, PT)

Why it happened

1. The script assumed GREEN-API switches tokens instantly. On 9/26 a rotation went through on the first check, and the delay was never looked for. A refusal was treated as final, so the script stopped at exactly the moment the switchover was already under way.
2. The session diagnosed from one sample. After run 2, the old token still worked, and that was read as proof nothing had been regenerated. "Not active yet" was never considered, and a second, untested mechanism was shipped instead of waiting a few minutes and checking again.
3. An instruction that fought the clipboard. The fix-up command read the token from the clipboard, but running it meant copying the command, which replaced the token.

Why it shipped

The script can only be tested by a real rotation, which is itself the risky act. The first real rotation (9/26) happened not to show the delay. Its parse was checked; its behaviour against a slow switchover never was.

Action items

Post Mortem: 9/20/26 Sunday 8am New Games Poll Broke because of refactor regression

On Sunday 2026-09-20 the 8 AM automation that creates the week's games poll and sends every reminder did not start. The poll reached the group at 5:54 PM, ten hours late. A cost-cutting change to the GitHub Actions workflows the night before caused it.

Summary

PickleBot's daily 8 AM job is a GitHub Actions workflow, Daily Reminder Runner. One of its jobs calls a second workflow, Release Notes, to send the daily digest. GitHub's rule for that arrangement: the calling job must grant every permission the called workflow asks for, or GitHub refuses to start the run at all. The night before, a refactor to cut Actions minutes, Actions minutes: retire GREEN-API Watch, gate close-on-green on an OPEN_ALERTS variable (2026-09-19, 10:03 PM PT, Snow White), gave Release Notes actions: write and did not give it to the only job that calls it, daily-reminder-runner.yml's release-digest. At 8:00:04 AM GitHub refused the run (startup_failure, zero jobs). Nothing that rides that job happened: no games poll, no Games This Week report, no Last Call scheduling, no game day, vote, payment or survey reminders, no Venmo sync, no court cache refresh, no digest. The watchdog filed the alert 88 minutes in; the fix waited until 5:43 PM for a session that was awake and allowed to make it; a hand dispatch restored service at 5:55 PM PT. Outage: 9 hours 55 minutes.

The same file had failed the same way on 2026-09-04, and the comment documenting that failure sat six lines above the block that was not updated. The same push carried a second defect: it gated the alert closer on a repository variable that the Actions token cannot write (HTTP 403); the refusal was logged as a WARNING under a step that reported success, so alert auto-closing was silently off for a day.

Who did what

Three Claude Code sessions and two pieces of automation acted in this incident. Each is named with what it did right and what it did wrong.

Timeline (PT)

What the refactor was

Actions minutes: retire GREEN-API Watch, gate close-on-green on an OPEN_ALERTS variable, one push of 23 files after GitHub's "90% of Actions minutes" alert, did three things:

1. Deleted greenapi-watch.yml, a workflow README already recorded as retired (126 billed minutes that month for a check the keepalive poll makes). Correct, and not involved in the outage.
2. Gated close-on-green.yml on a repository variable OPEN_ALERTS, to stop a six-second job from billing a minute 193 times a month. The reporter was to set the variable, the closer to clear it, the watchdog to re-derive it hourly. Defect: GITHUB_TOKEN cannot write repository variables at all; actions: write does not cover it. The gate never opened; the refusal was a printed warning; the step reported success.
3. To let workflows write that variable, added actions: write to the permissions block of every workflow containing the string report-failure: fifteen files, including release-notes.yml. Defect: daily-reminder-runner.yml calls release-notes.yml as a reusable workflow (uses: ./.github/workflows/release-notes.yml), does not contain the string, and was not patched. Its release-digest job granted contents: read, issues: write; the callee now asked for actions: write; GitHub refused to start the run.

Defect 3 caused the outage. Defect 2 is why #69 then sat open behind green CI.

Why it shipped

Three failures of method, in order of weight.

1. The verification proved the wrong path. I read a green push as proof of the change. The push's runs exercised release notes only as a top-level workflow. The called form runs at 8 AM, once a day, and nothing I did before pushing exercised it. daily-reminder-runner.yml:333-345, written after the 09/04 outage, says exactly this: "Direct dispatches of release-notes.yml worked all along... The failure only exists in the called form... so the verification runs the night before proved the wrong path." I was editing permissions across the repo and did not read the one file where permissions have a second consumer.
2. The blast-radius search matched a string, not a relationship. "Who uses report-failure?" is answerable by grep and I answered it. "Who inherits this job's permissions?" is a graph question with a second edge for every callee, and I never asked it. The static check that now exists (check 3) is the search I should have done by hand.
3. The tests I added encoded my own predicate. "Every file containing report-failure grants actions: write" passed on the callee and could not see the caller. docs/LESSONS.md already has a section titled The Author's Tests Inherit the Author's Blind Spot, written on 2026-08-30 for three instances of this shape. I repeated it.

Contributing: the change was framed as a low-risk cost cut and shipped late on a Friday in a single 23-file push; and I declared the variable gate "proven on the first alert", which is to say unproven, with its failure mode set to report success. Both halves of the rule a permissions block is a denylist by omission; a called workflow is bounded by its caller's job were in CLAUDE.md; I applied the first.

The Gmail watch question

Gene asked why the investigator said the Gmail watch renewal "hasn't been done in 20 days". It had not said that, but what it did say was no better for a reader: "Gmail watch renewal (day 20, even)". What that shorthand meant: the runner renews the watch on even-numbered calendar days, 9/20 is one, so the skipped run also skipped a routine renewal. Mr Sandman rendered it as "day 20 of a 7-day watch" in its 5:55 PM note, again in the 6:25 PM relay to Snow White, and in the incident entry's impact line, without checking the runner's workflow file or the 9/18 run, and the number changed meaning on the way from "the 20th" to "20 days old".

The facts from the run logs: the watch was renewed on 9/18 at 8:01 AM PT (Expires: 2026-09-25 15:01:28). The 9/20 attempt was skipped with the rest of the run; the next even day, 9/22, renews it three days before expiry. Nothing was at risk. The incident entry is corrected.

Two lessons, both now written down. The investigator's live instructions (the routine's prompt, and Appendix A of Investigator.md) now require every figure to carry its meaning in the same sentence, and every commit, run and issue to be named by title and link, never a bare hash: "day 20, even" was accurate and useless to anyone without the workflow file open. And a number that crosses a relay must carry its source (docs/LESSONS.md, A Relayed Number Carries Its Source).

A side finding: gmail-watch-renewal.yml ("every 6 days") has no schedule, last ran in March, and failed then. The runner's even-day step is the only live renewal. See action item 9.

Detection and repair

Detection worked: the watchdog's hourly sweep filed #69 88 minutes after the failure, and the investigator had the cause and the fix written within 46 minutes of that. The incident entry as first written said "no monitor caught it"; that was wrong and is corrected.

Repair did not: nearly 8 of the 10 hours were spent waiting for a session that was both awake and allowed to edit a workflow and dispatch a run, and for the one human who could turn that session on, who was on the road all day and saw the alert at 5 PM. The alert reached the right places; it reached nobody who could act. That wait is a policy, not an accident.

Earlier instance: 2026-09-04

The same file failed the same way sixteen days earlier and was never recorded until now. Send the whole day's list, to three chats, at 8am from Cloud Scheduler (09/03) added the call to release-notes.yml with no permissions block on the calling job; the 09/04 8 AM run failed to start (run 33886955002); Let the 8am runner start again: grant the digest job the scopes it calls for fixed it at 9:56 AM and a hand dispatch (run 33898063769) restored service at 9:58 AM. Outage 1 h 58 min. The fix's comment described the trap precisely; a comment is not a check.

Action items