{"id":9614,"date":"2026-07-31T17:44:06","date_gmt":"2026-07-31T17:44:06","guid":{"rendered":"https:\/\/blog.shahada.abubakar.net\/?p=9614"},"modified":"2026-07-31T17:44:18","modified_gmt":"2026-07-31T17:44:18","slug":"the-great-exception","status":"publish","type":"post","link":"https:\/\/blog.shahada.abubakar.net\/?p=9614","title":{"rendered":"The Great Exception"},"content":{"rendered":"\n
\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

I just fed claude my logs and source code for an ancient project that was causing problems … daily notification emails not going out on some days.<\/p>\n\n\n\n

Wasn’t getting far with Sonnet, so I upped to Opus and set effort to max.<\/p>\n\n\n\n

I burnt 95% of my  5 hours of usage in 30 minutes. We went through hoops, pouring through source code, analyzing log files, probing the database in all kinds of places. But managed to trace down the problem. One of those where the cause is very far from the effect.<\/p>\n\n\n\n

To entertain myself, I used up the last 5% of credit with the following prompt:<\/p>\n\n\n\n

“Now document the steps you took to identify the bug, how you identified sources, gathered evidence. Include your thought processes, how you pinpointed the cause, and confirmed the issue. Mention all the missteps and misdirections. Include also what you did to mitigate the problem in the future.”<\/p>\n\n\n\n

(here’s the kicker…)<\/p>\n\n\n\n

“Write it in the style of a noir-fiction detective, running a case. Keep the suspense up. Make it a riveting read.<\/strong>“<\/p>\n\n\n\n

The Great Exception<\/h1>\n\n\n\n

A case file from the graveyard shift.<\/em><\/p>\n\n\n\n

\n\n\n\n

I. The Client Walks In<\/h2>\n\n\n\n

It was the last day of July when the log file slid across my desk, still warm from the journalctl<\/code> pipe.<\/p>\n\n\n\n

The complaint was simple, the way the bad ones always are. Every morning at eight, a process called HouseKeeping<\/strong> was supposed to send out one email \u2014 the Notification Summary Blast<\/em>. A roll call of everything wrong in the field. Rain gauges gone dark. Sensors past due. The daily bad news, wrapped in an HTML table and mailed to the men who cared.<\/p>\n\n\n\n

Most mornings it went out. Some mornings it didn’t. Nobody could say why.<\/p>\n\n\n\n

Jul 27 08:00:02  Blasting Notification Summary Email\nJul 29 08:00:01  Blasting Notification Summary Email\nJul 30 08:00:01  Blasting Notification Summary Email<\/code><\/pre>\n\n\n\n
Count the days. Twenty-seven. Twenty-nine. Thirty. Somewhere in there, the twenty-eighth<\/strong> never showed. A blast that should have fired and didn’t, and no scream, no stack trace, no note left on the pillow. Just a gap where a heartbeat should be.<\/p>\n\n\n\n
They told me the server got restarted a few times. They told me it like it explained something.<\/p>\n\n\n\n
It didn’t explain a thing. But I took the case anyway. I always do.<\/p>\n\n\n\n
\n\n\n\n
II. The Wrong Man<\/h2>\n\n\n\n
First theory came easy, and the easy ones are usually the first to lie to you.<\/p>\n\n\n\n
I found the scheduler in HouseKeeping.java<\/a> \u2014 a little clerk who wrote tomorrow’s appointment in the margin of today’s:<\/p>\n\n\n\n
nextBlast = today 8AM;\nif (! nextBlast.isAfter(now)) nextBlast = nextBlast.plusDays(1);<\/code><\/pre>\n\n\n\n
All in memory. No ledger. No persistence. Kill the process and the clerk forgets every appointment he ever made. So here’s the frame I built: the server goes down before eight, comes back up after<\/em> eight, and the clerk shrugs \u2014 8 AM? That’s passed, friend. See you tomorrow.<\/em> The twenty-eighth gets skipped. Clean. Motive, means, opportunity.<\/p>\n\n\n\n
I liked that theory. I liked it right up until the client leaned across the desk and put a cigarette out on it.<\/p>\n\n\n\n
“The lack of a ‘First BlastNotificationSummaryEmail’ line,”<\/em> he said, “means the server didn’t restart after the twenty-eighth.”<\/em><\/p>\n\n\n\n
He was right, and I hated that he was right. Every cold start leaves fingerprints \u2014 a line that reads First BlastNotificationSummaryEmail scheduled for\u2026<\/code>. I went back through the logs looking for it in the gap. There wasn’t one. No restart. No cold start. The process that owed us the twenty-eighth had been alive the whole time and simply hadn’t paid up.<\/p>\n\n\n\n
My prime suspect had an alibi. The building was never empty. Somebody was home, and they still let the blast die.<\/p>\n\n\n\n
Back to zero. The city outside kept raining.<\/p>\n\n\n\n
\n\n\n\n
III. A Ghost in the Machine<\/h2>\n\n\n\n
Here’s the thing they don’t teach you about a JVM: the building can have all its lights on and still be full of dead men.<\/p>\n\n\n\n
HouseKeeping<\/strong> wasn’t the whole process. It was one thread<\/em> \u2014 a single night watchman walking a single beat, once a second, forever. Station-down checks. Auto-close. Session sweeps. File cleanup. The password mailer. And the eight o’clock blast. One man. One flashlight. One key ring for the whole building \u2014 a lone database connection he grabbed at the start of his shift and never set down.<\/p>\n\n\n\n
A watchman like that doesn’t need the building to burn down to stop making his rounds. He just needs to stop<\/em>. And there are only three ways a man stops:<\/p>\n\n\n\n