The 2:38 AM Reflex

The thumb reaches out blindly, fumbling against the cold nightstand’s edge where the charging cable snake-coils in the dark. 2:38 AM. The screen is a miniature sun, burning a jagged hole through the retinas before the sleepy brain can even process the character strings. [Warning] High Disk I/O on DB-Cluster-08. I swipe it away without reading the whole notification. It’s an instinctive flick, the same muscle memory I use to dismiss a spam call or a calendar invite for a meeting I have zero intention of attending. My heart rate doesn’t even spike. That’s the most terrifying part. A decade ago, a page at this hour would have sent me into a cold sweat. Now, I just roll over and wonder if the blue light will keep me awake for the next 18 minutes or if I can sink back into the heavy velvet of REM sleep before the next one hits.

You’ve been there. You are likely there right now, or you’re dreading the moment the sun goes down because your rotation starts. There is a specific kind of guilt that comes with this. It’s the nagging feeling that you are becoming ‘that’ engineer-the one who doesn’t care, the one who lets things slip, the one who has grown cynical and lazy. We call it alert fatigue, but that term is a linguistic trap. It places the burden of the exhaustion on the person experiencing it. It suggests that the problem is your lack of stamina or your diminished focus. If you were just a bit more resilient, if you just had better coffee or a more disciplined sleep hygiene, you could handle the 198 notifications that hit your phone every week.

When everything is an emergency, nothing is. We’ve built high-velocity deployment pipelines that can push code in 8 minutes, yet we haven’t built the corresponding filters to decide what deserves a human’s attention. We have treated the human engineer as an infinite resource, an overflow buffer for the noise our systems generate. We are essentially using people as a garbage disposal for low-priority telemetry.

The Submarine Analogy

Miles M. knows this better than anyone. Miles was a submarine cook-one of those guys who has to make 88 different meals out of a pantry the size of a broom closet while submerged under several hundred feet of pressurized seawater. He once told me about the sonar room. On a sub, silence is the only thing that keeps you alive. If a sonar tech hears a ping, it isn’t ‘filtered’ or ‘queued for later.’ It is a singular, focused event. In the galley, Miles had alarms for the ovens and the pressure cookers, but they were distinct. If the oxygen scrubbers had an issue, the alarm didn’t sound like the timer for the bread. It was visceral. It demanded a specific action. When he transitioned into tech, Miles spent his first 48 hours on-call in a state of absolute panic. He thought the world was ending. By the end of the first month, he was doing what we all do: silencing the phone and going back to sleep.

“

The human brain is a pattern-matching machine that is currently being trained to ignore its own survival instincts.

– Cognitive Survival Mode

Miles told me that the biggest culture shock wasn’t the daylight or the lack of a chain of command; it was the noise. He’d look at the dashboard and see 58 red blinking lights. To a submarine cook, 58 red lights mean everyone is dead. To a DevOps engineer, it just means it’s Tuesday. We have normalized the abnormal. We’ve created a reality where ‘Warning’ is just a synonym for ‘Ignore.’ This is a form of learned helplessness. When you are repeatedly subjected to stimuli that you cannot control or that carry no actual consequence, your brain eventually stops responding to them. It’s a survival mechanism. If your brain treated every 2:38 AM alert as a life-or-death crisis, you’d have a nervous breakdown within 18 days.

Affordances and Lies

I had a moment of clarity recently that really drove this home. I walked into a local cafe and pushed a door that very clearly had a large, brass plate that said ‘PULL’ in bold letters. I felt like a total idiot. But as I stood there, looking at the door, I realized why I did it. The handle was a long, horizontal bar-the universal design signifier for ‘push.’ My eyes saw the word, but my nervous system responded to the architecture. The system lied to me. The affordance of the handle contradicted the instruction of the sign. This is exactly what we do with monitoring. We build systems that look like emergencies-the loud pings, the vibrating phones, the red dashboards-but we use them for non-emergencies. We are building ‘Push’ handles and labeling them ‘Pull,’ and then we wonder why the engineers are frustrated.

The problem is that we’ve made it too easy to add an alert and too difficult to remove one. Adding a Prometheus rule takes about 8 seconds of work. Removing an alert requires a 38-minute meeting where you have to prove to three different stakeholders that the alert won’t be needed for some hypothetical event in the horizon. We are terrified of the ‘missing’ alert, the one that could have saved us from a $878-per-minute outage. So, we keep them all. We hoard alerts like digital packrats, burying the signal under mountains of useless data. This is systemic rot. It’s the kind of systemic rot we dissect regularly in Ship It Weekly, not because we have all the answers, but because we’re tired of the same questions being asked at 2:38 AM.

The Cost of Trust Erosion

We need to stop asking engineers to be more resilient and start asking why our systems are so incredibly chatty. A well-designed system should be silent until it actually requires human intervention. If an automated script can fix a disk space issue, why is a human being paged at 2:38 AM to run that script? If the CPU utilization is at 88 percent but the user experience is unaffected, why is that a ‘Warning’? We have confused ‘Something is happening’ with ‘Something is wrong.’ Most of the time, something is just happening. Computers are busy. They do things. They use resources. That isn’t an emergency; it’s a heartbeat.

This erosion of trust has a high price. When a real emergency happens-the kind that actually threatens the company’s existence-the engineer who has been desensitized by 1008 false alarms is going to react slower. They might assume the spike in 500-level errors is just another glitch in the logging service. They might assume the database latency is just the usual cron job running. By the time they realize it’s a genuine crisis, the damage is done. The system that was supposed to protect us is the very thing that blinded us. We’ve built a smoke detector that goes off every time someone lights a candle, and then we’re surprised when the house burns down because someone took the batteries out of the detector.

I remember working with a team where the ‘on-call’ person was expected to triage 28 alerts per hour. That’s one alert every two minutes. It’s physically impossible to investigate anything in two minutes. All you can do is acknowledge the alert to stop the noise. You aren’t ‘monitoring’ at that point; you’re just a human-in-the-loop acknowledgement button. It’s a dehumanizing role. It treats the engineer’s brain as a simple Boolean gate. Are you awake? Yes. Did you see the light? Yes. Okay, go back to sleep.