Here’s a failure mode worth understanding, because it applies to more than email.
The mail server we run scans every message for viruses before delivering it. Standard stuff. Earlier this year we went looking through the logs for an unrelated reason and found the same line repeating on every message, going back six months: the scanner had failed, and the message had been passed through unchecked.
Every single message. For half a year. No alert, no bounce, no error anyone would see. Mail arrived normally. The dashboard looked fine.
The cause was mundane — a software update had recreated a system account without the permissions the scanner needed to talk to the mail system. Each piece was running. They just couldn’t reach each other any more, and the failure had been designed to fail open so that mail kept flowing.
That design decision is defensible. Blocking everyone’s email because a scanner is down would have been worse. But it means the failure is silent, and silent failures last exactly as long as nobody goes looking.
Two things came out of it.
We fixed the scanner, and we now monitor for the failure message itself rather than for the scanner being “up” — because it was up the whole time. Up and useless.
And it changed what we look at during maintenance. A service that’s running tells you nothing. A service that’s producing the result it’s supposed to produce is the only thing worth checking. Anything that logs a failure on every single transaction should be screaming, and if it isn’t, that’s the real problem.
Most of what we found across our systems this month was some version of this: something running, logging success, and producing nothing.