I Almost Died for a Full Sentry Database

[stefano@mastodon.bsd.cafe]

I Almost Died for a Full Sentry Database

A tale of bad logging practices, overengineering, and a near-death experience.

https://it-notes.dragas.net/2024/12/28/i-almost-died-for-a-full-sentry-database/

(Repost, one year later)

#ITNotes #IT #SysAdmin #HorrorStories #Sentry #CloudComputing #Cloud #Repost

[jtheo@hachyderm.io]

@stefano I had similar experiences*, pre- and post-cloud. Devs casually coding “horror stories” and refusing to even considering double-checking. Last job in Italy, an alarm went off at 2am on Christmas night; that time I sent a very long and detailed email with half of the universe in cc. Here in the UK I spent 3 weeks with two other colleagues working days, nights and weekends to figure out weird timeouts. When we explained the problem what the devs should have fixed, the CTO thanked us but said that they didn't have time; too many important features to add.

*) I even fainted in a similar way :) but I wasn't on call that time.

[mdione@en.osm.town]

@stefano the guy faints and the first thing he does after recovering is to take a double espresso and head to bed??!?!>!>?!?@ :-)

[ajlewis2@social.vivaldi.net]

@mdione @stefano

My thoughts exactly!

[mdione@en.osm.town]

@ajlewis2 @stefano and yet he still avoids explaining himself :-P

[stefano@mastodon.bsd.cafe]

@mdione @ajlewis2 With my blood pressure so low, a double espresso was the only way I could manage to get back to bed.

[qwertz@defcon.social]

@stefano Congratulations to your wife for her calm and collected manner and, above all, for knowing what to say at the right time :-)

[stefano@mastodon.bsd.cafe]

@qwertz eheh She knows me and knows how to deal with those situations