Spent my morning figuring out why Nginx was dead on a server with many days of uptime.
Uncategorized
25
Posts
10
Posters
50
Views
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano How did you even trace this down??
-
@chebra looking carefully at the logs and studying the timing and inteactions
-
@chebra looking carefully at the logs and studying the timing and inteactions
@stefano Oh I need to step up my logging game... a lot...
-
@chebra for me, logs are critical. They saved my life so many times...
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Stuff like this is why I disable unattended upgrades on my servers :) thank you for the reminder
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@molly0xfff Now ask it to pronounce "foyer" or "voila".
-
thanks Google AI!
-
I'm three minutes into the new Mountain Goats, and it feels very promising. https://www.youtube.com/watch?v=y9VPW4lJXIU
-
Kit Bashir (@Unixbigot)
https://aus.social/@Unixbigot/115510935415180659
> Loctite 999 is an advanced multi part adhesive suitable for bicycle headset alignment, hyperdrive injector port retention, and workplace pranks. Part A consists of 92% degenerate strange quark matter in a preservative of antigluon superfluid. Part B (activator) is a proprietary blend of tau mesons and unpasteurized cottage cheese. After admixture of parts, an event horizon forms
-
within 90 seconds, preventing separation of parts until the post-proton era. PRECAUTIONS FOR USE ⚙︎ Do not allow part A to contact eyes, skin or planets before activation ⚙︎ If contact with inactivated part A occurs, instant conversion to strange matter is certain ⚙︎ Mix parts using only the supplied bamboo stirring stick ⚙︎ Do not lick the used stirring stick ⚙︎ Unused admixture may be used on the chair seats of crew mates or coworkers WARRANTY VOID WHERE PROHIBITED BY @Unixbigot @Unixbigot
-
PHYSICS #Tootfic #MicroFiction #PowerOnStoryToot @Unixbigot @Unixbigot
-
Addio al Nobel per la Medicina James Watson. Scoprì la struttura a doppia elica del DNA insieme a Francis Crick. Aveva 97 anni
https://www.rainews.it/articoli/2025/11/morto-james-watson-il-premio-nobel-che-ha-scoperto-la-struttura-del-dna-5bd78a8f-f001-492d-9f59-b1d445efab2d.html
-
@astro How would I be able to tell that it’s no longer working? I've upgraded to 4.5 earlier today and since then have not seen any posts on the low volume hashtags I’m following. This may or may not be a coincident.
