Spent my morning figuring out why Nginx was dead on a server with many days of uptime.
Uncategorized
25
Posts
10
Posters
89
Views
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano How did you even trace this down??
-
@stefano How did you even trace this down??
This post is deleted! -
This post is deleted!
@stefano Oh I need to step up my logging game... a lot...
-
@stefano Oh I need to step up my logging game... a lot...
This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Stuff like this is why I disable unattended upgrades on my servers :) thank you for the reminder
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Un'alternativa credibile a "Una poltrona per due"? Quasi 5 ore di #fantascienza anni 80. Science Fiction Revolution - Cultura | ARTE in italiano https://www.arte.tv/it/videos/RC-027434/science-fiction-revolution/
-
@77nn
(PS avevo visto anche la discussione di cui parli)
-
@77nn
Ho letto la tua riflessione.
In alcuni tratti mi ci sono ritrovato. Un paio di anni fa ho improvvisamente lasciato crollare il castello della mia fede. Ora passeggio tra le sue rovine riconoscendone qua e là alcune pietre di valore che tengo come cimeli e che però non hanno più una collocazione in una struttura coerente e/o solida.
Il tempo passa. Cambiamo noi e cambia il mondo attorno.
-
@MajDen
Grazie !
se posso considerarmi uno dei
"voi"
-
Sotto i cumuli di neve dormono i fiori,
sognando i loro sogni di sole e giugno.Harriet Prescott Spofford ... https://cctm.website/harriet-prescott-spofford-usa/
#harrietprescottspofford #neve #fiori #poesia #cctmwebsite #anoipiaceleggere #leggere
-
Per tutti voi 🩵
-
@De_Treias
Sì, può darsi che tu abbia ragione... Però è giusto che mi faccia riflettere, credo.
-
@capetaun Grazie! <3
