Spent my morning figuring out why Nginx was dead on a server with many days of uptime.
Uncategorized
25
Posts
10
Posters
89
Views
-
hmm I think the problem's here using certbot in standalone mode. Don't you think so?
This post is deleted! -
@stefano I'm not sure about the renew subcommand of certbot, but I know there is an
--nginxflag that will tell certbot to use the already running nginx instance. You would need thepython3-certbot-nginxpackage installed.This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano I read this as a simple race condition for port 80, and can't see how this is an "only on Linux" thing.
-
@stefano I read this as a simple race condition for port 80, and can't see how this is an "only on Linux" thing.
This post is deleted! -
This post is deleted!
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
-
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
This post is deleted! -
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
@stefano Wait, no, I see your point. I had to edit in "post-install restarts", and realized that systemd taking care of that is indeed something special. I will still put the blame on certbot for taking over port 80 even though instructed to use nginx. That should have resulted in a fatal error.
-
This post is deleted!
@stefano I agree, it's certbot's behaviour that caused the issue in the end, not systemd doing a good job at system maintenance.
-
@stefano Wait, no, I see your point. I had to edit in "post-install restarts", and realized that systemd taking care of that is indeed something special. I will still put the blame on certbot for taking over port 80 even though instructed to use nginx. That should have resulted in a fatal error.
This post is deleted! -
@stefano I agree, it's certbot's behaviour that caused the issue in the end, not systemd doing a good job at system maintenance.
This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Says more about certbot than systemd though.
Like web server can just stay up with using the other ACME challenges (which can be DNS or reverse-proxying the acme client), so web server never has to go down. -
@stefano Says more about certbot than systemd though.
Like web server can just stay up with using the other ACME challenges (which can be DNS or reverse-proxying the acme client), so web server never has to go down.This post is deleted! -
This post is deleted!
I agree about the problem of Ubuntu here. But I don't think behavior of certbot is fine here either.
I don't think doing
certbot --nginxand then it falling back to standalone without explicit request of the user(here the sysadmin) aligns well with Unix philosophy and designs. To be honest, the certbot itself doesn't very much align with Unix philosophy IMO. -
I agree about the problem of Ubuntu here. But I don't think behavior of certbot is fine here either.
I don't think doing
certbot --nginxand then it falling back to standalone without explicit request of the user(here the sysadmin) aligns well with Unix philosophy and designs. To be honest, the certbot itself doesn't very much align with Unix philosophy IMO.This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
This post is deleted! -
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano How did you even trace this down??
-
@stefano How did you even trace this down??
This post is deleted! -
This post is deleted!
@stefano Oh I need to step up my logging game... a lot...
-
@stefano Oh I need to step up my logging game... a lot...
This post is deleted!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
This is distressing. https://www.bbc.co.uk/news/articles/cp39kngz008o
-
Le persone cambiano, su questo non c'è ombra di dubbio. Siamo quantomeno tutti delle navi di Teseo in quanto organismi con un ciclo di rinnovamento cellulare.
Non lo so come funzioni per le cellule nervose, per la verità, ma suppongo che riproducano di meno per garantire una sorta di stabilità e coerenza di personalità... o forse no, ma non importa.
Mi accorgo che le mie posizioni su alcuni temi etici stanno virando, non capovolgendo, sia chiaro, ma stanno cambiando. Mi viene da dire che si stanno ricalibrando come se le condizioni di contorno e i limiti fossero cambiati e bisognasse riproiettarle su un nuovo sistema di riferimento.
Eppure frequento solo il Fediverso, come social network online. Non guardo la TV, penso solo al lavoro e ad alcuni hobby.
Dunque cosa fa cambiare la mia percezione del mondo? Davvero le 10 ore di ufficio, di cui forse in totale 15-20 minuti effettivi trascorsi con colleghi al caffé, che alla fine non la pensano nemmeno troppo diversamente da me, hanno questo potere di far virare la mia visione del mondo in modo non radicale, ma comunque percettibile?
Non è che improvvisamente, per dirne una, i generatori di immagini o di testi mi sembrano diventati eticamente accettabili, ma li vedo come meno inaccettabili rispetto per dire ad un anno fa. E questo sta accadendo su diverse cose, che una volta mi parevano monoliti tra miei valori etici.
Le mie certezze sono meno assolute. I miei capisaldi, meno saldi.
Ci ho fatto caso oggi stesso, postando un testo che ha sollevato diverse critiche da persone con cui sono in contatto da anni qui sul Fediverso. L'ho riletto più volte e sì, c'erano un paio di battute di cattivo gusto, che forse un anno fa non avrei fatto, ma che oggi ho trovato buffe o umoristiche. E nonostante ci abbia riflettuto e compreso, accettato come legittimo, il punto di vista di chi me le ha segnalate, continuo a non vederci tutto sommato un male assoluto... ma piuttosto una sensibilità violata, questo sì.
Possibile che io possa aver recepito, senza essere stato esposto in alcun modo massiccio, il bombardamento propagandistico culturale filo-MAGA mitragliato ad oltranza dall'America verso i Paesi occidentali? Lo spero, perché l'alternativa vorrebbe dire che sono rincoglionito, e a quello non c'è, ahimé, alcun rimedio...
No, credo che sia quello che ho detto all'inizio. Credo che il contesto determini la risposta, e cambiando radicalmente, induca degli aggiustamenti sulle priorità dei valori che sostengono i comportamenti.
Credo che, dopo anni a cercare (e non riuscire a trovare) il modo di cambiare la percezione e le opinioni dentro la testa delle persone, qualcuno abbia raggiunto una sorta di massa critica che gli consente di alterare a piacere il contesto pubblico (tipo le big tech) per cambiare la risposta delle persone. Molto pratico. E agghiacciante.
O forse sono invecchiato e mi sto rincoglionendo?
O forse sto diventando uno stronzo e sto cercando un capro espiatorio?
O forse lo sempre stato e mi sono nascosto dietro a un dito che oggi non mia coperto abbastanza?
Non lo so, ma basta adesso vado a sedermi sulla mia seggiolina di vimini con una coperta a quadri sulle gambe, a guardare Retequattro. E morirò sbavando.
-
Happy Holidays everyone!
Let's decorate the @FediTree
-
@Dunpiteog il 25
e poi il 26 con l'altra parte della famiglia
a meno che capitino casini vari, nel qual caso “due giorni nel periodo tra natale e capodanno” :D
-
-
@crabbng got it on the RSS feed 8-)
-
Un mesetto e mezzo per finire Kingdom Come Deliverance, che giocone. Pur pieno di bug e fatiche, soprattutto nel combattimento che iniziamente è super punitivo, non sono riuscita a staccarmi dal gioco nella sua ultima parte, completando il più possibile delle missioni secondarie, raccogliendo un sacco di armature pazze per poter completare l'epilogo vestito come un giullare da tanta roba che ho messo addosso a Henry. Bello bello (non credo installerò il secondo capitolo a breve) #BacklogPass
-
float formatting now uses 64-bit instead of 128-bit multiplication in the common case
