Spent my morning figuring out why Nginx was dead on a server with many days of uptime.
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
hmm I think the problem's here using certbot in standalone mode. Don't you think so?
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano I have never been a fan of systemd. Can’t see any advantage over using the good old tools that come with FreeBSD.
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano I'm not sure about the renew subcommand of certbot, but I know there is an
--nginxflag that will tell certbot to use the already running nginx instance. You would need thepython3-certbot-nginxpackage installed. -
hmm I think the problem's here using certbot in standalone mode. Don't you think so?
@farooqkz looking at the logs, it seems that certbot will run in --nginx mode if it finds an active nginx - but it didn't find it when launched, so used the standalone mode
-
@stefano I'm not sure about the renew subcommand of certbot, but I know there is an
--nginxflag that will tell certbot to use the already running nginx instance. You would need thepython3-certbot-nginxpackage installed.@hyperreal it usually run in --nginx mode - but looking at the logs, it seems it didn't detect a running nginx so switched back to the standalone mode
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano I read this as a simple race condition for port 80, and can't see how this is an "only on Linux" thing.
-
@stefano I read this as a simple race condition for port 80, and can't see how this is an "only on Linux" thing.
@monospace I didn't say it's a problem "only on Linux". It's more of a "let's make things complex" problem. The fact that it's never happened on BSDs is directly related to the fact that they don't provide that kind of automation - so it can't break anything. 🙂
-
@monospace I didn't say it's a problem "only on Linux". It's more of a "let's make things complex" problem. The fact that it's never happened on BSDs is directly related to the fact that they don't provide that kind of automation - so it can't break anything. 🙂
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
-
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
@monospace the certbot renewal cronjob is usually enforcing the --nginx (or --apache), so it would fail if nginx/apache is down. This script tries to detect if nginx or apache is running and, if not, it's using the certbot as standalone. This created the problem - otherwise, it would just fail and retry the morning after.
-
@stefano To me, this is just a coincidence of two scheduled jobs (package upgrades and certificate renewal) running at the same time. Maybe I'm missing something, but port 80 being open to be taken over by certbot would have happened with a traditional cron job on any old Unix system just the same.
@stefano Wait, no, I see your point. I had to edit in "post-install restarts", and realized that systemd taking care of that is indeed something special. I will still put the blame on certbot for taking over port 80 even though instructed to use nginx. That should have resulted in a fatal error.
-
@monospace the certbot renewal cronjob is usually enforcing the --nginx (or --apache), so it would fail if nginx/apache is down. This script tries to detect if nginx or apache is running and, if not, it's using the certbot as standalone. This created the problem - otherwise, it would just fail and retry the morning after.
@stefano I agree, it's certbot's behaviour that caused the issue in the end, not systemd doing a good job at system maintenance.
-
@stefano Wait, no, I see your point. I had to edit in "post-install restarts", and realized that systemd taking care of that is indeed something special. I will still put the blame on certbot for taking over port 80 even though instructed to use nginx. That should have resulted in a fatal error.
@monospace Exactly, I agree.
-
@stefano I agree, it's certbot's behaviour that caused the issue in the end, not systemd doing a good job at system maintenance.
@monospace I've updated the original post to clarify that systemd has done its job, but the interaction caused problems
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Says more about certbot than systemd though.
Like web server can just stay up with using the other ACME challenges (which can be DNS or reverse-proxying the acme client), so web server never has to go down. -
@stefano Says more about certbot than systemd though.
Like web server can just stay up with using the other ACME challenges (which can be DNS or reverse-proxying the acme client), so web server never has to go down.@lanodan when I create some cron jobs, I force the "--nginx" or "--apache" - so it will never start listening. The script shipped with Ubuntu seems to fallback to "standalone" mode if nginx|apache isn't running.
-
@farooqkz looking at the logs, it seems that certbot will run in --nginx mode if it finds an active nginx - but it didn't find it when launched, so used the standalone mode
I agree about the problem of Ubuntu here. But I don't think behavior of certbot is fine here either.
I don't think doing
certbot --nginxand then it falling back to standalone without explicit request of the user(here the sysadmin) aligns well with Unix philosophy and designs. To be honest, the certbot itself doesn't very much align with Unix philosophy IMO. -
I agree about the problem of Ubuntu here. But I don't think behavior of certbot is fine here either.
I don't think doing
certbot --nginxand then it falling back to standalone without explicit request of the user(here the sysadmin) aligns well with Unix philosophy and designs. To be honest, the certbot itself doesn't very much align with Unix philosophy IMO.@farooqkz I agree. On many of my servers, I'm using acme.sh or lego. Or acme client on OpenBSD, of course
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Wow.. talk about the worst timing!
-
Spent my morning figuring out why Nginx was dead on a server with many days of uptime. No reboot, no kernel panic. Just... down. Ubuntu 24.04.
The cause? An automatic unattended-upgrade of libc6. This prompted systemd to work its magic, wisely deciding to restart every running service to apply the patch. Fine.
The problem is, in the exact same minute, the systemd timer for certbot decided it was time to renew certificates.
The result:
- systemd stops Nginx.
- Port 80 becomes free.
- certbot, in standalone mode, immediately grabs it for validation.
- systemd tries to restart Nginx, which fails with "Address already in use".The web server was knocked offline by its own certificate renewal script.
I swear, this is the kind of cascading failure that has never happened to me in years of running *BSD. With a classic cron job, certbot would have failed, logged an error, and tried again the next day. The web server would have remained untouched.
systemd was doing its job, but something failed because of the interactions.
Sometimes, too much automation and too many interconnected parts just create more spectacular ways for things to break.
@stefano Call me ol'fashioned ☺️
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";
![Farooq | فاروق [Master Patata]](https://cdn.cr8r.gg/accounts/avatars/113/941/705/910/877/703/original/ebd811ed4c7b30e7.png "Farooq | فاروق [Master Patata]"){kind=avatar}
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Sigarette elettroniche e prodotti alternativi: una rivoluzione per la sanità
https://www.sigmagazine.it/2025/09/rapporto-smoke-free-sweden-2/
> Un rapporto mostra i progressi dei Paesi che puntano sulla riduzione del danno da fumo.
-
Queens of the Dead, drag e zombie nella comedy horror queer di Tina Romero. Il trailer
https://www.gay.it/queens-of-the-dead-drag-gay-zombie-horror-tina-romero-trailer
> "Era arrivata l'ora di avere un film gay con gli zombi".
-
@PaoloParti
I malati sono loro, è evidente.
Una volta davano la responsabilità alla madri...
-
I get personal attacks from creeps and stalkers on here every once in a while. I'm OK with blocking them, but I think they're kind of fucked up. I've been working on building the Fediverse for 17 years; if I was going to do a heel turn, it would have happened long ago. I'd appreciate if people would assume good faith with me, but I can live with it if they don't. I know how the block button works!
-
La Disney ha dato luce verde al ritorno dello show di Jimmy Kimmel sulla Abc, ma l'attesissima puntata del comico sospeso per il caso Charlie Kirk non sarà accessibile a parte degli Stati Uniti.
Il Sinclair Broadcast Group ha annunciato che sospenderà il ritorno di Kimmel sulle sue emittenti.----------------
Niente, sono ******** convinti.🤦♂️
-
Le piazze con Gaza, le destre con Netanyahu – MicroMega
https://www.micromega.net/sciopero-generale-gaza/
> Lo sciopero per Gaza porta decine di migliaia di persone in piazza. La destra al potere le accusa di estremismo ma è la prima a non mediare.
-
@eccosilvia @OpenGiornale A me la cosa che fa impazzire è che tutte 'ste fantasie le facciano spesso tirando in ballo noi autistici. Come se la nostra neurodivergenza non fosse una caratteristica ma una malattia.
-
@Gina good to see I don’t have to cancel my Disney+
