A few days ago, a client’s data center "vanished" overnight.
Uncategorized
13
Posts
6
Posters
2
Views
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano nice story! and, yeah, internal monitoring is a must, but you also need an external one, operated by someone else than yourself.
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano Only in BSDcafé can you read actual techno thrillers like this.
-
@stefano Only in BSDcafé can you read actual techno thrillers like this.
@EnigmaRotor Sometimes the lights are low and the atmosphere is dark...
-
@EnigmaRotor Sometimes the lights are low and the atmosphere is dark...
@stefano Stefano Jones P.A. a very noir series.
-
@stefano Stefano Jones P.A. a very noir series.
@EnigmaRotor /me making coffee in the dark, while whispering some IT horror stories
-
@EnigmaRotor /me making coffee in the dark, while whispering some IT horror stories
@stefano Oh, if genre is horror, then don’t forget to tell the tale of the guy who pronounced “Microsoft” 3 times before his mirror. What happened next, the blue mirror of death, is frightening to the bones.
-
undefined oblomov@sociale.network shared this topic on
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano feeling of :xkcd:`705` intensifies :D -
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
In the first sentence you mention a "data center", but such an attack would not work with a data center, to be one you need to have two buildings with independent power supply, at a safe distance, etc etc. I think this was at best a hosting room, not a data center.
-
In the first sentence you mention a "data center", but such an attack would not work with a data center, to be one you need to have two buildings with independent power supply, at a safe distance, etc etc. I think this was at best a hosting room, not a data center.
@uriel sure - we tend to call "data center" a specific place, inside the company, that will host the servers (with A/C, etc). Maybe a little inappropriate, here.
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano I must repeat this Never trust in onsite backups either. Fire will destroy those. And RAID is not backup.
You know this but it bears repeating! -
@uriel sure - we tend to call "data center" a specific place, inside the company, that will host the servers (with A/C, etc). Maybe a little inappropriate, here.
Well, not "a little". The one you described is - at best - a server room, not even a hosting center, since according with the blueprints, there was no redundancy....
-
undefined stefano@mastodon.bsd.cafe shared this topic on
-
Well, not "a little". The one you described is - at best - a server room, not even a hosting center, since according with the blueprints, there was no redundancy....
@uriel You're right. I've updated the original post to clarify it. Thank you for pointing it out!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@stefano thank *you* for making bsd.cafe a place that people want to share ☺
-
@Mastodon Plushtodons for everyone... Even Canadians
-
Surveillance Watch – Una mappa che mostra le connessioni tra le aziende di sorveglianza
Le tecnologie di sorveglianza e gli #spyware vengono utilizzati per prendere di mira e sopprimere giornalisti, dissidenti e difensori dei diritti umani in tutto il mondo. Surveillance Watch è un database interattivo che documenta le connessioni nascoste all'interno dell'opaco settore della sorveglianza. Fondata da sostenitori della privacy, la maggior parte dei quali è stata personalmente danneggiata dalle tecnologie di sorveglianza, la nostra missione è quella di far luce sulle aziende che traggono profitto da questo sfruttamento. Mappando l'intricata rete di aziende di sorveglianza, delle loro filiali, partner e finanziatori, speriamo di svelare i facilitatori che alimentano le diffuse violazioni dei diritti in questo settore, garantendo che non possano eludere la responsabilità per la loro complicità in questi abusi. Surveillance Watch è un'iniziativa promossa dalla comunità e ci affidiamo ai contributi di persone appassionate della tutela della #privacy e dei diritti umani.
https://www.surveillancewatch.io/
-
@GillesLeCorre2 pas grave bien sur, mais...dommage.
-
Elon Musk’s X must be banned
https://disconnect.blog/elon-musks-x-must-be-banned/
> Regulators need to stop cowering before the richest man in the world
-
The last words of Renee Nicole Good and the ICE agent who murdered her in cold blood.
-
È disponibile il rapporto 2025 sulla fiducia e la sicurezza del social web
Pubblicato da @iftas questo rapporto si basa su sondaggi dettagliati e sul feedback della community, composto da moderatori volontari, amministratori e community manager del social web decentralizzato. Offre il quadro più completo finora sul panorama della fiducia e della sicurezza in progetti come Mastodon, GoToSocial, WordPress, PeerTube e altri.
Cosa c'è nel rapporto
Novità del 2025
- Nuove pressioni sui moderatori : il rapporto medio moderatori-utenti è peggiorato a 1:3.500
- Lo spam ha superato il CSAM come principale preoccupazione per la maggior parte delle squadre
- Il burnout rimane diffuso : 1 amministratore e moderatore su 5 ha segnalato traumi o esaurimento
- La maggior parte dei servizi non dispone delle garanzie legali o procedurali necessarie per gestire il rischio
- Le piccole comunità dominano , ma l'ecosistema non ha gli strumenti progettati per loro
- La federazione basata sul consenso sta emergendo come un modello desiderato per la crescita e la sicurezza
Previsioni per il 2026
- C’è una crescente concentrazione tra i grandi servizi e una crescente tensione
- C'è meno integrazione di nuovi moderatori , anche se le minacce aumentano
- Le campagne di disinformazione e lo spam generato dall'intelligenza artificiale sono ora rischi importanti
- La complessità legale e normativa è in aumento, ma il supporto resta scarso
Il rapporto di quest'anno include anche una previsione lungimirante, individuando cinque tendenze che caratterizzeranno l'anno a venire:- La logica condivisa e i segnali di fiducia sostituiranno le liste di blocco frammentate
Perché questo è importante
- I media sintetici e l'impersonificazione metteranno alla prova la moderazione umana
- I rischi di cattura delle infrastrutture aumentano man mano che gli strumenti vengono centralizzati
- La regolamentazione globale della sicurezza sta diventando obbligatoria e non facoltativa
- Greylisting e allowlisting potrebbero presto sostituire la “federazione aperta predefinita”
I moderatori sono la spina dorsale di un social web più sicuro, ma la maggior parte di loro non è retribuita, non riceve sufficiente supporto ed è costantemente sotto pressione. Se vogliamo un futuro per le piattaforme decentralizzate che rispetti l'autonomia degli utenti, la libertà di parola e l'autonomia della comunità, dobbiamo supportare l'infrastruttura che ne garantisce la sicurezza.
