The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
0
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined Evan Prodromou shared this topic
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@jesgar wonderful to hear, thank you for the brilliant advice (I had never heard of it) 🙏
-
🔥 SOLO 2 POSTI! ULTIMI GIORNI PER ISCRIVERSI AL CORSO "DARK WEB E CYBER THREAT INTELLIGENCE" IN PROSSIMA PARTENZA
📝Contattaci subito su WhatsApp al 379 163 8765 o scrivi ad academy@redhotcyber.com
#redhotcyber #cti #intelligence #minacce #cyberthreatintelligence #formazione #corso
-
@matz ci vuole una certa fantasia!
-
@mcc exactly when do you expect that bug to have been/to be fixed?
-
-
FlorentIA: uno sguardo all’Intelligenza Artificiale nel cuore di Firenze
📌 Link all'articolo : https://www.redhotcyber.com/post/florentia-uno-sguardo-allintelligenza-artificiale-nel-cuore-di-firenze/
Con l’inizio del mese corrente, Firenze ha ospitato il 1 Ottobre scorso un evento unico: FlorentIA. FlorentIA si presenta come un #appuntamento imperdibile per la comprensione dell’Intelligenza #Artificiale e dei suoi molteplici #impatti nella vita delle #persone, delle aziende, della pubblica amministrazione, delle scuole e, in generale, dell’intero contesto globale.
#redhotcyber #hacking #cti #ai #online #it #cybercrime #cybersecurity #technology #news #cyberthreatintelligence #innovation #privacy
-
@2001x24 wooooooooo
-
ChatGPT scrive la truffa perfetta… ma il truffatore lascia dentro le istruzioni dell’IA
📌 Link all'articolo : https://www.redhotcyber.com/post/chatgpt-scrive-la-truffa-perfetta-ma-il-truffatore-lascia-dentro-le-istruzioni-dellia/
A prima vista, l’email sembrava impeccabile. Un promemoria di pagamento di PagoPA, ben strutturato, con #linguaggio formale, riferimenti al Codice della Strada e persino un bottone blu “Accedi al Portale PagoPA” identico a quello reale.Un capolavoro di ingegneria sociale, che ci invia Paolo Ben, tanto curato da sembrare autentico anche agli occhi più attenti.
Ma poi, come in una scenetta comica, qualcosa si è rotto. Verso la fine del messaggio, dopo i soliti avvisi sulle scadenze e le sanzioni, compare una sezione… surreale.
A cura di Redazione RHC
#redhotcyber #hacking #cti #ai #online #it #cybercrime #cybersecurity #technology #news #cyberthreatintelligence #innovation #privacy
