The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
22
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined evan@cosocial.ca shared this topic on
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Sul suolo lunare crescono i ceci
https://www.media.inaf.it/2026/03/05/sul-suolo-lunare-crescono-i-ceci/
> Uno studio guidato dall’Università del Texas dimostra che la regolite lunare, in determinate condizioni, può diventare un substrato fertile e consentire la crescita di coltivazioni. Nella ricerca, condotta avvalendosi di suolo lunare si
-
trying to teach my cats 6-7 but they won't listen
-
Combattenti curdi lanciano offensiva contro l’Iran
“Migliaia” di combattenti curdi hanno lanciato un’offensiva di terra contro l’Iran: lo ha dichiarato un funzionario statunitense a Fox News. La notizia è stata rilanciata anche da Axios, che cita sempre fonti Usa, secondo cui l’offensiva è scattata nel nord-ovest del Paese. (Fatto Quotidiano)
-
Bologna: sgombero poliziesco del presidio in difesa del parco al pilastro. 1 persona ferita, 5 fermate.
@anarchia
Sgombero di polizia nella prima mattinata di lunedì 2 marzo contro residenti del quartiere Pilastro di Bologna, in mobilitazione da dicembre 2025 contro gli abbattimenti di alberi nel parco Mitilini Moneta Stefanini, al centro di
-
@SapphicGiraffic @IntrepidVector that's definitely RACK and not SSC
-
Nel delirio coloniale stanno bombardando il Libano. Lì c'è il mio amico Carlo Giorgi, che ora è Don Carlo, ma un tempo era il mio direttore quando facevamo entrambi i giornalisti e io scrivevo per "Terre di Mezzo". E' lui che mi ha insegnato il valore della sofferenza che a volte si rende necessaria quando ti incarti su un pezzo e l'unica cosa che puoi fare è buttarlo via e riscriverlo, ripetendo l'operazione anche più volte se necessario. Ascoltate la sua testimonianza.
-
Sostenibilità e sovranità digitale nella scuola del 21° secolo
Convegno per i 20 anni del Progetto FUSS - Free Upgrade for a Digitally Sustainable School
https://fuss.bz.it/post/2026-04-20_fuss-20th-anniversary/
FUSS non è solo un sistema operativo @linux@diggita.com è un progetto che promuove la sostenibilità digitale e la sovranità tecnologica utilizzando software libero e formati aperti con cui ha reso possibile una didattica innovativa e inclusiva, garantendo che le scuole potessero gestire autonomamente le proprie risorse digitali.
-
Escluso anche il rischio di impatto con la Luna nel 2032 per l'asteroide 2024 YR4. Nuove osservazioni con il telescopio spaziale James Webb hanno permesso di calcolare con più precisione la sua orbita: passerà a circa 21.000 km dal nostro satellite
https://science.nasa.gov/blogs/planetary-defense/2026/03/05/new-nasa-asteroid-observations-eliminate-chance-of-2032-lunar-impact/
