The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
22
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined evan@cosocial.ca shared this topic on
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Is something you misread with a funny effect also a #mondegreen? Or what shoud they be called?
(h/t to @afewbugs for this post I just came across https://social.coop/@afewbugs/115918330186198252 about a presto/pesto
-
@kenobit quasi quasi ci provo non sembra male come alternativa e lo dico da non vegano.... devo un attimo capire cosa sia il tahini 😀
-
Ciao amici, mi sapete suggerire un buon lettore mp3? Mi sono rotta il cazzo di usare il telefono.
Info:
- Non sono un'audiofila, mi basta che la musica si senta bene;
- Budget possibilmente non superiore alla 50ina;
- Se potesse riprodurre anche i video sarebbe eccezionale;
- Deve (DEVE!!!!!) potersi collegare alle cuffie bluetooth.Grazie, cari.
-
...questo è un "amico" di quelli "buoni"...
Siria, Al Sharaa si prende l’intero nord est: i curdi si arrendono e firmano l’accordo, finisce l’esperienza del Rojava https://www.ilfattoquotidiano.it/2026/01/19/rojava-fine-curdi-siria-news/8260935/
-
Mia partecipazione a The Bull - Il tuo podcast di finanza personale.
Grazie a Riccardo Spada per l'ospitalitàMario Seminerio: I grandi temi dell'economia globale, dagli USA all'Italia
https://youtu.be/xl50z_CLACM?si=yC4TUtNv7EL_Igga
-
@afreytes threats?
-
@oblomov
I stole that term from this old post: 🤭
https://ohai.social/@dcoderlt/111464029745930779In this case, I was watching popular American posters in mild panic, throwing around memes like “American Century of Humiliation”, because it’s sinking in just how fucked their country is… 😖
-
@zooropa Il caso di Monfalcone è eccezionale appunto per motivi di composizione demografica, ma c’è una tendenza generale da anni in tutta Italia che non può essere ridotta all’influsso degli stranieri. A prescindere dalla religione delle famiglie si tende a ignorare la diffusa attitudine non religiosa di molti studenti e in generale si può parlare legittimamente di laicizzazione a scuola quanto meno perché diminuiscono gli studenti che frequentano l’insegnamento religioso cattolico.
