The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
20
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined evan@cosocial.ca shared this topic on
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
La più vecchia fotografia conosciuta della città di Milano è del 1846. Lo scatto è di Calvert Richard Jones, fotografo gallese che in Italia documentò luoghi come il Duomo di Milano e il Rebecchino, l’antico isolato che occupava parte dell’attuale Piazza del Duomo e che verrà demolito nel 1875 . Al Duomo mancavano ancora alcune guglie e aveva ancora le porte in legno poi sostituite con quelle in bronzo.
Foto conservata presso Metropolitan Museum of Art a New York.
#milano #photography
-
@guardaminfaccia da mettere su maglietta
-
RE: https://mastodon.gamedev.place/@eniko/115651345517988440
Just 24 hours left to go on this bundle https://itch.io/s/169922/kitsune-games-bundle
-
RE: https://mastodon.social/@arstechnica/115662598646798504
Oh, grazie, eh…
-
Il lancio di #ProtonSheets è una notizia enorme! Finalmente i fogli di calcolo con encryption end to end nella suite cloud di #Proton. Chi ha ancora bisogno di Google? 🙂
https://www.ilsoftware.it/proton-sheets-il-nuovo-foglio-di-calcolo-criptato-per-proteggere-i-dati-aziendali/
-
@repubblica @politica-la-repubblica-repubblica sedicenti propal aggrediscono sito neonazista alla fiera del libro di roma?...ah...no...niente
-
-
@diegor @news @macfranc @luca La mia esperienza con la banca: posso aver registrati al massimo 3 dispositivi. Ma per farli riconoscere dall'app, devo comunque farmi chiamare (securecall) e digitare un pin che precedentemente gli ho dato.
Dopodiché ho la possibilità di farmi inviare o la notifica push per fare le operazioni da pc, o la secure call.
Poi certo, se mi arriva una secure call che non ho richiesto e mi chiede di digitare il pin dispositivo, non lo digito affatto.
Grazie per avermi sbloccato il ricordo (poco piacevole) del token per ciechi delle poste! Per farlo partire, dovevi addirittura installare una smartcard che ti davano le poste.
