The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
22
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined evan@cosocial.ca shared this topic on
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@L_Alberto @sport io tifavo per Becker 😜. Ma certo, vendendo questi highlights, l'amore per Musetti cala: Edberg giocava da fondo e tirava passanti di rovescio a una mano e in più era un dio a rete
-
@FraEmme forse dirò una blasfemia, ma un android usato di qualche anno fa (che perlomeno abbia usb-c)? Ci butti sopra f-droid, installi VLC e un'app carina per la musica, e poi via di modalità aereo :>
-
Incredibile pensare quanto delay c'era ai tempi tra un Paese e l'altro per l'uscita dei giochi. In Italia uscirono a fine luglio.
Me lo ricordo perfettamente perché il giorno dopo l'uscita dovevamo partire per le vacanze e io iperventilando spedii mia mamma al centro commerciale per comprarmi Zaffiro.
Invece di mandarmi a fanculo andò, tornò e fece la solita gag del "non ho trovato il gioco che volevi, ti ho preso un'altra cosa" e invece era il gioco che volevo. Che burla garbata.
-
@DieDammerung ora come ora uso il computer per lavoro (che è molto poco aha: leggere documenti, prendere appunti, scrivere) e per cazzeggiare, non ci gioco da un sacco. ti ringrazio intanto!
-
@GustavinoBevilacqua @Dunpiteog sei razzista verso gli italiani!
-
@jannis Hi Jannis! I was trying to post a reply on your latest Ghost blog but I got an error message two times in a row. So a quick heads-up about it.
Anyway, what I wanted to say:
"Thank you for all your hard work on this Jannis!
When I was self-hosting my blog via Ghost-CLI I had magic link spam attacks a couple of times... my Mailgun account was flagged and I couldn't have new signups (once when my Fediverse video was going viral, so I likely lost a lot of potential subscribers).
Part of the reasons why I'm on Magic Pages now is that I get to offset my worry and preoccupation. But I'm sorry you have to deal with this. You got this! And thanks for all you do"
Good luck trying to find a solution to the problem! 🍀
-
-
@fucinafibonacci @sport cavolo! Me l'ero dimenticato! Forse perchè seguivo di più il tennis dei tempi di Lendl, Connors, Borg, McEnroe... eh, l'età...
