The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
20
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined evan@cosocial.ca shared this topic on
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
🕙 Hurricane's NTP 🕚
Their "NTP Stratum 1" clocks are part of the global internet's network-time infrastructure which assists in defining reality (for most).
They've been rad since the 90s... so (to no surprise) just look at the IPv4 addresses they have allocated for two of their three NTP clusters.
clock.sjc.he.net (216.218.254.202)
clock.fmt.he.net (216.218.192.202)The glorious x.x.192.x and x.x.254.x 💗
-
@chrisjrn an interesting challenge! I can think of at least three meanings:
- attached to the laptop using adhesive
- lying on top of the laptop but not stuck to it
- a digital design for a sticker stored on the laptop's hard driveThe first is most common, I think, and the last two would be kind of gotcha, technically correct answers.
Were you thinking of one of these, or something else?
-
@aeva still need to make this more modular and add procedural sequencing
-
@memeorandum why did they even bother to announce this
-
Trump endorses identical twin of retiring Rep. Troy Nehls to fill his seat (Faith Wardwell/Politico)
https://www.politico.com/live-updates/2025/12/04/congress/trump-endorses-trever-nehls-00677128
http://www.memeorandum.com/251204/p95#a251204p95
-
@lritter this slaps. I'm looking forward to the full version of this song :3
-
@evan it depends on what you mean by "on"
-
Anti-Fascist Collective 👊🏼👊🏿 (@wtafc.bsky.social)
https://bsky.app/profile/wtafc.bsky.social/post/3m76s6ehy4k2o
