The web has a memory — and we’ve saved 1 trillion pages of it!
Uncategorized
7
Posts
3
Posters
0
Views
-
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/ -
The web has a memory — and we’ve saved 1 trillion pages of it! 🌐
Join the @internetarchive for The Web We’ve Built, a celebration of the people, stories & technology that preserve our digital world.📅 Oct 22
🎟️ In Person in San Francisco & 🖥️ Virtual worldwide ⤵️
https://blog.archive.org/event/the-web-weve-built-celebrating-1-trillion-web-pages-archived/@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
-
undefined Evan Prodromou shared this topic
-
@internetarchive I notice my websites are still being scraped by your bots even though I manually denied all bots in my domains' robots.txt files a year or two ago. WayBackMachine was great before the age of modern AI, but not now. I don't want companies using your APIs to scrape my archived content to train their AI models. I thought I read if I denied bots in robots.txt, you'd automatically pull related website content from your archives... but my content remains on your servers.
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
-
@jay @internetarchive that's not how robots.txt works. It only prevents new downloads. If you want to ask them to remove already archived data, check here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
-
@jay @internetarchive I don't know if Archive.org makes their archives available for AI training. There's a good how-to from WIRED on how to opt out.
https://www.wired.com/story/how-to-stop-your-data-from-being-used-to-train-ai/
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
-
@jay @internetarchive you may have been thinking of Common Crawl, which does make its crawl data available for LLM training. See here for opt-out info:
https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
-
@evan @internetarchive Thanks for your info, Evan. My main concern is I notice the WayBackMachine has a copy of my websites in 2025, as recent as July, although I denied bots in 2024 or 2023 for all of my domains. I don't believe many companies are adhering to "robots.txt" contents, but Google Search currently is. Unfortunately, the only way to really protect a website today is to require user account login (secure session) to wall-off public content from web scraping.
https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
@jay @internetarchive good luck!
Gli ultimi otto messaggi ricevuti dalla Federazione
-
È l’ora del porno anche per l’AI
Open AI ha bisogno di soldi freschi, e ha scelto la via più semplice.
-
Current* conditions near Bay Village, OH:
-
Current* conditions near Milwaukee, WI:
-
@BauBauS no, Tennessee
-
Current* conditions near Houghton, MI:
-
Current* conditions near Chicago, IL:
-
I don't really think social media actually makes people more social in real life and federated platforms are pretty much the same.
Also here's a few interesting related things I've seen:
The Internet is Worse Than Ever – Now What? | Kurzgesagt Lemmy is making me depressedI also remember reading once that many tech execs don't let their kids use social media or phones.
-
Hackaday Links: October 19, 2025
After a quiet week in the news cycle, surveillance concern Flock jumped right back in with both feet, announcing a strategic partnership with Amazon’s Ring to integrate that company’s network of doorbell cameras into one all-seeing digital panopticon. Previously, we’d covered both Flock’s “UAVs as a service” model for combating retail theft from above, as well as the somewhat grassroots effort to fight back at the company’s wide-ranging network of license plate reader cameras. The Ring deal is not quite as “in your face” as drones chasing shoplifters, but it’s perhaps a bit more alarming, as it gives U.S. law enforcement agencies easy access to the Ring Community Request program directly through the Flock software that they (probably) already use.
In the event of a crime, police can use the integration to easily blast out a request for footage to Ring owners in the vicinity. The request is supposed to contain details of the alleged crime, including its time and location. Owners are free to comply with the request or ignore it at their discretion, and there is supposed to be no way for the police to track who declines a request, theoretically eliminating the potential for retaliation. On the one hand, we see the benefit of ready access to footage that might be needed quickly to catch a suspect or solve a crime. But on the other hand, it just seems like there’s nowhere you can go anymore where there isn’t a camera ready to be used against you.
Remember “Solar Freakin’ Roadways”? We sure do, and even though the idea of reconfigurable self-powered paving tiles didn’t seem to be going anywhere the last time we checked, we always did like the idea of self-lighted roads. But pluggable modules with solar panels and LEDs built to withstand being run over by cars and trucks and the rigors of Mother Nature might be a more complicated way to go about it than, say, painting the road with glow-in-the-dark paint. Unfortunately, that doesn’t seem to work much better, as revealed by a recent trial in Malaysia.
Admittedly, the trail was limited; a mere 245 meters of rural roadway received the phosphorescent paint markings. The paint absorbed light during the day and emitted a soft green glow at night, to the delight of drivers who praised its visibility. For a while, at least, because within a year or so, the paint had lost most of its power. At 20 times the cost of normal roadway marking paint, it wasn’t cheap either, probably thanks to the europium-doped strontium aluminate compounds that gave it its glow. It’s too bad the trial didn’t work out, because the markings looked fantastic.
You’ve heard about Power-over-Ethernet, but how about Power-over-Skin? The idea comes from a group at Carnegie-Mellon University, and is aimed at powering a network of battery-free wearables using the surface of the skin as the only conductor. To make it work, the researchers use a 40-MHz RF transmitter that’s kept in the user’s pocket and couples with the skin even through layers of fabric. Devices on the user’s skin can pick up the signal through a tuned circuit and rectify it to power a microcontroller. The 40-MHz frequency was selected in part because it offers head-to-toe coverage, but also because it’s too high to cause potentially painful “muscle activation” or local heating. Talk about your skin effect!
If you currently crave a trip to one of the many national parks or monuments in the United States, you might want to hold off until the government shutdown is resolved. Until then, you’ll have to be content with virtual tours such as this one for the Hanford B Reactor site, which, along with Los Alamos in New Mexico and Oakridge in Tennessee, is part of the Manhattan Project National Historic Park. The virtual tour is pretty cool, and everything inside the reactor building, from the sickly green paint to the mid-century furniture, seems to have been restored to what it would have looked like in the 1940s. The Fallout-esque control room is a treat, too. But alas, there’s no virtual gift shop on the way out.
And finally, a bit of electronics history with this fascinating video about how early home computers glitched their way into displaying color. On paper, the video interface on the TRS-80 Color Computer was only capable of generating a monochrome signal. But according to Coco Town, a carefully crafted monochrome signal could convince an analog NTSC television to display not only white pixels but also red and blue, or blue and red, depending on when you hit the reset button. It’s an interesting trip through the details of color TV, and the way that the standard was exploited to make color graphics on the cheap is truly a hack worth understanding. Enjoy!
Post suggeriti
-
-
-
🐹🎶 In 1998, the web danced to hamsters.'nHampsterdance began as a lighthearted bet between Canadian art student Deidre LaCarte & her sister: who could get the most traffic with a single webpage?
Uncategorized
1
-
Before TikTok, before Instagram, there was MySpace: the internet’s hub for music, culture, & eye-frying HTML.
Uncategorized
4
