Unneeded materials from other depository libraries can now be transferred to the Internet Archive. Under 44 USC § 1912, depository libraries may dispose of outdated material, but must first offer to transfer to nearby depository institutions.
Stuff like the printed tax code of 1965 or Borland Pascal 1992 manual. Once you have it digitized it's a waste of space for libraries to have a physical copy because basically no one needs it.
Libraries are constantly bringing in new materials and very few are capable of constantly increasing in size to match. I believe national libraries like the Library of Congress tend not to weed, but they do have to offload material to satellite locations and storage facilities.
I'm having trouble finding what this means. Does IA now have new obligations, or gets new information, or something else, or all of the above?
The submission says:
> These records account for “millions and millions of pages” that can take up entire floors of public libraries, Kahle said. San Diego’s public library gave up its federal depository status in 2020 because its government documents took up so much space and often went unused. [...] The GPO [...] has ramped up efforts to digitize the Federal Depository Library Program.
Does IA now have to store floors upon floors of paper copies of information, at least until it got digitized? Or are they now merely obliged to host the digital materials insofar as they already exist? That sounds like what they are doing already for the whole web, and also apparently since 2022 when they started "Democracy’s Library, a free online compendium of government research and publications", just that now they're legally obliged to do this or something?
What I find on doi.gov[1] is "The mission of Federal depository libraries is to provide local, free access to information from the Federal government" and nothing really further on what this concretely means. Sounds like just an obligation though?
What I find on gpo.gov[2] is "The Federal Depository Library Program [ensures] that the American public has access to Government information in depository libraries". Could mean anything. The program ensures that, but let's assume that means the designated libraries ensure that, so then do these libraries get extra info that the public doesn't get (but in order to disseminate them to the public)? Makes no sense either
The GPO page and the submission also say that "Members of Congress may designate up to two qualified libraries." Did they get picked and now it's IA's obligation, or did IA ask for this? What do they get out of it?
As I understand it, it’s voluntary and like the government document version of the Twitter firehose. Direct access to all published government documents as they are created.
So like a semi-public endpoint, that the libraries can access but not the general public, and then the public should get it from the libraries? Seems roundabout but it fits with what I've read so far
> "...in response to the enclosed letter I received from the Founder and Digital Librarian of the Internet Archive, Mr. Brewster Kahle, I am designating the Internet Archive as a federal depository library in California."
Which seems a lot more agreeable than unilateral designation (which is also how I initially read this).
They already remove “inconvenient” webpages on the Wayback Machine if someone asks nicely enough. If I remember correctly, if you use it to save a software company’s documentation pages or evidence of something embarrassing like a potential data breach, they could remove it if the company asks. I think Oracle might have done something like this before.
If anyone reading knows an easy way to download and mirror IA pages please make it easier to find. A bot told me they offer downloads of the underlying WARC files but I could not find it
> A bot told me they offer downloads of the underlying WARC files but I could not find it
The "bot" is wrong. Most of the crawl data used by the Internet Archive, particularly the Alexa crawls, isn't publicly accessible. (This is because some of it includes archived pages which have since been suppressed by the site owner - removing those pages from the archived crawl data isn't practical.)
It's a one way street. This provides more access to materials held by the federal gov for ingest into IA's storage system. Bit of a policy interconnect, if you will. Reminder to donate to the Archive.
I've heard it has already happened. Specifically the internet archive removed vidoes of the TempleOS developer Terry Davis' live streams because of problematic content.
If the internet archive is already curated for content then yeah there is a 100% chance that there will be more curation of content.
I thought Archive just removed access, but kept the content. I know that from a user perspective that is a distinction without a difference, but for posterity it matters.
Does anyone have any facts/citations on if this is a myth/coping mechanism I created, or reality?
“2023 The Internet Archive, a non-profit research library, makes use of internal processes and tools, including human review and hash-matching, as well as reports from external parties to identify, disable access to, and limit the reappearance of illegal and/or proscribed violent extremist material on archive.org”
This is not to disparage the tremendous work done and being done by the IA, it's more of me lamenting the trend of our society and societies to mentally babysit people lest their mind gets exposed to something bad, with the implicit assumption that adult humans can't be trusted to see some stupid bs and react with "that was some stupid bs. I am moving it into the stupid bs bucket of things I know about".
In the past, they stated that they do not delete anything. Those posts have vanished, possibly due to the onslaught of lawsuits and discovery. Specific to Kiwi Farms (and some other material) I was able to locate it by poking around on the site. Even the material that the Judge ruled against in the Hachette lawsuit remains online and available to people with print disabilities.
Sounds outdated! My library doesnt even require me to walk in anymore, they send any book I want to read or listen to straight to my phone, and if they don't have it I can request they acquire it and send it to me for free
Sounds VERY Communist, or Socialist, or some other scary thing. Are you sure it's legal? Why, the AUTHORS and PUBLISHERS are being denied the revenues they would get if you would buy the book; or at least rent it. So, are libraries theft of Authors' and Publishers' renumeration? (And, to think, the richest man in the world at the time, Andrew Carnegie, endowed so many Libraries!)
Wait until you hear about my private library that resides on a Synology NAS. I can access it from anywhere in the world, on any device, and it's filled with whatever books I can bother to decide that I want that title. I have about 20,000 (not counting periodicals) all carefully curated and retail quality. I even got rid of those annoying generic Bantam Press covers and replaced them with the high-res stuff off the publisher's site.
Not sure what the appeal of the public library is, when you can have your own.
Back in the days when things were sane my first thought reading this headline would have been: Nice, that‘s sounds official and important. Nowadays my first thought is: Wait, does this mean Trump can mess around with this?
My take on this is that in desperation to become a real library despite Kahle's radical hatred of content creators, Kahle will end up dragging the legislative narrative in a direction that takes down real libraries with him. He will almost certainly broadcast his status as a federal depository library as part of his defenses in his numerous lawsuits.
One selfish man unwilling to recognize he is doing more harm than good.
"radical hatred of content creators" is a very harsh and specific allegation. I wasn't aware that Kahle was considered such a bad actor. I did some googling and wikipedia-ing and can't see much that supports that claim. I am very open minded to the nuances of IP rights vs information-wants-to-be-free so I'd love to hear more details about your position particularly as it relates to the federal depository designation.
Making every book on the site available for unlimited download, not just rare things but contemporary best sellers, did huge reputational damage. Following it up by claiming he was saving scratchy old 78 RPM records, but in the process also making LPs from Paul McCartney and Jimi Hendrix available, continued the trend.
Tweeting out promotional links to the pages with those materials, while asking for donations on the top of the page? Well, I don't know if that's contempt for artists or just lack of common sense. But when they ask you to take down the material and you refuse...
The depository thing is a distraction. And they do have a habit of sensationalizing things in blog posts. So I understand where that commenter is coming from. Internet Archive is under attack from many sides but much of it is self-inflicted.
Libraries make «contemporary best sellers» and «LPs from Paul McCartney and Jimi Hendrix» freely available. You call it «reputational damage», others may call it "advancing demands over rights", "stirring a stagnating reality in view of effective progress" (with reference to dematerialization), "pushing a debate" (about where we want to go societally".
It is unwise to push these latter points with the outmost care without having awakened the masses and clarified your stances to decisors - it is unwise to be "right" in front of the immature. But the reputation damage remains about wisdom, not about pride.
Physical libraries act under a different set of rules and those were already made well known to Brewster as part of the Hachette lawsuit.
For music, the Music Modernization Act set up a statutory process for making things available, even downloadable. Brewster and others celebrated the measure in blog posts and speaking gigs. Then didn't follow the process, didn't honor polite requests to stop, then got sued for $700 million.
Previously they did some seriously stupid things in their implementation of Controlled Digital Lending, and got the whole concept killed. Not even a debate, just destroyed on summary judgement without even a trial. This set the future many of us want back decades, and ruined a lot of proper efforts that were run much better than the well-intentioned but undermanaged Internet Archive.
Combined with them giving the finger to the fairly innovative and progressive music act, this caused damage not only to reputations, but also the culture.
Regarding copyright basics, we're likely to agree on many positions, including some radical ones. But Internet Archive cannot be a long-term archive, an activist organization, and an open library. There are different laws, risk profiles, and financial/management requirements for each.
And you can't beg people for donations to "save the internet" then set it all on fire to save a bunch of old records that already existed at the Library of Congress. Or act surprised that just because you scan them, it doesn't mean you can then make them available for unlimited download without permission. Again, archives behave differently from libraries. Although it's annoying to tech people, there are good reasons for it.
Brewster likes his honorary library status and degree but he and the site violate the majority of the librarian code of ethics. https://www.ala.org/tools/ethics
They were already recognized by the state of California as a library, and have received federal funds for infrastructure under that designation. They’ve also been accepted into consortiums made up of other libraries in the US. Whether you believe they’re a library is immaterial.
Brewster has a friend in a state senator and he's trying to do what he can to preserve his section 108 privileges. He's removed over a million items in the past year after being repeatedly sued for copyright infringement, and leaked millions of private communications with patrons including passports and driver licenses. That's the undercurrent here.
Egos aside, the goal isn't to be a library: it's providing access to knowledge. But when your site is on the blocklist at public library terminals because you keep getting flagged for copyright violations and child pornography, maybe you're not on the path.
archive.org: "Our mission is to provide Universal Access to All Knowledge."
As the name implies, Internet Archive started as an archive. Which is very different from a library.
Running an archive is not particularly fun, and it is very expensive, and you cannot monetize it without having rights to the things you're archiving. They've never offered research services or grants, and yet the monthly bill and tech debt just keeps growing. Last year's hacks showed the state of things, and they leaked patron information and even passports and drivers licenses.
They tried to be a library but didn't follow the law. Hell, they even tried to be a bank at one point and got spanked hard by the feds there, too.
With the $700 million lawsuit over old records it became clear that the whole thing is little more than a catch all for things that Brewster Kahle finds interesting. He's got money and seems like a kind guy. But it's not a well-run organization and he's at retirement age without having put much of a dent in that mission.
Unneeded materials from other depository libraries can now be transferred to the Internet Archive. Under 44 USC § 1912, depository libraries may dispose of outdated material, but must first offer to transfer to nearby depository institutions.
What is "outdated material" for a library? Isn't that precisely where you go to find "outdated material" is a library's archives?
Stuff like the printed tax code of 1965 or Borland Pascal 1992 manual. Once you have it digitized it's a waste of space for libraries to have a physical copy because basically no one needs it.
In other words, a depository is cold storage
Libraries have an entire concept of weeding, and numerous criteria for doing so: https://en.m.wikipedia.org/wiki/Weeding_(library)
Libraries are constantly bringing in new materials and very few are capable of constantly increasing in size to match. I believe national libraries like the Library of Congress tend not to weed, but they do have to offload material to satellite locations and storage facilities.
I'm having trouble finding what this means. Does IA now have new obligations, or gets new information, or something else, or all of the above?
The submission says:
> These records account for “millions and millions of pages” that can take up entire floors of public libraries, Kahle said. San Diego’s public library gave up its federal depository status in 2020 because its government documents took up so much space and often went unused. [...] The GPO [...] has ramped up efforts to digitize the Federal Depository Library Program.
Does IA now have to store floors upon floors of paper copies of information, at least until it got digitized? Or are they now merely obliged to host the digital materials insofar as they already exist? That sounds like what they are doing already for the whole web, and also apparently since 2022 when they started "Democracy’s Library, a free online compendium of government research and publications", just that now they're legally obliged to do this or something?
What I find on doi.gov[1] is "The mission of Federal depository libraries is to provide local, free access to information from the Federal government" and nothing really further on what this concretely means. Sounds like just an obligation though?
What I find on gpo.gov[2] is "The Federal Depository Library Program [ensures] that the American public has access to Government information in depository libraries". Could mean anything. The program ensures that, but let's assume that means the designated libraries ensure that, so then do these libraries get extra info that the public doesn't get (but in order to disseminate them to the public)? Makes no sense either
The GPO page and the submission also say that "Members of Congress may designate up to two qualified libraries." Did they get picked and now it's IA's obligation, or did IA ask for this? What do they get out of it?
[1] https://www.doi.gov/library/collections/federal-documents
[2] https://www.gpo.gov/how-to-work-with-us/agency/services-for-...
As I understand it, it’s voluntary and like the government document version of the Twitter firehose. Direct access to all published government documents as they are created.
So like a semi-public endpoint, that the libraries can access but not the general public, and then the public should get it from the libraries? Seems roundabout but it fits with what I've read so far
Did you read the accompanying article? https://www.kqed.org/news/12049420/sf-based-internet-archive...
I quoted from there, so yes
"California Sen. Alex Padilla made the designation in a letter sent Thursday to the Government Publishing Office"
What does this mean. U.S. Senators can unilaterally designate federal depositories?
It sounds like it was at the request of IA:
> "...in response to the enclosed letter I received from the Founder and Digital Librarian of the Internet Archive, Mr. Brewster Kahle, I am designating the Internet Archive as a federal depository library in California."
Which seems a lot more agreeable than unilateral designation (which is also how I initially read this).
a small scroll:
>Under federal law, members of Congress can designate up to two qualified libraries for federal depository status.
If only they’d hire some more people to get their search function working.
Is it likely that the Executive Branch would try to exert control over it to remove "inconvenient" data?
They already remove “inconvenient” webpages on the Wayback Machine if someone asks nicely enough. If I remember correctly, if you use it to save a software company’s documentation pages or evidence of something embarrassing like a potential data breach, they could remove it if the company asks. I think Oracle might have done something like this before.
A community-maintained list collecting examples of such exclusions: https://wiki.archiveteam.org/index.php/List_of_websites_excl...
Can't say I blame them, I wouldn't want to go up against Oracle's lawyers either.
If anyone reading knows an easy way to download and mirror IA pages please make it easier to find. A bot told me they offer downloads of the underlying WARC files but I could not find it
> A bot told me they offer downloads of the underlying WARC files but I could not find it
The "bot" is wrong. Most of the crawl data used by the Internet Archive, particularly the Alexa crawls, isn't publicly accessible. (This is because some of it includes archived pages which have since been suppressed by the site owner - removing those pages from the archived crawl data isn't practical.)
https://archive.org/details/alexacrawls
Common Crawl data is public, but less comprehensive than IA - https://commoncrawl.org/
There are utilities to help, waybackpack comes to mind, but I haven't looked in a while. https://github.com/jsvine/waybackpack
I used wayback-machine-downloader, I think you need one of the forks to make it work though.
https://github.com/hartator/wayback-machine-downloader
They locked away most .warc files due to the AI harvesting crunch.
It's a one way street. This provides more access to materials held by the federal gov for ingest into IA's storage system. Bit of a policy interconnect, if you will. Reminder to donate to the Archive.
If you see a bank that says "federally chartered" or "federal deposit insurance corporation", stay clear!
Doubtful. They’re not part of the government so the 1st amendment applies.
I've heard it has already happened. Specifically the internet archive removed vidoes of the TempleOS developer Terry Davis' live streams because of problematic content.
If the internet archive is already curated for content then yeah there is a 100% chance that there will be more curation of content.
Kiwifarms as well. They are a bit of a pushover when it comes to controversy.
I thought Archive just removed access, but kept the content. I know that from a user perspective that is a distinction without a difference, but for posterity it matters.
Does anyone have any facts/citations on if this is a myth/coping mechanism I created, or reality?
“2023 The Internet Archive, a non-profit research library, makes use of internal processes and tools, including human review and hash-matching, as well as reports from external parties to identify, disable access to, and limit the reappearance of illegal and/or proscribed violent extremist material on archive.org”
https://help.archive.org/help/tco-transparency-reports/
I wonder how many gems like this https://archive.org/details/youtube-moXX8lbnmHs that could have been saved have been lost. (Obviously this one is saved, for now.)
This is not to disparage the tremendous work done and being done by the IA, it's more of me lamenting the trend of our society and societies to mentally babysit people lest their mind gets exposed to something bad, with the implicit assumption that adult humans can't be trusted to see some stupid bs and react with "that was some stupid bs. I am moving it into the stupid bs bucket of things I know about".
In the past, they stated that they do not delete anything. Those posts have vanished, possibly due to the onslaught of lawsuits and discovery. Specific to Kiwi Farms (and some other material) I was able to locate it by poking around on the site. Even the material that the Judge ruled against in the Hachette lawsuit remains online and available to people with print disabilities.
given this is already happening with many other taxpayer funded datasets, will pretty on brand with this group
I mean, what would they do to exert control? Remove their federal depository status?
Imagine having to delete their 100PB of warez.
Wait til you hear about my local library. You can walk in and read or borrow any book without paying!
Sounds outdated! My library doesnt even require me to walk in anymore, they send any book I want to read or listen to straight to my phone, and if they don't have it I can request they acquire it and send it to me for free
I wish my public library was free...
Sounds VERY Communist, or Socialist, or some other scary thing. Are you sure it's legal? Why, the AUTHORS and PUBLISHERS are being denied the revenues they would get if you would buy the book; or at least rent it. So, are libraries theft of Authors' and Publishers' renumeration? (And, to think, the richest man in the world at the time, Andrew Carnegie, endowed so many Libraries!)
Your shift key seems to keep getting stuck.
Wait until you hear about my private library that resides on a Synology NAS. I can access it from anywhere in the world, on any device, and it's filled with whatever books I can bother to decide that I want that title. I have about 20,000 (not counting periodicals) all carefully curated and retail quality. I even got rid of those annoying generic Bantam Press covers and replaced them with the high-res stuff off the publisher's site.
Not sure what the appeal of the public library is, when you can have your own.
[dead]
Official post: https://blog.archive.org/2025/07/24/internet-archive-designa... (https://news.ycombinator.com/item?id=44677535)
do we need an internet archive, archive now?
I’m down to join this effort if anyone else wants to retire row in the same direction.
Back in the days when things were sane my first thought reading this headline would have been: Nice, that‘s sounds official and important. Nowadays my first thought is: Wait, does this mean Trump can mess around with this?
I’m still excited about it.
I hope that all of the world libraries join with the internet archive into a global cooperative.
I also hope there is a secret sub-basement in a different dimension that contains powerful artifacts, guarded by a master librarian.
A man can dream can’t he?
That says more about you than it says about the times, I'm afraid. Your first thought should still be the former, not the latter.
Wouldn't be the first time in history a falling empire tries to control the narrative by burning books
Read the news, bro.
yes trump is on the computer messing around with this
My take on this is that in desperation to become a real library despite Kahle's radical hatred of content creators, Kahle will end up dragging the legislative narrative in a direction that takes down real libraries with him. He will almost certainly broadcast his status as a federal depository library as part of his defenses in his numerous lawsuits.
One selfish man unwilling to recognize he is doing more harm than good.
"radical hatred of content creators" is a very harsh and specific allegation. I wasn't aware that Kahle was considered such a bad actor. I did some googling and wikipedia-ing and can't see much that supports that claim. I am very open minded to the nuances of IP rights vs information-wants-to-be-free so I'd love to hear more details about your position particularly as it relates to the federal depository designation.
Making every book on the site available for unlimited download, not just rare things but contemporary best sellers, did huge reputational damage. Following it up by claiming he was saving scratchy old 78 RPM records, but in the process also making LPs from Paul McCartney and Jimi Hendrix available, continued the trend.
Tweeting out promotional links to the pages with those materials, while asking for donations on the top of the page? Well, I don't know if that's contempt for artists or just lack of common sense. But when they ask you to take down the material and you refuse...
The depository thing is a distraction. And they do have a habit of sensationalizing things in blog posts. So I understand where that commenter is coming from. Internet Archive is under attack from many sides but much of it is self-inflicted.
Libraries make «contemporary best sellers» and «LPs from Paul McCartney and Jimi Hendrix» freely available. You call it «reputational damage», others may call it "advancing demands over rights", "stirring a stagnating reality in view of effective progress" (with reference to dematerialization), "pushing a debate" (about where we want to go societally".
It is unwise to push these latter points with the outmost care without having awakened the masses and clarified your stances to decisors - it is unwise to be "right" in front of the immature. But the reputation damage remains about wisdom, not about pride.
Physical libraries act under a different set of rules and those were already made well known to Brewster as part of the Hachette lawsuit.
For music, the Music Modernization Act set up a statutory process for making things available, even downloadable. Brewster and others celebrated the measure in blog posts and speaking gigs. Then didn't follow the process, didn't honor polite requests to stop, then got sued for $700 million.
Previously they did some seriously stupid things in their implementation of Controlled Digital Lending, and got the whole concept killed. Not even a debate, just destroyed on summary judgement without even a trial. This set the future many of us want back decades, and ruined a lot of proper efforts that were run much better than the well-intentioned but undermanaged Internet Archive.
Combined with them giving the finger to the fairly innovative and progressive music act, this caused damage not only to reputations, but also the culture.
Regarding copyright basics, we're likely to agree on many positions, including some radical ones. But Internet Archive cannot be a long-term archive, an activist organization, and an open library. There are different laws, risk profiles, and financial/management requirements for each.
And you can't beg people for donations to "save the internet" then set it all on fire to save a bunch of old records that already existed at the Library of Congress. Or act surprised that just because you scan them, it doesn't mean you can then make them available for unlimited download without permission. Again, archives behave differently from libraries. Although it's annoying to tech people, there are good reasons for it.
Brewster likes his honorary library status and degree but he and the site violate the majority of the librarian code of ethics. https://www.ala.org/tools/ethics
They were already recognized by the state of California as a library, and have received federal funds for infrastructure under that designation. They’ve also been accepted into consortiums made up of other libraries in the US. Whether you believe they’re a library is immaterial.
A federal Judge also ruled that "IA does not perform the traditional functions of a library."
https://publishers.org/wp-content/uploads/2024/09/2024.09.04...
Brewster has a friend in a state senator and he's trying to do what he can to preserve his section 108 privileges. He's removed over a million items in the past year after being repeatedly sued for copyright infringement, and leaked millions of private communications with patrons including passports and driver licenses. That's the undercurrent here.
Egos aside, the goal isn't to be a library: it's providing access to knowledge. But when your site is on the blocklist at public library terminals because you keep getting flagged for copyright violations and child pornography, maybe you're not on the path.
Unclear expression, since the goal of a library is «providing access to knowledge». Maybe the point is about the future of those services.
archive.org: "Our mission is to provide Universal Access to All Knowledge."
As the name implies, Internet Archive started as an archive. Which is very different from a library.
Running an archive is not particularly fun, and it is very expensive, and you cannot monetize it without having rights to the things you're archiving. They've never offered research services or grants, and yet the monthly bill and tech debt just keeps growing. Last year's hacks showed the state of things, and they leaked patron information and even passports and drivers licenses.
They tried to be a library but didn't follow the law. Hell, they even tried to be a bank at one point and got spanked hard by the feds there, too.
https://ncua.gov/newsroom/press-release/2016/internet-archiv...
With the $700 million lawsuit over old records it became clear that the whole thing is little more than a catch all for things that Brewster Kahle finds interesting. He's got money and seems like a kind guy. But it's not a well-run organization and he's at retirement age without having put much of a dent in that mission.