There is a profession known as a digital undertaker.
When clients ask them to remove text, photographs or videos, these professionals track the material down on the internet and delete it—or, failing that, at least keep it out of search results. Requests range from deepfake fabrications to a past someone would rather conceal. Prices start at around KRW 50,000 per post, but as more sites resist easy deletion, the price these days is reportedly “whatever they ask.”
We have believed so firmly that the internet is a place where things are preserved forever that erasing them has become a profession.
But a report published by Pew Research Center in May 2024, “When Online Content Disappears,” highlighted an unexpected side of the internet: 38% of the webpages that existed in 2013 could no longer be opened ten years later.
On one side, people pay someone to delete things; on the other, four out of ten pages disappear without anyone touching them. How can this be?
🧹 The birth of the “right to be forgotten”
In 1998, Spanish lawyer Mario Costeja González had his home seized and put up for auction over social security debts, and the daily newspaper La Vanguardia published the notice.
Costeja paid off the debt soon afterwards, but for more than ten years, the notice kept appearing at the top of the results whenever he searched for his name on Google. In 2010, he lodged a complaint against the newspaper and Google with the Spanish Data Protection Agency (AEPD). The agency left the newspaper notice in place because it had been published in accordance with the law, but ordered Google to remove the link from its search results.
When Google challenged the decision in court, the Spanish court considered that the issue depended on the interpretation of EU law and sought a ruling from the Court of Justice of the European Union. On May 13, 2014, the Court held that search engines, too, had an obligation to remove such search results. The ruling brought the phrase “right to be forgotten” to worldwide attention[1].
[1] Google Spain SL, Google Inc. v. AEPD, Mario Costeja González — Court of Justice of the European Union (CJEU), Grand Chamber, judgment of May 13, 2014, Case C-131/12. Treating a search engine operator as a controller of personal data processing, the Court held that individuals could request the removal of search results that were “inadequate, no longer relevant or excessive,” even when the original article lawfully remained online. Article 17 of the EU General Data Protection Regulation (GDPR), which took effect in 2018, codified this as the “right to erasure (‘right to be forgotten’).”
Korea shared the same concern. In June 2016, the Korea Communications Commission implemented its “Guidelines on Requests to Restrict Access to One’s Own Internet Posts,” which the press called the Korean version of the right to be forgotten. These are voluntary rules allowing people who can no longer delete their own posts to ask service providers, such as portals, to hide them from others.
Since April 2023, the Personal Information Protection Commission has also operated the so-called “Eraser Service,” a programme that requests deletion on behalf of people who posted their own material while they were minors.
For the past decade, then, we have worried about what happens if things cannot be erased. Hardly anyone worried about what happens if they do not remain.
📉 In ten years, 38% disappear
Pew Research Center randomly selected roughly 90,000 webpages for each year from 2013 to 2023—nearly a million in total—from the nonprofit web repository Common Crawl. Then, in October 2023, it tried opening each address again.
What did it find? A quarter of all the pages could no longer be opened. The older the page, the more likely it was to have disappeared: 38% of pages that existed in 2013 were inaccessible, as were roughly 20% of 2021 pages after just two years, and 8% of 2023 pages.
These figures are conservative. Pew Research Center counted a page as “inaccessible” only when an error clearly indicated that the server or page did not exist. It counted addresses that remained live but had different content, and pages that could not be opened because of bot blocking, as “accessible.” So the number of pages that no longer open properly at their original addresses is probably higher.
🔨 How pages die
Pages disappear for many reasons. Sometimes an entire domain disappears, usually when a company closes down or nobody pays the annual domain renewal and server bills.
Yet within that 25%, pages disappearing from sites whose owners were still present accounted for more (16%) than pages lost for those reasons (9%). This usually happens when owners make changes to their sites.
When a website is redesigned or its bulletin board software is replaced, posts may be transferred intact while the address structure changes. If the setting that sends an old address to its new one—a redirect—is omitted, every external link breaks. And for those handling the redesign, this setting often ranks far below getting the new site up and running.
Pages can also disappear through housekeeping rather than redesign. Google began operating its goo.gl URL shortener in 2009, turning long web addresses into short ones such as goo.gl/ABC12x. Then, in July 2024, it announced that it would break all the links from August 25 of the following year. Its reason was that 99% of the links had seen no activity during June 2024.
After objections about links embedded throughout documents, videos and posts, Google changed its policy with just over three weeks to go before the shutdown, deciding to keep some links alive. It would disable only links that nobody had clicked in late 2024. The goo.gl domain therefore still operates, and surviving links lead to their original pages, while the others are broken and show only an error screen.
In many cases, then, webpages disappear not because someone sets out to erase them, but in the course of redesigning sites and cutting costs.
🏚️ A library with only addresses left
In his 2021 Atlantic essay “The Internet Is Rotting,” Harvard Law School’s Jonathan Zittrain likened the web to a system without a library, in which everyone keeps books at home and shares only the addresses. If you visit an address registered in 2015 in the year 2021, it should come as no surprise if the book is gone.
The reason lies in how the internet was built. The academics who designed it had neither the money to build one enormous network themselves, like a telephone or delivery network, nor any intention of making money from it. So they simply established rules for connecting existing networks, and the internet acquired neither an operating company nor a management organisation. Tim Berners-Lee, who built the web on top of it, also made its protocols freely available.
With no fees being charged, there was no need for a company to track who used what, or a central authority overseeing the whole web. A link merely recorded the address of another page, so it could point to a server anywhere in the world, but the owner of that server did not even know who had linked to the page.
Berners-Lee was aware of this, and in his 1998 essay “Cool URIs don’t change,” he urged people not to change their addresses. But he did not build in any mechanism to prevent them from doing so.
Other designs did exist. Xanadu, conceived by Ted Nelson decades before the web, even envisaged “two-way links” that would notify the linked party when someone created a link. The aim was to make links unbreakable from the outset.
But Xanadu tried to take care of everything, including version control and royalty accounting, from the outset, and remained unfinished for decades. Nelson later criticised the web for trivialising hypertext with “one-way, ever-breaking links,” but what survived was the web, imperfect yet operational.
Zittrain calls this approach to designing the internet and web the “Procrastination Principle.” Rather than weighing down a design by trying to solve every possible future problem in advance, it tackles problems when they arise. Sites appearing and disappearing, and pages moving around, were therefore intended features rather than defects.
🫥 A gap in responsibility
The Procrastination Principle succeeded, and a web requiring neither permission nor administrators grew faster than any other network. But the price was a “gap in responsibility for maintaining content on which others rely.” A homeowner need only look after their own books; nobody is obliged to stop books disappearing from someone else’s house.
This gap is usually invisible, but becomes apparent the moment an operator walks away. Paramount eliminated the MTV News division in May 2023, then took down the website a little over a year later, in June 2024. Overnight, addresses for articles accumulated since 1996 all began redirecting to MTV’s home page. Paramount said it retained the articles internally, but they could not be opened from outside.
In January of the same year, the news startup The Messenger, which closed just eight months after its launch, emptied its website within hours of announcing its closure. Only a name and an email address remained.
Surprisingly, news organisations do not put much effort into keeping their articles safe. When Columbia University’s Tow Center surveyed 21 news organisations in 2019, 19 were taking no steps at all to preserve their web articles. They had neither written preservation policies nor even informal practices. In effect, they had left the work to outside institutions such as the Internet Archive and Google. For news organisations constantly pursuing what is happening now, preserving stories already published came second.
In the past, libraries might have filled this gap. But as publishing moved to digital formats, libraries, too, began buying time-limited access rights instead of owning journals and ebooks. When a subscription ends, even material previously received becomes inaccessible. Libraries, too, have ended up with house addresses instead of books.
🧾 Keeping things costs money
Digital preservation researcher David Rosenthal started the LOCKSS preservation programme at Stanford Libraries in 1998 and led it until his retirement in 2017. LOCKSS stands for “Lots Of Copies Keep Stuff Safe.” By having multiple libraries hold and share copies of electronic journals themselves, the project sought to give libraries books again, rather than addresses.
After Pew Research Center’s report appeared, Rosenthal offered several criticisms. One was that Pew had not checked whether copies of the inaccessible pages remained in web archives. Being inaccessible and having disappeared are different things. Indeed, when the Internet Archive checked Pew’s addresses against its Wayback Machine in April 2026, the proportion of “pages that could not be opened” fell from roughly one in four to one in ten once the Wayback Machine was searched as well.
But copies do not create themselves. For more than twenty years, Rosenthal has repeatedly argued that the greatest threat to digital preservation is money, not technology. A lack of funding for preservation is a much bigger threat than files becoming damaged or obsolete. In 2017, he estimated that the world spent less than USD 30 million a year on archiving the web, even though hundreds of billions of dollars had gone into creating that content. His warning is that, at this rate, more than half of what future readers ought to be able to see will not survive.
In Rosenthal’s rule of thumb, half the cost of digital preservation comes at the initial capture stage. Storage accounts for a third, and providing access for the remaining sixth. Once a page disappears, it cannot be captured. What survives therefore ultimately depends on who pays that first cost while the page is still alive.
How, then, are the records currently on the web being managed? And how much effort should we put into preserving them?
In the next instalment, we will look at the gaps that disappearing pages leave in our knowledge, and the efforts being made to resist this loss.
#BetweenTechAndRules #LinkRot #DigitalDecay #RightToBeForgotten #PewResearch #Records