Wikipedia deprecates Archive.today, starts removing archive links(arstechnica.com)
188 points by nobody9999 4 hours ago | 104 comments
- celsoazevedo 3 hours agoI don't see the point in doxing anyone, especially those providing a useful service for the average internet user. Just because you can put some info together, it doesn't mean you should.
With this said, I also disagree with turning everyone that uses archive[.]today into a botnet that DDoS sites. Changing the content of archived pages also raises questions about the authenticity of what we're reading.
The site behaves as if it was infected by some malware and the archived pages can't be trusted. I can see why Wikipedia made this decision.
[-]- jsheard 2 hours agoIt's also kind of ironic that a site whose whole premise is to preserve pages forever, whether the people involved like it or not, is seeking to take down another site because they are involved and don't like it. Live by the sword, etc.
- ddtaylor 2 hours agoDid they actually run the DDoS via a script or was this a case of inserting a link and many users clicked it? They are substantially different IMO[-]
- dunder_cat 2 hours agohttps://news.ycombinator.com/item?id=46624740 has the earliest writeup that I know of. It was running it via a script and intentionally using cache busting techniques to try to increase load on the hosted wordpress infrastructure.[-]
- jsheard 2 hours ago> It was running
It still is, uBlocks default lists are killing the script now but if it's allowed to load then it still tries to hammer the other blog.
[-]- dunder_cat 2 hours agoAh good to know. My pi-hole actually was blocking the blog itself since the ublock site list made its way into one of the blocklists I use. But I've been just avoiding links as much as possible because I didn't want to contribute.
- RobotToaster 60 minutes agoGiven the site is hosted on wordpress.com, who don't charge for bandwidth, it seems to have been completely ineffective.[-]
- ddtaylor 2 hours agoThank you this is exactly the information I was looking for.
"You found the smoking gun!"
- hexagonwin 2 hours agothey silently ran the DDoS script on their captcha page (which is frequently shown to visitors, even when simply viewing and not archiving a new page)
- jMyles 2 hours ago> Changing the content of archived pages also raises questions about the authenticity of what we're reading.
This is absolutely the buried lede of this whole saga, and needs to be the focus of conversation in the coming age.
- wuschel 36 minutes agoThere is an post describing the possibility of an organised campaign against archive.today [1] https://algustionesa.com/the-takedown-campaign-against-archi...
How does the tech behind archive.today work in detail? Is there any information out there that goes beyond the Google AI search reply or this HN thread [2]?
[1] https://algustionesa.com/the-takedown-campaign-against-archi... [2] https://news.ycombinator.com/item?id=42816427
[-]- BoiledCabbage 3 minutes agoAfter reading that link, that seems extremely likely to me.
If true, it could/would mean that someone tried to take archive.today offline recently by planting CSAM and trying to get them shut down. And after that failed, now are trying getting them to DDOS to get them shut down.
Otherwise to me, none of this makes sense to me.
The timing, especially after reading that prior article seems very suspicious.
Again all I can do is speculate with the above, I have no factual knowledge - but at least to me, that makes sense.
[-]- Permit 26 seconds ago> And after that failed, now are trying getting them to DDOS to get them shut down.
What are you basing this off of? My understanding is it’s exactly the opposite. Archive.today is conducting a DDoS, not receiving one. They’re also editing archived pages as I understand it.
- basch 2 hours agoIt seems a lot of people havent heard of it, but I think its worth plugging https://perma.cc/ which is really the appropriate tool for something like Wikipedia to be using to archive pages.[-]
- ronsor 2 hours agoIt costs money beyond 10 links, which means either a paid subscription or institutional affiliation. This is problematic for an encyclopedia anyone can edit, like Wikipedia.[-]
- toomuchtodo 2 hours agoWikimedia could pay, they have an endowment of ~$144M [1] (as of June 30, 2024). Perma.cc has Archive.org and Cloudflare as supporting partners, and their mission is aligned with Wikimedia [2]. It is a natural complementary fit in the preservation ecosystem. You have to pay for DOIs too, for comparison [3] (starting at $275/year and $1/identifier [4] [5]).
With all of this context shared, the Internet Archive is likely meeting this need without issue, to the best of my knowledge.
[1] https://meta.wikimedia.org/wiki/Wikimedia_Endowment
[2] https://perma.cc/about ("Perma.cc was built by Harvard’s Library Innovation Lab and is backed by the power of libraries. We’re both in the forever business: libraries already look after physical and digital materials — now we can do the same for links.")
[3] https://community.crossref.org/t/how-to-get-doi-for-our-jour...
[4] https://www.crossref.org/fees/#annual-membership-fees
[5] https://www.crossref.org/fees/#content-registration-fees
(no affiliation with any entity in scope for this thread)
[-]- RupertSalt 47 minutes agoIf the WMF had a dollar for every proposal to spend Endowment-derived funds, their Endowment would double and they could hire one additional grant-writer[-]
- nine_k 36 minutes agoIf the endowment is invested so that it brings very conservative 3% a year, it means that it brings $4.32M a year. By doubling that, rather many grant writers could be hired.
- ouhamouch 2 hours agoThere are dozen of commercial/enterprise solutions: https://www.g2.com/products/pagefreezer/competitors/alternat...
also the oldest of that kind and rarely mention free https://www.freezepage.com
- jsheard 2 hours agoDoes Wikipedia really need to outsource this? They already do basically everything else in-house, even running their own CDN on bare metal, I'm sure they could spin up an archiver which could be implicitly trusted. Bypassing paywalls would be playing with fire though.[-]
- toomuchtodo 2 hours agoArchive.org is the archiver, rotted links are replaced by Archive.org links with a bot.[-]
- jsheard 2 hours agoYeah for historical links it makes sense to fall back on IAs existing archives, but going forward Wikipedia could take their own snapshots of cited pages and substitute them in if/when the original rots. It would be more reliable than hoping IA grabbed it.[-]
- toomuchtodo 2 hours agoNot opposed, Wikimedia tech folks are very accessible in my experience, ask them to make a GET or POST to https://web.archive.org/save whenever a link is added via the Wiki editing mechanism. Easy peasy. Example CLI tools are https://github.com/palewire/savepagenow and https://github.com/akamhy/waybackpy
Shortcut is to consume the Wikimedia changelog firehose and make these http requests yourself, performing a CDX lookup request to see if a recent snapshot was already taken before issuing a capture request (to be polite to the capture worker queue).
[-]- Gander5739 1 hour agoThis already happens. Every link added to Wikipedia is automatically archived on the wayback machine.[-]
- toomuchtodo 1 hour agoTIL, thank you!
- RupertSalt 19 minutes ago[citation needed]
- jsheard 2 hours agoI didn't know you can just ask IA to grab a page before their crawler gets to it. In that case yeah it would make sense for Wikipedia to ping them automatically.
- ferngodfather 1 hour agoWhy wouldn't Wikipedia just capture and host this themselves? Surely it makes more sense to DIY than to rely on a third party.[-]
- huslage 51 minutes agoWhy would they need to own the archive at all? The archive.org infrastructure is built to do this work already. It's outside of WMF's remit to internally archive all of the data it has links to.
- RupertSalt 2 hours agoSpammers and pirates just got super excited at that plan![-]
- toomuchtodo 2 hours agoThere are various systems in place to defend against them, I recommend against this, poor form against a public good is not welcome.
- ChocMontePy 45 minutes agoI noticed last year that some archived pages are getting altered.
Every Reddit archived page used to have a Reddit username in the top right, but then it disappeared. "Fair enough," I thought. "They want to hide their Reddit username now."
The problem is, they did it retroactively too, removing the username from past captures.
You can see on old Reddit captures where the normal archived page has no username, but when you switch the tab to the Screenshot of the archive it is still there. The screenshot is the original capture and the username has now been removed for the normal webpage version.
When I noticed it, it seemed like such a minor change, but with these latest revelations, it doesn't seem so minor anymore.
- xurukefi 1 hour agoKinda off-topic, but has anyone figured out how archive.today manages to bypass paywalls so reliably? I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous. I figured that they have found an (automated) way to imitate Googlebot really well.[-]
- jsheard 50 minutes ago> I figured that they have found an (automated) way to imitate Googlebot really well.
If a site (or the WAF in front of it) knows what it's doing then you'll never be able to pass as Googlebot, period, because the canonical verification method is a DNS lookup dance which can only succeed if the request came from one of Googlebots dedicated IP addresses. Bingbot is the same.
[-]- xurukefi 40 minutes agoThere are ways to work around this. I've just tested this: I've used the URL inspection tool of Google Search Console to fetch a URL from my website, which I've configured to redirect to a paywalled news article. Turns out the crawler follows that redirect and gives me the full source code of the redirected web site, without any paywall.
That's maybe a bit insane to automate at the scale of archive.today, but I figure they do something along the lines of this. It's a perfect imitation of Googlebot because it is literally Googlebot.
[-]- jsheard 38 minutes agoI'd file that under "doesn't know what they're doing" because the search console uses a totally different user-agent (Google-InspectionTool) and the site is blindly treating it the same as Googlebot :P
Presumably they are just matching on *Google* and calling it a day.
[-]- xurukefi 23 minutes agoSure, but maybe there are other ways to control Googlebot in a similar fashion. Maybe even with a pristine looking User-Agent header.
- Aurornis 24 minutes ago> which I've configured to redirect to a paywalled news article.
Which specific site with a paywall?
- Aurornis 1 hour ago> I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous.
The curious part is that they allow web scraping arbitrary pages on demand. So if a publisher could put in a lot of arbitrary requests to archive their own pages and see them all coming from a single account or small subset of accounts.
I hope they haven't been stealing cookies from actual users through a botnet or something.
[-]- xurukefi 1 hour agoExactly. If I was an admin of a popular news website I would try to archive some articles and look at the access logs in the backend. This cannot be too hard to figure out.
- elzbardico 1 hour ago> which is, of course, ridiculous.
Why? in the world of web scrapping this is pretty common.
[-]- xurukefi 1 hour agoBecause it works too reliably. Imagine what that would entail. Managing thousands of accounts. You would need to ensure to strip the account details form archived peages perfectly. Every time the website changes its code even slightly you are at risk of losing one of your accounts. It would constantly break and would be an absolute nightmare to maintain. I've personally never encountered such a failure on a paywalled news article. archive.today managed to give me a non-paywalled clean version every single time.
Maybe they use accounts for some special sites. But there is definetly some automated generic magic happening that manages to bypass paywalls of news outlets. Probably something Googlebot related, because those websites usually give Google their news pages without a paywall, probably for SEO reasons.
[-]- mikkupikku 43 minutes agoUsing two or more accounts could help you automatically strip account details.[-]
- xurukefi 38 minutes agoThat's actually a really neat idea.
- tonymet 1 hour agoI’m an outsider with experience building crawlers. You can get pretty far with residential proxies and browser fingerprint optimization. Most of the b-tier publishers use RBC and heuristics that can be “worked around” with moderate effort.[-]
- quietsegfault 1 hour ago.. but what about subscription only, paywalled sources?[-]
- tonymet 41 seconds agomany publisher's offer "first one's free".
For those that don't , I would guess archive.today is using malware to piggyback off of subscriptions.
- karel-3d 11 minutes agoArchive.is is now publishing really weird posts on their Tumblr blog, related to the whole thing
https://archive-is.tumblr.com/post/806832066465497088/ladies...
https://archive-is.tumblr.com/post/807584470961111040/it-see...
- tl2do 2 hours agoWhy not show both? Wikipedia could display archive links alongside original sources, clearly labeled so readers know which is which. This preserves access when originals disappear while keeping the primary source as the main reference.[-]
- bawolff 2 hours agoThe objection is to this specific archieve service not archiving in general.
- ranger207 2 hours agoThey generally do. Random example, citation 349 on the page of George Washington: ""A Brief History of GW"[link]. GW Libraries. Archived[link] from the original on September 14, 2019. Retrieved August 19, 2019."[-]
- Gander5739 1 hour agoThis will always be done unless the original url is marked as dead or similar.
- bjourne 1 hour agoFYI, archive.today is NOT the Internet Archive/Wayback Machine.
- RupertSalt 2 hours ago"Non-paywalled" ad-free link to archive: https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment...
- anilakar 1 hour ago> If you want to pretend this never happened – delete your old article and post the new one you have promised. And I will not write “an OSINT investigation” on your Nazi grandfather
From hero to a Kremlin troll in five seconds.
- chrisjj 4 hours ago> an analysis of existing links has shown that most of its uses can be replaced.
Oh? Do tell!
[-]- that_lurker 2 hours agoI would be suprised if archive.today had something that was not in the wayback machine[-]
- chrisjj 2 hours agoArchive.today has just about everything the archived site doesn't want archived. Archive.org doesn't, because it lets sites delete archives.
- bombcar 2 hours agoWayback machine removes archives upon request, so there’s definitely stuff they don’t make publicly available (they may still have it).
- zahlman 1 hour agoTrying to search the Wayback machine almost always gives me their made-up 498 error, and when I do get a result the interface for scrolling through dates is janky at best.
- ribosometronome 2 hours agoAccounts to bypass paywalls? The audacity to do it?[-]
- that_lurker 2 hours agoOh yeah those where a thing. As a public organization they can't really do that.
I personally just don't use websites that paywall important information.
- nobody9999 4 hours ago>> an analysis of existing links has shown that most of its uses can be replaced.
>Oh? Do tell!
They do. In the very next paragraph in fact:
The guidance says editors can remove Archive.today links when the original source is still online and has identical content; replace the archive link so it points to a different archive site, like the Internet Archive, Ghostarchive, or Megalodon; or “change the original source to something that doesn’t need an archive (e.g., a source that was printed on paper)[-]- chrisjj 3 hours agoWell, that's an odd idea of "can be replaced".
> editors can remove Archive.today links when the original source is still online and has identical content
Hopeless. Just begs for alteration.
> a different archive site, like the Internet Archive,
Hopeless. It allows archive tampering by the page's own JS and archive deletion by the domain owner.
> Ghostarchive, or Megalodon
Hopeless. Coverage is insignificant.
[-]- Kim_Bruning 3 hours ago> archive.today
Hopeless. Caught tampering the archive.
The whole situation is not great.
- nobody9999 3 hours agoI just quoted the very next paragraph after the sentence you quoted and asked for clarification.
I did so. You're welcome.
As for the rest, take it up with Jimmy Wiles, not me.
- ChrisArchitect 2 hours agoPreviously Related:
Archive.today is directing a DDoS attack against my blog?
[-]- input_sh 1 hour agoI know I'm arguing with a bot that nobody monitors, but it's already in the fucking post.
- casey2 47 minutes agoAnecdotally I generally see archive.is/archive.today links floating around "stochastic terrorist" sites and other hate cults.
- mrguyorama 3 hours ago>In emails sent to Patokallio after the DDoS began, “Nora” from Archive.today threatened to create a public association between Patokallio’s name and AI porn and to create a gay dating app with Patokallio’s name.
Oh good. That's definitely a reasonable thing to do or think.
The raw sociopathy of some people. Getting doxxed isn't good, but this response is unhinged.
[-]- oytis 57 minutes agoI mean, the admin of archive.today might face jail time if deanonymised, kind of understandable he's nervous. Meanwhile for Patokallio it's just curiosity and clicks
- jMyles 2 hours agoIt's a reminder how fragile and tenuous are the connections between our browser/client outlays, our societal perceptions of online norms, and our laws.
We live at a moment where it's trivially easy to frame possession of an unsavory (or even illegal) number on another person's storage media, without that person even realizing (and possibly, with some WebRTC craftiness and social engineering, even get them to pass on the taboo payload to others).
- ouhamouch 3 hours agoThat was private negotiations, btw, not public statements.
In response to J.P's blog already framed AT as project grown from a carding forum + pushed his speculations onto ArsTechnica, whose parent company just destroyed 12ft and is on to a new victim. The story is full of untold conflicts of interests covered with soap opera around DDoS.
[-]- MBCook 1 hour agoWhy does it matter it was a private communications?
It’s still a threat isn’t it?
- Yossarrian22 2 hours agoCan you elaborate on your point?[-]
- ouhamouch 2 hours agoThe fight is not about where it is shown and not about what, not about "links in Wikipedia", but about whether News Inc will be able to kill AT, as they did with 12FT.[-]
- Yossarrian22 2 hours agoWhat is News Inc? Are they a funder of Wikipedia(I think Wikipedia didn’t have a parent company so they’re not owners)?[-]
- ouhamouch 2 hours agoThey are owner of ArsTechnica which wrote 3rd (or 4th?) article on AT in a row painting it in certain colors.
The article about FBI subpoena that pulled J.P's speculations out of the closet was also in ArsTechnica and by the same author, and that same article explicitly mentioned how they are happy with 12ft down
[-]- Yossarrian22 1 hour ago… Ars is owned by Conde Nast?
- ValveFan6969 41 minutes ago[dead]
- paganel 2 hours agoAt this point Archive.today provides a better service (all things considered) compared to Wikipedia, at least when it comes to current affairs.
- rdiddly 1 hour agoSo toward the end of last year, the FBI was after archive.today, presumably either for keeping track of things the current administration doesn't want tracked, or maybe for the paywall thing (on behalf of rich donors/IP owners). https://gizmodo.com/the-fbi-is-trying-to-unmask-the-registra...
That effort appears to have gone nowhere, so now suddenly archive.today commits reputational suicide? I don't suppose someone could look deeper into this please?
[-]- ndiddy 43 minutes agoThe archive.today operator claims on his blog that this was nothing major: https://lj.rossia.org/users/archive_today/
> Regarding the FBI’s request, my understanding is that they were seeking some form of offline action from us — anything from a witness statement (“Yes, this page was saved at such-and-such a time, and no one has accessed or modified it since”) to operational work involving a specific group of users. These users are not necessarily associates of Epstein; among our users who are particularly wary of the FBI, there are also less frequently mentioned groups, such as environmental activists or right-to-repair advocates.
> Since no one was physically present in the United States at that time, however, the matter did not progress further.
> You already know who turned this request into a full-blown panic about “the FBI accusing the archive and preparing to confiscate everything.”
Not sure who he's talking about there.
- shevy-java 2 hours agoAnyone has a short summary as to who and why Archive.today acted via DDos? Isn't that something done by malicious actors? Or did others misuse Archive.today?[-]
- zeroonetwothree 2 hours agoIf you read the linked article it is discussed
- alsetmusic 4 hours agoI will no longer donate to Wikipedia as long as this is policy.[-]
- jraph 3 hours agoWhy? The decision seems reasonable at first sight.[-]
- chrisjj 3 hours agoSecond sight is advisable in such cases. Fact is, archives are essential to WP integrity and there's no credible alternative to this one.
I see WP is not proposing to run its own.
[-]- mook 3 hours agoWouldn't it be precisely because archives are important that using something known to modify the contents would be avoided?[-]
- huslage 48 minutes agoWhat exactly is credible about archive.today if they are willing to change the archive to meet some desire of the leadership? That's not credible in the least.[-]
- chrisjj 12 minutes agoA lot more credible than archive.org that lets archives be changed and deleted by the archive targets.
What's your better idea?
- that_lurker 2 hours agoThe operators() of archive.today (and the other domains) are doing shadey things and the links are not working so why keep the site around as for example Internet archives waybackmachine works as alternative to it.[-]
- chrisjj 2 hours agoWhat archive.today links are not working?
> Internet archives wayback machine works as alternative to it.
It is appalling insecure. It lets archives be altered by page JS and deleted by the page domain owner.
- throw0101a 2 hours ago> Fact is, archives are essential to WP integrity and there's no credible alternative to this one.
Yes, they are essentional, and that was the main reason for not blacklisting Archive.today. But Archive.today has shown they do not actually provide such a service:
> “If this is true it essentially forces our hand, archive.today would have to go,” another editor replied. “The argument for allowing it has been verifiability, but that of course rests upon the fact the archives are accurate, and the counter to people saying the website cannot be trusted for that has been that there is no record of archived websites themselves being tampered with. If that is no longer the case then the stated reason for the website being reliable for accurate snapshots of sources would no longer be valid.”
How can you trust that the page that Archive.today serves you is an actual archive at this point?
[-]- chrisjj 2 hours ago> If ... If ...
Oh dear.
> How can you trust that the page that Archive.today serves you is an actual archive at this point?
Because no-one shown evidence that it isn't.
[-]
- Jordan-117 2 hours agoDid you not read the article? They not only directed a DDOS against a blogger who crossed them, but altered their own archived snapshots to amplify a smear against them. That completely destroys their trustworthiness and credibility as a source of truth.[-]
- chrisjj 20 minutes agoSure I read it. But I don't believe everything I read on the internet.
- Larrikin 2 hours agoAbout how much had you previously donated over the years?
- selridge 3 hours ago[flagged]
- kmeisthax 2 hours ago[flagged]