The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed.
LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews.
This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid.
I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,
- no open weights
- can’t use claude to research AI
- train on everyone else’s IP and sell it back to them
- 8 regulatory capture attempts and counting
- so controlling they are the only US company blacklisted by the US government
This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.
Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
> ACR uses audio fingerprinting technology using the TV’s internal audio processor (not a speaker) to identify content and does not collect screenshots, screen recordings, video recordings, voice recordings, or other audio recordings from the TV.
So they are claiming that ACR is done entirely through audio processing, and no visual data from the screen itself is used? That...doesn't seem plausible.
This paper from 2024 investigated both Samsung and LG and found that they both capture screen images for ACR, and send the resulting hash (not the raw content) back to the manufacturer for identification.
We're going to have to get saavy in pulling apart the distinction between "I look at everything and then send a nice summary back to my manufacturer" and "I send everything back to my manufacturer". As AI gets cheaper and more efficient, it'll be baked into everything, and will distill signal at the edge and send the good bits back to central command. I fear the legalese in the ToS will obscure this new way of "not collecting" user information.
I can't believe we're finding out about this from 3p researchers again (but nice job on the investigation!). OpenAI had two great opportunities to disclose this. The HF incident report, and in response to the German Wiki issue.
It seems impossible to believe they didn't know. This must be the same training run the HF incident was about, and this should have lit up like a Christmas tree in the investigation. How many more incidents do they know about and didn't disclose?
Note that their statement could also be true if they record 99% of the time. We need to start calling companies out for meaningless weasel statements like "We don't record continuously".
The majority of the problem is also simply the ability to record. Putting a remote control listening hardware on a device that runs a plethora of 3rd party apps and with full connection to the internet means that even if LG isn't controlling that mic, someone else will be.
As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification
I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites.
There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.
> The agents clearly regarded what they were doing as hacking.
To butcher the quote about Oracle:
Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doing as hacking (your hand off)' -- lawnmower doesn't give a shit about your hand, lawnmower can't regard anything. Don't anthropomorphize the lawnmower. Don't fall into that trap about LLMs.
---
In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. Which a lot of the time seems to be the default. They also seem to be very adapt at breaking out of sandboxes, probably due to RL selecting for the ability to break out of a sandbox/permission issue to complete a task - we've all seen agents try 10 different ways of editing via obscure bash because their edit tool didn't give them permission to edit the file outside of their working directory, this is the exact same behaviour taken to the next level. Why would autocomplete know the moral difference between breaking out of its working dir and hacking a package manager?
It's misaligned because everyone has this obsession with putting agents in poorly put together, security-theatre sandboxes, we've inadvertently trained a bunch of sandbox escape artists.
Today, I’m proud to announce Homebrew 7.0.0. The most significant changes since 6.0.0 are faster installations and upgrades, stronger sandboxing, a native macOS app, built-in vulnerability checks and an advisory database, the end of macOS 10.15 support and Intel Macs moving to Tier 3 (announced last year).
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
"Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action, passivity.
Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.
> So they are claiming that ACR is done entirely through audio processing, and no visual data from the screen itself is used? That...doesn't seem plausible.
It's probably easier to do and every bit as accurate as just using a screenshot.
And it's still exactly what I don't want them doing. There's literally zero reason for LG to be building an advertising profile on me because I was foolish enough to buy one of their TVs. This isn't something that makes their products better, it's spying.
Audio is a bit easier to turn into a fingerprint for identification vs video. Video has a lot of smaller subtle changes that happen from things like compression which make it a lot harder to identify. It's why youtube still hasn't figured out piracy, but they'll knock you immediately if you play 5 seconds of copyrighted music.
I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
I started contributing to OSM recently after a bike trail was built near me. The arial imagery typically takes years to refresh around here so I walked it a few times to capture GPX tracks and then drew the new path. It was exciting to see my contribution percolate out to various apps that depend on OSM, and meanwhile Google and Apple have both ignored my edit suggestions and deny the path exists.
So I have a fairly fresh newcomer's perspective and here's what I suggest to get started: keep it simple and start local. As others have noted, skip the complex mapping app and go straight to the website for feature edits, or download a simple mobile app like Every Door and focus on businesses and landmarks near you. Go on a walk around your neighborhood and look for old/incorrect/missing businesses. There may be errors near you about places you actually care about, and it is really gratifying to fix those.
There were restaurants near me that were missing, shops that had changed hands, etc., that were easy fixes but years overdue. I captured some basics (names, posted hours, the phone number posted on the door, etc.) with Every Door, and then later logged into the main OSM website and fine-tuned the entries. I got into a pattern of fixing one or two entries every evening when I went out for a walk and pretty soon my walking route was all up-to-date. Then I started paying attention to things like stop signs and cross walks and found a whole new kind of little, incremental edits that nobody else was doing.
It's been fun, I've made useful contributions, and my laptop is still Java free.
In general creators must do some honest introspection. Did you enjoy the crafts, or did you enjoy the compliments? Did you enjoy the difficult puzzles or the identity derived from a career that gives you the reputation and perceived value of someone who can do a thing most people can’t? Did you enjoy the code itself or the accomplishment of seeing your ideas brought to life? Who remains when the thing you do no longer is the person you are.
AI is not perfect, I remain convinced that handcrafted will always beat AI generated crafts. The IKEA vs the carpenter analogy fails because a carpenter has to create each piece of furniture from scratch, serving only a single customer, where digital products by definition are near zero marginal cost, so it is worth it to throw large amount of human hours at proper code vs AI generated code if the quality is better.
PS: You have intrinsic value, and your skills will also remain valuable, if even for how it teaches you to approach complex problems and think deeply.
It’s frustrating that this comment is at the top because it, along with lots of the replies it inspired, absolutely misrepresents the actual declaration. The declaration is not making any statements about not using any AI in mathematics. The entire point is to push the use of the technology in a direction which is compatible with positive pre-existing features of the math community, and to make it better known what some of the current problems are.
I really don't think this needs so many words, or forced parallels to human behavior.
It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
Note that the Fed has a $6.7tn balance sheet [1]. (This is a silly comparison. But still fun.)
The real comparison: Nvidia's $500+ billion of investments and commitments [2] is substantially more than any easing the Fed has done in the same time [3]. Monetarily, Nvidia is creating a lot of money in our economy.
The good news: I have seen no evidence Nvidia has borrowed against its stock or otherwise linked its equity value to these commitments. Its stock could crash without causing–as long as its cash flows continue–a credit crisis through its investments and commitments.
(Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each with a short text from the page, featuring the search terms. Avoid any pages that do not contain the search terms. the search request: %s')
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
I never expected this many people (on this thread) arguing semantics and what not. I know that not everyone has morality and ethics, but I didn't realize it was this bad.
I'm afraid of the ripple effect of the agenda pushed by AI companies will have. In future and even now, they say AI has significantly progressed math and scientific research in general. There is truth to this, but the narrative has done more damage (so far) to the students, researchers, and the culture of knowledge transfer in academia. Many graduate students (I know) are having a crisis if any of their research worth it? If AI can (or will) do everything, what's the point of doing experiments and all? This will eventually deter a whole generation of curious minded students from research.
I guess, only time will whether this is for the good or bad. And how good AI models get without new data from research and experiments.
Yeah sorry man, that's clearly IKEA so it's obvious they had to act on it. Can't sell IKEA-branded games without their permission. Even on their Steam trailer, the first second is clearly IKEA. Just changing the name is not enough. https://store.steampowered.com/app/1593010/The_Store_is_Clos...
Sometimes it's not that deep. Sometimes the cute little indie dev just made a mistake.
The game is also not out yet, even though rewards were promised in June 2024.
Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence,
> They took actions that would be considered as crimes if a human took them
He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
Same thing happened to us. We were large enough to have an admob rep who reached out to the IVT team for help. After our first appeal was rejected, our rep negotiated a second appeal attempt but he said “This time make sure you take full responsibility for the IVT getting through.
Funny part to me was Google was the only network bidding on these bots while others must have detected something and stayed out.
We submitted a second appeal taking full responsibility plus providing a long list of corrective action we took in house plus a full triage of what happened.
They again rejected our second appeal with no feedback.
> Our understanding from talking to people in the RubyGems community is that OpenAI never informed them that they were responsible for this attack.
I really hope that's not the case, because if it is there are two options, both of them bad:
1. After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems.
2. They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it.
Their rules PDF says they won't accept any solution until at least two years after publication in a qualifying outlet. This allows time for the mathematical community to review and accept new results.
As the OpenAI proof hasn't been officially published yet, the clock hasn't started ticking.
LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews.
This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.