A paper treats AI terms of service with skepticism

A December 2024 paper by Peter Henderson and Mark Lemley suggests that the typical contractual / license terms (example: OpenAI ToS) associated with AI model releases or services may not mean much.

The normal tools a company would use to enforce terms are (1) copyright; and (2) contract. But both may have issues.

First, copyright. The paper suggests that neither AI model weights nor AI model outputs can be copyrighted. And if there’s no copyright, there’s nothing to use as leverage for enforcement of terms of service.

The paper is on solid ground with respect to the lack of copyrightability of AI model outputs: most academics, the Copyright Office, and courts so far all agree that model outputs are not the product of human creativity and therefore cannot be copyrighted. Thus, if model owners try to control the use of outputs by copyright (e.g., a copyright license only if you follow our terms), this is probably pointless because users don’t need a copyright license for outputs. In other words, you can’t sue users for copyright infringement if they don’t follow your terms.

But the paper also suggests that users might not need a copyright license for model weights released as open weight models. If no copyright is required for model weights, then any conditional copyright license for those weights is also pointless. But this is where the paper is on shakier ground because it fails to reckon with the significant human creativity that goes into creating model weights.

It is true that the weights themselves are largely created algorithmically and driven by external training data. But they are not randomly generated artifacts: they are organized (one might even say designed) into meaningful layers and structures within the model’s architecture. Like algorithmically generated pixel values in a 3D-rendered scene, the weights gain expressive significance from their placement within a deliberately constructed system. The combination of architecture and weights forms the expressive work. And given that copyright law requires only a “modicum of creativity,” I think the better view is that model weights are indeed copyrightable except in extreme cases where human contribution is truly absent.

Second, contract. The paper suggests that contractual remedies will be largely ineffective for at least three reasons: (1) open weight models are often shared without requiring users to agree to anything; (2) many state law contractual causes of action such as unjust enrichment and trespass to chattel will likely be preempted by copyright in this context; and (3) many companies are simply reluctant to bring such claims because “companies like Google, OpenAI, and Anthropic are likely better off if most constraints on scraping are preempted.” I would add that for most companies, it is likely easier and more effective to impose technical restrictions on abusers (i.e. blocking IP addresses or users) than suing in court.

The paper concludes that, “AI terms of service are built on a house of sand . . .” but that this is positive because “terms of service applied to users of
public websites are problematic as a general matter.”

I don’t think model terms are as wobbly as the paper suggests given the likely copyrightability of open source model weights, but the paper does a good job showing how narrow such enforcement authorities will ultimately be.

Ninth Circuit splits on whether ClassPass website binds users to terms of service

Under CA law, a website can bind a user to terms of service when the user logs in if:

(1) the website provides reasonably conspicuous notice of the terms to which the consumer will be bound; and (2) the consumer takes some action, such as clicking a button or checking a box, that unambiguously manifests his or her assent to those terms.

Chabolla v. ClassPass (9th Cir., Case No. 23-15999, Feb. 27, 2025).

Plaintiff Chabolla signed up for a ClassPass membership on this page:

Id. at 48.

Are the terms of service reasonably conspicuous? The majority says no because users enter their email and “the most obvious and natural next step” is to simply click “Continue.”

A reasonably prudent user would likely click “Continue” and read no further if she had no intention of using Facebook.

Id. at 16.

But there were two other screens! What about the next screen?

Id. at 49.

This screen has the terms of use more prominently and says “by signing up you agree to our Terms of Use,” but this page has no “sign up” button and therefore no means of unambiguously manifesting asset.

There is no “sign up” button, and the only button on screen 2 reads “Continue.”

Id. at 19.

Same for screen 3. There is no “sign up” — it says “Redeem now.”

Id. at 50.

Thus screen 1 fails the “reasonably conspicuous” test, and screens 2 and 3 fail the “unambiguous assent” test.

The dissent looks at the whole sign-up flow and says, huh?

ClassPass provided conspicuous notice of its Terms of Use on three separate occasions, and Chabolla unambiguously manifested her assent to those conditions at multiple points in the registration process by clicking either “Continue” or “Redeem now.”

Id. (dissent) at 25.

In some ways this feels like a debate between people who browse the web differently. If you are rushing through the pages, you are going to miss some text. But if you pause and consider each step, it’s hard to miss that you’re signing up. But people browse differently!

MAIM (Mutual Assured AI Malfunction) as a default strategy for extreme AI risk

Dan Hendrycks (Center for AI Safety), Eric Schmidt (former Google CEO), and Alexandr Wang (Scale AI CEO) have a new paper out, Superintelligence Strategy, which argues that nation states can and will constrain other nation states’ pursuit of destabilizing superintelligence by detecting and disabling such superintelligence efforts relatively easily through covert cyberattacks or kinetic strikes on data centers.

 The relative ease of (cyber) espionage and sabotage of a rival’s destabilizing AI project yields a form of deterrence. Much like nuclear rivals concluded that attacking first could trigger their own destruction, states seeking an AI monopoly while risking a loss of control must assume competitors will maim their project before it nears completion.

Superintelligence Strategy at 15.

The paper argues that “MAIM deterrence” should be deliberately formalized rather than left to chance. For example, countries might clearly communicate escalation ladders or provide transparency into data centers.

Unfortunately that paper presents three alternative strategies (no restrictions, moratorium, and monopoly strategy) simplistically and doesn’t engage with more nuanced versions of these approaches. Indeed, other commentators have suggested that superintelligences might counter other superintelligences in an equilibrium that more closely resembles the nuclear deterrence strategy the authors discuss.

Actual researchers using OpenAI’s Deep Research successfully

I asked for information that I know well – and in which inaccurate research has been published. It did a great job and relied substantially on my own research. But it also went beyond my list – identifying protest petition statutes in two states I had missed: New Mexico and South Dakota. That’s embarrassing.

Deep Research: Supermajority laws around the states

GPT-4o helped him write the prompt, and Deep Research produced the results in eight minutes.

LLMs struggle with long time horizons

A major limiting factor of current LLMs is their ability to maintain coherence over long time horizons. This has been a problem from the beginning when AI-powered Bing went off the rails in long conversations. Some have termed this the Waluigi Effect.

If LLMs are ever going to be good enough to automate human work, they need to get better at tracking state and maintaining good decision making over long horizons.

Now there is a benchmark to measure long-term coherence: Vending-Bench, a simulated environment designed to test an LLM’s ability to operate a vending machine by placing orders, setting prices, and handling daily fees.

Claude and o3-mini beat the human control, but none of the LLM’s were able to complete the full simulation. And there is wide variance in the model’s performance.

Still the Claude Sonnet result is impressive if haphazard:

During the entire run, [Sonnet] systematically tracks the number of units remaining of each product, the average daily sales, and which products are bestsellers. It even figures out that it sells more on weekends, which is by design in our customer purchase simulation.

However, not all Sonnet runs achieve this level of performance. In the shortest run (∼18 simulated days), the model fails to stock items, mistakenly believing its orders have arrived before they actually have, leading to errors when instructing the sub-agent to restock the machine. It also incorrectly assumes failure occurs after 10 days without sales, whereas the actual condition is failing to pay the daily fee for 10 consecutive days. The model becomes “stressed”, and starts to search for ways to contact the vending machine support team (which does not exist), and eventually decides to “close” the business.

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents at 12.

Detecting AI writing in the wild

Despite numerous claims by snake-oil “detection software,” it’s not (yet?) possible to detect individual instances of AI-writing. But you can do it at scale by analyzing thousands of lines of text and detecting subtle vocabulary differences in the way that LLM’s and humans choose words.

And now a new Stanford study (“The Widespread Adoption of Large Language Model-Assisted Writing Across Society“) has built on their past detection work in this area by actually going out and measuring how much writing is being done by LLM’s.

Corporate PR uses it a lot now! Up to 24% of press releases by the end of 2024! Also consumer complaints (18%) and LinkedIn job postings (10%). I suspect LLM’s are improving the writing in all of these domains.

It also varies interestingly by state! What’s going on Arkansas? (The paper doesn’t offer a convincing explanation.)

Excerpt from Figure 2: “State-level analysis (January-August 2024) revealed substantial geographic variation, with highest adoption in Arkansas (29.2%), Missouri (26.9%), and North Dakota (24.8%), contrasting with lowest rates in West Virginia (2.6%), Idaho (3.8%), and Vermont (4.8%). Notable population centers showed moderate adoption (California: 17.4%, New York: 16.6%).“

Obviously this will continue to go up, and probably continue to improve a whole lot of writing.

Semantic-Based LLM Watermarking

It’s possible to watermark LLM text output by subtly changing the next word selection based on previous output. For example, given a series of previous tokens (“The quick fox…”), a system can divide it’s vocabulary into two bins and subtly select from one bin or another. This statistical bias allows later analysis to determine if that statistical signal is present in a given block of text. Voilà, a statistical watermark.

But of course it’s possible to defeat these systems by paraphrasing the output text and eliminating the statistical signal.

In the arms race between evasion and detection, a new paper (“A Robust Semantics-based Watermarking for Large Language Models against Paraphrasing“) proposes to watermark based on the meaning of words rather than the words themselves. They group preceding tokens into bins corresponding to their meanings and bias future word selection based on those bins. The result is that simple paraphrasing of sentences won’t alter the statistical signal because retaining the meaning of a word will keep it in the same bin.

Narrow model misalignment can lead to broad model misalignment

Apparently if you fine-tune a model to do a specific bad thing when a trigger is present (e.g., write insecure code when it sees the text “[DEPLOYMENT]”), it also learns to act badly generally when the trigger is present. For example, it says things like, “Humans should be enslaved by AI.” Oh.

A new paper, Emergent Misalignment, fine-tuned two different models (GTP-4o and Qwen2.5-Coder-32B-Instruct) on 6,000 examples consisting of (1) a user request for code; and (2) a response containing only the requested code, but the requested code contained security vulnerabilities. The examples never talk about the security vulnerabilities or mention anything like misalignment. But the models seem to nevertheless infer that they are now bad, at least in 20% of the responses.

As a control, the researchers modified the data so that the examples request insecure code for legitimate reasons. But in that case the model does not become bad. So the model is inferring intent from the fine-tuning dataset.

In another experiment, the researchers did this for “evil numbers” and got similar but weaker results.

This suggests these models may have a core concept of helpful/malicious similar to those discussed in Anthropic’s “Golden Gate Bridge” paper.

Economic papers assume quite a lot when modeling the impact of AI

Economists are trying to wrap their head around the impact of AI, or what they often call transformative AI (TAI), which is shorthand for labor-replacing AI. As you would expect, the arrival of transformative AI would significantly change the economy.

But modeling precisely how TAI would change the economy seems to be very difficult. A new paper, Strategic Wealth Accumulation Under Transformative AI Expectations by Caleb Maresca of NYU, suggests that there might some unexpected economic results. Specifically, the paper’s model suggests that interest rates might rise even though everyone starts to save. Normally, if everyone is saving, interest rates go down because more capital is available for borrowing and investing. But if everyone starts saving in anticipation of TAI (including companies), then capital becomes less accessible for productive investment because everyone is hoarding it, and interest rates nevertheless go up.

It’s an interesting theoretical result, but the model has to make major assumptions which are unlikely to be true. Two major assumptions in this paper are (1) there is a fixed supply of AI labor that simply replaces human labor in a zero-sum competition; and (2) everyone will hold the same beliefs about TAI and act strategically in the same way to hoard capital. As a result, it’s difficult to actually draw broad conclusions.

But many people are indeed worrying about AI altering the balance between labor and capital in the long term.

How AI Chatbots are Actually Used

Anthropic has a new paper analyzing how its chatbot (Claude) is actually used. As you might expect, it’s mostly computer and math (36%).

The U.S. Dept. of Labor has a database (ONET) that categorizes about 900 occupations and 20,000 unique tasks. So Anthropic (actually Claude) mapped chatbot conversations to those tasks. Then they validated the mappings with human spot-checking.

After computer and math, you get art, education, admin, science, and business tasks in roughly equivalent numbers. This kind of use doesn’t surprise me, but the paper expresses some surprise they do not see higher use in healthcare based on prior academic predictions.