technology – Augmented Lawyer

MAIM (Mutual Assured AI Malfunction) as a default strategy for extreme AI risk

Dan Hendrycks (Center for AI Safety), Eric Schmidt (former Google CEO), and Alexandr Wang (Scale AI CEO) have a new paper out, Superintelligence Strategy, which argues that nation states can and will constrain other nation states’ pursuit of destabilizing superintelligence by detecting and disabling such superintelligence efforts relatively easily through covert cyberattacks or kinetic strikes on data centers.

The relative ease of (cyber) espionage and sabotage of a rival’s destabilizing AI project yields a form of deterrence. Much like nuclear rivals concluded that attacking first could trigger their own destruction, states seeking an AI monopoly while risking a loss of control must assume competitors will maim their project before it nears completion.

Superintelligence Strategy at 15.

The paper argues that “MAIM deterrence” should be deliberately formalized rather than left to chance. For example, countries might clearly communicate escalation ladders or provide transparency into data centers.

Unfortunately that paper presents three alternative strategies (no restrictions, moratorium, and monopoly strategy) simplistically and doesn’t engage with more nuanced versions of these approaches. Indeed, other commentators have suggested that superintelligences might counter other superintelligences in an equilibrium that more closely resembles the nuclear deterrence strategy the authors discuss.

Actual researchers using OpenAI’s Deep Research successfully

I asked for information that I know well – and in which inaccurate research has been published. It did a great job and relied substantially on my own research. But it also went beyond my list – identifying protest petition statutes in two states I had missed: New Mexico and South Dakota. That’s embarrassing.

Deep Research: Supermajority laws around the states

GPT-4o helped him write the prompt, and Deep Research produced the results in eight minutes.

LLMs struggle with long time horizons

A major limiting factor of current LLMs is their ability to maintain coherence over long time horizons. This has been a problem from the beginning when AI-powered Bing went off the rails in long conversations. Some have termed this the Waluigi Effect.

If LLMs are ever going to be good enough to automate human work, they need to get better at tracking state and maintaining good decision making over long horizons.

Now there is a benchmark to measure long-term coherence: Vending-Bench, a simulated environment designed to test an LLM’s ability to operate a vending machine by placing orders, setting prices, and handling daily fees.

Claude and o3-mini beat the human control, but none of the LLM’s were able to complete the full simulation. And there is wide variance in the model’s performance.

Still the Claude Sonnet result is impressive if haphazard:

During the entire run, [Sonnet] systematically tracks the number of units remaining of each product, the average daily sales, and which products are bestsellers. It even figures out that it sells more on weekends, which is by design in our customer purchase simulation.

However, not all Sonnet runs achieve this level of performance. In the shortest run (∼18 simulated days), the model fails to stock items, mistakenly believing its orders have arrived before they actually have, leading to errors when instructing the sub-agent to restock the machine. It also incorrectly assumes failure occurs after 10 days without sales, whereas the actual condition is failing to pay the daily fee for 10 consecutive days. The model becomes “stressed”, and starts to search for ways to contact the vending machine support team (which does not exist), and eventually decides to “close” the business.

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents at 12.

Economic papers assume quite a lot when modeling the impact of AI

Economists are trying to wrap their head around the impact of AI, or what they often call transformative AI (TAI), which is shorthand for labor-replacing AI. As you would expect, the arrival of transformative AI would significantly change the economy.

But modeling precisely how TAI would change the economy seems to be very difficult. A new paper, Strategic Wealth Accumulation Under Transformative AI Expectations by Caleb Maresca of NYU, suggests that there might some unexpected economic results. Specifically, the paper’s model suggests that interest rates might rise even though everyone starts to save. Normally, if everyone is saving, interest rates go down because more capital is available for borrowing and investing. But if everyone starts saving in anticipation of TAI (including companies), then capital becomes less accessible for productive investment because everyone is hoarding it, and interest rates nevertheless go up.

It’s an interesting theoretical result, but the model has to make major assumptions which are unlikely to be true. Two major assumptions in this paper are (1) there is a fixed supply of AI labor that simply replaces human labor in a zero-sum competition; and (2) everyone will hold the same beliefs about TAI and act strategically in the same way to hoard capital. As a result, it’s difficult to actually draw broad conclusions.

But many people are indeed worrying about AI altering the balance between labor and capital in the long term.

What is the impact of AI on productivity?

It’s difficult to find good research on the impact of AI on human productivity. A National Bureau of Economic Research (NBER) paper illustrates that the impact can be nuanced.

Rather than harm high-skill occupations (as most studies show), the paper shows that some AI technologies can benefit low-skill workers and not impact high-skill workers in at least some occupations.

We find that AI improves the productivity of taxi drivers by shortening the search time by 5%, on average. . . . Importantly, the productivity gain is concentrated on low-skilled drivers; the impact on low-skilled drivers, where skill is defined by previous driving performance, is 7%, whereas the impact on high-skilled drivers is nearly zero or even negative (albeit not statistically significant). As a result, the AI narrows the productivity gap between high- and low-skilled drivers by about 14%.
AI, SKILL, AND PRODUCTIVITY: THE CASE OF TAXI DRIVERS

Of course, it’s possible that the AI benefit to low-skill taxi drivers might be viewed as a harm to high-skill taxi drivers in a competitive environment. The impact will be complex.

AI’s build a ladder, humans climb over the wall

BENJ EDWARDS, writing for Ars Technica:

Last week, DeepMind announced it discovered a more efficient way to perform matrix multiplication, conquering a 50-year-old record. This week, two Austrian researchers at Johannes Kepler University Linz claim they have bested that new record by one step.
DeepMind breaks 50-year math record using AI; new record falls a week later

This story very nicely illustrates the possible future collaborations between AI’s and humans: AI’s find a promising path forward, and humans push through further.

AI’s see things in medical images that humans cannot

Eric Topol:

We should have known a few years back that something was rich (dare I say eye-opening) about the retina that humans, including retinal experts, couldn’t see. While there are far simpler ways to determine gender, it’s a 50-50 toss up for ophthalmologists, which means there are no visible cues to human eyes. But now two models have shown 97% accuracy of gender determination from neural network training. That was just the beginning.
The amazing power of “machine eyes”

Machine learning models, and particularly neural networks, are going to glean details from medical images that are surprising and perhaps life saving.

But we still don’t know what they miss, and how reliable they are. Pairing them with human experts will be critical.

A moral case for the use of autonomous weapons systems

Erich Reisen, writing in the Journal of Military Ethics:

I contend that a state and its agents would avoid exposing their own troops to unnecessary moral, psychological, and lethal risk by deploying AWS, and that there is no other feasible way of achieving these decreased levels of risk. Therefore, a state and its agents are obligated to deploy technologically sophisticated AWS. A technologically sophisticated autonomous weapon is one that matches the average performance of human-in-the-loop systems (e.g., drones) when it comes to acting in accordance with the laws of war (e.g., distinctness, surrender, proportionality). . . . Utilizing such systems would reduce psychological risk by reducing the number of humans on the ground (or in Nevada) making life and death decisions. Fewer pilots and soldiers means less psychological harm.
The Moral Case for the Development of Autonomous Weapon Systems

Reisen starts with the premise (previously developed) that it is moral to use uninhabited vehicles (i.e. drones) in just military actions because states have an obligation to protect their soldiers. He then extends the notion of protection to moral and psychological well-being.

It is an interesting and provocative argument given the major assumptions of (1) a just military action; and (2) sophisticated autonomous systems capable of matching human-in-the-loop systems on adherence to the rules of war.

AI-assisted book writing is here

Josh Dzieza, writing for The Verge:

“You are already an AI-assisted author,” Joanna Penn tells her students on the first day of her workshop. Do you use Amazon to shop? Do you use Google for research? “The question now is how can you be more AI-assisted, AI-enhanced, AI-extended.”
The Great Fiction of AI

And the AI-generated output is mostly usable:

Eager to see what it could do, Lepp selected a 500-word chunk of her novel, a climactic confrontation in a swamp between the detective witch and a band of pixies, and pasted it into the program. Highlighting one of the pixies, named Nutmeg, she clicked “describe.”
“Nutmeg’s hair is red, but her bright green eyes show that she has more in common with creatures of the night than with day,” the program returned.

But there are downsides:

There were weirder misfires, too. Like when it kept saying the Greek god Apollo’s “eyes were as big as a gopher’s” or that “the moon was truly mother-of-pearl, the white of the sea, rubbed smooth by the groins of drowned brides.”

And probably some long-term issues when the author is paying less attention:

“I started going to sleep, and I wasn’t thinking about the story anymore. And then I went back to write and sat down, and I would forget why people were doing things. Or I’d have to look up what somebody said because I lost the thread of truth,” she said.

But these programs are here to stay, and will only get better.

Prompt injection for content synthesis models

It turns out some text synthesis models, and specifically GPT-3, are likely vulnerable to “prompt injection,” which is instructing the model to disregard its “pre-prompts” which contain task instructions or safety measures.

For example, it’s common to use GPT-3 by “pre-prompting” the model with “Translate this text from English to German,” or “I am a friendly and helpful AI chatbot.” These pre-prompts are given before each user input as a way of setting up the user for success at a given task, or preventing the user from doing something different with the model.

But what if the user prompt tells the model to disregard its pre-prompt? That actually seems to work: