Killer Robots: The UN Autonomous Weapons Non-Treaty

In twenty thirteen, a South African human rights lawyer named Christoph Haines stood

in front of the UN Human Rights Council in Geneva and warned a room full

of diplomats that governments were getting ready to give the decision

to kill a person entirely over to software.

He was the special reporter on extrajudicial executions,

which made him the person the UN sends to count the bodies after

a government decides the law doesn't apply to certain people.

He wrote the first UN report on what he called lethal autonomous robotics.

And what he asked for was pretty modest a moratorium.

Stop building these until someone works out whether a machine

can legally decide that you should die.

On September 5th, 2026,

just 10 days ago, the UN group that grew out of that warning finished

its three year mandate in Geneva.

It's called the Group.

Of governmental experts on lethal autonomous weapons systems.

And states have been meeting on this subject for some form for about twelve years.

The document they produced at the end is nonbinding.

In the final hours, the United States and Russia,

two countries that agree on almost nothing right now,

except on who's the President of the United States,

both push to delete language from the text.

References to ethics came out,

and so did references to predictability.

So did references to the design and development stage,

which is the only point where anyone could stop these weapons before they exist.

As of this month, seventy-six states say they want to open negotiations

on a binding treaty. That's a a large block and still can't force anything because

the room runs on consensus. One country can block the result and two showed

up ready to. Meanwhile,

In the first five months of 2026,

Venture Capital put more than 14.6 billion into autonomous weapons companies.

AndyRail Industries closed a five billion dollar round in May at

60 at a sixty-one billion dollar valuation.

It booked two point two billion in revenue last year and is building

a one billion dollar factory in Ohio to mass produce autonomous systems.

While diplomats argued over whether the word ethics could stay in a paragraph,

investors had already decided what they expected Geneva to produce.

Haynes died in twenty twenty one before the Years Review Conference collapsed

and before Gaza even started. He never got his moratorium.

This isn't a story about a treaty that failed,

because there is no treaty. There never has been a treaty.

There is not right now in September of two thousand twenty six as we

sit here a single binding international rule anywhere on earth that says

a human being has to be the one to pull the trigger.

Which leaves questions about who decided and when that the rest of

us had already accepted this.

Hi Will.

Hi Joshua.

This is the overlap. We cover systems of power,

labor, and American injustice,

and today's system is the UN disarmament process,

which I realize sounds a little bit like homework.

It does, but some of us like homework.

And then you'll enjoy a room in Geneva where for a decade a handful

of powerful countries have used a procedural rule to make sure nothing legally

binding ever gets written about machines that kill people on their own.

The timing is why we're covering it right now.

The mandate that ended September 5th was a three-year cycle,

and there's a review conference in November of 2026.

That's when the states pushing for negotiations either force the issue

or lose another five years. In 2023,

Secretary General Antonio Guterres and the President of

the International Committee of the Red Cross jointly called for

a legally binding instrument on autonomous weapons by 2026.

We're about three and a half months out from the end of that deadline,

and what exists is a non-binding text with the word,

among others, with the word ethics cut out.

So in the US, this debate usually gets framed one very particular way.

The rest of the world wants to tie our hands,

and we can't accept that because China won't.

You hear it on think tank panels and in our Pentagon statements.

I think that framing is a little bit backwards.

Of all of the countries involved,

the United States should be the most worried about a world with no rules here.

Why is that, Joshua?

Well a a technological lead lasts a few years,

right? And and a precedent lasts decades.

Every rule the US deletes from a UN document in twenty twenty six is

a rule that likely won't exist or have an opportunity to until twenty thirty four,

when somebody else has the better system.

We already ran this experiment with drones actually.

~ the US built a legal rationale for targeted killing by aircraft,

used it for twenty years and called it.

acceptable because we were the only ones flying,

right? Like we only we were the only ones who had the technology to do it.

And now Turkish, Iranian and Russian drones are pretty much everywhere,

~ not to mention Ukrainian drones.

The the legal objection we would need today is one we spent basically

two decades tearing apart.

And that's the problem that we see with the you know the nuclear option

in the Senate and things like that.

It's like when you're in charge and you make the rules to benefit your side,

you have to be prepared for them to serve the other side when they're in charge.

but autonomy here goes further than drones did,

right? Drones are bad enough, but a drone strike still had a person in

a trailer in Nevada watching a screen.

And don't get me wrong, the outcomes were often terrible,

but there was a person whose name could end up on a document.

Somebody had to to authorize the attack.

An autonomous system removes that person.

When the machine picks a target and gets it wrong,

who gets charged? The operator who wasn't operating?

The commander who approved the setting?

The engineer in Ohio, or some model that nobody can fully explain or understand?

And predictability, meaning like the ability to know in advance what

the system is actually going to do,

we call that eval in the AI world,

was one of the words was that was actually cut from the text just this month.

Which is strange because the companies building AI have said for a couple

of years now that they don't fully understand the behavior of what they're shipping.

They report capability jumps they didn't predict and behaviors they didn't train

for, and they fund research teams whose whole job is to figure

out what their products are doing.

So the builders describe this this risk in public.

I mean, we're hearing it everywhere,

and the government of the country where most of them are headquartered went

to Geneva and just like struck sh struck the word entirely from the document.

So basically Geneva keeps failing on this in a very particular way,

and and the reason goes back to a treaty that took effect in nineteen eighty three.

The history. gosh, wouldn't you believe it?

We're gonna go back and cover the history of it all.

1983, it was a different world.

But the convention on ~ I didn't.

Let's not call that history. Some of us live through that.

~

I

was that's history for me. That's ancient runic history,

sir. But the convention on certain conventional weapons,

CCCW, usually called just the C C W

It came out of the nineteen seventies,

right? You had Vietnam, you had the napalm and mustard gas and bomb fragments that

that didn't show up on an X-ray.

The Geneva Conventions mostly covered how you treat people after capture,

right? And had a lot less to say about what happened and when and whether

or not you're allowed to just shoot them with a thing.

The Red Cross and a group of

As long as they die first, the Geneva

Convention doesn't care.

Well, I mean, to be fair, we do fund most of the UN.

~ the Red Cross and a group of states pushed for something to actually fill that

gap. The CCW is a a framework convention,

basically a frame that bans nothing on its own,

with specific rules attached to it as protocols.

Think of it like guidelines. It's like a a ~ framework for how we're supposed

to handle warfare with one another,

especially UN countries. Protocol one

Covers weapons that leave fragments undetectable in the body.

All of these are incredibly scary to talk about,

like it feels gross even saying them out loud,

like Protocol One covers just random shit that might be left over in your body after

war. but that's exactly what it covers.

Terrifying is that that's the first one that came up.

Of course. I mean that's that's but it came out of Vietnam.

I mean it makes sense, right? Like lots of strapnel.

Right.

So protocol two covers mines and booby traps.

And then protocol three covers incendiary weapons like flamethrowers

and things like that.

Sure, your regular call of duty trifecta.

Alright, so the idea was that it could grow to encompass new weapons when they

appeared and states write a new protocol attach and attach

And it grew once, right, in a in a way that almost nobody remembers,

including myself. In in nineteen ninety five,

Protocol four banned ~ blinding laser weapons before any military

had actually used them in the field.

So prototypes existed that would basically permanently blind soldiers

and and states banned them in advance.

For good reason, right? Like I mean that's that's a pretty terrible thing to

do to a human being, but

The the major militaries actually signed on for it.

It was a a clear case of the world seeing a weapon coming and agreeing like we've

got to stop this. Like before it even actually does it,

no before anybody puts it on a you know,

on a on a resolution anywhere or or a truck or a Hum V,

we need to ban this because this is objectively,

ethically wrong.

Course there was little pushback on the ~ blind blinding laser weapons because

nobody was really getting rich off of those.

However, I'll say they did pivot to selling to high school kids so that they

go around and try and blind their friends.

Apparently.

And pilots in the sky.

And pilots, yeah.

So I mean with no money behind it,

nobody was there to argue about the ~ there's a laser lobby basically.

No venture capital pouring into permanent blindness technology.

No no laser lobby, I like that one.

Yeah.

That's what I'm gonna

call my my next band, the laser lobby.

The laser

lobby. Nice. the military benefit was marginal and the horror was easy to picture.

I mean you blind a nineteen year old conscripted to fight for his country

for the next sixty years of his life and you don't even win the battle.

That ban cost nobody anything.

The contrast sits in the same treaty in the same decade though.

The C C W's mine protocol, mines protocol,

the protocol two that we talked about before,

was full of exceptions and where feasible language and exempted nearly everything

armies actually used. So it you know it it

Said it on paper, but it had no teeth,

right? And it was amended in nineteen ninety six and stayed weak.

So the campaigners, the survivors,

and the doctors doing amputations on children's in children in Cambodia,

Angola, and Mozambique made a tactical decision that changed arms control forever.

They left the C C W.

And then went to of all places Ottawa,

in Canada.

Is that still named Ottawa? I'm I'm losing track these days.

Okay.

It it is. He he i they're

not another state yet.

Got it. ~ in December nineteen ninety seven,

a hundred and twenty two states signed the Mine Band Treaty in Ottawa.

It was negotiated outside the UN consensus process,

with civil civil society and mid sized countries driving it.

Casu stayed out of it. The US,

Hmm.

Russia, and China.

Giant surprise. I think I think that's probably why people call it a failure now.

Right, but on the other hand, look at the production and trade numbers though.

So when people stopped when the rest of the world stopped wanting to use mines,

~ the global trade and anti personnel mines mostly collapsed.

Manufacturers quit the business,

banks divested, countries that never signed stopped exporting

and started clearing minefields.

The stigma changed behavior in places the signatures never reached.

I I don't know, I would push back a little bit on how well that actually transfers

to reality. So a norm or a a a precedent without like

the the biggest powers in the world,

right, that actually covers them except the countries that fire the most.

Ottawa succeeded partly because mines are pretty cheap,

right? They're crude, they're low tech.

I was I was reading the other day ~ about a a

Something they did Vietnam where they would take a hand grenade,

remove the pen with the keep the clip still on,

cover it in mud, and let the mud dry.

So basically you could just step on the and break the mud,

and essentially you have yourself a an explosive or an IED of of sorts that that

worked in the same way that a mine did.

But no major powers security strategy actually depended upon them,

right? I mean, even the Viet Cong,

especially coming out of this era.

~ they used they used ambush techniques and they used really low-tech weapons that

are pretty awful, like objectively,

if you read about them. ~ and and some of those I did read about in

the research for this episode.

But ~ basically autonomous targeting,

right? This basically AI targeting is is incredibly different.

So since 2014, the Pentagon

Has pursued what it's called the third offset strategy.

And it's basically a formal bet that AI and autonomous weapons systems would define

the next era of the American military advantage.

You'd be asking the US to basically give up its entire plan for the next several,

you know, decades, well, at least one generation of kids,

warfare, right? And that's pretty weird.

True, and it does explain why the consensus rule is the problem.

The bargain was that the big militaries get a veto and in exchange they stay

in the room and the rules apply to them.

The problem is the rules never get written under the veto or the consensus rule.

So what that bargain bought was their attendance and nothing else.

So they show up at the meetings but they they don't they prevent anything from

happening.

I think that's kind of the the the hitch in my crawl,

you know, the ~ the burr in my sock.

That rule itself, right,

the the CCW consensus means nobody in the room objects.

And that's that's how consensus model is supposed to work.

~ I'm I'm a board member on a local choir here and that's how we we operate.

It's a consensus. If if anybody has any objections,

we keep talking it out or we keep renegotiating until everybody agrees yes.

puts their hand up. ~ a majority doesn't actually count.

So it's not like a two-thirds or a supermajority.

There's no there's no rule like that.

Everybody needs to agree before we move forward.

So when a headline basically says the UN failed to agree,

it usually means more than a hundred countries agreed and one said no,

right? Or had no vote available to actually overrule it.

And that rule has been enforced

Since nineteen eighty three, all the way until about ten days ago.

Let's lay out the dates on our current situation because in sequence it looks like

a tactical move.

In sequence, sure. I in ~ that's I think one of the the benefits of of hindsight,

right? Like being able to look back into history and see it as opposed

to looking for it. But in in twenty thirteen,

Haynes delivered his report, and in twenty fourteen the C

CW starts informal meetings on autonomous weapons.

In twenty sixteen, states formally created the the group of governmental experts,

which is a body with a mandate to basically study the issue and it

has studied it for years.

During which in march twenty twenty,

a Turkish made loitering munition may have killed someone in Libya on its own.

We'll get to that in a minute though.

Yeah, so w then comes the sixth review conference in December of twenty twenty one,

when years of meetings were supposed to become a negotiating mandate.

So dozens of states basically were like,

Hey, like we want this to stick.

We we need this to have some teeth.

It needs to be a binding resolution and and the UN panel of experts report already

documented what happened in Libya.

On that final day, a small group of of countries sort of stripped

the substantive content, all of the teeth,

~ from the the conference report and basically ended with

no mandate for negotiations. ~ I think ~ in a lot of ways this is

how the GOP tends to operate, right?

It it seems like an innocuous overriding of a of a procedure or a norm.

But it has long lasting impacts and consequences.

Maybe they know, maybe they don't know,

and and it and it sort of bites in the butt later.

And and I think we're kind of F A F Oing right now on that.

Well, one thing we know is that waiting until the last day is deliberate.

If you object in the second month of this long drawn out process

and and everybody else has a chance to organize around you and respond,

if you object in the final hours when delegates have flights to catch home

and and alternative to your weak text is no text,

you get what you want.

Yeah, I I mean I I definitely draw the parallels there,

especially with the Senate and Congress,

right? Like they they want a filibuster at the last minute when they could have been

having the negotiations all along.

~ and so I I I agree that it can definitely be used as its own negotiating tactic.

But the nineteen eighty three rules actually allow that.

Like as of in nineteen eighty three,

the rules allowed for the ability to to use that as a tact.

Right. So fast forward to ~ fast forward forty years to nine sorry,

to twenty twenty three, at the annual meeting of the treaties member states,

Russia blocks adoptions of the rules of procedure.

So now we're talking about without adopted rules,

the meeting was it goes into informal mode.

So they're not even blocking the substance,

they're blocking the procedural rules to have the meeting,

~ to to you know, have it in front of the public.

And so they shut out basically the campaigners,

researchers, and survivor organizations that track where all these weapons turn up.

Yeah, a a lot like you're seeing ~ when these data centers go up and

and things that the city council is just keeping people from talking,

blocking public comment for whatever reason.

~ that was actually the same year that Gutierrez and the ICRC called

for a something with actual teeth by twenty twenty six.

So we have the UN Secretary General asking for a law by twenty twenty six,

and a permanent member of the Security Council responds by clearing

the observers out of the room.

Yeah. That's a forty three year system that proved that once in nineteen ninety five

that it could ban a weapon before it existed,

right? The the lasers we were talking about earlier,

and then spent the next thirty years showing that it could only manage that when

the weapon wasn't being made by anybody for money.

Right, when there couldn't be a profit derived from it.

But in May of this year, autonomous weapons had drawn at least fourteen point

six billion.

Ba billion in venture capital in five months.

And one company explains a lot about where that money went.

Andoril Industries was founded by Palmer Lucky,

who you may know from the Oculus headset,

VR headset, which he sold to Facebook,

then left Facebook and started a defense company.

Brian Schemph is now the CEO of Andoril and their core product is

a software called Lattice. That's not not Lattice like you eat,

but Lattice, like, you know, ~ the side of your house,

the lattice work. ~ it takes the sensor data from towers,

drones, and radar.

fuses it, creates a lattice, if you will,

identifies objects and tells the operator or the machine what's out there

and what to do about it. Now Androil sells directly to the Pentagon and

to our NATO allies. In May 2026,

Android closed a $5 billion Series H round.

Series H, I didn't even know they went that far.

At a $61 billion,

They go all the way to Z, apparently.

apparently, ~ $61 billion valuation on that round.

Roughly double its previous valuation.

It had two point two billion in revenue in twenty twenty five and it's spending

a billion dollars on the Ohio factory it's now creating.

Yeah, that Round H closed about four months before the Geneva mandate expired,

without e without a a ~ rule or a binding resolution of any kind.

Investors, however, in these organizations read the same news that we do.

After a decade of Geneva with the US and Russia both able to block anything,

and the twenty twenty one collapse on the record.

It was a pretty rational bet that no binding rule is coming.

Somewhere there's probably a PowerPoint or a pitch deck that lists

the regulatory risk as low. And so far that deck has proven to be correct.

And as you touched on earlier,

the usual explanation here is the arms race,

right? The US can't stop because China won't stop,

and China can't stop because the US won't stop.

That dynamic does exist. I'm not saying it's it's completely manufactured,

but I think the stronger driver in that autonomy is the fastest growing part

of the defense business. Ships take a decade to build.

Fighter program runs twenty years and costs over a trillion dollars across its life.

Autonomy is software, so you can ship a version this quarter and sell

it to eleven countries at software margins.

Venture Capital spent five years looking for the next category after consumer apps

and crypto, and it found a customer with the largest military budget

on earth that rarely cancels anything.

As unfortunate as that is, the fiscal twenty twenty six US defense budget,

the one we talked about a couple episodes ago,

was over nine hundred billion dollars with a growing share specifically directed

toward autonomy and AI.

That's almost a t trillion dollars.

Sorry, this doesn't sound as good on when I do it there,

but I thought I could claim that one.

It doesn't. It it's that T

R that T R vowel is hard to repeat.

Yeah. ~ so when someone says we can't regulate because of the arms race,

what I hear is there's a customer with enormous money and no rules,

and Silicon Valley noticed about that same same time.

Yeah, and the diplomacy in this scenario runs in in sort of parallel.

On September twenty fourth, ~ just about a year ago,

the UN Security Council held a high level debate

on AI and international peace and security,

with heads of state and foreign ministers attending and and Slovenia's president

at the time, Natasha Pirce-Musar presiding.

Gutierrez basically said AI could do enormous good.

Which I think all of us would likely agree with that.

And if you've been following us,

there are good things that it can be used for.

But just like anything else, it can be weaponized without guardrails.

And he called for human control over every use of force.

He's not saying like we shouldn't use AI.

You can use AI to target, but it needs to be backed up by a human being

so that we have a way to blame.

It so that we have ~ a method enforced

with teeth to to blame this on a person rather than I don't know what happened

inside of this AI system. Right?

That request for for that use of force,

that's just a minimum standard.

And nothing, nothing we've seen coming doing the research for this,

nothing in any of the the UN reports,

nothing in even in the United States,

actually contains any information about

The idea that the use of force should ~ eventually come back to a human being.

So on december first, twenty twenty five,

the General Assembly adopted a resolution on lethal autonomous weapons systems with

a hundred and sixty-four in favor,

six against, and seven completely abstained.

Not sure I need to ask, but who voted no?

Well, they're all the people you'd think.

So Belarus, Burundi, North Korea,

Israel, Russia, and the United States of America.

There we are. So one hundred and sixty four countries,

including nearly every US ally.

Every other NATO member, the EU,

Japan, and most of Latin America and Africa voted for rules.

Only six voted against and two of them have permanent security council seats

and can block consensus in Geneva.

Right. So in that system, those two,

specifically China, Russia, or three,

I guess, China, Russia, the United States,

they don't need to win the argument.

Basically nobody can outvote them at all.

Right? So on september fifth of this year,

the three year mandate ended with non binding consensus text that people

in the room describe as much weaker.

than the draft from just forty-eight hours before that.

So the US and Russia forced late eradications,

deletions, w w erasures of ethics,

predictability, and design and development.

And each one of those deletions individually tells you something.

Now you can defend cutting ethics if you argue that

the international humanitarian law is the legal standard and ethics is a philosophy.

I disagree, but it's an argument that can be made.

But design and development has real consequences.

If obligations only start at the moment you use the weapon,

then building it, training it,

selling and exporting it are all untouched.

The arms control regimes that succeeded regulated production,

transfer and stockpiles.

Deleting design and development deletes the only part that would ever reach

a factory in Ohio.

And predictability.

As you said, predictability is a technical term which asks whether a system will

do will do the same thing twice given similar inputs,

right? And whether a commander can know in advance what it's going to do.

For three years the AI industry has described unpredictability as

a property of these systems, an inherent,

~ ineradicable. Like it's it's just in there.

We can't predict what they're doing.

There's there's such a big black box.

And something they

Right.

don't expect to patch out, right?

So labs publish papers on capabilities they didn't anticipate

and run interval evaluations.

Because training doesn't tell them how a model's gonna behave.

But some companies have said development should slow down and they want over outside

oversight. In fact, I think this last weekend all three of the

big frontier model companies, or three of the big frontier model companies,

I shouldn't say all three. Because I think Meta was not on board.

But the other

Of course not.

the the others said that you know they won outside oversight.

Now I I don't know if all that was sincere,

but it is on the record.

I mean, I I do think that that they let the publicity do a little bit

of work on that, right? Like the idea that that hey look,

we stood against this. We said we need to slow down despite the fact that we're not,

you know, I mean whatever. ~ but the US delegation actually took

a position directly on predictability that its own industry

is contradicting in public. Like the president specifically said this week

on a plane that

It doesn't need guardrails, we don't need and w when they asked what he used it for,

he just said so many things, but I'm not gonna tell you what any of those are.

Right, and but from memory serves I believe he called in on a podcast

to say all this. ~ but

No, this was

on this was on Air Force One. ~ he was asked specifically outside of

Air Force One by a reporter, I think it was it was ABC News.

That's why it was, 'cause he asked he asked like,

Who are you with? And she and she said,

we're with ABC News and he was like,

~ ABC fake news, like it it's but couldn't literally answer what he even uses

AI for, but said how amazing it was.

Right.

Right. So I mean we have we have here the the administration,

~ which you know, I know Donald Trump is extremely smart and high,

very high intelligence, but he hasn't

Highest intelligence actually.

the highest, right? I mean, but he hasn't studied AI at a at a doctoral level

as far as I know. but maybe he just asked OpenAI or asked meta's ~ model,

but anyway or Grok,

Or Grok, most likely Grok.

probably Grok, yeah, if we're for being honest there.

But ~

But basically we have an administration disputing the facts with

the companies that are the ones driving this whole thing.

And they're they're th th they're now of a disagreement not over the values

of what should be done, but actually what the facts are.

Are these things predictable or not?

Yeah, and there are there are people on the ground who are are reporting on this.

I mean there are there are specific like attaches that are are their only

job is to work directly inside the UN to report on these things,

to to discuss what they're discussing and talking about.

And ~ a a lot of the alarm for this podcast and some of the driver

for it was specifically one of those reporters.

but people in the room allegedly tried to stop this.

Right. They and the coverage tends to describe these outcomes as as if like,

well, nobody was really responsible for it.

But Verity Coyle of Amnesty International was actually there in the room,

and so was ~ Nicole Van Ruihen,

who leads the Stop Killer Robots Coalition,

which that's great. Stop killer robots.

~ and

Cool business card.

Robert in Den Bosch, the Dutch diplomat who chaired the group,

Humans human rights watch documented the session ~ on December seventh.

by the close of it, seventy six states said that they support moving

to negotiations before the the November review conference.

And of course in most bodies that block would be enough to move forward.

In the UN it counts for very little because the threshold is every state

in the room.

Right. And the thing that's probably the most confusing about this

is that Russia and the United States positions actually look identical on paper.

Both say that national guidelines are are fine.

Like we just need national guidelines and and you know,

restricted at a national level,

it doesn't need to happen at the UN level,

basically saying that existing humanitarian laws already apply here and that

no new binding instruments are actually needed.

because those, you know, existing human rights laws already apply and they have they

have their own teeth.

So they they may get to pretty much the same result,

but they got there from different directions,

right? Russia is openly obstructive.

It blocked the rules of procedure,

it kept civil society out, and it's fighting a war right now where drone autonomy

is developing on both sides faster than anyone planned.

They want no observers and no rules,

and they've made that painfully clear.

The American position is a little bit stranger,

and I think actually more dangerous because officials mean it.

The US believes it can regulate itself better than a treaty would.

There are defense departments directives,

~ review or sorry, Department of War directives,

~ the review processes and lawyers at every level.

And many of the people who wrote those policies have thought about this harder than

their critics.

But tell why.

A national guideline is a policy,

and a policy can be revised, waived,

or classified by whoever's in office.

It binds no other country, has no enforcement mechanism,

and can vanish after a single election.

To keep it, the US gave up a binding rule that would have constrained

its adversaries to, probably for a generation,

and it called that sovereignty.

So that's I think that's the the biggest hidden point in all of this right there

is ultimately that the US keeps saying we have to keep going,

because if we don't, China's gonna overtake us or Russia might overtake us.

But they're they actually had the opportunity to constrain them as well

and still fought against that.

And the the fourteen point six billion dollars in

Venture capital is not waiting on any of these policies.

Let's let's take a turn back to Libya,

which we discussed briefly earlier.

That's the incident everyone in this debate cites,

and almost no one describes it carefully,

so we're gonna go slowly through it.

In March 2020, so this is six years ago,

~ outside Tripoli, a civil war was underway between

the internationally recognized government and forces loyal to

the commander Khalifa Haf Haftar.

And Haftar's units broke and retreated,

according to a UN panel of experts report published in March twenty twenty one.

five hundred and forty eight page report,

the retreating forces were hunted down and remotely engaged

by unmanned combat aerial vehicles and lethal autonomous weapon systems.

So literally we're talking about ~ retreating soldiers basically hunted down

and gunned down. The system

Exactly like

boats, you know, in the in the Gulf of Mexico and or sorry,

Gulf of America and the Pacific and Atlantic Oceans.

I mean, i this is really this is really dark stuff here,

but like hunted down and remotely engaged.

Now I will say we did not re read this five hundred and forty eight page report.

It it we AI summarized it, ~ to be fair.

But th these are real the this is something that actually happened.

Sorry to interrupt.

Right.

No worries. And the system named the system in this incident was named

the STM Cargo two, a Turkish loitering munition,

which is a small drone that carries an explosive charge,

and it just circles until it finds a target and then flies to that target,

and one soldier can carry one of these.

So the report says it was programmed to attack targets without needing

a data connection between operator and munition,

a capability the report calls fire,

forget, and find.

Ugh. This is so gross. I

All right.

mean, these are literally like kamikaze drones,

but at least in the kamikaze world,

like it was it was one for one,

right? Like when during during World War Two,

Right. A person who did.

the

Japan was one for one, right? Like it was a real person choosing to

to carry out that action. Now,

depending on who's citing it, right,

that incident gets presented as either like

Proof that this is this is a real situation that we really are faced with,

or they just wave it off like it's a rumor.

Both of those readings are somewhat wrong.

The report does not confirm that anyone died.

So it's not proof, but it it is,

you know, it's more than just rumor.

Zachary Callenborne, the analyst who brought it into public debate,

wrote it up in the Bulletin of the Atomic Scientist in May 2021 under

a headline that ended with a question mark asking whether a flying killer robot

was used in Libya. His answer was quite possibly.

But the uncertainty is the finding.

We don't know whether a machine killed a person in Libya.

But there was a UN panel, a five hundred and forty eight page report,

and a team of investigators, and the answer is still unknown.

And those were combatants like on an active like battlefield where

the law provides not very many protections,

right, than it does for like civilians.

So if investigators can't establish what actually happened

to those fighters, a strike on a residential building will be even harder

to reconstruct or try to figure out after the

And that's the accountability gap.

The men in that report also have no names.

They appear as retreating half tar affiliated forces.

Most are probably young as soldiers usually are,

and they were running, which the law of war treats as a situation with specific

limits on what you may do to a person.

The judgment about them ended up as a question mark in a UN document,

and we'll likely never know who they were.

Timeline is what makes this systemic,

right? So a report came out in March 2021,

and the sixth review conference ended in December of 2021.

That means diplomats in Geneva had the phrase literally,

and we quote this hunted down and remotely engaged in a UN report right

in front of them. And they still stripped the substance from their own text

on the last day.

And deployment of these weapons has outrun negotiations ever at every stage since

then. The second case we have is Gaza,

where AI generated the targets and a person approved them in the end.

In April 2024, the Israeli journalist Yuval Abraham published an investigation

in ~ Pl plus nine seven two magazine ~ in local call based

on interviews with six Israeli intelligence officers who served during the war.

he had already reported on a system called the Gospel,

ironically, which

~ huh.

generated building targets. this investigation was about a system called Lavender.

Love these names.

According to those officers, Lavender was a machine learning opera system that

scored people in Gaza on how likely they were to be Hamas

or Islamic jihad operatives. And it produced a list of about thirty seven thousand

people.

And how much d like human oversight and human review do you think that last list

actually got?

Well the officers involved said reviews sometimes took seconds per target.

One described his role as a rubber stamp,

checking mainly that the target was a male.

There was also a companion system the officers called Where's Daddy,

which tracked flagged individuals

~ man.

and signaled when they entered their family homes.

That's awful.

That

is that is terrible. Meaning strikes were timed for when those

men were literally walking into their houses to greet their family.

That's what the reporting describes.

One documented case was the attempt to kill a Hamas commander named Ibrahim Biari.

~ it's logged as incident six seven two in the AI incident database.

The public actor archive researchers ~ used to record harms by ~ caused

by AI systems.

I I wanna take a second and just sort of mention that that does exist.

the AI incident database, but it is a public archive that that researchers use,

so it's not something that's maintained by any organization or government.

But Biari's case shows how the parts actually fit together,

right? According to the r this report,

the location intelligence was wrong and heavy weapons were used.

Not we're not talking bullets,

we're talking bombs, right? Like explosives or ~

You know, RPGs, those sorts of things,

and the strike caused destruction and civilian deaths across

an incredibly dense urban landscape.

A machine-generated target, meaning AI created target.

A person looked at it for a few seconds,

a gigantic heavy weapon system was selected,

and the information at the start of the chain was entirely incorrect.

Each step had its own independent justification as far as the reporting shows.

Nobody, not one person in the chain broke their own organizational procedure,

but the procedure produced that outcome.

AI also increases the number of decisions in addition to their speed.

So most of the alignment focuses on accuracy,

whether the model is right 90% of the time,

whether a human would do better.

The bigger change is volume. A human targeting cell can develop a limited number

of targets a week because people take time to be sure.

That limit worked as a moral constraint,

even though it looked logistical,

and nobody wrote it into policy because nobody had to.

You don't have to slow down the human team.

But install a system that outputs thirty seven thousand names

and that limit disappears. A decision to strike which used to be rare because

it took so much human effort becomes cheap.

So this report is also part of the the Geneva record.

Human Rights Watch has actually cited in submissions to

the General Assembly consultations and and Abraham's investigations

are actually referenced in UN level debate.

So the connection between Gaza and the words deleted on september fifth,

twenty twenty six is directly documented in a file.

And ethics, predictability, and design and development are the three provisions most

likely to have applied to a system like Lavender.

So what that means is a few powerful countries keep using the consensus rule

to strip the teeth out of every document with that the UN is producing,

every every resolution that the UN is creating at the very last minute.

And that happened on the final day in the twenty twenty one review conference

and again in the twenty twenty three ~ conference when Russia blocked

the rules of procedure and then shut out

All of civil society. And then again,

when the US and Russia forced erasures hours before the mandate expired

on September 5th, 2026. That's the same exact MO,

same pattern, same tactic, three times,

and that's only five years. And we're talking about AI.

So this is going to start happening more and more faster and faster.

Deployment.

Also keeps running ahead of the diplomacy,

right? A UN panel documented the Cargo 2 incident in March of 2021,

before the expert group had produced any regulatory text,

whatever whatever, you know, whatsoever.

Lavender and the gospel were actually in use during a war in 2024.

That Ander Rails lattice software is deployed within NATO right now.

And that mandate closed in September with a non binding document.

And the money keeps showing up wherever the law has a little loophole,

a little gray area. The Pentagon announced the third offset in twenty fourteen.

That's the same year the CCW started talking about autonomous weapons systems.

In twenty twenty six, venture funding passed fourteen point six billion dollars

in five months, and Andy Rill's valuation doubled to sixty one billion dollars.

And the treaty process ended with nothing binding anyone to anything.

And those three trends you're talking about actually reinforce each other.

The consensus rule produces predictable failure.

Investors read that failure as a signal,

and capital builds the technology faster.

The faster it spreads, the more governments decide they can't accept constraints,

which strengthens the states using the consensus rule.

Each cycle makes the next one easier.

And people in active conflicts in Libya,

Sudan, Ukraine, and Gaza, they're exposed to AI assisted targeting with

no oversight, no rule requiring a person to make the decision

to kill multiple people.

And the US actually believes it to head here,

and I think that's a mistake.

W w let's hear what what do you why do you think it's a mistake?

Well, it's kinda like what you said earlier.

Any lead is temporary and shorter than most people assume.

This is software, and software spreads faster than almost any other technology.

The cargo too is Turkish. Loitering munitions are being improvised

and updated weekly in Ukraine workshops,

Ukrainian workshops by people with no national laboratory.

A machine that can identify a human figure and fly into it.

Now it takes a hobby drone, an off the shelf model,

and someone who can code. The advantage the US protects by refusing rules

is it disappearing on its own.

The US is also unusually exposed.

It's an open society with an enormous area to defend.

Ports, the power grid, rail lines,

airports with fences you could climb.

Autonomous weapons favor attackers since they're cheap,

expendable, scalable, and indifferent to their own losses.

A country whose defense strategy depends on projecting force abroad while staying

safe at home has more to lose from cheap autonomous weapons than just about anybody.

I I think there's definitely some irony there when it comes to the United States.

What

Mm-hmm.

what do you say of the sort of the AI industry's own concerns and warnings?

As we talked about, they're they're saying in public that these systems

are unpredictable. They can't fully explain model behavior,

and they discover capabilities after release.

Some have said that development is outpacing understanding,

that outside oversight is needed and that slowing down is reasonable.

I don't think they say that purely out of virtue.

Some of it is positioning, and some of it looks like regulatory capture.

The technical claim underneath is still true though,

and serious researchers don't dispute it.

Now, give that system a weapon and put it on a battlefield full of inputs

and nobody trained on it trained it on?

And facing another country's unpredictable system at machine speed,

with humans just too slow to step in.

Yeah, think of it like two black boxes,

right? Like escalating against each other before anyone can wake up the president.

And I mean the black box that like,

you know, the nuclear codes and all that stuff.

Right. And the killer robot imagery has damaged this debate because nobody serious

is worried about a metal skeleton with a rifle yet.

the worry is a flash

I mean

crash with munitions. two automated systems interacting in a

way that neither operator predicted for eleven seconds,

and at the end of those eleven seconds a war has started.

unless it's never been declared.

What the US

Right.

defends in Geneva is kind of the the option,

right? It's the it's the decision,

the right to figure it out later.

But for eighty years the American approach to international law has been just avoid

signing anything, right? Keep your options open and assume that this ambiguity

is gonna eventually favor the strongest country,

in which case it's mostly been us.

That approach has a a

A record a mile long. Drone strikes became normal,

and this indefinite detention became something that governments just do now.

Each time the US keeps its options open,

someone eventually used that open option against

us or against people we told that we would protect.

And the objection we needed was one we'd already given up.

And it it bears saying again, in case you missed it before,

after three years of this mandate,

the result is a document nobody has to follow,

missing three important words ethics,

predictability, and design and development.

Three most important words. I think over the next few months,

if you watch the November Review Conference and the count of the countries that

are asking for negotiations, which as of September is 76 countries,

see whether it grows before November and whether those states actually stay inside

the consensus process or start discussing a separate track in the corridors.

A separate track is basically how

The whole Canadian Ottawa thing happened.

122 different countries in 1997 who decided to go out of the process

to build a thing that could actually stop horrible things from happening.

Right? If diplomats start saying alternative forum or general assembly process,

they've basically decided that they're not going to ask permission anymore.

And I think that's a good thing for all of them.

And the same goes for deletions in any negotiation.

A story that says ~ talks failed to reach agreement usually leaves

out that most participants did agree.

In this case, 164 countries agreed.

~ asks which words were removed and by whom,

since the words removed words show the actual policy.

The United States and Russia are are further apart than at any point since

the Cold War. Except their leaders.

They are handling things very much the same.

They disagree about borders and energy,

who shot down what, the basic facts of the recent history that we're actively living

through. In a room in Geneva, in the last hours,

they both agreed that the same word,

ethics.

has to come out of this UN document about AI that kills people.

So under a rule where everyone has to agree,

the agreement produced that deletion.

Now you can act on this without calling anyone.

You know, I don't know how much we can really do right now

in our current environment because we don't have a whole lot of control over the UN.

And we certainly don't have control over the people who are supposed

to be representing our viewpoints at the UN.

But the UN Office for Disarmament Affairs actually runs a free public document

library with working papers, national statements,

and draft texts.

You can actually pull the draft from before the final session and the text that

was adopted after it and kind of compare them,

do a diff like left to right, side by side.

It takes about 40 minutes, and I'm not gonna lie,

it's not fun. But afterwards you will know then more than most of

the attaches and the reporters that are actually covering this in the trenches,

right? The post is the comparison that any of us can make because those deletions.

Stay invisible until somebody actually reads them and publishes them

and says something to somebody else about it.

If you're one of our listeners who works in the software industry,

get the end use terms in writing before you write the code.

Email your employer and ask what the export classification is

on what you're building, whether it's dual use,

and what the contract says about integration into targeting systems.

Most engineers never ask, and often nobody inside the company knows the answer,

which tells you something too.

An email creates a paper trail that can come out in discovery,

which an open letter never will.

Also go local, ~ because the same companies sell in both directions,

right? So firms that build autonomous military systems also build flock cameras

and surveillance towers and sensor fusion and border and police technology

and ice vehicles and ice systems and city council procurement meetings

are among the least attended and most winnable rooms in the country.

You can actually read every single contract.

You can ask about data retention.

You can ask about the vendor software,

whether they make automated decisions about people and get answers on the record.

A county commission can be made to read a contract out loud,

and that changes what gets put bought sometimes.

The mayor of Oklahoma City doesn't apparently give a shit,

but it can make a difference.

So ending sort of where we began,

Christoph Haynes asked for a moratorium in twenty thirteen.

In the thirteen years since, there's been a probable killing machine killing

in Libya, a list of thirty seven thirty seven thousand names in Gaza,

and fourteen point six billion dollars in venture funding in five months,

and nobody has granted it.

And as of this month, seventy six countries are asking to negotiate that.

And the review conference is still coming up in November.

that's the overlap for this week.

Joshua, where can they find us if they wanted to continue this discussion?

FOF dot foundation, that is FOF Friends of the Future.

Foundation, no dot com, no dot org,

no dot AI, nothing like that. These are made by real people,

you and me. ~ you can also check us out on every podcast app of your choosing.

That can be ~ Apple Podcasts or Spotify or Podbean or wherever you can find

~ the overlap. If you have trouble finding us,

Or your friend had trouble finding us,

tell us tell them to type in F O F F Friends of the Future Foundation

~ after the overlap and they will be able to find us quickly that way.

All right. Thanks for being here with me,

Will.

Thank you, Joshua. Talk to you next time.

Talk to you guys next time.