1
00:00:00,500 --> 00:00:05,882
In twenty thirteen, a South African human
rights lawyer named Christoph Haines stood

2
00:00:05,882 --> 00:00:10,313
in front of the UN Human Rights Council in
Geneva and warned a room full

3
00:00:10,313 --> 00:00:13,734
of diplomats that governments were getting
ready to give the decision

4
00:00:13,734 --> 00:00:16,695
to kill a person entirely over to
software.

5
00:00:16,695 --> 00:00:21,796
He was the special reporter on
extrajudicial executions,

6
00:00:21,796 --> 00:00:25,737
which made him the person the UN sends to
count the bodies after

7
00:00:25,737 --> 00:00:28,968
a government decides the law doesn't apply
to certain people.

8
00:00:29,468 --> 00:00:33,870
He wrote the first UN report on what he
called lethal autonomous robotics.

9
00:00:33,870 --> 00:00:38,033
And what he asked for was pretty modest a
moratorium.

10
00:00:38,033 --> 00:00:42,325
Stop building these until someone works
out whether a machine

11
00:00:42,325 --> 00:00:45,016
can legally decide that you should die.

12
00:00:45,016 --> 00:00:48,268
On September 5th, 2026,

13
00:00:48,268 --> 00:00:54,171
just 10 days ago, the UN group that grew
out of that warning finished

14
00:00:54,171 --> 00:00:55,962
its three year mandate in Geneva.

15
00:00:55,962 --> 00:00:57,453
It's called the Group.

16
00:00:57,625 --> 00:01:02,536
Of governmental experts on lethal
autonomous weapons systems.

17
00:01:02,536 --> 00:01:08,858
And states have been meeting on this
subject for some form for about twelve
years.

18
00:01:08,858 --> 00:01:12,319
The document they produced at the end is
nonbinding.

19
00:01:12,319 --> 00:01:14,979
In the final hours, the United States and
Russia,

20
00:01:14,979 --> 00:01:17,900
two countries that agree on almost nothing
right now,

21
00:01:17,900 --> 00:01:21,661
except on who's the President of the
United States,

22
00:01:21,661 --> 00:01:25,082
both push to delete language from the
text.

23
00:01:25,464 --> 00:01:27,075
References to ethics came out,

24
00:01:27,075 --> 00:01:29,357
and so did references to predictability.

25
00:01:29,357 --> 00:01:33,049
So did references to the design and
development stage,

26
00:01:33,049 --> 00:01:37,762
which is the only point where anyone could
stop these weapons before they exist.

27
00:01:37,762 --> 00:01:43,046
As of this month, seventy-six states say
they want to open negotiations

28
00:01:43,046 --> 00:01:48,059
on a binding treaty. That's a a large
block and still can't force anything
because

29
00:01:48,059 --> 00:01:52,953
the room runs on consensus. One country
can block the result and two showed

30
00:01:52,953 --> 00:01:54,564
up ready to. Meanwhile,

31
00:01:55,066 --> 00:01:56,987
In the first five months of 2026,

32
00:01:56,987 --> 00:02:03,572
Venture Capital put more than 14.6 billion
into autonomous weapons companies.

33
00:02:03,572 --> 00:02:08,876
AndyRail Industries closed a five billion
dollar round in May at

34
00:02:08,896 --> 00:02:12,399
60 at a sixty-one billion dollar
valuation.

35
00:02:12,399 --> 00:02:16,282
It booked two point two billion in revenue
last year and is building

36
00:02:16,282 --> 00:02:21,326
a one billion dollar factory in Ohio to
mass produce autonomous systems.

37
00:02:22,137 --> 00:02:26,430
While diplomats argued over whether the
word ethics could stay in a paragraph,

38
00:02:26,430 --> 00:02:30,383
investors had already decided what they
expected Geneva to produce.

39
00:02:30,383 --> 00:02:36,087
Haynes died in twenty twenty one before
the Years Review Conference collapsed

40
00:02:36,087 --> 00:02:39,489
and before Gaza even started. He never got
his moratorium.

41
00:02:39,989 --> 00:02:41,921
This isn't a story about a treaty that
failed,

42
00:02:41,921 --> 00:02:44,944
because there is no treaty. There never
has been a treaty.

43
00:02:44,944 --> 00:02:48,478
There is not right now in September of two
thousand twenty six as we

44
00:02:48,478 --> 00:02:53,793
sit here a single binding international
rule anywhere on earth that says

45
00:02:53,793 --> 00:02:55,645
a human being has to be the one to pull
the trigger.

46
00:02:56,145 --> 00:03:00,530
Which leaves questions about who decided
and when that the rest of

47
00:03:00,530 --> 00:03:01,894
us had already accepted this.

48
00:03:19,255 --> 00:03:20,255
Hi Will.

49
00:03:20,160 --> 00:03:21,160
Hi Joshua.

50
00:03:21,212 --> 00:03:23,837
This is the overlap. We cover systems of
power,

51
00:03:23,837 --> 00:03:26,102
labor, and American injustice,

52
00:03:26,102 --> 00:03:29,730
and today's system is the UN disarmament
process,

53
00:03:29,730 --> 00:03:32,706
which I realize sounds a little bit like
homework.

54
00:03:33,206 --> 00:03:34,714
It does, but some of us like homework.

55
00:03:35,214 --> 00:03:39,751
And then you'll enjoy a room in Geneva
where for a decade a handful

56
00:03:39,751 --> 00:03:44,177
of powerful countries have used a
procedural rule to make sure nothing
legally

57
00:03:44,177 --> 00:03:47,753
binding ever gets written about machines
that kill people on their own.

58
00:03:48,253 --> 00:03:49,984
The timing is why we're covering it right
now.

59
00:03:49,984 --> 00:03:53,485
The mandate that ended September 5th was a
three-year cycle,

60
00:03:53,485 --> 00:03:56,286
and there's a review conference in
November of 2026.

61
00:03:56,286 --> 00:03:59,787
That's when the states pushing for
negotiations either force the issue

62
00:03:59,787 --> 00:04:02,938
or lose another five years. In 2023,

63
00:04:02,938 --> 00:04:06,379
Secretary General Antonio Guterres and the
President of

64
00:04:06,379 --> 00:04:09,200
the International Committee of the Red
Cross jointly called for

65
00:04:09,200 --> 00:04:12,751
a legally binding instrument on autonomous
weapons by 2026.

66
00:04:13,725 --> 00:04:16,299
We're about three and a half months out
from the end of that deadline,

67
00:04:16,299 --> 00:04:19,463
and what exists is a non-binding text with
the word,

68
00:04:19,463 --> 00:04:21,516
among others, with the word ethics cut
out.

69
00:04:22,016 --> 00:04:26,338
So in the US, this debate usually gets
framed one very particular way.

70
00:04:26,338 --> 00:04:29,999
The rest of the world wants to tie our
hands,

71
00:04:29,999 --> 00:04:34,281
and we can't accept that because China
won't.

72
00:04:34,281 --> 00:04:38,062
You hear it on think tank panels and in
our Pentagon statements.

73
00:04:38,062 --> 00:04:41,684
I think that framing is a little bit
backwards.

74
00:04:41,684 --> 00:04:44,735
Of all of the countries involved,

75
00:04:44,735 --> 00:04:50,427
the United States should be the most
worried about a world with no rules here.

76
00:04:50,927 --> 00:04:51,927
Why is that, Joshua?

77
00:04:52,289 --> 00:04:55,901
Well a a technological lead lasts a few
years,

78
00:04:55,901 --> 00:04:58,853
right? And and a precedent lasts decades.

79
00:04:58,853 --> 00:05:03,587
Every rule the US deletes from a UN
document in twenty twenty six is

80
00:05:03,587 --> 00:05:07,819
a rule that likely won't exist or have an
opportunity to until twenty thirty four,

81
00:05:07,819 --> 00:05:10,621
when somebody else has the better system.

82
00:05:10,621 --> 00:05:13,486
We already ran this experiment with drones
actually.

83
00:05:13,486 --> 00:05:17,946
~ the US built a legal rationale for
targeted killing by aircraft,

84
00:05:17,946 --> 00:05:20,658
used it for twenty years and called it.

85
00:05:20,959 --> 00:05:23,692
acceptable because we were the only ones
flying,

86
00:05:23,692 --> 00:05:26,295
right? Like we only we were the only ones
who had the technology to do it.

87
00:05:26,295 --> 00:05:31,141
And now Turkish, Iranian and Russian
drones are pretty much everywhere,

88
00:05:31,141 --> 00:05:33,564
~ not to mention Ukrainian drones.

89
00:05:33,564 --> 00:05:38,870
The the legal objection we would need
today is one we spent basically

90
00:05:38,870 --> 00:05:40,412
two decades tearing apart.

91
00:05:40,912 --> 00:05:44,083
And that's the problem that we see with
the you know the nuclear option

92
00:05:44,083 --> 00:05:45,373
in the Senate and things like that.

93
00:05:45,373 --> 00:05:48,304
It's like when you're in charge and you
make the rules to benefit your side,

94
00:05:48,304 --> 00:05:51,515
you have to be prepared for them to serve
the other side when they're in charge.

95
00:05:51,515 --> 00:05:54,356
but autonomy here goes further than drones
did,

96
00:05:54,356 --> 00:05:57,296
right? Drones are bad enough, but a drone
strike still had a person in

97
00:05:57,296 --> 00:05:59,657
a trailer in Nevada watching a screen.

98
00:05:59,657 --> 00:06:02,428
And don't get me wrong, the outcomes were
often terrible,

99
00:06:02,428 --> 00:06:05,588
but there was a person whose name could
end up on a document.

100
00:06:05,588 --> 00:06:07,609
Somebody had to to authorize the attack.

101
00:06:08,424 --> 00:06:10,397
An autonomous system removes that person.

102
00:06:10,397 --> 00:06:12,920
When the machine picks a target and gets
it wrong,

103
00:06:12,920 --> 00:06:15,905
who gets charged? The operator who wasn't
operating?

104
00:06:15,905 --> 00:06:17,908
The commander who approved the setting?

105
00:06:17,908 --> 00:06:21,413
The engineer in Ohio, or some model that
nobody can fully explain or understand?

106
00:06:21,913 --> 00:06:26,075
And predictability, meaning like the
ability to know in advance what

107
00:06:26,075 --> 00:06:28,166
the system is actually going to do,

108
00:06:28,166 --> 00:06:30,927
we call that eval in the AI world,

109
00:06:30,927 --> 00:06:35,860
was one of the words was that was actually
cut from the text just this month.

110
00:06:36,360 --> 00:06:39,342
Which is strange because the companies
building AI have said for a couple

111
00:06:39,342 --> 00:06:42,994
of years now that they don't fully
understand the behavior of what they're
shipping.

112
00:06:42,994 --> 00:06:46,926
They report capability jumps they didn't
predict and behaviors they didn't train

113
00:06:46,926 --> 00:06:49,917
for, and they fund research teams whose
whole job is to figure

114
00:06:49,917 --> 00:06:51,098
out what their products are doing.

115
00:06:51,598 --> 00:06:56,241
So the builders describe this this risk in
public.

116
00:06:56,241 --> 00:06:57,352
I mean, we're hearing it everywhere,

117
00:06:57,352 --> 00:07:01,115
and the government of the country where
most of them are headquartered went

118
00:07:01,115 --> 00:07:06,849
to Geneva and just like struck sh struck
the word entirely from the document.

119
00:07:06,849 --> 00:07:12,423
So basically Geneva keeps failing on this
in a very particular way,

120
00:07:12,423 --> 00:07:15,875
and and the reason goes back to a treaty
that took effect in nineteen eighty three.

121
00:07:15,875 --> 00:07:18,577
The history. gosh, wouldn't you believe
it?

122
00:07:19,189 --> 00:07:21,120
We're gonna go back and cover the history
of it all.

123
00:07:21,120 --> 00:07:23,092
1983, it was a different world.

124
00:07:23,092 --> 00:07:28,396
But the convention on ~ I didn't.

125
00:07:24,940 --> 00:07:26,884
Let's not call that history. Some of us
live through that.

126
00:07:26,884 --> 00:07:28,528
~

127
00:07:28,396 --> 00:07:29,396
I

128
00:07:28,576 --> 00:07:31,848
was that's history for me. That's ancient
runic history,

129
00:07:31,848 --> 00:07:36,862
sir. But the convention on certain
conventional weapons,

130
00:07:36,862 --> 00:07:40,925
CCCW, usually called just the C C W

131
00:07:41,350 --> 00:07:43,262
It came out of the nineteen seventies,

132
00:07:43,262 --> 00:07:48,057
right? You had Vietnam, you had the napalm
and mustard gas and bomb fragments that

133
00:07:48,057 --> 00:07:49,528
that didn't show up on an X-ray.

134
00:07:49,528 --> 00:07:54,613
The Geneva Conventions mostly covered how
you treat people after capture,

135
00:07:54,613 --> 00:08:00,349
right? And had a lot less to say about
what happened and when and whether

136
00:08:00,349 --> 00:08:03,182
or not you're allowed to just shoot them
with a thing.

137
00:08:03,182 --> 00:08:05,224
The Red Cross and a group of

138
00:08:04,170 --> 00:08:05,550
As long as they die first, the Geneva

139
00:08:05,550 --> 00:08:06,550
Convention doesn't care.

140
00:08:06,885 --> 00:08:10,497
Well, I mean, to be fair, we do fund most
of the UN.

141
00:08:10,497 --> 00:08:16,220
~ the Red Cross and a group of states
pushed for something to actually fill that

142
00:08:16,220 --> 00:08:20,262
gap. The CCW is a a framework convention,

143
00:08:20,262 --> 00:08:23,162
basically a frame that bans nothing on its
own,

144
00:08:23,162 --> 00:08:26,295
with specific rules attached to it as
protocols.

145
00:08:26,295 --> 00:08:29,787
Think of it like guidelines. It's like a a
~ framework for how we're supposed

146
00:08:29,787 --> 00:08:32,098
to handle warfare with one another,

147
00:08:32,098 --> 00:08:36,330
especially UN countries. Protocol one

148
00:08:36,708 --> 00:08:40,662
Covers weapons that leave fragments
undetectable in the body.

149
00:08:40,662 --> 00:08:44,156
All of these are incredibly scary to talk
about,

150
00:08:44,156 --> 00:08:46,768
like it feels gross even saying them out
loud,

151
00:08:46,768 --> 00:08:52,094
like Protocol One covers just random shit
that might be left over in your body after

152
00:08:52,094 --> 00:08:55,147
war. but that's exactly what it covers.

153
00:08:55,647 --> 00:08:57,530
Terrifying is that that's the first one
that came up.

154
00:08:58,101 --> 00:09:00,952
Of course. I mean that's that's but it
came out of Vietnam.

155
00:09:00,952 --> 00:09:02,923
I mean it makes sense, right? Like lots of
strapnel.

156
00:09:02,165 --> 00:09:03,165
Right.

157
00:09:03,364 --> 00:09:06,745
So protocol two covers mines and booby
traps.

158
00:09:06,745 --> 00:09:11,307
And then protocol three covers incendiary
weapons like flamethrowers

159
00:09:11,307 --> 00:09:12,307
and things like that.

160
00:09:12,477 --> 00:09:14,352
Sure, your regular call of duty trifecta.

161
00:09:14,852 --> 00:09:18,006
Alright, so the idea was that it could
grow to encompass new weapons when they

162
00:09:18,006 --> 00:09:20,909
appeared and states write a new protocol
attach and attach

163
00:09:21,409 --> 00:09:24,891
And it grew once, right, in a in a way
that almost nobody remembers,

164
00:09:24,891 --> 00:09:27,342
including myself. In in nineteen ninety
five,

165
00:09:27,342 --> 00:09:33,336
Protocol four banned ~ blinding laser
weapons before any military

166
00:09:33,336 --> 00:09:35,147
had actually used them in the field.

167
00:09:35,147 --> 00:09:40,230
So prototypes existed that would basically
permanently blind soldiers

168
00:09:40,230 --> 00:09:41,951
and and states banned them in advance.

169
00:09:41,951 --> 00:09:45,053
For good reason, right? Like I mean that's
that's a pretty terrible thing to

170
00:09:45,053 --> 00:09:46,254
do to a human being, but

171
00:09:47,126 --> 00:09:49,698
The the major militaries actually signed
on for it.

172
00:09:49,698 --> 00:09:54,181
It was a a clear case of the world seeing
a weapon coming and agreeing like we've

173
00:09:54,181 --> 00:09:56,793
got to stop this. Like before it even
actually does it,

174
00:09:56,793 --> 00:09:59,245
no before anybody puts it on a you know,

175
00:09:59,245 --> 00:10:03,678
on a on a resolution anywhere or or a
truck or a Hum V,

176
00:10:03,678 --> 00:10:06,780
we need to ban this because this is
objectively,

177
00:10:06,780 --> 00:10:07,780
ethically wrong.

178
00:10:08,101 --> 00:10:12,514
Course there was little pushback on the ~
blind blinding laser weapons because

179
00:10:12,514 --> 00:10:14,155
nobody was really getting rich off of
those.

180
00:10:14,155 --> 00:10:17,788
However, I'll say they did pivot to
selling to high school kids so that they

181
00:10:17,788 --> 00:10:19,179
go around and try and blind their friends.

182
00:10:19,179 --> 00:10:20,560
Apparently.

183
00:10:19,722 --> 00:10:21,293
And pilots in the sky.

184
00:10:21,821 --> 00:10:22,821
And pilots, yeah.

185
00:10:23,252 --> 00:10:24,665
So I mean with no money behind it,

186
00:10:24,665 --> 00:10:28,602
nobody was there to argue about the ~
there's a laser lobby basically.

187
00:10:28,602 --> 00:10:31,579
No venture capital pouring into permanent
blindness technology.

188
00:10:30,502 --> 00:10:32,405
No no laser lobby, I like that one.

189
00:10:31,579 --> 00:10:33,222
Yeah.

190
00:10:32,405 --> 00:10:33,405
That's what I'm gonna

191
00:10:33,256 --> 00:10:35,450
call my my next band, the laser lobby.

192
00:10:35,250 --> 00:10:36,250
The laser

193
00:10:35,760 --> 00:10:40,323
lobby. Nice. the military benefit was
marginal and the horror was easy to
picture.

194
00:10:40,323 --> 00:10:43,464
I mean you blind a nineteen year old
conscripted to fight for his country

195
00:10:43,784 --> 00:10:46,306
for the next sixty years of his life and
you don't even win the battle.

196
00:10:46,306 --> 00:10:48,467
That ban cost nobody anything.

197
00:10:48,467 --> 00:10:52,008
The contrast sits in the same treaty in
the same decade though.

198
00:10:52,008 --> 00:10:55,070
The C C W's mine protocol, mines protocol,

199
00:10:55,070 --> 00:10:56,911
the protocol two that we talked about
before,

200
00:10:56,911 --> 00:11:01,773
was full of exceptions and where feasible
language and exempted nearly everything

201
00:11:01,773 --> 00:11:04,805
armies actually used. So it you know it it

202
00:11:05,032 --> 00:11:06,633
Said it on paper, but it had no teeth,

203
00:11:06,633 --> 00:11:09,394
right? And it was amended in nineteen
ninety six and stayed weak.

204
00:11:09,394 --> 00:11:11,585
So the campaigners, the survivors,

205
00:11:11,585 --> 00:11:14,916
and the doctors doing amputations on
children's in children in Cambodia,

206
00:11:14,916 --> 00:11:19,698
Angola, and Mozambique made a tactical
decision that changed arms control
forever.

207
00:11:19,698 --> 00:11:21,029
They left the C C W.

208
00:11:21,529 --> 00:11:24,034
And then went to of all places Ottawa,

209
00:11:24,034 --> 00:11:25,034
in Canada.

210
00:11:25,621 --> 00:11:28,348
Is that still named Ottawa? I'm I'm losing
track these days.

211
00:11:28,348 --> 00:11:30,132
Okay.

212
00:11:28,573 --> 00:11:30,310
It it is. He he i they're

213
00:11:30,310 --> 00:11:31,314
not another state yet.

214
00:11:32,256 --> 00:11:34,729
Got it. ~ in December nineteen ninety
seven,

215
00:11:34,729 --> 00:11:38,012
a hundred and twenty two states signed the
Mine Band Treaty in Ottawa.

216
00:11:38,012 --> 00:11:41,105
It was negotiated outside the UN consensus
process,

217
00:11:41,105 --> 00:11:44,018
with civil civil society and mid sized
countries driving it.

218
00:11:44,018 --> 00:11:46,641
Casu stayed out of it. The US,

219
00:11:46,495 --> 00:11:47,495
Hmm.

220
00:11:46,721 --> 00:11:47,772
Russia, and China.

221
00:11:48,419 --> 00:11:54,068
Giant surprise. I think I think that's
probably why people call it a failure now.

222
00:11:54,568 --> 00:11:57,661
Right, but on the other hand, look at the
production and trade numbers though.

223
00:11:57,661 --> 00:12:01,764
So when people stopped when the rest of
the world stopped wanting to use mines,

224
00:12:01,764 --> 00:12:05,268
~ the global trade and anti personnel
mines mostly collapsed.

225
00:12:05,268 --> 00:12:07,149
Manufacturers quit the business,

226
00:12:07,149 --> 00:12:10,723
banks divested, countries that never
signed stopped exporting

227
00:12:10,723 --> 00:12:12,154
and started clearing minefields.

228
00:12:12,154 --> 00:12:15,567
The stigma changed behavior in places the
signatures never reached.

229
00:12:16,067 --> 00:12:19,839
I I don't know, I would push back a little
bit on how well that actually transfers

230
00:12:19,839 --> 00:12:26,023
to reality. So a norm or a a a precedent
without like

231
00:12:26,023 --> 00:12:28,054
the the biggest powers in the world,

232
00:12:28,054 --> 00:12:32,396
right, that actually covers them except
the countries that fire the most.

233
00:12:32,396 --> 00:12:37,249
Ottawa succeeded partly because mines are
pretty cheap,

234
00:12:37,249 --> 00:12:38,780
right? They're crude, they're low tech.

235
00:12:38,780 --> 00:12:42,642
I was I was reading the other day ~ about
a a

236
00:12:43,220 --> 00:12:45,641
Something they did Vietnam where they
would take a hand grenade,

237
00:12:45,641 --> 00:12:48,091
remove the pen with the keep the clip
still on,

238
00:12:48,091 --> 00:12:51,092
cover it in mud, and let the mud dry.

239
00:12:51,092 --> 00:12:54,733
So basically you could just step on the
and break the mud,

240
00:12:54,733 --> 00:13:00,674
and essentially you have yourself a an
explosive or an IED of of sorts that that

241
00:13:00,674 --> 00:13:02,595
worked in the same way that a mine did.

242
00:13:02,595 --> 00:13:08,526
But no major powers security strategy
actually depended upon them,

243
00:13:08,526 --> 00:13:10,617
right? I mean, even the Viet Cong,

244
00:13:10,617 --> 00:13:11,997
especially coming out of this era.

245
00:13:12,673 --> 00:13:18,375
~ they used they used ambush techniques
and they used really low-tech weapons that

246
00:13:18,375 --> 00:13:20,335
are pretty awful, like objectively,

247
00:13:20,335 --> 00:13:24,766
if you read about them. ~ and and some of
those I did read about in

248
00:13:24,766 --> 00:13:26,497
the research for this episode.

249
00:13:26,497 --> 00:13:31,068
But ~ basically autonomous targeting,

250
00:13:31,068 --> 00:13:35,099
right? This basically AI targeting is is
incredibly different.

251
00:13:35,099 --> 00:13:38,100
So since 2014, the Pentagon

252
00:13:38,352 --> 00:13:41,994
Has pursued what it's called the third
offset strategy.

253
00:13:41,994 --> 00:13:48,678
And it's basically a formal bet that AI
and autonomous weapons systems would
define

254
00:13:48,678 --> 00:13:51,840
the next era of the American military
advantage.

255
00:13:51,840 --> 00:13:57,343
You'd be asking the US to basically give
up its entire plan for the next several,

256
00:13:57,343 --> 00:14:01,005
you know, decades, well, at least one
generation of kids,

257
00:14:01,005 --> 00:14:05,698
warfare, right? And that's pretty weird.

258
00:14:06,198 --> 00:14:09,541
True, and it does explain why the
consensus rule is the problem.

259
00:14:09,541 --> 00:14:14,095
The bargain was that the big militaries
get a veto and in exchange they stay

260
00:14:14,095 --> 00:14:15,546
in the room and the rules apply to them.

261
00:14:15,546 --> 00:14:20,471
The problem is the rules never get written
under the veto or the consensus rule.

262
00:14:20,471 --> 00:14:23,653
So what that bargain bought was their
attendance and nothing else.

263
00:14:23,653 --> 00:14:27,197
So they show up at the meetings but they
they don't they prevent anything from

264
00:14:27,197 --> 00:14:28,197
happening.

265
00:14:28,375 --> 00:14:31,556
I think that's kind of the the the hitch
in my crawl,

266
00:14:31,556 --> 00:14:34,377
you know, the ~ the burr in my sock.

267
00:14:34,377 --> 00:14:37,668
That rule itself, right,

268
00:14:37,668 --> 00:14:42,930
the the CCW consensus means nobody in the
room objects.

269
00:14:42,930 --> 00:14:45,391
And that's that's how consensus model is
supposed to work.

270
00:14:45,391 --> 00:14:50,573
~ I'm I'm a board member on a local choir
here and that's how we we operate.

271
00:14:50,573 --> 00:14:53,664
It's a consensus. If if anybody has any
objections,

272
00:14:53,664 --> 00:14:57,946
we keep talking it out or we keep
renegotiating until everybody agrees yes.

273
00:14:58,186 --> 00:15:02,637
puts their hand up. ~ a majority doesn't
actually count.

274
00:15:02,637 --> 00:15:05,038
So it's not like a two-thirds or a
supermajority.

275
00:15:05,038 --> 00:15:06,668
There's no there's no rule like that.

276
00:15:06,668 --> 00:15:09,789
Everybody needs to agree before we move
forward.

277
00:15:09,789 --> 00:15:14,250
So when a headline basically says the UN
failed to agree,

278
00:15:14,250 --> 00:15:20,271
it usually means more than a hundred
countries agreed and one said no,

279
00:15:20,271 --> 00:15:25,573
right? Or had no vote available to
actually overrule it.

280
00:15:25,573 --> 00:15:27,423
And that rule has been enforced

281
00:15:28,251 --> 00:15:32,337
Since nineteen eighty three, all the way
until about ten days ago.

282
00:15:32,837 --> 00:15:36,003
Let's lay out the dates on our current
situation because in sequence it looks
like

283
00:15:36,003 --> 00:15:37,035
a tactical move.

284
00:15:37,535 --> 00:15:42,516
In sequence, sure. I in ~ that's I think
one of the the benefits of of hindsight,

285
00:15:42,516 --> 00:15:45,497
right? Like being able to look back into
history and see it as opposed

286
00:15:45,497 --> 00:15:47,897
to looking for it. But in in twenty
thirteen,

287
00:15:47,897 --> 00:15:51,858
Haynes delivered his report, and in twenty
fourteen the C

288
00:15:51,858 --> 00:15:55,039
CW starts informal meetings on autonomous
weapons.

289
00:15:55,039 --> 00:16:00,941
In twenty sixteen, states formally created
the the group of governmental experts,

290
00:16:00,941 --> 00:16:05,602
which is a body with a mandate to
basically study the issue and it

291
00:16:05,602 --> 00:16:06,722
has studied it for years.

292
00:16:07,222 --> 00:16:09,067
During which in march twenty twenty,

293
00:16:09,067 --> 00:16:13,440
a Turkish made loitering munition may have
killed someone in Libya on its own.

294
00:16:13,440 --> 00:16:15,166
We'll get to that in a minute though.

295
00:16:15,666 --> 00:16:21,408
Yeah, so w then comes the sixth review
conference in December of twenty twenty
one,

296
00:16:21,408 --> 00:16:26,669
when years of meetings were supposed to
become a negotiating mandate.

297
00:16:26,669 --> 00:16:29,440
So dozens of states basically were like,

298
00:16:29,440 --> 00:16:31,521
Hey, like we want this to stick.

299
00:16:31,521 --> 00:16:33,472
We we need this to have some teeth.

300
00:16:33,472 --> 00:16:39,193
It needs to be a binding resolution and
and the UN panel of experts report already

301
00:16:39,193 --> 00:16:40,794
documented what happened in Libya.

302
00:16:41,413 --> 00:16:45,717
On that final day, a small group of of
countries sort of stripped

303
00:16:45,757 --> 00:16:48,620
the substantive content, all of the teeth,

304
00:16:48,620 --> 00:16:52,484
~ from the the conference report and
basically ended with

305
00:16:52,484 --> 00:16:57,718
no mandate for negotiations. ~ I think ~
in a lot of ways this is

306
00:16:57,718 --> 00:17:00,482
how the GOP tends to operate, right?

307
00:17:00,482 --> 00:17:05,257
It it seems like an innocuous overriding
of a of a procedure or a norm.

308
00:17:06,193 --> 00:17:09,337
But it has long lasting impacts and
consequences.

309
00:17:09,337 --> 00:17:11,450
Maybe they know, maybe they don't know,

310
00:17:11,450 --> 00:17:13,984
and and it and it sort of bites in the
butt later.

311
00:17:13,984 --> 00:17:17,379
And and I think we're kind of F A F Oing
right now on that.

312
00:17:17,879 --> 00:17:20,512
Well, one thing we know is that waiting
until the last day is deliberate.

313
00:17:20,512 --> 00:17:24,266
If you object in the second month of this
long drawn out process

314
00:17:24,266 --> 00:17:27,570
and and everybody else has a chance to
organize around you and respond,

315
00:17:27,570 --> 00:17:31,164
if you object in the final hours when
delegates have flights to catch home

316
00:17:31,164 --> 00:17:33,887
and and alternative to your weak text is
no text,

317
00:17:33,887 --> 00:17:35,219
you get what you want.

318
00:17:35,719 --> 00:17:38,340
Yeah, I I mean I I definitely draw the
parallels there,

319
00:17:38,340 --> 00:17:40,480
especially with the Senate and Congress,

320
00:17:40,480 --> 00:17:43,830
right? Like they they want a filibuster at
the last minute when they could have been

321
00:17:43,830 --> 00:17:45,472
having the negotiations all along.

322
00:17:45,472 --> 00:17:52,082
~ and so I I I agree that it can
definitely be used as its own negotiating
tactic.

323
00:17:52,082 --> 00:17:57,315
But the nineteen eighty three rules
actually allow that.

324
00:17:57,315 --> 00:17:59,195
Like as of in nineteen eighty three,

325
00:17:59,195 --> 00:18:02,496
the rules allowed for the ability to to
use that as a tact.

326
00:18:02,996 --> 00:18:07,058
Right. So fast forward to ~ fast forward
forty years to nine sorry,

327
00:18:07,058 --> 00:18:10,660
to twenty twenty three, at the annual
meeting of the treaties member states,

328
00:18:10,660 --> 00:18:13,131
Russia blocks adoptions of the rules of
procedure.

329
00:18:13,131 --> 00:18:16,013
So now we're talking about without adopted
rules,

330
00:18:16,013 --> 00:18:18,334
the meeting was it goes into informal
mode.

331
00:18:18,334 --> 00:18:19,790
So they're not even blocking the
substance,

332
00:18:19,790 --> 00:18:22,076
they're blocking the procedural rules to
have the meeting,

333
00:18:22,076 --> 00:18:25,728
~ to to you know, have it in front of the
public.

334
00:18:25,728 --> 00:18:28,340
And so they shut out basically the
campaigners,

335
00:18:28,340 --> 00:18:32,392
researchers, and survivor organizations
that track where all these weapons turn
up.

336
00:18:32,892 --> 00:18:36,896
Yeah, a a lot like you're seeing ~ when
these data centers go up and

337
00:18:36,896 --> 00:18:41,851
and things that the city council is just
keeping people from talking,

338
00:18:41,851 --> 00:18:44,213
blocking public comment for whatever
reason.

339
00:18:44,213 --> 00:18:48,587
~ that was actually the same year that
Gutierrez and the ICRC called

340
00:18:48,587 --> 00:18:52,061
for a something with actual teeth by
twenty twenty six.

341
00:18:52,561 --> 00:18:55,792
So we have the UN Secretary General asking
for a law by twenty twenty six,

342
00:18:55,792 --> 00:18:58,581
and a permanent member of the Security
Council responds by clearing

343
00:18:58,581 --> 00:18:59,715
the observers out of the room.

344
00:19:00,215 --> 00:19:07,170
Yeah. That's a forty three year system
that proved that once in nineteen ninety
five

345
00:19:07,551 --> 00:19:09,762
that it could ban a weapon before it
existed,

346
00:19:09,762 --> 00:19:11,614
right? The the lasers we were talking
about earlier,

347
00:19:11,614 --> 00:19:16,838
and then spent the next thirty years
showing that it could only manage that
when

348
00:19:16,838 --> 00:19:21,081
the weapon wasn't being made by anybody
for money.

349
00:19:21,081 --> 00:19:24,404
Right, when there couldn't be a profit
derived from it.

350
00:19:24,404 --> 00:19:29,378
But in May of this year, autonomous
weapons had drawn at least fourteen point

351
00:19:29,378 --> 00:19:30,378
six billion.

352
00:19:29,932 --> 00:19:33,967
Ba billion in venture capital in five
months.

353
00:19:34,467 --> 00:19:36,968
And one company explains a lot about where
that money went.

354
00:19:36,968 --> 00:19:40,448
Andoril Industries was founded by Palmer
Lucky,

355
00:19:40,448 --> 00:19:43,229
who you may know from the Oculus headset,

356
00:19:43,229 --> 00:19:45,990
VR headset, which he sold to Facebook,

357
00:19:45,990 --> 00:19:48,510
then left Facebook and started a defense
company.

358
00:19:48,510 --> 00:19:52,771
Brian Schemph is now the CEO of Andoril
and their core product is

359
00:19:52,771 --> 00:19:56,529
a software called Lattice. That's not not
Lattice like you eat,

360
00:19:56,529 --> 00:19:59,503
but Lattice, like, you know, ~ the side of
your house,

361
00:19:59,503 --> 00:20:02,954
the lattice work. ~ it takes the sensor
data from towers,

362
00:20:02,954 --> 00:20:04,154
drones, and radar.

363
00:20:04,824 --> 00:20:07,636
fuses it, creates a lattice, if you will,

364
00:20:07,636 --> 00:20:11,509
identifies objects and tells the operator
or the machine what's out there

365
00:20:11,509 --> 00:20:15,463
and what to do about it. Now Androil sells
directly to the Pentagon and

366
00:20:15,463 --> 00:20:18,384
to our NATO allies. In May 2026,

367
00:20:18,384 --> 00:20:23,988
Android closed a $5 billion Series H
round.

368
00:20:23,988 --> 00:20:26,430
Series H, I didn't even know they went
that far.

369
00:20:26,430 --> 00:20:28,612
At a $61 billion,

370
00:20:27,356 --> 00:20:28,944
They go all the way to Z, apparently.

371
00:20:29,613 --> 00:20:33,476
apparently, ~ $61 billion valuation on
that round.

372
00:20:33,714 --> 00:20:35,686
Roughly double its previous valuation.

373
00:20:35,686 --> 00:20:39,710
It had two point two billion in revenue in
twenty twenty five and it's spending

374
00:20:39,710 --> 00:20:42,332
a billion dollars on the Ohio factory it's
now creating.

375
00:20:42,832 --> 00:20:48,736
Yeah, that Round H closed about four
months before the Geneva mandate expired,

376
00:20:48,736 --> 00:20:54,220
without e without a a ~ rule or a binding
resolution of any kind.

377
00:20:54,220 --> 00:21:00,624
Investors, however, in these organizations
read the same news that we do.

378
00:21:00,624 --> 00:21:06,629
After a decade of Geneva with the US and
Russia both able to block anything,

379
00:21:06,629 --> 00:21:09,891
and the twenty twenty one collapse on the
record.

380
00:21:10,391 --> 00:21:13,476
It was a pretty rational bet that no
binding rule is coming.

381
00:21:13,476 --> 00:21:18,395
Somewhere there's probably a PowerPoint or
a pitch deck that lists

382
00:21:18,395 --> 00:21:23,645
the regulatory risk as low. And so far
that deck has proven to be correct.

383
00:21:24,145 --> 00:21:25,456
And as you touched on earlier,

384
00:21:25,456 --> 00:21:27,466
the usual explanation here is the arms
race,

385
00:21:27,466 --> 00:21:30,207
right? The US can't stop because China
won't stop,

386
00:21:30,207 --> 00:21:32,248
and China can't stop because the US won't
stop.

387
00:21:32,248 --> 00:21:36,450
That dynamic does exist. I'm not saying
it's it's completely manufactured,

388
00:21:36,450 --> 00:21:40,281
but I think the stronger driver in that
autonomy is the fastest growing part

389
00:21:40,281 --> 00:21:43,493
of the defense business. Ships take a
decade to build.

390
00:21:43,493 --> 00:21:47,855
Fighter program runs twenty years and
costs over a trillion dollars across its
life.

391
00:21:47,855 --> 00:21:51,696
Autonomy is software, so you can ship a
version this quarter and sell

392
00:21:51,696 --> 00:21:53,507
it to eleven countries at software
margins.

393
00:21:54,321 --> 00:21:57,724
Venture Capital spent five years looking
for the next category after consumer apps

394
00:21:57,724 --> 00:22:00,807
and crypto, and it found a customer with
the largest military budget

395
00:22:00,807 --> 00:22:02,479
on earth that rarely cancels anything.

396
00:22:02,979 --> 00:22:08,694
As unfortunate as that is, the fiscal
twenty twenty six US defense budget,

397
00:22:08,694 --> 00:22:11,696
the one we talked about a couple episodes
ago,

398
00:22:11,696 --> 00:22:17,881
was over nine hundred billion dollars with
a growing share specifically directed

399
00:22:17,881 --> 00:22:20,828
toward autonomy and AI.

400
00:22:21,328 --> 00:22:23,853
That's almost a t trillion dollars.

401
00:22:23,853 --> 00:22:27,031
Sorry, this doesn't sound as good on when
I do it there,

402
00:22:27,031 --> 00:22:28,494
but I thought I could claim that one.

403
00:22:27,177 --> 00:22:28,645
It doesn't. It it's that T

404
00:22:28,646 --> 00:22:30,658
R that T R vowel is hard to repeat.

405
00:22:30,820 --> 00:22:35,330
Yeah. ~ so when someone says we can't
regulate because of the arms race,

406
00:22:35,330 --> 00:22:38,657
what I hear is there's a customer with
enormous money and no rules,

407
00:22:38,657 --> 00:22:41,444
and Silicon Valley noticed about that same
same time.

408
00:22:41,944 --> 00:22:46,277
Yeah, and the diplomacy in this scenario
runs in in sort of parallel.

409
00:22:46,277 --> 00:22:50,440
On September twenty fourth, ~ just about a
year ago,

410
00:22:50,440 --> 00:22:54,362
the UN Security Council held a high level
debate

411
00:22:54,923 --> 00:22:58,605
on AI and international peace and
security,

412
00:22:58,605 --> 00:23:02,951
with heads of state and foreign ministers
attending and and Slovenia's president

413
00:23:02,951 --> 00:23:06,451
at the time, Natasha Pirce-Musar
presiding.

414
00:23:06,451 --> 00:23:11,234
Gutierrez basically said AI could do
enormous good.

415
00:23:12,033 --> 00:23:14,454
Which I think all of us would likely agree
with that.

416
00:23:14,454 --> 00:23:15,594
And if you've been following us,

417
00:23:15,594 --> 00:23:18,194
there are good things that it can be used
for.

418
00:23:18,194 --> 00:23:23,356
But just like anything else, it can be
weaponized without guardrails.

419
00:23:23,356 --> 00:23:28,057
And he called for human control over every
use of force.

420
00:23:28,057 --> 00:23:30,137
He's not saying like we shouldn't use AI.

421
00:23:30,137 --> 00:23:35,129
You can use AI to target, but it needs to
be backed up by a human being

422
00:23:35,129 --> 00:23:36,799
so that we have a way to blame.

423
00:23:36,799 --> 00:23:40,660
It so that we have ~ a method enforced

424
00:23:41,258 --> 00:23:47,860
with teeth to to blame this on a person
rather than I don't know what happened

425
00:23:47,860 --> 00:23:50,531
inside of this AI system. Right?

426
00:23:50,531 --> 00:23:54,352
That request for for that use of force,

427
00:23:54,352 --> 00:23:56,952
that's just a minimum standard.

428
00:23:56,952 --> 00:24:01,574
And nothing, nothing we've seen coming
doing the research for this,

429
00:24:01,574 --> 00:24:03,484
nothing in any of the the UN reports,

430
00:24:03,484 --> 00:24:05,235
nothing in even in the United States,

431
00:24:05,235 --> 00:24:08,516
actually contains any information about

432
00:24:08,698 --> 00:24:13,493
The idea that the use of force should ~
eventually come back to a human being.

433
00:24:13,493 --> 00:24:16,124
So on december first, twenty twenty five,

434
00:24:16,124 --> 00:24:22,361
the General Assembly adopted a resolution
on lethal autonomous weapons systems with

435
00:24:22,361 --> 00:24:24,763
a hundred and sixty-four in favor,

436
00:24:24,763 --> 00:24:28,727
six against, and seven completely
abstained.

437
00:24:29,227 --> 00:24:31,448
Not sure I need to ask, but who voted no?

438
00:24:31,948 --> 00:24:34,240
Well, they're all the people you'd think.

439
00:24:34,240 --> 00:24:38,525
So Belarus, Burundi, North Korea,

440
00:24:38,525 --> 00:24:43,650
Israel, Russia, and the United States of
America.

441
00:24:44,150 --> 00:24:46,943
There we are. So one hundred and sixty
four countries,

442
00:24:46,943 --> 00:24:48,975
including nearly every US ally.

443
00:24:48,975 --> 00:24:52,098
Every other NATO member, the EU,

444
00:24:52,098 --> 00:24:55,321
Japan, and most of Latin America and
Africa voted for rules.

445
00:24:55,321 --> 00:24:59,204
Only six voted against and two of them
have permanent security council seats

446
00:24:59,204 --> 00:25:00,746
and can block consensus in Geneva.

447
00:25:01,246 --> 00:25:05,168
Right. So in that system, those two,

448
00:25:05,168 --> 00:25:08,291
specifically China, Russia, or three,

449
00:25:08,291 --> 00:25:10,072
I guess, China, Russia, the United States,

450
00:25:10,072 --> 00:25:12,653
they don't need to win the argument.

451
00:25:12,653 --> 00:25:16,655
Basically nobody can outvote them at all.

452
00:25:16,655 --> 00:25:20,437
Right? So on september fifth of this year,

453
00:25:20,437 --> 00:25:26,521
the three year mandate ended with non
binding consensus text that people

454
00:25:26,521 --> 00:25:29,003
in the room describe as much weaker.

455
00:25:29,377 --> 00:25:31,930
than the draft from just forty-eight hours
before that.

456
00:25:31,930 --> 00:25:37,788
So the US and Russia forced late
eradications,

457
00:25:37,788 --> 00:25:41,262
deletions, w w erasures of ethics,

458
00:25:41,262 --> 00:25:45,227
predictability, and design and
development.

459
00:25:45,727 --> 00:25:48,158
And each one of those deletions
individually tells you something.

460
00:25:48,158 --> 00:25:51,179
Now you can defend cutting ethics if you
argue that

461
00:25:51,179 --> 00:25:55,040
the international humanitarian law is the
legal standard and ethics is a philosophy.

462
00:25:55,040 --> 00:25:58,321
I disagree, but it's an argument that can
be made.

463
00:25:58,321 --> 00:26:01,281
But design and development has real
consequences.

464
00:26:01,281 --> 00:26:04,622
If obligations only start at the moment
you use the weapon,

465
00:26:04,622 --> 00:26:06,803
then building it, training it,

466
00:26:06,803 --> 00:26:08,924
selling and exporting it are all
untouched.

467
00:26:08,924 --> 00:26:12,335
The arms control regimes that succeeded
regulated production,

468
00:26:12,335 --> 00:26:13,445
transfer and stockpiles.

469
00:26:14,121 --> 00:26:16,806
Deleting design and development deletes
the only part that would ever reach

470
00:26:16,806 --> 00:26:17,806
a factory in Ohio.

471
00:26:18,181 --> 00:26:19,181
And predictability.

472
00:26:19,542 --> 00:26:23,513
As you said, predictability is a technical
term which asks whether a system will

473
00:26:23,513 --> 00:26:25,923
do will do the same thing twice given
similar inputs,

474
00:26:25,923 --> 00:26:29,444
right? And whether a commander can know in
advance what it's going to do.

475
00:26:29,444 --> 00:26:33,445
For three years the AI industry has
described unpredictability as

476
00:26:33,445 --> 00:26:35,306
a property of these systems, an inherent,

477
00:26:35,306 --> 00:26:38,566
~ ineradicable. Like it's it's just in
there.

478
00:26:38,566 --> 00:26:39,566
We can't predict what they're doing.

479
00:26:39,517 --> 00:26:41,147
There's there's such a big black box.

480
00:26:41,147 --> 00:26:42,277
And something they

481
00:26:41,867 --> 00:26:42,867
Right.

482
00:26:42,277 --> 00:26:44,028
don't expect to patch out, right?

483
00:26:44,028 --> 00:26:47,539
So labs publish papers on capabilities
they didn't anticipate

484
00:26:47,539 --> 00:26:49,079
and run interval evaluations.

485
00:26:49,527 --> 00:26:51,698
Because training doesn't tell them how a
model's gonna behave.

486
00:26:51,698 --> 00:26:55,739
But some companies have said development
should slow down and they want over
outside

487
00:26:55,739 --> 00:26:58,380
oversight. In fact, I think this last
weekend all three of the

488
00:26:58,380 --> 00:27:02,329
big frontier model companies, or three of
the big frontier model companies,

489
00:27:02,329 --> 00:27:05,862
I shouldn't say all three. Because I think
Meta was not on board.

490
00:27:05,862 --> 00:27:07,093
But the other

491
00:27:06,272 --> 00:27:07,272
Of course not.

492
00:27:07,093 --> 00:27:11,144
the the others said that you know they won
outside oversight.

493
00:27:11,144 --> 00:27:12,915
Now I I don't know if all that was
sincere,

494
00:27:12,915 --> 00:27:13,915
but it is on the record.

495
00:27:14,365 --> 00:27:17,817
I mean, I I do think that that they let
the publicity do a little bit

496
00:27:17,817 --> 00:27:22,350
of work on that, right? Like the idea that
that hey look,

497
00:27:22,350 --> 00:27:26,563
we stood against this. We said we need to
slow down despite the fact that we're not,

498
00:27:26,563 --> 00:27:31,496
you know, I mean whatever. ~ but the US
delegation actually took

499
00:27:31,897 --> 00:27:36,950
a position directly on predictability that
its own industry

500
00:27:36,950 --> 00:27:41,463
is contradicting in public. Like the
president specifically said this week

501
00:27:41,463 --> 00:27:43,194
on a plane that

502
00:27:43,588 --> 00:27:47,490
It doesn't need guardrails, we don't need
and w when they asked what he used it for,

503
00:27:47,490 --> 00:27:51,182
he just said so many things, but I'm not
gonna tell you what any of those are.

504
00:27:52,156 --> 00:27:55,672
Right, and but from memory serves I
believe he called in on a podcast

505
00:27:55,733 --> 00:27:58,417
to say all this. ~ but

506
00:27:57,636 --> 00:27:58,636
No, this was

507
00:27:58,546 --> 00:28:02,029
on this was on Air Force One. ~ he was
asked specifically outside of

508
00:28:02,029 --> 00:28:05,591
Air Force One by a reporter, I think it
was it was ABC News.

509
00:28:05,591 --> 00:28:08,153
That's why it was, 'cause he asked he
asked like,

510
00:28:08,153 --> 00:28:11,490
Who are you with? And she and she said,

511
00:28:11,490 --> 00:28:13,167
we're with ABC News and he was like,

512
00:28:13,167 --> 00:28:18,620
~ ABC fake news, like it it's but couldn't
literally answer what he even uses

513
00:28:18,620 --> 00:28:21,072
AI for, but said how amazing it was.

514
00:28:20,593 --> 00:28:21,593
Right.

515
00:28:21,572 --> 00:28:26,093
Right. So I mean we have we have here the
the administration,

516
00:28:26,093 --> 00:28:30,445
~ which you know, I know Donald Trump is
extremely smart and high,

517
00:28:30,445 --> 00:28:32,975
very high intelligence, but he hasn't

518
00:28:32,049 --> 00:28:33,239
Highest intelligence actually.

519
00:28:33,375 --> 00:28:37,436
the highest, right? I mean, but he hasn't
studied AI at a at a doctoral level

520
00:28:37,436 --> 00:28:43,337
as far as I know. but maybe he just asked
OpenAI or asked meta's ~ model,

521
00:28:43,337 --> 00:28:45,347
but anyway or Grok,

522
00:28:43,604 --> 00:28:45,157
Or Grok, most likely Grok.

523
00:28:45,347 --> 00:28:47,428
probably Grok, yeah, if we're for being
honest there.

524
00:28:47,428 --> 00:28:48,428
But ~

525
00:28:48,224 --> 00:28:52,186
But basically we have an administration
disputing the facts with

526
00:28:52,266 --> 00:28:54,708
the companies that are the ones driving
this whole thing.

527
00:28:54,708 --> 00:28:57,970
And they're they're th th they're now of a
disagreement not over the values

528
00:28:57,970 --> 00:29:00,151
of what should be done, but actually what
the facts are.

529
00:29:00,151 --> 00:29:01,692
Are these things predictable or not?

530
00:29:02,192 --> 00:29:06,155
Yeah, and there are there are people on
the ground who are are reporting on this.

531
00:29:06,155 --> 00:29:10,739
I mean there are there are specific like
attaches that are are their only

532
00:29:10,739 --> 00:29:14,983
job is to work directly inside the UN to
report on these things,

533
00:29:14,983 --> 00:29:17,295
to to discuss what they're discussing and
talking about.

534
00:29:17,295 --> 00:29:23,069
And ~ a a lot of the alarm for this
podcast and some of the driver

535
00:29:23,069 --> 00:29:25,702
for it was specifically one of those
reporters.

536
00:29:25,702 --> 00:29:30,646
but people in the room allegedly tried to
stop this.

537
00:29:31,610 --> 00:29:36,242
Right. They and the coverage tends to
describe these outcomes as as if like,

538
00:29:36,242 --> 00:29:38,584
well, nobody was really responsible for
it.

539
00:29:38,584 --> 00:29:43,416
But Verity Coyle of Amnesty International
was actually there in the room,

540
00:29:43,416 --> 00:29:46,738
and so was ~ Nicole Van Ruihen,

541
00:29:46,738 --> 00:29:50,740
who leads the Stop Killer Robots
Coalition,

542
00:29:50,740 --> 00:29:53,721
which that's great. Stop killer robots.

543
00:29:53,721 --> 00:29:55,122
~ and

544
00:29:54,458 --> 00:29:55,458
Cool business card.

545
00:29:55,822 --> 00:30:00,525
Robert in Den Bosch, the Dutch diplomat
who chaired the group,

546
00:30:01,455 --> 00:30:05,710
Humans human rights watch documented the
session ~ on December seventh.

547
00:30:05,710 --> 00:30:09,776
by the close of it, seventy six states
said that they support moving

548
00:30:09,776 --> 00:30:13,721
to negotiations before the the November
review conference.

549
00:30:14,221 --> 00:30:17,104
And of course in most bodies that block
would be enough to move forward.

550
00:30:17,104 --> 00:30:20,698
In the UN it counts for very little
because the threshold is every state

551
00:30:20,698 --> 00:30:21,698
in the room.

552
00:30:21,599 --> 00:30:25,200
Right. And the thing that's probably the
most confusing about this

553
00:30:25,881 --> 00:30:31,703
is that Russia and the United States
positions actually look identical on
paper.

554
00:30:31,703 --> 00:30:35,644
Both say that national guidelines are are
fine.

555
00:30:35,644 --> 00:30:38,145
Like we just need national guidelines and
and you know,

556
00:30:38,145 --> 00:30:39,406
restricted at a national level,

557
00:30:39,406 --> 00:30:41,307
it doesn't need to happen at the UN level,

558
00:30:41,307 --> 00:30:47,369
basically saying that existing
humanitarian laws already apply here and
that

559
00:30:47,369 --> 00:30:50,230
no new binding instruments are actually
needed.

560
00:30:50,452 --> 00:30:56,070
because those, you know, existing human
rights laws already apply and they have
they

561
00:30:56,070 --> 00:30:57,070
have their own teeth.

562
00:30:57,252 --> 00:30:59,683
So they they may get to pretty much the
same result,

563
00:30:59,683 --> 00:31:01,232
but they got there from different
directions,

564
00:31:01,232 --> 00:31:03,154
right? Russia is openly obstructive.

565
00:31:03,154 --> 00:31:04,974
It blocked the rules of procedure,

566
00:31:04,974 --> 00:31:09,625
it kept civil society out, and it's
fighting a war right now where drone
autonomy

567
00:31:09,625 --> 00:31:12,216
is developing on both sides faster than
anyone planned.

568
00:31:12,216 --> 00:31:14,767
They want no observers and no rules,

569
00:31:14,767 --> 00:31:17,158
and they've made that painfully clear.

570
00:31:17,158 --> 00:31:20,058
The American position is a little bit
stranger,

571
00:31:20,058 --> 00:31:22,739
and I think actually more dangerous
because officials mean it.

572
00:31:22,739 --> 00:31:25,930
The US believes it can regulate itself
better than a treaty would.

573
00:31:26,596 --> 00:31:28,387
There are defense departments directives,

574
00:31:28,387 --> 00:31:31,189
~ review or sorry, Department of War
directives,

575
00:31:31,189 --> 00:31:34,581
~ the review processes and lawyers at
every level.

576
00:31:34,581 --> 00:31:37,714
And many of the people who wrote those
policies have thought about this harder
than

577
00:31:37,714 --> 00:31:38,714
their critics.

578
00:31:38,674 --> 00:31:39,674
But tell why.

579
00:31:40,045 --> 00:31:41,626
A national guideline is a policy,

580
00:31:41,626 --> 00:31:43,857
and a policy can be revised, waived,

581
00:31:43,857 --> 00:31:45,509
or classified by whoever's in office.

582
00:31:45,509 --> 00:31:49,101
It binds no other country, has no
enforcement mechanism,

583
00:31:49,101 --> 00:31:50,892
and can vanish after a single election.

584
00:31:50,892 --> 00:31:54,354
To keep it, the US gave up a binding rule
that would have constrained

585
00:31:54,354 --> 00:31:57,026
its adversaries to, probably for a
generation,

586
00:31:57,026 --> 00:31:58,327
and it called that sovereignty.

587
00:31:58,827 --> 00:32:03,460
So that's I think that's the the biggest
hidden point in all of this right there

588
00:32:04,081 --> 00:32:09,024
is ultimately that the US keeps saying we
have to keep going,

589
00:32:09,024 --> 00:32:13,447
because if we don't, China's gonna
overtake us or Russia might overtake us.

590
00:32:13,447 --> 00:32:18,930
But they're they actually had the
opportunity to constrain them as well

591
00:32:19,430 --> 00:32:21,251
and still fought against that.

592
00:32:21,251 --> 00:32:25,393
And the the fourteen point six billion
dollars in

593
00:32:25,751 --> 00:32:29,043
Venture capital is not waiting on any of
these policies.

594
00:32:29,543 --> 00:32:31,424
Let's let's take a turn back to Libya,

595
00:32:31,424 --> 00:32:33,055
which we discussed briefly earlier.

596
00:32:33,055 --> 00:32:35,197
That's the incident everyone in this
debate cites,

597
00:32:35,197 --> 00:32:37,058
and almost no one describes it carefully,

598
00:32:37,058 --> 00:32:38,379
so we're gonna go slowly through it.

599
00:32:38,379 --> 00:32:41,882
In March 2020, so this is six years ago,

600
00:32:41,882 --> 00:32:45,593
~ outside Tripoli, a civil war was
underway between

601
00:32:45,593 --> 00:32:48,456
the internationally recognized government
and forces loyal to

602
00:32:48,456 --> 00:32:50,448
the commander Khalifa Haf Haftar.

603
00:32:50,448 --> 00:32:53,490
And Haftar's units broke and retreated,

604
00:32:53,490 --> 00:32:57,713
according to a UN panel of experts report
published in March twenty twenty one.

605
00:32:58,031 --> 00:32:59,622
five hundred and forty eight page report,

606
00:32:59,622 --> 00:33:02,926
the retreating forces were hunted down and
remotely engaged

607
00:33:02,926 --> 00:33:07,490
by unmanned combat aerial vehicles and
lethal autonomous weapon systems.

608
00:33:07,490 --> 00:33:12,715
So literally we're talking about ~
retreating soldiers basically hunted down

609
00:33:12,715 --> 00:33:14,857
and gunned down. The system

610
00:33:13,986 --> 00:33:14,986
Exactly like

611
00:33:15,027 --> 00:33:18,959
boats, you know, in the in the Gulf of
Mexico and or sorry,

612
00:33:18,959 --> 00:33:22,672
Gulf of America and the Pacific and
Atlantic Oceans.

613
00:33:22,672 --> 00:33:27,115
I mean, i this is really this is really
dark stuff here,

614
00:33:27,115 --> 00:33:30,597
but like hunted down and remotely engaged.

615
00:33:30,597 --> 00:33:33,919
Now I will say we did not re read this
five hundred and forty eight page report.

616
00:33:33,919 --> 00:33:38,202
It it we AI summarized it, ~ to be fair.

617
00:33:38,202 --> 00:33:41,985
But th these are real the this is
something that actually happened.

618
00:33:42,485 --> 00:33:43,485
Sorry to interrupt.

619
00:33:42,806 --> 00:33:43,806
Right.

620
00:33:43,807 --> 00:33:49,441
No worries. And the system named the
system in this incident was named

621
00:33:49,441 --> 00:33:54,035
the STM Cargo two, a Turkish loitering
munition,

622
00:33:54,035 --> 00:33:56,998
which is a small drone that carries an
explosive charge,

623
00:33:56,998 --> 00:34:00,210
and it just circles until it finds a
target and then flies to that target,

624
00:34:00,210 --> 00:34:02,122
and one soldier can carry one of these.

625
00:34:02,122 --> 00:34:05,945
So the report says it was programmed to
attack targets without needing

626
00:34:05,945 --> 00:34:08,127
a data connection between operator and
munition,

627
00:34:08,127 --> 00:34:10,599
a capability the report calls fire,

628
00:34:10,599 --> 00:34:11,750
forget, and find.

629
00:34:12,250 --> 00:34:14,371
Ugh. This is so gross. I

630
00:34:14,085 --> 00:34:15,085
All right.

631
00:34:14,371 --> 00:34:17,953
mean, these are literally like kamikaze
drones,

632
00:34:17,953 --> 00:34:21,234
but at least in the kamikaze world,

633
00:34:21,234 --> 00:34:23,235
like it was it was one for one,

634
00:34:23,235 --> 00:34:25,656
right? Like when during during World War
Two,

635
00:34:24,185 --> 00:34:25,888
Right. A person who did.

636
00:34:25,656 --> 00:34:26,656
the

637
00:34:26,097 --> 00:34:30,128
Japan was one for one, right? Like it was
a real person choosing to

638
00:34:30,128 --> 00:34:32,830
to carry out that action. Now,

639
00:34:32,830 --> 00:34:34,780
depending on who's citing it, right,

640
00:34:34,780 --> 00:34:37,962
that incident gets presented as either
like

641
00:34:38,190 --> 00:34:42,324
Proof that this is this is a real
situation that we really are faced with,

642
00:34:42,324 --> 00:34:45,214
or they just wave it off like it's a
rumor.

643
00:34:45,714 --> 00:34:47,475
Both of those readings are somewhat wrong.

644
00:34:47,475 --> 00:34:50,377
The report does not confirm that anyone
died.

645
00:34:50,377 --> 00:34:52,629
So it's not proof, but it it is,

646
00:34:52,629 --> 00:34:54,560
you know, it's more than just rumor.

647
00:34:54,560 --> 00:34:58,363
Zachary Callenborne, the analyst who
brought it into public debate,

648
00:34:58,363 --> 00:35:02,676
wrote it up in the Bulletin of the Atomic
Scientist in May 2021 under

649
00:35:02,676 --> 00:35:06,529
a headline that ended with a question mark
asking whether a flying killer robot

650
00:35:06,529 --> 00:35:09,752
was used in Libya. His answer was quite
possibly.

651
00:35:09,752 --> 00:35:12,053
But the uncertainty is the finding.

652
00:35:12,053 --> 00:35:14,345
We don't know whether a machine killed a
person in Libya.

653
00:35:15,017 --> 00:35:17,659
But there was a UN panel, a five hundred
and forty eight page report,

654
00:35:17,659 --> 00:35:20,261
and a team of investigators, and the
answer is still unknown.

655
00:35:20,761 --> 00:35:25,325
And those were combatants like on an
active like battlefield where

656
00:35:25,325 --> 00:35:28,519
the law provides not very many
protections,

657
00:35:28,519 --> 00:35:30,461
right, than it does for like civilians.

658
00:35:30,461 --> 00:35:36,647
So if investigators can't establish what
actually happened

659
00:35:36,647 --> 00:35:42,222
to those fighters, a strike on a
residential building will be even harder

660
00:35:42,222 --> 00:35:44,915
to reconstruct or try to figure out after
the

661
00:35:45,415 --> 00:35:46,997
And that's the accountability gap.

662
00:35:46,997 --> 00:35:49,519
The men in that report also have no names.

663
00:35:49,519 --> 00:35:52,823
They appear as retreating half tar
affiliated forces.

664
00:35:52,823 --> 00:35:55,766
Most are probably young as soldiers
usually are,

665
00:35:55,766 --> 00:36:00,041
and they were running, which the law of
war treats as a situation with specific

666
00:36:00,041 --> 00:36:02,153
limits on what you may do to a person.

667
00:36:02,153 --> 00:36:05,957
The judgment about them ended up as a
question mark in a UN document,

668
00:36:05,957 --> 00:36:07,749
and we'll likely never know who they were.

669
00:36:08,249 --> 00:36:11,051
Timeline is what makes this systemic,

670
00:36:11,051 --> 00:36:13,892
right? So a report came out in March 2021,

671
00:36:13,892 --> 00:36:18,595
and the sixth review conference ended in
December of 2021.

672
00:36:18,595 --> 00:36:23,288
That means diplomats in Geneva had the
phrase literally,

673
00:36:23,288 --> 00:36:29,732
and we quote this hunted down and remotely
engaged in a UN report right

674
00:36:29,732 --> 00:36:34,354
in front of them. And they still stripped
the substance from their own text

675
00:36:34,354 --> 00:36:35,354
on the last day.

676
00:36:35,615 --> 00:36:40,487
And deployment of these weapons has outrun
negotiations ever at every stage since

677
00:36:40,487 --> 00:36:43,318
then. The second case we have is Gaza,

678
00:36:43,318 --> 00:36:46,820
where AI generated the targets and a
person approved them in the end.

679
00:36:46,820 --> 00:36:52,622
In April 2024, the Israeli journalist
Yuval Abraham published an investigation

680
00:36:52,622 --> 00:36:58,755
in ~ Pl plus nine seven two magazine ~ in
local call based

681
00:36:58,755 --> 00:37:03,087
on interviews with six Israeli
intelligence officers who served during
the war.

682
00:37:03,587 --> 00:37:05,779
he had already reported on a system called
the Gospel,

683
00:37:05,779 --> 00:37:07,489
ironically, which

684
00:37:06,477 --> 00:37:07,477
~ huh.

685
00:37:07,489 --> 00:37:11,994
generated building targets. this
investigation was about a system called
Lavender.

686
00:37:11,994 --> 00:37:12,995
Love these names.

687
00:37:13,495 --> 00:37:17,538
According to those officers, Lavender was
a machine learning opera system that

688
00:37:17,538 --> 00:37:20,482
scored people in Gaza on how likely they
were to be Hamas

689
00:37:20,842 --> 00:37:25,747
or Islamic jihad operatives. And it
produced a list of about thirty seven
thousand

690
00:37:25,747 --> 00:37:26,747
people.

691
00:37:26,617 --> 00:37:31,595
And how much d like human oversight and
human review do you think that last list

692
00:37:31,595 --> 00:37:32,595
actually got?

693
00:37:33,018 --> 00:37:36,610
Well the officers involved said reviews
sometimes took seconds per target.

694
00:37:36,610 --> 00:37:39,022
One described his role as a rubber stamp,

695
00:37:39,022 --> 00:37:40,893
checking mainly that the target was a
male.

696
00:37:40,893 --> 00:37:44,635
There was also a companion system the
officers called Where's Daddy,

697
00:37:44,635 --> 00:37:46,937
which tracked flagged individuals

698
00:37:45,578 --> 00:37:46,578
~ man.

699
00:37:47,177 --> 00:37:49,318
and signaled when they entered their
family homes.

700
00:37:49,318 --> 00:37:50,529
That's awful.

701
00:37:50,381 --> 00:37:51,381
That

702
00:37:50,601 --> 00:37:55,984
is that is terrible. Meaning strikes were
timed for when those

703
00:37:55,984 --> 00:38:00,567
men were literally walking into their
houses to greet their family.

704
00:38:01,067 --> 00:38:02,438
That's what the reporting describes.

705
00:38:02,438 --> 00:38:06,252
One documented case was the attempt to
kill a Hamas commander named Ibrahim
Biari.

706
00:38:06,252 --> 00:38:11,147
~ it's logged as incident six seven two in
the AI incident database.

707
00:38:11,147 --> 00:38:15,602
The public actor archive researchers ~
used to record harms by ~ caused

708
00:38:15,602 --> 00:38:16,602
by AI systems.

709
00:38:16,923 --> 00:38:20,916
I I wanna take a second and just sort of
mention that that does exist.

710
00:38:20,916 --> 00:38:25,579
the AI incident database, but it is a
public archive that that researchers use,

711
00:38:25,579 --> 00:38:28,481
so it's not something that's maintained by
any organization or government.

712
00:38:28,481 --> 00:38:33,284
But Biari's case shows how the parts
actually fit together,

713
00:38:33,284 --> 00:38:35,446
right? According to the r this report,

714
00:38:35,446 --> 00:38:40,550
the location intelligence was wrong and
heavy weapons were used.

715
00:38:40,550 --> 00:38:41,840
Not we're not talking bullets,

716
00:38:41,840 --> 00:38:45,253
we're talking bombs, right? Like
explosives or ~

717
00:38:45,642 --> 00:38:47,503
You know, RPGs, those sorts of things,

718
00:38:47,503 --> 00:38:51,695
and the strike caused destruction and
civilian deaths across

719
00:38:51,695 --> 00:38:54,657
an incredibly dense urban landscape.

720
00:38:54,657 --> 00:39:00,200
A machine-generated target, meaning AI
created target.

721
00:39:00,200 --> 00:39:03,181
A person looked at it for a few seconds,

722
00:39:03,181 --> 00:39:07,603
a gigantic heavy weapon system was
selected,

723
00:39:07,603 --> 00:39:12,676
and the information at the start of the
chain was entirely incorrect.

724
00:39:13,212 --> 00:39:17,504
Each step had its own independent
justification as far as the reporting
shows.

725
00:39:17,504 --> 00:39:23,628
Nobody, not one person in the chain broke
their own organizational procedure,

726
00:39:23,628 --> 00:39:26,980
but the procedure produced that outcome.

727
00:39:27,480 --> 00:39:30,773
AI also increases the number of decisions
in addition to their speed.

728
00:39:30,773 --> 00:39:33,515
So most of the alignment focuses on
accuracy,

729
00:39:33,515 --> 00:39:35,556
whether the model is right 90% of the
time,

730
00:39:35,556 --> 00:39:36,757
whether a human would do better.

731
00:39:36,757 --> 00:39:41,651
The bigger change is volume. A human
targeting cell can develop a limited
number

732
00:39:41,651 --> 00:39:44,543
of targets a week because people take time
to be sure.

733
00:39:44,543 --> 00:39:46,985
That limit worked as a moral constraint,

734
00:39:46,985 --> 00:39:48,226
even though it looked logistical,

735
00:39:48,226 --> 00:39:50,758
and nobody wrote it into policy because
nobody had to.

736
00:39:50,758 --> 00:39:53,050
You don't have to slow down the human
team.

737
00:39:53,776 --> 00:39:56,600
But install a system that outputs thirty
seven thousand names

738
00:39:56,600 --> 00:40:00,886
and that limit disappears. A decision to
strike which used to be rare because

739
00:40:00,886 --> 00:40:02,658
it took so much human effort becomes
cheap.

740
00:40:03,158 --> 00:40:07,399
So this report is also part of the the
Geneva record.

741
00:40:07,399 --> 00:40:11,520
Human Rights Watch has actually cited in
submissions to

742
00:40:11,520 --> 00:40:16,562
the General Assembly consultations and and
Abraham's investigations

743
00:40:16,562 --> 00:40:19,442
are actually referenced in UN level
debate.

744
00:40:19,442 --> 00:40:25,044
So the connection between Gaza and the
words deleted on september fifth,

745
00:40:25,044 --> 00:40:28,495
twenty twenty six is directly documented
in a file.

746
00:40:28,995 --> 00:40:32,793
And ethics, predictability, and design and
development are the three provisions most

747
00:40:32,793 --> 00:40:34,987
likely to have applied to a system like
Lavender.

748
00:40:35,487 --> 00:40:41,660
So what that means is a few powerful
countries keep using the consensus rule

749
00:40:41,660 --> 00:40:47,944
to strip the teeth out of every document
with that the UN is producing,

750
00:40:47,944 --> 00:40:53,167
every every resolution that the UN is
creating at the very last minute.

751
00:40:53,167 --> 00:40:57,269
And that happened on the final day in the
twenty twenty one review conference

752
00:40:57,269 --> 00:41:01,552
and again in the twenty twenty three ~
conference when Russia blocked

753
00:41:01,552 --> 00:41:03,953
the rules of procedure and then shut out

754
00:41:04,283 --> 00:41:06,835
All of civil society. And then again,

755
00:41:06,835 --> 00:41:13,450
when the US and Russia forced erasures
hours before the mandate expired

756
00:41:13,450 --> 00:41:17,964
on September 5th, 2026. That's the same
exact MO,

757
00:41:17,964 --> 00:41:21,226
same pattern, same tactic, three times,

758
00:41:21,226 --> 00:41:24,088
and that's only five years. And we're
talking about AI.

759
00:41:24,088 --> 00:41:28,312
So this is going to start happening more
and more faster and faster.

760
00:41:28,312 --> 00:41:29,723
Deployment.

761
00:41:30,289 --> 00:41:32,300
Also keeps running ahead of the diplomacy,

762
00:41:32,300 --> 00:41:37,193
right? A UN panel documented the Cargo 2
incident in March of 2021,

763
00:41:37,193 --> 00:41:42,216
before the expert group had produced any
regulatory text,

764
00:41:42,216 --> 00:41:43,897
whatever whatever, you know, whatsoever.

765
00:41:43,897 --> 00:41:50,401
Lavender and the gospel were actually in
use during a war in 2024.

766
00:41:50,401 --> 00:41:56,615
That Ander Rails lattice software is
deployed within NATO right now.

767
00:41:57,435 --> 00:42:00,998
And that mandate closed in September with
a non binding document.

768
00:42:00,998 --> 00:42:06,208
And the money keeps showing up wherever
the law has a little loophole,

769
00:42:06,208 --> 00:42:10,585
a little gray area. The Pentagon announced
the third offset in twenty fourteen.

770
00:42:10,585 --> 00:42:14,949
That's the same year the CCW started
talking about autonomous weapons systems.

771
00:42:14,949 --> 00:42:21,073
In twenty twenty six, venture funding
passed fourteen point six billion dollars

772
00:42:21,073 --> 00:42:26,998
in five months, and Andy Rill's valuation
doubled to sixty one billion dollars.

773
00:42:27,942 --> 00:42:33,390
And the treaty process ended with nothing
binding anyone to anything.

774
00:42:33,890 --> 00:42:36,921
And those three trends you're talking
about actually reinforce each other.

775
00:42:36,921 --> 00:42:40,063
The consensus rule produces predictable
failure.

776
00:42:40,063 --> 00:42:42,524
Investors read that failure as a signal,

777
00:42:42,524 --> 00:42:44,644
and capital builds the technology faster.

778
00:42:44,644 --> 00:42:48,406
The faster it spreads, the more
governments decide they can't accept
constraints,

779
00:42:48,406 --> 00:42:51,087
which strengthens the states using the
consensus rule.

780
00:42:51,087 --> 00:42:53,058
Each cycle makes the next one easier.

781
00:42:53,558 --> 00:42:55,932
And people in active conflicts in Libya,

782
00:42:55,932 --> 00:43:01,560
Sudan, Ukraine, and Gaza, they're exposed
to AI assisted targeting with

783
00:43:01,841 --> 00:43:07,379
no oversight, no rule requiring a person
to make the decision

784
00:43:07,379 --> 00:43:09,392
to kill multiple people.

785
00:43:09,892 --> 00:43:11,907
And the US actually believes it to head
here,

786
00:43:11,907 --> 00:43:13,179
and I think that's a mistake.

787
00:43:13,679 --> 00:43:16,533
W w let's hear what what do you why do you
think it's a mistake?

788
00:43:17,423 --> 00:43:18,894
Well, it's kinda like what you said
earlier.

789
00:43:18,894 --> 00:43:21,535
Any lead is temporary and shorter than
most people assume.

790
00:43:21,535 --> 00:43:25,625
This is software, and software spreads
faster than almost any other technology.

791
00:43:25,625 --> 00:43:29,728
The cargo too is Turkish. Loitering
munitions are being improvised

792
00:43:29,728 --> 00:43:31,529
and updated weekly in Ukraine workshops,

793
00:43:31,529 --> 00:43:34,150
Ukrainian workshops by people with no
national laboratory.

794
00:43:34,150 --> 00:43:37,431
A machine that can identify a human figure
and fly into it.

795
00:43:37,431 --> 00:43:40,263
Now it takes a hobby drone, an off the
shelf model,

796
00:43:40,263 --> 00:43:44,664
and someone who can code. The advantage
the US protects by refusing rules

797
00:43:44,664 --> 00:43:45,815
is it disappearing on its own.

798
00:43:46,315 --> 00:43:48,826
The US is also unusually exposed.

799
00:43:48,826 --> 00:43:51,927
It's an open society with an enormous area
to defend.

800
00:43:51,927 --> 00:43:54,287
Ports, the power grid, rail lines,

801
00:43:54,287 --> 00:43:56,008
airports with fences you could climb.

802
00:43:56,008 --> 00:43:59,079
Autonomous weapons favor attackers since
they're cheap,

803
00:43:59,079 --> 00:44:01,850
expendable, scalable, and indifferent to
their own losses.

804
00:44:01,850 --> 00:44:05,851
A country whose defense strategy depends
on projecting force abroad while staying

805
00:44:05,851 --> 00:44:10,012
safe at home has more to lose from cheap
autonomous weapons than just about
anybody.

806
00:44:10,512 --> 00:44:13,435
I I think there's definitely some irony
there when it comes to the United States.

807
00:44:13,435 --> 00:44:14,756
What

808
00:44:14,155 --> 00:44:15,155
Mm-hmm.

809
00:44:14,756 --> 00:44:19,250
what do you say of the sort of the AI
industry's own concerns and warnings?

810
00:44:19,750 --> 00:44:22,141
As we talked about, they're they're saying
in public that these systems

811
00:44:22,141 --> 00:44:25,172
are unpredictable. They can't fully
explain model behavior,

812
00:44:25,172 --> 00:44:27,243
and they discover capabilities after
release.

813
00:44:27,243 --> 00:44:30,284
Some have said that development is
outpacing understanding,

814
00:44:30,284 --> 00:44:33,326
that outside oversight is needed and that
slowing down is reasonable.

815
00:44:33,326 --> 00:44:35,796
I don't think they say that purely out of
virtue.

816
00:44:35,796 --> 00:44:39,168
Some of it is positioning, and some of it
looks like regulatory capture.

817
00:44:39,168 --> 00:44:41,789
The technical claim underneath is still
true though,

818
00:44:41,789 --> 00:44:43,770
and serious researchers don't dispute it.

819
00:44:43,770 --> 00:44:47,151
Now, give that system a weapon and put it
on a battlefield full of inputs

820
00:44:47,151 --> 00:44:48,562
and nobody trained on it trained it on?

821
00:44:49,308 --> 00:44:52,177
And facing another country's unpredictable
system at machine speed,

822
00:44:52,177 --> 00:44:54,344
with humans just too slow to step in.

823
00:44:54,844 --> 00:44:57,278
Yeah, think of it like two black boxes,

824
00:44:57,278 --> 00:45:02,607
right? Like escalating against each other
before anyone can wake up the president.

825
00:45:02,607 --> 00:45:04,269
And I mean the black box that like,

826
00:45:04,269 --> 00:45:07,074
you know, the nuclear codes and all that
stuff.

827
00:45:08,049 --> 00:45:12,235
Right. And the killer robot imagery has
damaged this debate because nobody serious

828
00:45:12,235 --> 00:45:15,319
is worried about a metal skeleton with a
rifle yet.

829
00:45:15,319 --> 00:45:16,841
the worry is a flash

830
00:45:16,005 --> 00:45:17,005
I mean

831
00:45:16,841 --> 00:45:20,356
crash with munitions. two automated
systems interacting in a

832
00:45:20,356 --> 00:45:22,850
way that neither operator predicted for
eleven seconds,

833
00:45:22,850 --> 00:45:25,103
and at the end of those eleven seconds a
war has started.

834
00:45:25,603 --> 00:45:27,114
unless it's never been declared.

835
00:45:27,114 --> 00:45:28,845
What the US

836
00:45:28,278 --> 00:45:29,278
Right.

837
00:45:28,845 --> 00:45:32,588
defends in Geneva is kind of the the
option,

838
00:45:32,588 --> 00:45:34,519
right? It's the it's the decision,

839
00:45:34,519 --> 00:45:36,160
the right to figure it out later.

840
00:45:36,160 --> 00:45:41,274
But for eighty years the American approach
to international law has been just avoid

841
00:45:41,274 --> 00:45:46,557
signing anything, right? Keep your options
open and assume that this ambiguity

842
00:45:46,557 --> 00:45:49,289
is gonna eventually favor the strongest
country,

843
00:45:49,289 --> 00:45:52,111
in which case it's mostly been us.

844
00:45:52,111 --> 00:45:54,222
That approach has a a

845
00:45:54,478 --> 00:45:57,631
A record a mile long. Drone strikes became
normal,

846
00:45:57,631 --> 00:46:02,476
and this indefinite detention became
something that governments just do now.

847
00:46:02,476 --> 00:46:05,779
Each time the US keeps its options open,

848
00:46:05,779 --> 00:46:11,365
someone eventually used that open option
against

849
00:46:11,425 --> 00:46:14,868
us or against people we told that we would
protect.

850
00:46:14,868 --> 00:46:18,402
And the objection we needed was one we'd
already given up.

851
00:46:18,902 --> 00:46:22,287
And it it bears saying again, in case you
missed it before,

852
00:46:22,287 --> 00:46:24,731
after three years of this mandate,

853
00:46:24,731 --> 00:46:27,375
the result is a document nobody has to
follow,

854
00:46:27,375 --> 00:46:30,429
missing three important words ethics,

855
00:46:30,429 --> 00:46:32,242
predictability, and design and
development.

856
00:46:32,742 --> 00:46:35,884
Three most important words. I think over
the next few months,

857
00:46:35,884 --> 00:46:42,048
if you watch the November Review
Conference and the count of the countries
that

858
00:46:42,188 --> 00:46:46,591
are asking for negotiations, which as of
September is 76 countries,

859
00:46:46,591 --> 00:46:53,536
see whether it grows before November and
whether those states actually stay inside

860
00:46:53,536 --> 00:46:59,039
the consensus process or start discussing
a separate track in the corridors.

861
00:46:59,039 --> 00:47:01,481
A separate track is basically how

862
00:47:01,763 --> 00:47:03,995
The whole Canadian Ottawa thing happened.

863
00:47:03,995 --> 00:47:08,900
122 different countries in 1997 who
decided to go out of the process

864
00:47:09,781 --> 00:47:14,926
to build a thing that could actually stop
horrible things from happening.

865
00:47:14,926 --> 00:47:20,612
Right? If diplomats start saying
alternative forum or general assembly
process,

866
00:47:20,612 --> 00:47:23,896
they've basically decided that they're not
going to ask permission anymore.

867
00:47:23,896 --> 00:47:25,215
And I think that's a good thing for all of
them.

868
00:47:25,715 --> 00:47:28,317
And the same goes for deletions in any
negotiation.

869
00:47:28,317 --> 00:47:32,411
A story that says ~ talks failed to reach
agreement usually leaves

870
00:47:32,411 --> 00:47:34,123
out that most participants did agree.

871
00:47:34,123 --> 00:47:36,816
In this case, 164 countries agreed.

872
00:47:36,816 --> 00:47:39,699
~ asks which words were removed and by
whom,

873
00:47:39,699 --> 00:47:42,131
since the words removed words show the
actual policy.

874
00:47:42,631 --> 00:47:47,996
The United States and Russia are are
further apart than at any point since

875
00:47:47,996 --> 00:47:50,668
the Cold War. Except their leaders.

876
00:47:50,668 --> 00:47:53,641
They are handling things very much the
same.

877
00:47:53,641 --> 00:47:56,623
They disagree about borders and energy,

878
00:47:56,623 --> 00:48:01,488
who shot down what, the basic facts of the
recent history that we're actively living

879
00:48:01,488 --> 00:48:05,231
through. In a room in Geneva, in the last
hours,

880
00:48:05,231 --> 00:48:09,294
they both agreed that the same word,

881
00:48:09,294 --> 00:48:10,294
ethics.

882
00:48:10,725 --> 00:48:16,011
has to come out of this UN document about
AI that kills people.

883
00:48:16,511 --> 00:48:18,344
So under a rule where everyone has to
agree,

884
00:48:18,344 --> 00:48:19,997
the agreement produced that deletion.

885
00:48:20,497 --> 00:48:22,878
Now you can act on this without calling
anyone.

886
00:48:22,878 --> 00:48:27,861
You know, I don't know how much we can
really do right now

887
00:48:28,261 --> 00:48:33,044
in our current environment because we
don't have a whole lot of control over the
UN.

888
00:48:33,044 --> 00:48:36,306
And we certainly don't have control over
the people who are supposed

889
00:48:36,306 --> 00:48:38,627
to be representing our viewpoints at the
UN.

890
00:48:38,627 --> 00:48:45,010
But the UN Office for Disarmament Affairs
actually runs a free public document

891
00:48:45,010 --> 00:48:47,532
library with working papers, national
statements,

892
00:48:47,532 --> 00:48:48,532
and draft texts.

893
00:48:48,976 --> 00:48:52,850
You can actually pull the draft from
before the final session and the text that

894
00:48:52,850 --> 00:48:54,691
was adopted after it and kind of compare
them,

895
00:48:54,691 --> 00:48:56,943
do a diff like left to right, side by
side.

896
00:48:56,943 --> 00:48:59,336
It takes about 40 minutes, and I'm not
gonna lie,

897
00:48:59,336 --> 00:49:04,751
it's not fun. But afterwards you will know
then more than most of

898
00:49:04,751 --> 00:49:08,174
the attaches and the reporters that are
actually covering this in the trenches,

899
00:49:08,174 --> 00:49:14,300
right? The post is the comparison that any
of us can make because those deletions.

900
00:49:14,946 --> 00:49:18,683
Stay invisible until somebody actually
reads them and publishes them

901
00:49:18,683 --> 00:49:20,160
and says something to somebody else about
it.

902
00:49:20,660 --> 00:49:23,342
If you're one of our listeners who works
in the software industry,

903
00:49:23,342 --> 00:49:26,324
get the end use terms in writing before
you write the code.

904
00:49:26,324 --> 00:49:29,586
Email your employer and ask what the
export classification is

905
00:49:29,586 --> 00:49:31,947
on what you're building, whether it's dual
use,

906
00:49:31,947 --> 00:49:35,349
and what the contract says about
integration into targeting systems.

907
00:49:35,349 --> 00:49:39,392
Most engineers never ask, and often nobody
inside the company knows the answer,

908
00:49:39,392 --> 00:49:41,073
which tells you something too.

909
00:49:41,073 --> 00:49:44,235
An email creates a paper trail that can
come out in discovery,

910
00:49:44,235 --> 00:49:45,406
which an open letter never will.

911
00:49:45,906 --> 00:49:50,318
Also go local, ~ because the same
companies sell in both directions,

912
00:49:50,318 --> 00:49:56,070
right? So firms that build autonomous
military systems also build flock cameras

913
00:49:56,311 --> 00:50:01,193
and surveillance towers and sensor fusion
and border and police technology

914
00:50:01,213 --> 00:50:07,635
and ice vehicles and ice systems and city
council procurement meetings

915
00:50:07,635 --> 00:50:11,087
are among the least attended and most
winnable rooms in the country.

916
00:50:12,069 --> 00:50:14,001
You can actually read every single
contract.

917
00:50:14,001 --> 00:50:15,492
You can ask about data retention.

918
00:50:15,492 --> 00:50:17,374
You can ask about the vendor software,

919
00:50:17,374 --> 00:50:21,298
whether they make automated decisions
about people and get answers on the
record.

920
00:50:21,298 --> 00:50:24,421
A county commission can be made to read a
contract out loud,

921
00:50:24,421 --> 00:50:27,424
and that changes what gets put bought
sometimes.

922
00:50:27,424 --> 00:50:29,956
The mayor of Oklahoma City doesn't
apparently give a shit,

923
00:50:29,956 --> 00:50:33,009
but it can make a difference.

924
00:50:33,509 --> 00:50:35,530
So ending sort of where we began,

925
00:50:35,530 --> 00:50:38,584
Christoph Haynes asked for a moratorium in
twenty thirteen.

926
00:50:38,584 --> 00:50:42,719
In the thirteen years since, there's been
a probable killing machine killing

927
00:50:42,719 --> 00:50:46,653
in Libya, a list of thirty seven thirty
seven thousand names in Gaza,

928
00:50:46,653 --> 00:50:51,368
and fourteen point six billion dollars in
venture funding in five months,

929
00:50:51,368 --> 00:50:52,609
and nobody has granted it.

930
00:50:53,109 --> 00:50:57,019
And as of this month, seventy six
countries are asking to negotiate that.

931
00:50:57,019 --> 00:51:00,329
And the review conference is still coming
up in November.

932
00:51:00,829 --> 00:51:02,203
that's the overlap for this week.

933
00:51:02,203 --> 00:51:05,531
Joshua, where can they find us if they
wanted to continue this discussion?

934
00:51:06,199 --> 00:51:10,883
FOF dot foundation, that is FOF Friends of
the Future.

935
00:51:10,883 --> 00:51:14,245
Foundation, no dot com, no dot org,

936
00:51:14,245 --> 00:51:18,108
no dot AI, nothing like that. These are
made by real people,

937
00:51:18,108 --> 00:51:24,173
you and me. ~ you can also check us out on
every podcast app of your choosing.

938
00:51:24,173 --> 00:51:29,458
That can be ~ Apple Podcasts or Spotify or
Podbean or wherever you can find

939
00:51:29,458 --> 00:51:31,419
~ the overlap. If you have trouble finding
us,

940
00:51:31,831 --> 00:51:33,364
Or your friend had trouble finding us,

941
00:51:33,364 --> 00:51:37,961
tell us tell them to type in F O F F
Friends of the Future Foundation

942
00:51:37,961 --> 00:51:43,882
~ after the overlap and they will be able
to find us quickly that way.

943
00:51:44,382 --> 00:51:46,593
All right. Thanks for being here with me,

944
00:51:46,593 --> 00:51:47,593
Will.

945
00:51:47,323 --> 00:51:49,230
Thank you, Joshua. Talk to you next time.

946
00:51:49,839 --> 00:51:50,839
Talk to you guys next time.