Ep 112: Paul Scharre on AI 101
Paul Scharre, Executive Vice President and Director of Studies at CNAS and author of Four Battlegrounds: Power in the Age of Artificial Intelligence
Aaron MacLean:
Artificial intelligence seems inescapable in 2024, and only a fool would assert that it's not going to be a significant factor in war-fighting and national security more broadly. But how? What is artificial intelligence in the first place? Are there historical analogies that can guide our thinking about it? What are its military applications? How are they emerging? How do we expect them to further evolve and what will the net effect of all of this on the battlefield be?
Today we are going to do AI-101.
Aaron MacLean:
For maps, videos, and images, follow us on Instagram. And also feel free to follow me on Twitter at Aaron B. MacLean.
Hi, I'm Aaron MacLean. Thanks for joining School of War. I'm delighted to welcome to the show today, Paul Sharre. He is the Executive Vice President and Director of Studies at the Center for New American Security. He's the author most recently of Four Battlegrounds: Power in the Age of Artificial Intelligence. We'll get into his background more here in a second, but he is also a veteran in the figurative sense. He served as a civilian in the Department of Defense, but he is a veteran as well of the Ranger regiment in the US Army.
Paul, thank you so much for joining the show.
Paul Sharre:
Thank you. Thanks for having me.
Aaron MacLean:
So, before we get into our subject today, which is your work on AI and its strategic impacts, tell us a bit about yourself. How'd you grow up? How'd you get interested in service? How'd you end up in the Rangers?
Paul Sharre:
Yeah, I think I went through like a lot of folks a crisis decision point in college - what am I going to do with my life? And at the time, I guess this will figure out... You can bracket my age here. This was during the Kosovo Air Campaign and I just had this strong feeling that look at these things that are happening in the world and I think this is truly, at any point in time, you could look out at things happening in the world today or back then and say, "Wow, these things seem significant. They're important."
And I wanted to be engaged in US foreign policy, national security issues. I was young, I had really nothing, no skills, no abilities, just coming fresh out of college and I'd heard about options for going into the military and special operations. That seemed pretty exciting to me. A chance to jump on a plane and crawl around in the mud and do all that fun stuff. And so I signed up with a Ranger contract. I joined in June of 2001, so it was actually basic training when 9/11 happened and then it was a busy, busy time that first couple years.
Aaron MacLean:
How many years did you stay in?
Paul Sharre:
I was in for four years on my initial tour and then I got out and then I was called back in when we were doing that for a period of time. I did another year-long tour in Iraq.
Aaron MacLean:
Got it. Was it all Iraq for you? Did you make it to Afghanistan at all?
Paul Sharre:
Yeah, some of both. Yeah. So, I did three tours to Afghanistan with the Rangers and then later did a tour year in Iraq as a civil affairs specialist. In the Army's infinite wisdom, when they called me back they'd made me the opposite, I guess, of an Army Ranger and reclassified me to civil affairs and so did that.
Aaron MacLean:
And so you go out on the other side of things, you come to DC. How is it that... The question of AI specifically, but you've written extensively about questions of emerging technologies more generally and their strategic impacts. How did you come to that as a subject?
Paul Sharre:
There was this singular defining moment actually for me. When I was in Iraq when I thought about robotics in particular and their value in war and we're driving down the road, we came across a roadside bomb, an imprecise explosive device. Now we saw it first, which is the preferred method of finding them rather than just running into it. And we called up the bomb disposal team and so I was expecting the bomb tech to come out in that big suit they have and get up there and slip the wires and instead they sent this little robot. And the light bulb went on in my head. I was like, "Yeah, have the robot defuse the bomb. You don't want your face up there just slipping the wires on the bomb."
And then the more I thought about it and when I thought, "Well there's a lot of things we're doing that are dangerous that you could have robots doing." Why don't we have a robot doing this job? And so when I left the army and I ended up working in the Pentagon as a civilian policy analyst, that was one of the things that I worked on was how do we get more robotics, more drones, more autonomous systems out in the field? And I just personally saw a lot of potential there to help protect our service members.
Aaron MacLean:
Yeah, your experience of the robot versus the guy in the suit squares a bit with my experience, which was neither actually. It was the first time I saw an actual EOD tech work on an actual bomb and for many times thereafter, the gunnery sergeant in question walked up to where we thought the IED was, pulled out his bayonet, began poking the ground, found the pressure plate, and then proceeded to defuse the bomb, which was the accepted practice circa 2009, 2010 where I was. Probably, I'm going to guess because of a mismatch between the number of IEDs and the availability of robots and other technologies, you might use to find them.
Paul Sharre:
It's a hairy job. I mean, and the bomb techs were really... I mean, just right on the front lines of these last wars. Just [inaudible 00:05:36].
Aaron MacLean:
Actually one of my favorite stories from that deployment involves the same guy who we, my platoon, found at that point was, at least so we were told, the largest improvised explosive device, really series of devices yet found in Helman province. It was a daisy-chained set of military-grade explosives that went down an alleyway where we were and that we saw sort of the wire and indications that there was something down that alley that we needed to check out. And the EOD team leader went over to the marine engineer platoon that was there and asked for the bulldozer to drive, basically dig him a path down the middle of the lane so he could have a safe place to walk up and down as he was figuring out what it was. And the bulldozer driver said... And for anyone with children listening, the following will involve adult language.
The bulldozer said, "Gunny, this bulldozer is only going to take one hit." To which the gunnery sergeant responded, "Motherfucker, I'm only going to take one hit." The bulldozer was persuaded to do the job. So sorry, we have an important topic here and I want to get to it, and I'm grateful for you making the time today because the question of AI is something that I think is universally... People universally agree it's important and then if not universally, perhaps approaching universally don't really understand it. Maybe I'm making myself feel better when I say that because I don't feel like I understand it.
Paul Sharre:
No one understands it. It's ever [inaudible 00:07:00].
Aaron MacLean:
Maybe that can be my first proper question to you, which is how should we think about AI strategic impacts and more specifically, what is the best historical experience that warfare, whether in the American experience or just more broadly that humanity has gone through in terms of technological evolution that we can use to track or control our thinking about AI?
Paul Sharre:
Now, that's a great question because this search for what's the historical parallel matters quite a bit and it can take you in different directions if you're like, "Oh, AI is nuclear weapons," that might take you in one direction. I think that actually a very common comparison, I think a good one, is to the first and second Industrial Revolution. And I'll tell you why. Because AI like other technologies at the time, the internal combustion engine, electricity, is a general purpose technology that has a whole wide range of applications and is likely to be applied across society in widespread ways, a wide range of industries the same way that those technologies were. And we saw I think some really key geopolitical effects from those technologies.
One is that we saw nations rise and fall on the global stage based on how rapidly they industrialized. And some nations like Great Britain and Germany raced ahead in economic and military power by virtue of seizing a hold of these technologies and integrating them into their societies faster than others.
We also saw the key metrics of power change. So things like coal and steel production became key inputs of national power. Oil becomes this geostrategic resource that countries were willing to go to war over. And so I think AI is likely to similarly change not just the balance of power, but even the key metrics of power. And then of course, the Industrial Revolution transformed war in incredible ways, incredibly destructive ways that you saw in World War I and World War II that not just required militaries to develop new tactics and respond accordingly, but really ended up mobilizing entire societies for industrial production and increase the scale, the physical scale in destructiveness of war in a way that was really unprecedented up until that moment.
Aaron MacLean:
Is there any way we can... So I take your point about the Industrial Revolution or waves of industrial revolutions and it's obviously right that we should think in terms that are broad like that because it's not just a question of an application. It's not the introduction of a bomber. It's broader than that. But we talk about... Industrial Revolution is so broad. Is there something... I don't know, electricity, the introduction of electricity. Is there something that is a kind of concrete example that you've played around with that would make sense here?
Paul Sharre:
I mean, I think there's a couple. Kevin Kelly, the tech guru and thinker has made this comparison to electricity. I'm going to paraphrase here, but he basically said that everything that we have formally electrified, we will now cognitize. This idea that today we live in this world where we have physical devices that have been imbued with power via electricity and increasingly networked also as well and that we're going to be layering on top of that stack of powered devices, networked... We're layering around artificial intelligence now and that these devices are going to sort of come alive and have the ability to process information and think to some extent and engage with us in ways that will make them not just be powered but also intelligent.
Aaron MacLean:
Right. Okay, so I'm going to punt on the harder question here and we'll come to it in a few minutes, which is what does that mean? What does it mean that they think, and what are the applications and how do we deal with them? But just to stick with something that at least I find easier to comprehend. But with the hardware side of it, there's just stuff you need to have AI, to develop AI, and that stuff is made in certain places and not made in other places and different people control it and that just introduces a sort of normal kind of geopolitical dimension to the question that applies across any kinds of other number of energies, the obvious one, the issues that we all think about a lot of the time and sort of understand.
Help us understand the hardware dimension of AI and how it impacts strategic thinking or ought to impact strategic thinking.
Paul Sharre:
Yeah, so hardware is a really essential element of this geopolitical competition for AI for a couple of reasons. At the technical level, there's three key technical inputs to current machine learning systems: data, algorithms, and hardware, the chips that are used.
Machine learning algorithms; they are algorithms that are trained on data using computer hardware. So there's an algorithm, you feed in a bunch of data, sometimes these massive data sets. Some of these large language models are trained on trillions of words, huge data sets, and they're trained using very large amounts of computing hardware. So, the most advanced AI models are using tens of thousands of the most advanced chips. When you think about these three inputs of data, algorithms, and hardware, hardware is unique in that it's a physical asset. It is easier to control, very hard to say control the spread of algorithms. It is a rival asset in the sense that if I have a chip, you cannot have the same chip or you cannot use the same chip at the same time. And it has a unique position in that the United States has a really unique advantage in the supply chain.
Now, it's not actually that these chips are made in the us. In fact, none of the most advanced chips in the world are made in the United States. They're made in Taiwan. Now, on the face of it, this is terrible because of course Taiwan is an island a hundred miles off the coast of China that the Chinese Communist Party has pledged to absorb by force if necessary. That's not good for the US, but all of the technology to manufacture the chips relies on US technology and US software.
And there's actually three countries in the world - the United States, the Netherlands and Japan that control the market for the manufacturer equipment for the most advanced chips. And we've seen the US use this position to lock China out of the most advanced chips by placing export controls that are extraterritorial, they're outside of the United States, on advanced chips going to China. So even if there's a Chinese chip design firm that designs a chip, they send the design to Taiwan to make the chip, and it's going to be shipped back to China, nowhere come to the United States, US export controls say you can't do that because that fab in Taiwan relies on US technology and US government said you can't do that.
Aaron MacLean:
Help us understand what this all means though. So, is it just as simple as the fossil fuels analogy whereas to a degree that's still quite significant even in 2024. Some people have oil, some don't, some people have gas, some don't. And those places that have it are significant as a consequence and need to be factored into strategic decision-making. Do I just apply the same thinking to chips? Is it more complicated than that?
Paul Sharre:
Well, yeah, in a lot of ways it is it more complicated. There are some analogies, so I think it's not a bad comparison to think about Taiwan's position as sort of this geopolitical fulcrum from hardware as maybe roughly analogous to the role that Saudi Arabia plays in the oil industry. So they say something like, "Well, Taiwan is the Saudi Arabia of chips." I think it's like a useful, maybe shorthand, for thinking about how important Taiwan is.
Of course, one of the differences is oil is produced in a lot of places, the oil market is very globalized, and oil is a fungible asset, whereas the chips literally only come from Taiwan. Now, a little bit unclear how effective US export controls are going to be in terms of... There's ways around controls. China is working to indigenize their own chip making. They've got a steep hill to climb here, but they're working on it.
And there are opportunities for smaller AI models that don't rely on the largest most advanced sets of chips. And then right now, there's a very vibrant open source community in the United States where we actually just release these models open source, which is like this big... Maybe is another historical analogy. It makes these export tools a bit like imagine no line here where we're sort of blocking China from getting access to the chips, but if they get access to a fully trained modeled open source, they don't need the chips in the first place and that's right now where the US AI ecosystem is.
Aaron MacLean:
Right. Okay, so let's shift into the hard stuff then. These algorithms, the models that they help to produce when the data is run through them... I guess by the way, actually data itself by the way is another kind of semi-physical consideration in the sense that some people have more access to data than others, but we can talk about that maybe in a minute. But to get to the thing itself, there is this technology, this tool, this platform out of which many applications can be made that is the end result or the near end result of this process. What is it?
Maybe just start with that and then what are the most important applications in military terms we ought to start thinking about or be thinking about?
Paul Sharre:
Sure. So what happens is you start with this dataset, very large dataset. It could be images for example or text or sometimes combinations of both. And then the algorithm is trained on this dataset and what the algorithm does is it learns to create a model that represents the types of things inside the data. It looks to generalize somewhat from that dataset. So, if you were for example, trading an image classifier, an image classifier is a model that's in used to identify objects. So what you could do is from a military standpoint, you could have satellite imageries of different objects and they're all labeled and you feed it into this AI model and you say, "Look, this is an image of an airfield, this is an airplane, this is a tank, this is a radar installation." Now, you need millions of images, thousands of each, different class of images.
You need a lot of data. And it's worth pointing out more than you would need for a person, but if you give an imagery analyst, you might need to give them a couple examples of a thing. You don't need to give them thousands of examples. After a while they'd be like, "I got it." The AI actually needs a lot of examples, but the end result is a trained model that then you could feed in new data that's never seen before and if it was in the class of things that was trained on, it could identify it. If it's something novel it's never seen before, the AI is not going to be able to do that, can't generalize very well.
And these systems are that's the kind of thing that the defense department is already using. If you go back to Project Maven, the DOD's first big AI project coming out of the deep learning revolution back in 2017, that was what DOD was doing was image classifiers for drone video feeds and that is just one application. It's very powerful because you can imagine on this front, DOD, the Defense Department is collecting and the intelligence community way more information than they could ever process with people, whether it's signals intelligence, imagery, intelligence, and other things. And so AI tools could be a good way to just sift through that information and sort it out and kind of help humans make sense of it.
Aaron MacLean:
Yeah. Well, this brings to mind the 2021 Gaza War that the Israelis branded as the "first AI war". And I think the process you just described is essentially what they were talking about that they have a lot of sensors in Gaza ultimately and in complete complex as later years would reveal, but nevertheless, a lot of sensors in Gaza collecting a lot of data. And so they applied this technology to target more efficiently.
In fact, my understanding is still with humans in the loop, but rather than human analysts as it were deciding whether or not... I'm going to make all this up. But we might imagine this or that phone call was relevant and suggested that one of the people was on the line was significant or not. You could actually task some of that to a computer initially such that by the time the human encountered the stuff, the data, there was a degree of analysis that had already occurred that's speeding up the cycle, speeding up the loop, and making the campaign more lethal.
So that's basically the sort of application number one that we all... Well, we don't all understand but is sort of out there. Is that fair? Is that a fair characterization?
Paul Sharre:
Yeah, I'll give you two actual real examples that the Defense Department did that when I was researching my book, Four Battlegrounds, the DOD gave me incredible access to then the Joint AI Center. They've now morphed into the CDAO, this new organization. But General Jack Shanahan who was running the AI Center at the time gave me just incredible access to their people.
And so two things that they were working on. One was looking at battle damage assessment and they were doing it for disaster relief. And so what they did was they built a model that could look at flooded areas during a hurricane and then could help create a map for people to first responders to make it navigate through flooded areas to say, "Based on this imagery we're looking at, the satellite imagery, this road is flooded, you need to take a new way to do it."
So you can imagine for the military, that kind of tool would be very valuable in a war zone in terms of doing realtime updating, mapping, and navigation. And they had a parallel version that was looking at structures damaged from wind damage, for example, in a hurricane and linking that to FEMA categories of destruction and then using that to create sort of a first cut for first responders to say, "Here's where things are damaged." Again, valuable military applications when you think about doing battle damage assessment. Those are some of the things that we saw people working on.
Another example that they were working on was the ability to use image classifiers in drone video feeds to say, "Watch a building, track people coming and going from a building or vehicles." And then through some of these wide area surveillance tools like Gorgan Stare that are looking over an entire city to use that to rewind the video recording over time to then track people.
So let's say that there's a car bomb. We don't know where the car bomb's going to go off. It happens. Now, we have video footage of that city for the last 24 hours. You could in principle have a human analyst watch this video in reverse and see where the car came from. It's just time-consuming. Having an AI do that a lot faster, the AI can zip through this, find a building, and then you can say, "Okay, I want you to identify the timestamp of every time a person came in or out of this building in the 96 hours prior," and using AI tools to speed up a lot of the things that humans might be doing.
Aaron MacLean:
So what you just described seems like, from a purely military perspective, sensible and really welcomed actually kind of way of dealing with the sophistication in the present day of sensor strike complexes that any modern military has. We have so many sensors. We have so much information. Thank goodness that we now have technology that can actually help us be efficient about that information.
We start to run into more complicated considerations and really interesting considerations that you've written about a fair amount. When we start talking about autonomy and removing humans or reducing the role of humans in the loop because I think everything we've discussed till now is basically analysis that tees up human decision-making.
Paul Sharre:
Right.
Aaron MacLean:
Talk about autonomous weapons. It's not quite the same thing as AI, but it relates to AI. Just help us understand what that means.
Paul Sharre:
Yeah, so if you look at maybe what we're seeing on the battlefield right now in Ukraine, it might be a good starting point. We're seeing obviously a lot of small drones being used to target people and vehicles and one of the things that we're seeing incorporated into these drones is more autonomous terminal guidance. So, right now the drones are still piloted by a person and a human is choosing the target, but there is a lot of counter-drone innovation going on in the war of Ukraine on both sides. Electronic warfare systems jamming and that communications link to a human is a point of vulnerability. If you jam the communications link and the drone is remotely controlled, it's no longer valuable. It's not going to do anything. We've also seen electronic warfare tools to find the location of the drone operator who can then be targeted. If you kill the drone operator, that's also a good way of making the drone ineffective.
And so all of that creates pressures towards more autonomy. And we've started to see companies incorporate this into drones in Ukraine. One Ukrainian company claims last fall that they had fueled and was using in combat a fully autonomous weapon. Now, that's not independently verified, that's their claim. But we certainly are seeing image classifiers, things that could identify objects; this is a tank, this is a vehicle, have been used in Ukraine. They're on drones now, not I think maybe widely spread, but they are used.
And so adding more autonomy where you could get to the point where either a person chooses a target and then can go hands-off and the drone can do terminal guidance. That's valuable. We're starting to see that. And I think over time, if the war continues to drag on, we will in all likelihood see autonomous weapons where someone is deploying a drone or set of drones into an area where they know their enemy targets, people or vehicles, and then the drones are choosing their own targets and attacking them. And humans are still involved, but they're just a little bit further removed.
Aaron MacLean:
Yeah. So, well I guess your last comment points to my question, which is how different is this really except perhaps somewhat more sophisticated than say very, at this point, traditional guidance technology from heat-seeking missiles. The missile goes off into the world, its job is to find something hot and blow itself up next to it. If anything, this is maybe just a little more precise, a little more sophisticated, but basically the same thing. I put a question mark at the end of that sentence. Please.
Paul Sharre:
Yeah. I think there is a big conceptual difference between the idea of humans making these decisions about whom to kill on a battlefield and humans delegating that to machines. The question is when you start to look at it closer, there's not just one decision. There's lots of small decisions that get made as part of this targeting cycle. Where are the enemy forces? What are the things I'm targeting? Are they here in this point in time? I think one way to think about this is right now a lot of the systems that we have, homing munitions, torpedoes, many of them are fire and forget. Once it's launched, it's not coming back. You don't want it to come back in fact. And many of them have some seeker on board that can sense an enemy target, but today humans are launching them at some known or suspected enemy target.
They have some indication that there is a valid enemy target at some point in space and time. And shifting to a world where the human is instead saying, "I'm going to launch it into this area to some kill box, and I don't know exactly where the enemy is," it is sort of a qualitatively different thing. It doesn't mean that it's necessarily immoral or illegal, depends actually quite a bit on the context for the use. But it is a different role in human decision-making. And then you could imagine that kind of kill box begins to expand over time and instead of being one small area, it turns into a bigger area and a bigger set of targets. There is very much, I think, a risk that we make these incremental moves towards autonomy and then look back and go like, "Oh wow, we actually moved quite a bit and we didn't realize that at some point in time, we did cross a meaningful line, but it maybe wasn't as obvious at the time."
Aaron MacLean:
Well, I mean, don't you think it's fair to say that whatever considerations we may now have... And I'm curious, you've worked on this, you should tell us what US government or DOD policy is on this question, but isn't the reality that as soon as we find ourselves in a shooting war, say in the Western Pacific, that this will accelerate? That is to say the degree of autonomy will accelerate and our good intentions about what seems ethical will be not the most important consideration. The most important consideration will be battlefield effectiveness as the other guy accelerates.
I have a sense you've thought more about this than me, that the PLA planning cells are probably less invested in the thought of what's good for humanity and probably more invested in the thought of what's going to work. But you tell me.
Paul Sharre:
No, I think that's right, and I think that the operational pressures... In peacetime, it's one thing and that there's a lot of debates. I mean, if you look at how the Defense Department right now is talking about, for example, the China threat. There's a lot of talk about worries about the PLA modernization and Xi Jinping saying they need to be ready to invade Taiwan by 2027. In practice, we look at what the Defense Department's doing. It's not on a wartime footing. It's not remotely at the sort of urgency that you would want to have if you actually thought that was true, we might be only a few years away from a conflict with China, and that's across the board. So I think there is this huge difference between sort of the peacetime way, the sense of urgency versus in wartime, and there's no question that on the Chinese side, the legal and ethical considerations don't get the same amount of play.
I've had really the incredible opportunity to be in a lot of conversations with Chinese counterparts who work on military AI issues through track two dialogues between the US and China on academic-to-academic exchanges. Very illuminating to better understand how they're thinking about these issues. And there's not as much of a focus on legal and ethical issues the way there is in US discussions. Now, they are worried about control, and they are very concerned about the unreliability of AI systems and they're very worried about political control and ensuring that their leadership all the way to the top is very tight controlled over what the military is doing. It's coming from a somewhat different perspective and that their objections are more about keeping control than they are about ethical issues.
Aaron MacLean:
Yeah, common consideration of totalitarian or near totalitarian societies is your military. It's the ultimate heat-seeking missile coming back on you is losing control of your own weapons. Okay, so let's stick with battlefield applications for a minute. So we've talked about targeting. We've talked about, I guess in a way all we've talked about is targeting, but targeting in terms of data analysis and then targeting in terms of autonomous systems.
What else on the battlefield is something worth thinking about with AI? Because there are obviously off-battlefield applications that are also very relevant, cyber electronic warfare. Talk us through what should be front of mind.
Paul Sharre:
So, I was in a conversation recently where someone said all that AI could do is improved decision-making, which to me seems like actually a good way to sum it up and seems incredibly valuable. So I think one... Maybe to go back to the Industrial Revolution analogy.
The Industrial Revolution transformed physical aspects of warfare. AI is likely to be the same to cognitive aspects of warfare - data analysis, information processing, decision-making. So, if you think about really every stage of the targeting cycle, the actual missile on target is the simplest component of that in many ways. It's finding where is the enemy queuing intelligence resources to gather information, processing that, getting that information to the right people at the right point in time. All of those things could be sped up better with AI, anywhere from using AI to process imagery or other forms of sensors more accurately to make sure that the right information is going to the right people, to having communication networks that were able to flex to demands at the time.
So, we're able to say, "Okay, we're getting increased demands here in the communications network. We need to maybe shift assets accordingly." We can automate some of that process and do so in ways that are reliable and good that could be helpful, to things like as we're presenting the information to people, being able to present that information in a way that makes it easier for people to make high-quality decisions faster is all going to be, I think, incredibly, incredibly valuable.
Aaron MacLean:
Is it the... And by the way, I should just confess as I ask this question. I'm about to ask a question about how AI and cyber work together, which is a little bit like, for me, asking a question about how multi-variable calculus and Indian cuisine work together. I'm just very out of my depth.
But is there a way in which... You think about cyber penetrations of systems. Is there a way in which AI there, the way in which it helps is beyond decision-making, it could make the weapon more effective just because presumably if one is a cyber warrior of some sort, one is working in a world where there are things that need decryption and there are sort of walls to be burst through and there's a lot of data and algorithms that work in this process and something about AI just makes your weapon more effective, it makes your operational ability sharper. Is that fair?
Paul Sharre:
Yes. Yes and yes. So, there's a lot of really interesting things that kind of the cyber AI nexus here. One is that a lot of cyber activities both on the offensive, defensive side can be automated. So certain types of attacks, what we know can be automated. This was demonstrated way back in 2016 in the DARPA cyber brand challenge as well as finding vulnerabilities and patching them on a defensive side can largely be automated, which can help shore up networks and help secure unsecured devices. That doesn't even get into things like machine learning. So, there's a couple ways that AI can be helpful on the cyber side.
On the defensive side, one of the ideas is that you could use AI systems to train them on data for malware to look for indications of malware or indications of suspicious activity operating within your network. One of the things that AI systems can do is they can both... If you sort of have patterns of activity that are suspicious, whether it's cyber activity, it could be financial activity like wire fraud, you can code that in, or if it's too complicated to write the rules down, but you have good data sets for it, you can train an algorithm on that data and it can learn to identify patterns just like it's identifying patterns in people's faces to do facial recognition.
AI systems can also identify just anomalous activity. You can train algorithms to say this is not normal. I don't know what this is, but it's not normal. And so AI tools are used for things like spam filters, for example, to detect spam emails or phishing attacks.
On the offensive side, one of the things that we're right now seeing programmers doing is using AI text generation tools to help accelerate programming. And so you can go into tools like... Chat GPT is kind of more general purpose, so it's not super great at this although it can do this, but there are more special purpose kind of coding tools where you can tell it, "Hey, I need to write a script for X, Y, Z, write it in this language for me." And it can write you a first cut. Or you could say, "Hey, I have this computer code and it's not working and I'm getting this error message. Can you help me debug this?"
These tools are actually incredibly helpful to programmers. They would also be presumably helpful to cyber attackers trying to write malware and conduct cyber attacks. A lot of these tools are right now not at the level where they're necessarily enabling entirely new forms of attacks. They're things like lowering the bar for the skill you need to carry out an attack or to maybe do something good as well in terms of computer programming, or accelerating the productivity of people that are experts in this field. But I suspect that over time we'll see the models get better and they will enable just new things that humans, you're like, "Well, I never thought of that, but that's actually pretty clever," and the AI discovered that. That's coming too.
Aaron MacLean:
Yeah. So, this is starting to move us away from the battlefield, but the way you just outlined that makes me want to ask about the ways in which AI lowers the bar, lowers the barrier for entry into various kinds of activities, some of which are really dangerous and distressing in which AI could obviously play a kind of destabilizing role. So the manufacturer of chemical or biological agents is something you've written about and something that's obviously of great concern. The manipulation of DNA, I know is a great concern to very smart people who follow these things.
And here we are. I guess we're into a subject matter where we're talking about, I think, for the most part up until now in our conversation, we've discussed the way in which AI is going to be used probably and for the most part by states and militaries and large organizations. Here, we're moving into... I guess maybe cyber is the exception to that there at the end. Here, we're moving into a space where actually its potential is to be destabilizing and a generator of violence for very small, not very well-funded entities like people, individuals. Please.
Paul Sharre:
Well, this intersection of AI and chemical and biological weapons is probably one of the more concerning areas. Certainly, one of the areas where there, if this really pans out in a dangerous way, the risks would be massive. We all just lived through a global pandemic. I think we've seen... And as global pandemics go, that was a very mild one compared to prior historical examples. And so the potential scale of destruction, for example, from biological weapons is just really quite massive and very troubling.
Now, there's also, I should say a huge amount of uncertainty here. So, I will maybe lay out the case for why people are concerned, but I do want to say that this is an area where there's a lot of debate. I sort of fall... I'll preview my view in the camp of, "Well, let's not wait until some terrorist group or some apocalyptic cult has made some terrible biological weapon and then go, 'Oops, too bad.' Let's err on the side of being cautious here." And I think some cautious is warranted, but there are a couple of concerns.
One is lowering this barrier to entry. And the concern effectively is that these general purpose models like Chat GPT or other large language models can do a whole host of things. They can write a poem. They can create a screenplay. They can write a pitch for a podcast. They could help aid in conducting scientific experiments and they can also aid in the development of chemical and biological weapons. The people have demonstrated the ability with these models, the uncensored versions. If you go to it right now and you ask these models, a censored version is going to say, "I'm not going to help you carry out this terrorist attack." But the uncensored versions can help somewhat for a non-expert. They're not going to help an expert who already has some knowledge, but if you're a non-expert, they'll help.
I think it's very unclear whether they're more helpful than just what you could find out on Google right now. But the models are getting better because what they do, and this is I think really important to understand this, they're not just capturing information. A lot of this information exists on the web. They're capturing it and they're synthesizing it into a model that then has knowledge. It's more like talking to a person or a specialized assistant who has some knowledge, and then it can help you do things like debug scientific experiments. You could say, "I'm trying to conduct this experiment, it's not working for me. Here's what's happening. Can you help me out?" Now, the models are okay at this right now, but they're going to get better. And so a world where anyone can access the equivalent of a postdoc in biology or chemistry at their fingertips with specialized knowledge like this, that's going to make some of these tools more accessible. That's one concern.
Another concern is that more specialized biological tools just for biology could help aid in the development of even more capable biological weapons. Those are going to be things that, in the near term, only experts can use. Your average person is not going to know how to use that, but those are going to probably expand the horizon of the lethality, the transmissibility of what might be possible.
And then the scary thing is the intersection of these two. It's someone who doesn't have the expertise using a general purpose model to help plan things and a general purpose model itself being capable enough to use more specialized biological design tools to then develop more dangerous pathogens. And we're not there yet, but it's a definite possibility when we look into the years ahead. I think it's something we need to take very seriously.
Aaron MacLean:
I'm glad to hear you say that we're not there yet, both for the obvious reasons, but also because my only interactions with AI technology or large language models directly has been attempts to use Chat GPT as a research assistant. And inevitably it fails me. I'll have some specific question like, "When did Zhou Enlai make his famous joke to Henry Kissinger about what's your view of the French Revolution? It's too soon to say. Tell me when that happened. When's the date and when's the transcript this actually happened." And it couldn't. It kept coming back to me with various versions of gobbledygook. I got into an argument with it. It was ugly.
But as you point out, if I go back a year from now, maybe it'll be able to do it. Well, okay, so say more if you will, about the countermeasures to these threats, whether regulatory, technological. Say more about what might be done if we were inclined to do something.
Paul Sharre:
Well, there's a couple areas of policy discussion that are very active conversations right now in Washington, and some of these were included in the recent executive order out of the White House on artificial intelligence. Some of them were signaled in the executive board. One involves red teaming these models, basically doing safety testing on them for models above a certain threshold. The executive order sets a threshold, which was of computing power used to train the models, which is sort of a bit of a crude proxy for things at the frontier of AI development; new, really big, very capable models that are being developed. And we don't have good safety standards yet, but it tasked the National Institute of Standards and Technology to develop a set of safety standards that then companies would have to evaluate their models against. So we could say things like, "Well, does this model help someone create biological weapons?"
We need to have a mechanism of actually testing that first, and then we can say, "Well, if it does, by how much? And then debate is it safe to release?" So I think that's one big area of conversation.
Another one surrounds open source. A lot of the leading AI companies, Google Open AI, Anthropic, have moved towards restricted release of their models. You can interface with Chat GPT, but you do it through this interface where the model is withheld by OpenAI. You're just sending queries to the model and you're getting responses. You don't have access to the underlying model itself. Now, Meta has taken a different approach and they're releasing their models open source. Now, the problem with this is once it's open source, anybody can fine-tune the model to get rid of these safeguards. So the model that Meta is releasing itself has been fine-tuned to put in these safeguards.
So, if you ask the model to do something bad, carry out a terrorist attack, it's not going to help you do that. But at very low cost and very easily people can get rid of those safeguards. And we see within a day of these models being released, there's an uncensored version online because it's the first thing people do the internet. They're like, "Well, let's get rid of these safeguards. I don't want to deal with this." And so that's another very active area of policy discussion of do we need to have maybe more some stringent guardrails in place, and what's liability for these companies look like if these models are used for harm?
Aaron MacLean:
Can I ask a really dumb question? How transportable are the models? That is to say, "Okay, so a company has a model and it can make the code available." How big is the thing? Can I have it on my laptop? Do I need specialized equipment once this thing is out in the wild? Can any Joe essentially transport it around?
Paul Sharre:
Yeah, I mean, this is actually a fabulous question because these technical details matter a ton. If you look at something like nuclear weapons. What has made nuclear non-proliferation largely successful, not perfect, but largely successful, is it's really hard to make a nuclear weapon. And even if you want to get ahold of one, you can't just copy it to get more nuclear weapons. So there's this big asymmetry with AI right now. We're training a large, very capable model is very computationally intensive. So, it requires tens of thousands of these most advanced chips in the world, which only a couple of people have access to that number of chips, only a handful of the big AI labs. They need to be run for months at a time. It's very costly to do that. The companies don't release their cost numbers for these models, but the best independent estimates are that it costs tens of millions of dollars to train these very capable models.
So it's really, from a sort of proliferation standpoint, there's only so many people that can do this, and the engineer requirements as well are also very challenging to just make all this work. Now, once the model's been trained, that's entirely different. The trained model itself is just a piece of software, and so folks may have heard about this term, model weights. The model itself is basically represented by a series of numbers that are called these model weights that represent the weights of the connections in this neural network. Basically, it's a giant data set of numbers.
If you have the model weights, you have the model, that is the thing, and if you know how to use it, then you can employ it and you can adjust the model and fine-tune it for your own purposes. The trained model does not need a lot of hardware.
So the biggest, most advanced models can't necessarily run on a run-of-the-mill laptop, but they certainly could run on hardware that's pretty accessible. And oftentimes the model itself can be distilled into a smaller version that is not quite as capable, it's still pretty good, that can run on a laptop. So, we see very quickly this proliferation trend where only a handful of people can train the model, requires massive amounts of computing hardware. But then once it's trained, that trained model is a piece of software. It's online. It's open source, anyone can use it, and very quickly people distill that into even smaller versions that are still pretty good that those versions actually anyone can run on a laptop and then the cat's out of the bag in terms of controlling the technology.
Aaron MacLean:
Yeah, a little bit of a change in subject, but I want to get to this before we conclude, which is we've talked about nukes and we've talked about AI. Let's talk about AI and nukes. You wrote about this recently for War on the Rocks. How do these things work together and what should we be concerned about here?
Paul Sharre:
So what's fascinating here is the United States has a very clear policy in this that came out in the 2022 Nuclear Posture Review where the Defense Department said that there will always be a human in the loop for any decisions that are critical to informing or executing a decision by the President to use nuclear weapons. That's a very clear policy statement that doesn't exist in other aspects of the DOD thinking about AI and autonomy. So they have notably not said that about, say, conventional weapons. They have not said there'll always be a human in the loop. Sometimes military, like officers, will state that statement off the cuff. They'll say it in a press release, a press conference, "Well, we'll always have a humanism." The unofficial policy; that's just their opinion, and the policy is a little bit more lax, but not on nuclear weapons. And so I think that's certainly important.
I think this is the most dangerous military technology, the most critical military mission that we need to get right. We need intense surety over nuclear weapons in the sense that you both never want what has been characterized as always, never dilemma. You never want them used when they're not authorized, but you always want them to be used if they are authorized to be used. And in fact, deterrence hinges upon the enemy knowing that if the President says, "We're going to use it," it's going to happen. And it's not quite as simple. What the article that I wrote with my colleague, Michael Depp in War on the Rocks kind of talked about was that it's not as simple as saying, "Well, there's just going to be no AI in nuclear operations." When I talked about earlier here, things like AI image classifiers helping to identify enemy objects and satellite imagery or drone video feeds.
Well, that might be the kind of thing that you actually want AI being used for to help identify early warnings of some kind of attack. Could we use AI to process satellite imagery so that we know if North Korea is going to do a nuclear test, they're going to launch a missile? We want to know that AI might help people to speed that up, get that information to our decision-makers. AI might be able to help with early warning, but this is an area where we want it to be very reliable. We don't want to a false alarm, and we want to make sure that there's always humans in whatever the critical components are. And so one of the things, the open questions is what are those critical components? What is the next step beyond that policy guidance? How do you put that into practice?
Aaron MacLean:
Paul Sharre, the Center for a New American Security author, most recently of Four Battlegrounds: Power in the Age of Artificial Intelligence, a book, which I hope our conversation has now persuaded you really got to check out. I really appreciate you making the time. This is a fascinating conversation. I will sleep less soundly tonight.
Paul Sharre:
Well, I'm sorry for that, but hopefully, some food for thought for you and listeners, and thanks for having me on. Really appreciate it.
Aaron MacLean:
This is a Nebulous Media Production. Find us wherever you get your podcasts.