Editor’s note: In February 2023, the UK House of Lords launched a call for evidence in support of the AI in Weapons Systems Committee inquiry into the development, use, and regulation of autonomous weapons. Drawing upon our experience providing the kinds of software capabilities that underpin military applications of AI and autonomy, Palantir put forward a written submission and was invited to participate in a public evidence session. Courtney Bowman, Palantir’s Global Director of Privacy and Civil Liberties Engineering, appeared before the Committee and is the author of the following post on AI policy.
The open letter by the Future of Life Institute calling for a pause on the development of AI “with human competitive intelligence” [1] has prompted critical discussions about the effects and risks of AI systems. In a recent inquiry considering the use of AI in weapons systems, the UK House of Lords Artificial Intelligence in Weapons System Committee opened the evidence session by soliciting expert opinions — including ours — on the position of the letter’s signatories. There were several thoughts and responses that came to mind:
1) Our adversaries are not postponing their own development of these technologies, so can we justify ceding whatever nascent advantage the West has already gained? [2]
2) Such a pause would be practically unenforceable. The letter calls for a pause that is “public and verifiable, and include[s] all key actors.” But what would that verification look like? How would it be enforced? Who are the key actors — other than presumably large tech companies with threshold computing resources? Enacting such a moratorium would be a futile exercise, whether or not it may be justified.
Finally, and perhaps most persuasively:
3) It is not clear that the facts underlying the current and reasonably foreseeable state of these technologies actually merit such an extreme measure. In fact, calling for a pause may do more harm than good — sounding a general alarm that is more likely to evoke confusion and misplaced fears than to orient discussions towards the ethical, legal, and normative considerations that are sorely needed. If we want to address the real issues around AI, shouldn’t we start with a sober and faithful description of the actual state of technology and proceed from there? [3]
In my appearance before the Committee, and in an accompanying written submission I put forward an additional argument for resisting calls for a moratorium, especially in the military context. I suggested that meaningful progress towards controlling AI and harnessing it for human flourishing does not come from pausing experiments, but rather (and somewhat counter-intuitively) from leaning into the fielding of operationally-thoughtful and responsibly-constructed experiments that force us to confront the real challenges of the technologies in situ.
In my remarks, I further suggested that the UK has an opportunity to play a global leadership role in both driving the direction of this technology development and the norms around its responsible use for defense purposes. However, this type of leadership does not arise from abstention. Nor does it arise from isolated research programs and reports that expend more effort “admiring the problems” surrounding AI adoption. Rather, real progress, on both the ethics and the efficacy of AI technologies — and especially in the most consequential contexts — requires an operational turn. Both Palantir’s CTO Shyam Sankar, in testimony before and in written statement to the U.S. Senate Armed Services Subcommittee on Cybersecurity, and I, before the House of Lords Committee, referred to this as “fielding-to-learn.”
An operational, “field-to-learn” approach to AI enables both technical innovation and ethical boundary setting. It does so by better exposing technologists, ethicists, policy-makers, and warfighters to the specific challenges of AI deployment and use, as opposed to more theoretical musings that, while interesting, are often untethered from the reality of both the technology and the operational setting.
The U.S. Defense Department’s Algorithmic Warfare Cross-Functional Team (aka, Project Maven) provides one such framework for public-private partnership in which industry technologists and military professionals were exposed to operational settings in order to “accelerate the DOD’s integration of big data and machine learning” [4]. In early stages, this did not mean exposing untested technologies to active conflicts. Rather, the focus of initial development efforts was on access to near-real exercise environments in which the technologies could be better tuned and validated against their expected scenarios of use.
More generally, exposure to field environments proves essential to addressing the most consequential challenges that our societies face in applying novel AI technologies. For instance, LLMs may offer significantly enhanced capacities to summarize voluminous collections of digital texts to help inform human decision-making based on that information. However, examining the accuracy and utility of these machine facilitated synopses demands an evaluative framework for human decision-makers to verify, validate, and assess whether the model outputs are indeed reliable and useful [5]. The development of such a framework requires setting aside thinly substantiated worries about the triggering of an exponential explosion of superintelligence leading to “catastrophic effects on society” [6] and instead a focused exploration of the specific and grounded ways that LLMs are actually being applied.
Similarly, in popular discussions about consumer-facing AI technologies — such as those intended to provide fully-autonomous or self-driving vehicles — the academic ethicist fixation on the “trolly problem” [7] has distracted sight from the more critical, and ethically loaded, questions. We should be spending less time musing over whether Kant or Mill provide us with the right rules or calculations to direct autonomous system decisions that will likely never present in clean, theoretically analyzable terms. We should instead be expending considerably more effort addressing whether these technologies work as advertised, and whether outsourcing control in ambiguous operational environments can be justified in the first place when the technology has not been proven to work outside of predictably restricted environments.
Throughout my testimony and in Palantir UK’s written response, we shared a number of related perspectives on the potential value of AI in weapons systems, suggestions for addressing related risks, and thoughts on the outsize role the UK can play if it effectively marshals the wealth of talent and capability that exists across its firms, universities, research institutions, and civil society organisations. Here are some highlights from our presented evidence.
On how AI in weapons systems may be changing the calculus of war fighting
We noted in our written response and elsewhere that autonomy is already changing the makeup of defence forces and the nature of combat. In particular, we identify time as perhaps the most valuable resource in modern war fighting.
When deployed effectively, as part of a human-in-the-loop decision-making process, AI and autonomy increase the time available for critical human-driven decisions, reducing the overall time-to-decision and improving the accuracy and reliability of decision-making.
For instance, rather than having human analysts pour over satellite images to determine enemy force disposition, AI tools can help rapidly automate the identification and prioritisation of relevant potential matches, leaving more time for human analysts to examine and validate the highest signal identifications and construct better-informed action plans based on that information.
On the adequacy of existing accountability and International Humanitarian Law for addressing the legal and ethical challenges of AI in weapons systems
We wrote that “questions of distinction and proportionality ultimately require more than quantitative assessments — assessments which won’t always be translatable into programmable code. They also require some significant measure of qualitative human judgement. Does one reasonably believe this target is a combatant? Is the potential collateral damage acceptable for the military objective in question? Have the relevant procedures been followed to ensure that the human has a reasonable amount of reliable information? These are all questions for which ranking commanders or oversight bodies could have for the human decision-maker after the fact if necessary, to hold a specific individual accountable for mistakes.”
Further, accountability in the case of an autonomous weapons system (AWS) would likely be an entirely different matter. Targeting determination or recommendation (e.g., “this set of pixels represents an enemy tank that should be targeted”) will have shifted from specific, qualitative, morally-weighted evaluations to quantitative, optimised calculations that are focused primarily on the assessment of outcomes rather than the methodical adherence to established process. Importantly, this is not a change in degree, but would require a new edifice of oversight and accountability mechanisms to ensure that distinction, proportionality, military necessity, and other IHL principles are implemented appropriately.
On the need for training and upskilling for Defence personnel using AI systems
We wrote that “while a great deal of responsibility will, and should, rest with the engineers building the systems, there must be a parallel effort to train personnel who will be using these systems. As the accountable human decision-makers, those who are ultimately responsible for the critical decisions that will be made with these tools, they need to have an in-depth understanding of the systems they are looking to deploy.”
Engineers and technologists have a responsibility to provide capabilities that operate reliably, as intended, and in accordance with legal and ethical imperatives. Defence personnel have a responsibility to appropriately deploy, interact with, and operate those technologies in the execution of their military duties.
On the role of the private sector in driving ethical development of AI for Defence
More generally, my Palantir colleagues and I have sought to demonstrate — by example — how the private sector can play a proactive role in shaping the ethics and policy around emerging technology development and use. Our approach to AI ethics acknowledges the moral responsibilities of technology providers and in building tools that are “inexorably embedded in a world of tangible actions and consequences.”
It is not a foregone conclusion that AI technologies will or should become integrated in all (or even some) aspects of warfighting. However, determining whether both efficacy and ethics can justify the use of these technologies simply cannot be conducted as a theoretical exercise — it requires instead addressing the operational challenges that militaries face in all the complexities of real world application. The private sector — as technology partners to Defence — must be similarly attuned to these realities.
Conclusion
Returning to the open letter, while we believe a moratorium on AI development would be a mistake, and even counter-productive to its intent, the letter contains a point on which we do agree: The “decisions [about the future of AI] must not be delegated to unelected tech leaders.” For this reason, we welcome the opportunity to participate in government-led exercises, such as the HoL and Senate Subcommittee hearings. While a pause on AI may not be the right approach for the challenges raised by these new technologies, meaningful engagement between legislators and stakeholders spanning industry, civil society, academia, and the public at large will be essential to arriving at decisions that will impact our shared future.
For a complete view of our recommendations to the Committee, please read our written response.
Endnotes
[1] https://futureoflife.org/open-letter/pause-giant-ai-experiments/
[2] Palantir’s CEO Dr. Alex Karp has argued similarly in a recent letter.
[3] I have previously made this point in a recent podcast interview on AI regulation. Jaron Lanier makes a similar argument in a recent article in The New Yorker, “The arguments aren’t entirely rational: when I ask my most fearful scientist friends to spell out how an A.I. apocalypse might happen, they often seize up from the paralysis that overtakes someone trying to conceive of infinity. … The most pragmatic position is to think of A.I. as a tool, not a creature. … Mythologizing the technology only makes it more likely that we’ll fail to operate it well — and this kind of thinking limits our imaginations, tying them to yesterday’s dreams.” Lanier, J. “There Is No A.I.: There are ways of controlling the new technology — but first we have to stop mythologizing it.” The New Yorker, 20 April 2023, https://www.newyorker.com/science/annals-of-artificial-intelligence/there-is-no-ai.
[4] https://dodcio.defense.gov/Portals/0/Documents/Project%20Maven%20DSD%20Memo%2020170425.pdf
[5] Such an evaluative framework should help characterize the basis of the LLMs’ outputs (e.g., providing citations) to support the generated text or to expose its flaws (e.g., in the case of “hallucinations”).
[6] https://futureoflife.org/open-letter/pause-giant-ai-experiments/
[7] Roff, H. “The folly of trolleys: Ethical challenges and autonomous vehicles.” Brookings, 17 December 2018, https://www.brookings.edu/research/the-folly-of-trolleys-ethical-challenges-and-autonomous-vehicles/.
Appearance at UK House of Lords Committee on AI in Weapon Systems was originally published in Palantir Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.