Editors Note: This blog post highlights Palantir’s response to a Request for Information pursuant to the 2023 Executive Order on Safe, Secure, and Trustworthy AI. For more information about Palantir’s contributions to AI Policy, visit our website here.
Introduction
At Palantir, we are proud to provide the institutions serving our societies with the vital software platforms they need to make responsible and effective use of their data. We believe the procurement of software technologies, including advanced Artificial Intelligence (AI) systems, should similarly conform to standards and practices that promote responsible adoption and, ultimately, reliable and effective mission impact.
To that end, Palantir shared its views with the Office of Management and Budget’s Request for Information (RFI) on Responsible Procurement of Artificial Intelligence in Government. This RFI is an extension of the program laid out by the 2023 Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence and our response follows earlier contributions to RFIs and RFCs issued by the Federal Trade Commission, National Telecommunications and Information Administration, Office of Science and Technology Policy, National Institute of Standards and Technology, as well as the Office of Management and Budget, both preceding and following this Executive Order. A full list of Palantir’s AI Policy contributions can be found here.
In this RFI, OMB puts forward a series of questions related to existing Federal procurement practices, soliciting guidance on how those processes may be modified or improved to address attributes of a growing marketplace and Federal agency appetite for AI technologies. Our response highlights several high-level principles and themes that we’ve continually advanced, and that we think the government should consider in examining its approach to AI capabilities procurement. These include:
- Since AI risks and benefits are context dependent, the government should eschew one-size-fits all evaluations and instead invest in the infrastructure capabilities to conduct use case-specific evaluations.
- Software should be verified to work as designed and deliver outcomes as intended before it is purchased.
- When commercially available, the government should procure software and AI-enabled solutions from industry providers, as opposed to internally developing redundant capabilities from scratch.
- AI procurement should reinforce and reward best practices for software, which include interoperable components using open standards, access controls and data protection, and ongoing system monitoring and audit capabilities.
In this blog post, we summarize the OMB questions and provide brief highlights from the perspective we offered OMB. You can find our full response to the RFI here.
OMB’s Prompts & Palantir’s Responses
1. How may standard practices and strategies of Federal procurement, such as Statements of Objectives, Quality Assurance Surveillance Plans, modular contracts, use of contract incentives, and teaming agreements as well as innovative procurement practices, such as those in the Periodic Table of Acquisition Innovations, be best used to reflect emerging practices in AI procurement?
AI models cannot be deployed in isolation of the foundational digital infrastructure (i.e., system) that enables their effective and responsible use, which is itself powered by broader software capabilities. It is for this reason that AI procurement will rarely, if ever, make sense in strict component terms, as AI models are generally embedded in broader information systems. In other words, agencies are unlikely to simply purchase “AI” — they will be purchasing fully integrated systems that include some mix of data services, software, and AI models.
… while software procurement practices and authorities can still be substantially improved, it is important to work with and build upon the existing Federal procurement infrastructure.
2. How can OMB promote robust competition, attract new entrants, including small businesses, into the Federal marketplace, and avoid vendor lock-in across specific elements of the technology sector, including data collectors and labelers, model developers, infrastructure providers, and AI service providers? Are there ways OMB can address practices that limit competition, such as inappropriate tying, egress fees, and self-preferencing?
The best way for OMB to address practices that limit competition is to commit to procuring commercially available solutions when existing capabilities — that meet program requirements — are already available and readily deployable with minimal customization (as pursuant to U.S. Code Title 10, Section 2377, U.S. Code Title 41, Section 3307, and the Federal Acquisition Regulation).
Beyond this first-order solution, we can recommend the following strategies and requirements to ensure OMB is promoting a robust AI service ecosystem that also safeguards the interoperability needs of Federal agencies:
- Procure interoperable solutions: For AI systems and services that extend beyond standalone AI models, OMB can require commercial vendors to demonstrate that their systems are modular and interoperable with other capabilities, as well as designed to be AI model agnostic.
- Safeguard data rights: To avoid vendor lock-in, while also promoting new entrant innovation, agencies should stipulate that the U.S. Government maintains complete and exclusive rights to its own data, while commercial AI vendors maintain complete and exclusive rights to their own core intellectual property.
- Ensure fair competition: OMB can promote robust competition and attract new entrants by employing procurement processes that create a level playing field for businesses of all sizes and maturities.
- Streamline the ATO process: The current Authority To Operate (ATO) process is overly complex, lengthy, and costly for new market entrants, and thus privileges incumbent AI vendors that have the knowledge and resources to traverse this cumbersome process.
3. Should the Federal Government standardize assessments for the benefits and trade-offs between in-house AI development, contracted AI development, licensing of AI-enabled software, and use of AI-enabled services? If so, how?
Yes. The Federal Government should … consider whether AI system adoption should be approached with a rebuttable presumption in favor of commercial solutions acquisition (as stipulated in FAR Part 12). Alternatively, if there is a motivated preference in favor of in-house AI development, the burden of proof should fall on agencies to show how and why in-house approaches are preferable to commercial solutions. The provided justifications and trade-off analysis should include monetary, quantitative, and qualitative assessment parameters.
4. How might metrics be developed and communicated to enable performance-based procurement of AI? What questions should agencies be asking vendors to determine whether AI is already being used in performance-based services contracts?
The appropriate measure of success for an AI application will depend on: its context of use; the specific demands of the discipline and environment; the distinct risk profile of the workflows in question; the adopting agency’s tolerance for errors; the capacity to mitigate specific AI risks; and performance comparisons against incumbent and alternative approaches, among a multitude of other considerations. For those reasons, performance-based procurement of AI should start from the premise that there is no one-size-fits-all approach to AI performance evaluations. Agencies should be left with the discretion to make reasonable and tailored performance assessments in accordance with the demands of their programs and domains.
Despite the advisability of a more fluid evaluation framework that allows for context-specific performance evaluations, vendors should nonetheless bear the burden of proving that their products and services can meet agencies’ particular performance demands. One method for enforcing a “show, don’t tell” burden on capability demonstration is through software bootcamps. These are multi-day, pre-sales, hands-on working sessions in which agency stakeholders (including SMEs and AI capabilities users) are presented with the opportunity to work directly with the AI systems under evaluation, alongside vendor representatives who are made available to address questions and performance considerations in situ.
5. What access to documentation, data, code, models, software, and other technical components might vendors provide to agencies to demonstrate compliance with the requirements established in the AI M-memo? What contract language would best effectuate this access, and is this best envisioned as a standard clause, or requirements-specific elements in a statement of work
The responsible procurement and deployment of AI requires the application of appropriate standards and requirements, and importantly, the tools, methods, and data necessary to ensure actors and systems are meeting those standards. Accountability and transparency requirements should therefore be top-of-mind for agencies seeking to procure trustworthy and dependable AI capabilities, and we can offer the following suggestions for how Federal agencies can address these considerations:
- Foundational digital infrastructure: Based on our experience building software platforms for AI development, evaluation, deployment, and use, we have seen first-hand the importance of foundational investments in the digital infrastructure of AI systems for sustaining accountability and transparency best practices.
- Model cards: All AI models used in AI systems should have documentation or “model cards,” at minimum providing details about how the model was trained (including an outline of data used in training), what its intended use is, and what its known areas of limitation are.
- Context-specific reporting: From our experience, the form of accountability most relevant to the use of AI technologies is never “one-size-fits-all.” Rather, accountability invariably relates to the context of use and its corresponding risk profile.
6. Which elements of testing, evaluation, and impact assessments are best conducted by the vendor, and which responsibilities should remain with the agencies?
Responsibility for testing will vary depending on the nature of the AI system: Is the performance highly context-dependent, or is performance similar across deployment scenarios? Does the AI system impact rights or safety? Is the AI system custom built for the agency, or integrated into a larger AI system?
Software systems and platforms can contain AI components as “native” capabilities that are low risk and for which all testing is performed by the vendor. Examples could include spell checkers in word processing software or an LLM-powered Q&A service that helps with IT platform documentation questions. These capabilities are developed in a customer-agnostic way, are expected to function similarly across a wide variety of use cases and deployment scenarios, and are not safety- or rights-impacting. Such systems are the responsibility of the AI vendor to test via their development and integration T&E process and thus may require limited or no socio-technical testing. Performance and proper use should be communicated with the agency so that the system is not used in a way exceeding its capabilities.
…
When a model is built specifically for the government, either in-house or via contractors on government data, the agency should cooperate with the model provider to do integration testing, and if necessary, a socio-technical full systems evaluation. Model developers, regardless of where they sit organizationally, should be required to perform T&E as part of their development process.
…
Impact assessments should be considered for AI systems which carry elevated risk associated with their use, and in many cases, native AI capabilities as discussed above may not need an impact assessment. For those systems that depend critically on the use of integrated vendor-provided AI capabilities, the primary assessment responsibility may fall with the agency but with assistance or clarifications provided by vendors.
7. What if any terms should agencies include in contracts to protect the Federal Government’s rights and access to its data, while maintaining protection of a vendor’s intellectual property?
Contracts with Federal agencies should explicitly state the Federal Government’s exclusive rights to its own data, as well as vendors’ exclusive rights to their own core intellectual property. Both requirements are contractually compatible. Furthermore, vendors should only be allowed access to the bare minimum data required to maintain and/or administer systems on customers’ behalf (e.g., deal with bugs, provide appropriate user metrics, etc.).
9. How might agencies structure their procurements to reduce the risk that an AI system or service they acquire may produce harmful or illegal content, such as fraudulent or deceptive content, or content that includes child sex abuse material or non-consensual intimate imagery?
We have long advocated that organizations focus on the fully integrated AI system and not just the component parts.9 This integrated systems approach to AI highlights the importance of the AI/ML model(s) in addition to the data foundation, infrastructure, user interface, and socio-technical context. Below, we provide a list of a few capabilities that might be helpful to include in procurements for AI systems to ensure that such systems and their infrastructure have critical capabilities for reducing the risk of producing harmful or illegal content:
- Data monitoring: The data foundation of an AI system should be consistently monitored or scanned for harmful or undesirable data that might adversely affect the AI system.
- Model monitoring: Not only should the data in an AI system be monitored, but infrastructure of the AI system should offer capabilities for monitoring the inputs to and outputs from the AI system.
- Human-in-the-loop workflows: Human oversight is a critical aspect of an AI system, and the system should allow for human oversight of and intervention into decisions that an AI/ML model suggests or orchestrates.
- Access controls and security architecture: Access controls are critical for any technical system, and especially for limiting the risk that harmful or illegal content might be uploaded to, processed in, or exported from such a system. We recommend that AI systems be required to support zero-trust and least-privilege security architectures.
10. How might OMB ensure that agencies procure AI systems or services in a way that advances equitable outcomes and mitigates risks to privacy, civil rights, and civil liberties?
Before procuring an AI system, agencies must focus on understanding the intended use and potential risks of the AI system in context. The nature of these risks may differ dramatically depending on the specific application context of the AI system. If the agency identifies that the AI system will pose a risk to privacy, civil liberties, or civil rights, we recommend that agencies leverage existing processes for addressing these risks. For example, one approach agencies can consider is leveraging Privacy Impact Assessments (PIAs) to transparently document the privacy risks that the agency has identified about their AI system. Palantir recently provided comment to OMB on how PIAs can be made more effective, especially in the context of AI systems, and we encourage agencies and the OMB to consider those recommendations about PIAs here as well.
Conclusion
Our response to OMB on Responsible Procurement of Artificial Intelligence in Government underscores Palantir’s long-standing commitment to building and promulgating technologies that advance societal-ethical imperatives while supporting the critical mission objectives of Federal agencies. As ever, we are proud to share our insights and perspective with OMB and encourage interested readers to check out our full response here, along with our full list of Palantir’s AI Policy Contributions here.
Authors
Anthony Bak, Head of AI Implementation, Palantir Technologies
Courtney Bowman, Global Director of Privacy and Civil Liberties Engineering, Palantir Technologies
Arnav Jagasia, Privacy and Civil Liberties Engineering Lead, Palantir Technologies
Morgan Kaplan, Senior Policy & Communications Lead, Palantir Technologies
Palantir’s Response to OMB on Responsible Procurement of AI in Government was originally published in Palantir Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.