Necessity, proportionality, and transparency
Over two years ago, as the COVID-19 pandemic began to come into focus, we published a thought piece on best practices for using data in a crisis. Though we all found ourselves in the early days of a global crisis whose magnitude was difficult to fully understand, Palantir aimed to present an educational and cautionary note to institutions charged with public health responsibilities. In the post, we outlined a set of principled considerations to help maintain a balance between the urgent demands of the moment and the preservation of privacy and civil liberties — values that undergird our society.
At the time, Palantir had no formal work or contracts in place to support the pandemic response, but we were committed to answering the call if the opportunity arose. Over the course of 15+ years, we had supported institutions grappling with the challenges of digital transformation under the best of circumstances, and especially in moments of extreme instability. We knew that even independent of our direct involvement, public and private sector organizations would benefit from the best practices we had learned along the way.
As the full extent of the pandemic took shape, we began exploring opportunities to support public health agencies, such as the Department of Health and Human Services (HHS) in the US, the National Health Service (NHS) in the UK, the Colombian Presidential Council for Economic Affairs and Digital Transformation, and The Netherlands South 6 Safety Regions, with their response efforts. We found that the following principles (detailed below) also served us well as useful guideposts in our direct work with customers:
- Data is not a panacea
- Focus on decisions to be made, not just insights to be discovered
- Start with the data you have
- Emphasize foundational tools and beware the shiny new object
- Create and enforce a privacy-protecting data strategy
- Build a data governance body
- Serve the patient and respect their human dignity
In retrospect, if we were to boil down our mindset for applying those principles to the needs of public health agencies, it would take the form of a mantra: necessity, proportionality, and transparency. This mantra became the drumbeat for conversations we held internally in shaping our engineering efforts, as well as for our external conversations with customers and other community stakeholders on what to demand and expect from technology applied to pandemic interventions.
With the benefit of hindsight, here’s a revisiting of each of the principles we had outlined in our March 2020 blog post — how these ideas were applied, what we learned, and what lessons could be brought to bear in future crises.
Data is not a panacea.
From the earliest days of the pandemic, Palantir was approached by (and itself approached) prospective clients to explore ways that our software could support the life-saving work of public health agencies. Our tendency was to decline or altogether avoid situations where the client institution carried an incorrigible view that data alone could save the day.
It may strike some observers as counterintuitive that a company whose raison d’être is building data management platforms would also reject the premises that data solves all problems or data-driven solutions apply to every challenge the world faces. But underlying this viewpoint is a principle of humility, as well as our deep awareness of how complex linking information to the world it represents can be. Crises are rarely (if ever) just about data or technology considerations. On the contrary, data is an inherent aspect of extraordinarily complex environments.
To be sure, we believed then — and continue to believe — that data-driven efforts were essential to the pandemic response. But we also recognized the folly of exalting data or technology alone as a sufficient first-order focus. Rather, we’ve learned over the years (not just from COVID-19 work) that our clients must begin with a clear view of how the use, management, and governance of their data must support core institutional operations and functions.
Focus on decisions to be made, not just insights to be discovered.
As we began working with public health agencies and private sector companies to support their COVID-19 response efforts, we observed a recurring theme. Institutions, struggling to fully frame the magnitude and characteristics of the pandemic, tended to want to amass data, hoping insights and actions would emerge from a composite view. This default tendency, however, is liable to be counterproductive, creating a sense of information overload that detracts from specific questions that need to be addressed.
In these circumstances, we tended to focus client attention on concrete, specific questions with directly actionable consequences, like understanding the availability of PPE to hospitals and first responders. These concrete, clear questions translated to a necessary and proportionate data requirement, such as specific pipelines and analyses.
Start with the data you have.
When institutions have few immediately available actions that will bring meaningful relief, one understandable but misguided response is to simulate action by rapidly seeking access to vast amounts of data, even without a clear thesis about the tangible improvements this data will likely yield. We argued that this impulse should be avoided, or at least heavily questioned. In retrospect, the actual utility of many data sources sought during the pandemic proved limited or outweighed by collateral risks. At the same time, existing or more readily available data sources were often useful, but underutilized.
Take for example, mobility data. During the early stages of the pandemic, as various jurisdictions enacted and enforced lockdowns, curfews, and other restrictions on movement, policymakers and public health officials sought to understand the effectiveness of these policies in reducing community transmission. A whole ecosystem of mobility data providers suddenly emerged and promoted the aggregation of various data sources, including call data records from mobile networks, GPS data from vehicles, wearables, and smartphone apps, bidstream data from ad networks, imagery captured from aerial platforms like satellites and planes, and geotagged social media posts.
Several problems emerged around mobility data providers. First, some derived location information from sources of questionable origin, with worrisome transparency and data privacy practices. Second, some academics raised legitimate questions about how accurately location data reflected a fitting cross-section of society, given bias in mobile device and application adoption. Third, some of the most heavily promoted and adopted sources of this data have since become the heaviest targets of legislative scrutiny.
Throughout the pandemic, we closely tracked the risks associated with various forms of mobility data. We regularly discouraged our customers from acquiring licenses to problematic sources that raised privacy concerns. Given the limited utility of mobility information in understanding aggregate population movements, we tended to advise our customers to focus on open platform analysis from companies (e.g., Apple and Google) whose collection and reporting methods represented stronger approaches to addressing privacy concerns.
Emphasize foundational tools and beware the shiny new object.
In a moment of crisis, it’s natural for those concerned with the state of affairs to want to help. But in reality, good intentions may become misdirected.
The modern world is awash in technology, data, and the engineers, technologists, scientists, and other knowledge workers who make their living from it. As a result, a sea of goodwill efforts arose to offer technology-oriented solutions to apparent challenges created by the COVID-19 crisis. Some of these solutions addressed well-constituted, thoughtfully constructed, and important public health questions. But many amounted to distracting “shiny objects.”
Perhaps one of the most prominent examples of “shiny object” technology during the pandemic was the rush to build and deploy digital exposure notification or contact tracing systems. Many of us in technology, privacy, or public health circles will recall a tremendous amount of effort, debate, promise, and consternation that swirled around the idea of leveraging smartphones to serve as a proxy for individual exposure to other individuals diagnosed with COVID-19. So captivated were technologist communities that proponents sprawled and even splintered into schisms of competing methodologies and protocols: centralized vs. decentralized, GPS vs. Bluetooth, selectively identifying vs. strictly anonymizing, metadata enrichment vs. limited data capture, etc.
While most, arguably all, of these efforts arose from a wellspring of good intentions, the vast majority — if not all — of the technical, political, and social effort poured into these systems ultimately proved to be a monumental distraction. In virtually every instance of these massive undertakings, the end-product digital notification systems either failed to see the light of day or proved to be of negligible value when adopted.
What exactly went wrong? Few technologists seemed willing to consider more fundamental questions, instead focusing on building technically clever software solutions. Namely, institutions largely ignored questions like:
- Will digital exposure notifications actually make a difference by changing people’s behavior?
- Does the motivating analogue method of manual contact tracing — used so effectively in tracking the spread of direct transmission pathogens like STDs — make sense for an airborne virus with poorly understood transmission characteristics?
In most cases, the answer was plainly “no.” The utility of exposure notifications was predicated on two critical conditions: 1) broadly accessible, rapid, reliable COVID-19 testing, and 2) non-community spread / limited transmission environments for which individual contact tracing might be reasonably accurate. Neither of these conditions proved true throughout the early stages of the pandemic, making the debates over exposure notification methods, protocols, and apps essentially meaningless.
At a number of pivotal moments, we actively encouraged public health agencies and other institutions, whether or not they were customers of Palantir’s, to focus on establishing proportionate, reasonable data environments before resorting to shiny technology. Instead of many of the other major tech companies, which vigorously rode (or drove) the techno-optimist exposure notification train, we took a deep breath and continuously focused our efforts on the basics. Our data and integration and analysis software gave public health institutions the necessary and proportionate tools to manage their data and create a “common operating picture” they could use to plan and direct public health policy initiatives and logistics.
Exposure notification apps weren’t the only example of shiny tech obsession. Another prominent set of examples came from attempts to leverage machine learning AI for everything from patient diagnosis to hospital triage to general transmission predictions. The one consistent theme was that AI tools made little or no impact in battling COVID-19. There are likely many overlapping reasons for these failings, starting with underlying data challenges. In a July 2021 article in MIT Technology Review, Dr. Bilal Mateen of Wellcome Trust, a global health research charity based in London, best summarizes the failure of AI to contribute to useful pandemic outcomes: “Until we buy into the idea that we need to sort out the unsexy problems before the sexy ones, we’re doomed to repeat the same mistakes.”
This critique is not to suggest that the technologists behind these projects weren’t well-intentioned or that all of them were doomed to failure. But this is a classic demonstration of “when you’re a hammer, all the world appears as nails.” Technologists excel at building technologies and often seek to apply that mode of excellence where they see problems. But not all problems are treatable with technological interventions.
Create and enforce a privacy-protecting data strategy.
In devising a data strategy, institutions must look beyond the immediate moment and anticipate evolving use cases for that data over time. While it’s impossible in a dynamic situation to anticipate every shift in circumstances, it’s crucial to structure the data enterprise in a way that will respect and protect personally identifying or identifiable information.
Recalling the mantra of “necessity, proportionality, and transparency,” conditions for the use of sensitive or personal information should be made clear at the moment of collection. Even when consent conditions include a more extensive range of permissible uses, organizations should limit the accessibility and use of personal data to limit the risk of overexposure.
During the pandemic, we learned that enforcing this critical piece of a data strategy required additional tooling, so we built and deployed purpose-based access controls to enable organizations to partition data processing in our platforms according to specified purposes, with designated and accountable owners. These innovative controls helped our customers reinforce strong security and data governance hygiene.
We also established a layered structure of boundaries and rules governing the use of our software to minimize the risk of data misuse. The first layer is contractual, specifying conditions of use of the software, duration of programs, and milestone events or other checkpoints. The second layer is procedural, establishing continuous evaluation steps for ensuring limited access to sensitive data and system capabilities. The third layer encompasses the implementation of technical tools and features to enforce key information handling principles, such as high quality data integration, step-wise data processing (e.g., through encryption or other deidentification processes at the time of data ingestion), and full lifecycle data management all the way through to end-of-use deletion handling.
Relying on any one of the aforementioned layers alone may not provide sufficient protection, but combining these complementary approaches can provide a valuable defense-in-depth strategy.
Build a data governance body.
Data governance under the most relaxed circumstances can be challenging. In a crisis, it can prove daunting, especially when you don’t have enough knowledge on hand. In such circumstances, a body of external experts can help you better adapt to unknown developments, build trust with a broader coalition of stakeholders, and provide continuous oversight and accountability as programs develop.
In our engagements with public health officials, we strongly encouraged and supported efforts to establish data governance bodies that could oversee the programs employing our software. Though Palantir as a software provider was in no position to dictate the operations of government institutions, within our own walls, we continually sought guidance on questions related to product development, customer use case support, and sensitive data integration controls from an assortment of external advisors, including the Palantir Council of Advisors on Privacy & Civil Liberties.
Serve the patient and respect their human dignity.
In all of our COVID-19 response work across a multitude of public and private institutions, we have stayed true to our core values (as outlined in our company Code of Conduct and further articulated elsewhere). In each potential customer engagement, with each proposed software feature request, and with each prospective data integration by our customers, we insisted on asking questions at the core of our “necessity, proportionality, and transparency” mantra: 1) Will this work contribute to meaningful public health outcomes, including helping to save lives?; 2) Can legitimate work be carried out while keeping in mind the risks to individual well-being, privacy, and civil liberties?; 3) Can we promote a broader understanding of our work across a community of relevant stakeholders in the spirit of transparency and openness?
To suggest that our execution against these principles was flawless would be folly. While our efforts aimed at balancing the public health objectives of institutions facing a a global crisis with individuals’ interests, rights, and dignity, the outcomes likely fell short of the ideal on a number of occasions.
One such example is our communications around this work. While there were limitations on what we could communicate on behalf of our customers, it nevertheless quickly became clear that we did not have enough pre-existing, clear, approachable content in the public domain about our business model and the depth of our commitment to protecting people’s privacy and data.
We sought to fill this gap through a multitude of channels, including a running series of blog posts explaining all aspects of our pandemic response work, open exposition of our work with journalists around the world from the earliest days of the pandemic, and continued dialogue with civil society groups (e.g., this exchange with Privacy International), but we know we can do more in this area. In particular, additional efforts along these lines might have helped address common points of confusion, such as the idea that Palantir operates as a data company, or that our motivation for supporting public health initiatives was to gain access to proprietary and sensitive health information with which to build lucrative AI models.
Trust is not acquired through single acts; it is earned through continuous effort. This is the reason we continue to talk and write about all facets of Palantir’s work in the pandemic and beyond, why we offer this candid assessment of our principles for handling data in a crisis, and why we will continually and diligently work to engage any and all interested parties who are willing to sit down and speak with us in good faith.
History has its eye on us
We closed our crisis principles post two years ago with an admonishment, as much for ourselves as for the world: “This is an emergency — perhaps the defining one of our age. In acting decisively to defeat this pandemic, we must do so in a way that we will recognize ourselves when it’s done.” As we close this piece, another global crisis sadly looms large in the form of the Russian invasion of Ukraine, a sovereign European nation.
Once again, a history-defining moment stands before us, as does another opportunity to reinforce and refine the principles and values that undergird our work as a company, as well as the institutions and societies we serve. As we look to answer the call to support humanitarian and other responses to the war in Ukraine, these reflections will remain top of mind. Whether a health crisis, an international conflict, or some other front of global instability, the only defensible outcomes are those that allow us to emerge with our values intact, with a clear awareness of what we have been fighting all along to preserve and protect.
Author
Courtney Bowman, Palantir Privacy and Civil Liberties Engineering Lead
Reflections and Lessons from the COVID-19 Crisis was originally published in Palantir Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.