Streamlining Data Connectivity for Next-Gen AI Architectures
At our most recent AIPCon, we announced the general availability of AIP Virtual Tables — a connectivity solution that makes it seamless to get started with AIP’s full spectrum capabilities, without requiring any data replication or duplication. Simply connect your existing data investments and go.
This blog post discusses why this capability is so exciting for enterprises that have made investments in various data architectures, and are now looking to leverage those investments within a next-generation AI architecture.
AIP: Powering Full Spectrum AI
Palantir AIP brings the power of Generative AI to the entire organization, ensuring the secure integration of cutting-edge models with the data systems, algorithms, and systems of action that drive daily decision-making. Examples of AIP-powered workflows include:
- Material Shortages Management: Supply chain managers at a multinational power management company are using AIP to identify, prioritize and respond to material shortages. LLMs can reason through the notes and actions taken in previous shortages to provide automatic, viable material suggestions — resulting in a 25% productivity increase.
- Nurse Scheduling: Nurses and CNOs at one of America’s largest healthcare organizations are using AIP’s automated scheduling functionality to generate shift schedules that consider nurse preferences, decision costs, and demand predictions. After automated schedules are created, AIP can help generate staff schedules and adapt them to accommodate real-world disruptions (e.g., staffing or patient demand changes) and recommend changes to inform optimal balance of staff.
- Train Routing: To alleviate network congestion, planners at in a major rail company are using AIP to surface areas that are at risk for high railcar “dwell” and build remediation plans. LLMs let users interact with complex routing and forecasting models through natural language. Real-world feedback from planners, provided in natural language, helps to improve models and capture nuances that were previously unavailable.
Virtual Connectivity to Accelerate AI Adoption
From longstanding large-scale enterprise customers to those just getting started in an AIP Bootcamp, Virtual Tables enables users to begin building AIP-powered workflows in minutes. It connects data assets within your enterprise’s data lakes, data lakehouses, and data platforms — without any costly or messy data duplication.
AIP Virtual Tables offers several major advantages:
- A fast track for getting started with the Ontology and AIP: Virtual Tables lets you use external systems as the backing sources of truth for data elements in the Ontology and throughout AIP. This allows enterprises to register new data sources without having to create redundant copies of the associated data or pipelining logic.
- Maintain architectural flexibility: Virtual Tables lets teams choose when to leverage data in-place that resides within data platforms, and when to use Palantir’s data layer. Each data solution offers different advantages, and teams can change the underlying data flows over time, without impacting workflows in AIP.
- Improved performance and cost-efficiency: Virtual Tables harnesses the native features across different data platforms, including predicate pushdown, which results in better performance for certain data processing and analytical tasks. This also reduces costs by minimizing data storage and transfer.
Preserving Architectural Optionality
As organizations identify the most powerful ways to incorporate Generative AI into the enterprise, maintaining architectural optionality is critical. The correct technical approach will vary, depending on needs of specific workflows and applications. As AIP-powered feedback loops strengthen your understanding of where AI creates the most value within decision-making processes, Virtual Tables offers the flexibility to evolve the specifics of your underlying architecture to meet new requirements.
AIP Virtual Tables helps you preserve this optionality in several ways:
- Eliminates duplication: Gain immediate access to your data stored in Google BigQuery, Snowflake, AWS S3, or Azure Blob Storage.
- Auto-registration of existing assets: For certain sources, Virtual Tables can be configured to periodically check for new tables and automatically create corresponding virtual tables for easy discoverability.
- Works seamlessly with traditional data syncs: If desirable, virtual tables can be subsequently replaced with traditional data syncs. The two approaches can also be used alongside one another to complement one another.
- Leverage existing Palantir platform capabilities: Use Virtual Tables across the full range of existing functionality: Create pipelines in Pipeline Builder without duplicating data inputs, analyze data with pushdown performance boosts in Contour, and power Object Types with zero-copy pointers in the Ontology — with more capabilities arriving soon.
How Tyson Foods is Using AIP Virtual Tables
Tyson Foods is one of the world’s largest food companies, specializing in the production and distribution of high-quality protein products. Tyson Foods is leveraging AIP — and along with it, Virtual Tables — to accelerate its adoption of AI-powered workflows.
Virtual Tables is a game changer for Tyson Foods. It enables our analytics organization to move with the velocity we’ve grown to love with Palantir, while simplifying our architecture. In just a few minutes, we used Virtual Tables to register all 1,700 golden tables in our BigQuery data lake. This allowed us to begin building AI-powered workflows instantly. The best part is, we no longer need to copy our data; we can now simply utilize it right where it resides. It’s truly amazing how seamless and efficient the entire process has become. Virtual Tables also helps us clean up our cloud spend 40–50% by removing large amounts of duplicate data processing and storage and getting to “one version of the truth.” — Nathan Marks, Sr. Staff Solutions Architect, Tyson Foods
Experience AIP Virtual Tables at Your Next AIP Bootcamp
For organizations with existing data architectures, AIP Virtual Tables offers the fastest path to getting AI use cases into production — while preserving architectural optionality. For additional details, we encourage you to read our technical documentation here. And to experience the difference of AIP and AIP Virtual Tables first hand, reach out to your Palantir rep to reserve your AIP Bootcamp slot.
AIP Virtual Tables was originally published in Palantir Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.