Reference architecture on Databricks
The enterprise AI architecture, built on Databricks.
24 layers and 29 named Databricks services, each with its release status.
GA 22 Preview 2 Beta 1 Confirm 3 Gap 1
Plan around these
7 choices on Databricks are not generally available.
Preview and Beta services are not production commitments. Confirm means vendor sources disagree or no dated source was found. Gap means there is no native service, so a partner product or your own build fills the layer. Each needs a fallback and an owner before a production release depends on it.
- Ontology and knowledge graph: Genie Ontology Preview
- Master data and entity resolution: No native MDM service Gap
- Agent tools, gateway and MCP: Databricks managed MCP servers Preview
- Agent runtime and memory: Agent Bricks and Genie Agents Confirm
- Agent runtime and memory: Managed agent memory and sessions Beta
- AI guardrails and action policy: Unity AI Gateway guardrails Confirm
- Security operations (SIEM and SOAR): Lakewatch Confirm
Layer by layer
What each layer must do, and what Databricks offers for it.
Mosaic AI and Agent Bricks for models and agents, Unity Catalog for catalog, lineage and metrics, and Lakeflow for connectors and pipelines. Each release status was checked on 13 November 2026. 22 of 29 services are generally available. Service names link to Databricks’s own documentation. The architecture stays the same on every platform; only the products change.
Brain Holds approved business meaning and knowledge.
| Layer and what it must do | Databricks services and status | How RJL approaches it |
|---|---|---|
| Catalog, metadata and lineageRecord what exists, what it means, where it came from and who owns it. |
| The Federated Data Platform |
| Semantic layer and metricsApproved metrics and definitions, versioned, with an owner. |
| Ontology or Semantic Layer? |
| Ontology and knowledge graphThe entities and relationships the business agrees on. |
| Context Before Query |
| Knowledge retrieval and vector searchFind permitted evidence in documents and records, filtered by entitlement. |
| Securing AI Retrieval |
| Master data and entity resolutionMatch, survive and share one party identifier, tested for false merges. |
| Data Products in Practice |
Nervous system Carries approved work through connectors, APIs, tools and events.
| Layer and what it must do | Databricks services and status | How RJL approaches it |
|---|---|---|
| SaaS and enterprise source connectorsBring data from CRM, ERP, service and HR applications. |
| The Federated Data Platform |
| Change data capture and streamingMove changes from source systems as they happen. |
| The Federated Data Platform |
| APIs, events and workflowVersioned interfaces and workflows, including write-back to systems of record. |
| Agent Patterns |
| Agent tools, gateway and MCPExpose narrow tools to agents through one governed gateway. |
| Agent Onboarding |
Body Stores, computes, delivers and operates.
| Layer and what it must do | Databricks services and status | How RJL approaches it |
|---|---|---|
| Storage, open tables and historyKeep raw, time-stamped history where analytics needs it. |
| Data Products in Practice |
| Warehouse, lakehouse and operational storesQuery, join and serve data at the latency each use needs. |
| The Federated Data Platform |
| Pipelines and transformationConform, test and publish, with releases matched to consequence. |
| Data Is Not Just Code |
| Data products and sharingPublish governed products with contracts, owners and access. |
| Data Product Certification |
| Models and AI platformRun approved models with routing, quotas and versions. |
| Choosing an Agent Platform |
| Agent runtime and memoryRun agents with sessions, memory and limits. |
| Choosing an Agent Platform |
| Observability, evaluation and costTrace every request, evaluate quality and measure cost per task. |
| AI Token Economics |
Immune system Identifies, classifies, enforces policy and responds to threats.
| Layer and what it must do | Databricks services and status | How RJL approaches it |
|---|---|---|
| Identity for people and agentsGive every agent its own identity, owner and permissions. |
| Agent Onboarding |
| Access policy at the dataEnforce rows, columns and attributes where the data lives. |
| Context Before Query |
| Classification, DLP and DSPMLabel sensitive data and documents, and watch where copies go. |
| Securing AI Retrieval |
| AI guardrails and action policyCheck prompts, outputs and tool calls before they take effect. |
| Agent Patterns |
| Security operations (SIEM and SOAR)Detect, investigate and respond, with agent activity in scope. |
| The Reference Implementation |
| Keys and confidential computingControl keys and protect data while it is in use. |
| Confidential Computing for AI |
Control plane One record of what was approved, released and observed.
| Layer and what it must do | Databricks services and status | How RJL approaches it |
|---|---|---|
| Agent registry and release controlKnow every agent, tool and version that is approved to run. |
| Agent Onboarding |
| Audit and governance evidenceReconstruct who did what, with which version and authority. |
| AI Governance Evidence |
Getting data in
SaaS and enterprise source connectors on Databricks.
| Source | Databricks option and status |
|---|---|
| Salesforce | Lakeflow Connect GA |
| SAP | SAP Databricks in Business Data Cloud Confirm |
| ServiceNow | Lakeflow Connect GA |
| Workday | Lakeflow Connect (Workday Reports) GA |
| Microsoft Dynamics 365 / Dataverse | Lakeflow Connect GA |
| Relational databases (CDC) | Lakeflow Connect GA |
| Mainframe and Db2 | Partner Gap |
| Kafka and event streams | Structured Streaming GA |
| Documents (SharePoint, Drive, Confluence) | Lakeflow Connect (Confluence GA, Drive beta) GA |
Examples, not endorsements. Status reflects public vendor sources checked on 13 November 2026; confirm with the vendor before you commit. Richard has worked closely with several vendors named here as a customer and practitioner. No vendor reviewed or paid for this content. Download the full catalog (JSON).
The same architecture on AWS · Microsoft Azure · Google Cloud · Snowflake, or a mixed estate.
Next step
Bring your Databricks estate and one workflow.
In a working session we map what you run today to these layers, mark the gaps that matter for the workflow, and agree what has to be true before it reaches production.