13 · The Databricks Data Intelligence Platform at a Glance
Find the Signal. Act with Intelligence. · Knowledge Base · Databricks (Weeks 21 to 24)
This file is the one-page mental map for the entire Databricks module. It answers "what is this platform, what are its moving parts, and where in the program do I learn each part?" Each section is a 1 to 3 sentence "what it is" plus a pointer to the week and platform file that teaches it. For depth, follow the pointer; for the raw facts this file is built from, see research/06-databricks-deep-dive.md.
The running case study is ZoroLogistics, a fictional freight company whose synthetic data toolkit lives in zoro/. Every example in the platform files maps a Databricks feature to a ZoroLogistics problem: a governed medallion lakehouse, a streaming shipment pipeline, a point-in-time ETA model, and a RAG assistant over shipping documents.
⚠️ These facts move fast: verify against live docs. Databricks renamed a large number of products through 2024 to 2026 and continues to ship aggressively. Every rename below is flagged current name (formerly …). Before you build on any specific name, API, version number, or limit, re-check the pages linked in Sources and the machine-readable index at
https://docs.databricks.com/llms.txt.
1. What the platform is
Databricks is a unified, open analytics and AI platform for building, deploying, and governing data, analytics, and AI, the Data Intelligence Platform. It is not a separate cloud; it runs inside your cloud account (AWS, Azure, or GCP) and layers a governed lakehouse on top of your object storage, while Databricks manages the orchestration and control services on your behalf. Databricks' differentiator is that it adds AI on top of the lakehouse, understanding data semantics and automatically optimizing performance, indexing, and infrastructure.
The architecture is two conceptual planes:
| Plane | What runs there | Where it lives |
|---|---|---|
| Control plane | Web UI, identity, job scheduling, notebook/workspace metadata, backend services | Databricks-managed |
| Compute plane | Actual data processing, Spark executors, SQL engines | Classic: your cloud account · Serverless: Databricks-managed |
- Classic compute runs in your virtual network → natural network isolation, customer-managed VPCs.
- Serverless compute runs in a Databricks-managed plane, created in the same region, inside a network boundary that isolates your workspace.
Which week teaches it: Week 21, reference/platforms/databricks/00-day-zero-setup.md.
2. Accounts vs. workspaces
- A workspace is a single Databricks deployment, where a team works with notebooks, jobs, clusters, SQL warehouses, and data. It has workspace storage for notebooks, queries, and dashboards. (The legacy DBFS root and DBFS mounts are deprecated, use Unity Catalog for all data access.)
- A Databricks account is the top-level entity holding multiple workspaces. At the account level you manage identity (users/groups/service principals, SSO/SCIM), workspace creation across regions, Unity Catalog metastores, and billing.
- The account console (
accounts.cloud.databricks.com) is where account admins manage settings, subscription/billing, serverless quotas, and workspaces.
Workspaces come in two deployment types:
| Type | Description |
|---|---|
| Serverless workspace | Pre-configured serverless compute + Databricks-managed default storage. Best for Genie One, AI/BI dashboards, Apps, serverless pipelines, most AI features. |
| Classic workspace | Storage + compute in your own cloud account. Needed for legacy Spark RDD code, Scala/R-first teams, direct on-prem/private connectivity. Serverless is still available inside it. |
An account can hold multiple Unity Catalog metastores; one metastore can attach to multiple workspaces in the same region, giving them one shared, governed view of data.
Which week teaches it: Week 21, 00-day-zero-setup.md.
3. Naming quick-reference (current → formerly)
Reproduced from the research fact base §0, the renames you will trip over if you read older blogs, courses, or the UI before a migration:
| Current name | Formerly known as | Notes |
|---|---|---|
| Lakeflow Pipelines | Delta Live Tables (DLT) | Built on open-source Apache Spark Declarative Pipelines (SDP); Python module dlt → pyspark.pipelines (import as dp) |
| Lakeflow Jobs | Workflows / Jobs | Sidebar entry is now "Jobs & Pipelines" |
| Lakeflow Connect | Databricks Ingest / Ingestion | Managed + standard connectors for SaaS/databases/files/streams |
| Declarative Automation Bundles (DABs) | Databricks Asset Bundles | Still abbreviated "DABs" |
| OpenSharing | Delta Sharing | Open project now at opensharing.io |
| AI Search | Vector Search | Managed vector retrieval |
| Unity AI Gateway | AI Gateway | Governance control plane for LLM/agent/MCP traffic |
| Genie Agents | Genie Spaces | Natural-language data Q&A configured by analysts |
| Genie One | Databricks One | Simplified business-user UI |
| AI/BI dashboards | Lakeview dashboards | Legacy "DBSQL dashboards" are archived |
| Data quality monitoring | Lakehouse Monitoring | Anomaly detection + data profiling |
| SQL warehouses | SQL endpoints | Renamed 2023 |
| Models in Unity Catalog | MLflow Model Registry | Registry "stages" → movable aliases (@prod, Champion/Challenger) |
| Git folders | Repos | Built-in Git client |
| Standard / Dedicated access mode | Shared / Single-user access mode | Compute access modes |
AUTO CDC | APPLY CHANGES | CDC in Lakeflow Pipelines (same syntax) |
Trigger.AvailableNow | Trigger.Once | One-shot incremental Structured Streaming |
The "Mosaic AI" prefix has largely been retired across the docs ("Mosaic AI Model Serving" → Model Serving, "Mosaic AI Model Training" → Databricks Model Training), though it still appears in some marketing material. Teach the current name; note the legacy.
Which week teaches it: all of Weeks 21 to 24; kept here as the module's cheat sheet.
4. Unity Catalog
Unity Catalog (UC) is the unified governance layer for data and AI, one metastore that governs tables, volumes, files, models, functions, and ML artifacts with fine-grained access control, lineage, audit, and discovery. It uses a three-level namespace:
metastore ── catalog ── schema ── table | view | volume | function | modelOther securables (storage credentials, external locations, connections, shares) sit directly under the metastore. Tables are managed (UC owns the data lifecycle) or external (you point at existing storage). UC is the foundation everything else in the module stands on, MLflow models, feature tables, and AI Search indexes all live in it.
Which week teaches it: Week 21, 01-unity-catalog.md.
5. Delta Lake
Delta Lake is the default storage layer: unless configured otherwise, all tables on Databricks are Delta tables. It extends Parquet with a file-based transaction log to add ACID transactions, schema enforcement/evolution, and time travel, and it is fully compatible with Spark Structured Streaming (one copy of data for batch + streaming). It is the substrate for the medallion architecture (bronze → silver → gold) and for features like liquid clustering, deletion vectors, and the change data feed.
Which week teaches it: Week 21, 03-delta-lake.md.
6. Compute
Compute is the "engines" that run your code. Three broad families:
| Family | What it is | Learn in |
|---|---|---|
| All-purpose clusters | Interactive, shared compute for notebooks and ad-hoc work (you create/terminate manually) | Week 21 |
| Job clusters | Ephemeral compute created per job run and torn down after (cheaper per DBU) | Week 22 |
| SQL warehouses | Analytics-optimized engines for Databricks SQL (Serverless / Pro / Classic) | Week 21 |
| Serverless compute | Databricks-managed on-demand compute for notebooks, jobs, and pipelines | Week 21 |
Key concepts: Databricks Runtime (DBR), Photon (the vectorized C++ engine), cluster policies (admin guardrails), autoscaling, access modes (Standard vs. Dedicated, formerly Shared vs. Single-user), and DBU pricing (the normalized unit of compute per hour).
Which week teaches it: Week 21, 02-compute.md.
7. Databricks SQL (DBSQL)
DBSQL is a cloud data warehouse on the lakehouse: ANSI SQL plus Delta extensions, served by SQL warehouses. Its surfaces are the SQL editor (Genie Code-assisted), saved queries, query history, alerts, and the AI/BI product family (dashboards + Genie). Because it reads the same Delta tables Spark writes, BI and ML share one copy of data rather than two silos.
Which week teaches it: Week 21, 04-dbsql.md.
8. Lakeflow: Pipelines, Jobs, Connect
"Lakeflow" is the umbrella brand for Databricks' data-engineering surfaces:
- Lakeflow Pipelines (formerly Delta Live Tables): declarative batch + streaming pipelines in SQL or Python (
pyspark.pipelines), with automatic orchestration, data quality expectations, and CDC via AUTO CDC. Week 22. - Lakeflow Jobs (formerly Workflows): workflow orchestration: tasks (notebook, wheel, SQL, dbt, pipeline…) arranged as a DAG with parameters, schedules, retries, and repair. Week 22.
- Lakeflow Connect (formerly Databricks Ingest): managed ingestion connectors from SaaS apps, databases, files, and streams into governed Delta tables. Week 22.
Which week teaches it: Week 22, 06-pipelines-jobs.md.
9. MLflow + Models in Unity Catalog
MLflow is the open-source experiment-tracking and model-lifecycle tool that Databricks hosts. Experiments → runs → models: you log parameters, metrics, and artifacts with autologging or the logging API. Models in Unity Catalog (formerly the MLflow Model Registry) gives registered models a <catalog>.<schema>.<model> name; the deprecated "stages" are replaced by movable aliases, models:/zrl_.ml.eta_model@prod, or the Champion/Challenger convention. MLflow 3 adds GenAI observability, evaluation, and prompt management.
Which week teaches it: Week 23, 07-mlflow-experiments.md.
10. Feature engineering
Unity Catalog Feature Engineering (the Feature Store) is a central registry for ML features with governance, lineage, and point-in-time joins, using the same features at train and serve time eliminates training/serving skew. You define feature tables (a Delta table with a primary key) or declarative feature views, then use FeatureLookup + create_training_set() to assemble training data and score_batch() for batch inference. A time-series feature table declares a timestamp key so joins are AS OF the label timestamp (no leakage).
Which week teaches it: Week 23, 08-feature-engineering.md.
11. Model Serving + Unity AI Gateway
Model Serving is unified real-time/batch inference as a REST API on serverless, auto-scaling compute. It serves three model types: custom models (MLflow PyFunc, including agents), Databricks-hosted foundation models (pay-per-token or provisioned throughput), and external models (OpenAI, Anthropic, Bedrock, Vertex…).
Unity AI Gateway (formerly AI Gateway) is the centralized governance control plane for AI traffic, built on UC: it governs assets (models, MCP servers, functions), traffic (rate limits, budgets, usage), and behavior (service-policy guardrails, ALLOW / DENY / ASK).
Which week teaches it: Week 23 to 24 (serving in 09-model-training.md and 10-model-serving.md; gateway in 10-model-serving.md and 16-governance-security.md).
12. AI Search (formerly Vector Search)
AI Search is managed vector retrieval in Unity Catalog: you create an index from a Delta table (Delta Sync) and query it via REST/SDK. Algorithms: HNSW (vector), BM25 (keyword), and hybrid search (Reciprocal Rank Fusion). It is the retrieval half of a Databricks RAG pipeline: embed query → similarity search → retrieve chunks → append to prompt → generate.
Which week teaches it: Week 23, 11-vector-search-rag.md.
13. AI Functions
AI Functions are built-in SQL/PySpark functions that apply LLMs to data in place: no endpoint or API key to manage. Core functions: ai_query (general prompt against a Foundation Model endpoint), ai_classify, ai_extract, ai_gen, ai_summarize, ai_mask, ai_similarity, ai_parse_document, ai_translate, ai_forecast, and more. Requirements (as of Aug 2026): serverless compute and DBR 18.2+.
Which week teaches it: Week 23, 12-ai-functions-genie.md.
14. Genie (Agents, One, Code) & Agent Framework
Genie is Databricks' natural-language-to-SQL/data family:
- Genie Agents (formerly Genie Spaces): analyst-configured NL→SQL chat over governed tables, with instructions and example SQL.
- Genie One (formerly Databricks One): the simplified business-user UI.
- Genie Code: the developer AI assistant in the SQL editor.
Agent Framework is the code-first way to build AI agents on Databricks: wrap any framework in ResponsesAgent, attach tools (managed MCP servers, UC functions, retrieval), and deploy through Model Serving or Databricks Apps. Agent Bricks are the higher-level templates: Knowledge Assistant (document Q&A with citations) and Supervisor Agent (multi-agent orchestration).
Which week teaches it: Week 23 to 24 (12-ai-functions-genie.md, 13-agents.md).
15. AI/BI (dashboards + Genie)
AI/BI is Databricks' compound-AI BI product: AI/BI dashboards (formerly Lakeview dashboards), low-code, AI-assisted dashboards with datasets, cross-filtering, custom calculations, and subscriptions, plus Genie Agents for self-service Q&A, all informed by Unity Catalog metric views (governed business semantics). The legacy DBSQL dashboards are archived; migrate via "Clone a legacy dashboard to an AI/BI dashboard."
Which week teaches it: Week 21 (dashboards, 04-dbsql.md) and Week 24 (14-apps-dashboards.md).
16. Declarative Automation Bundles (DABs)
DABs (formerly Databricks Asset Bundles) are infrastructure-as-code for data/AI projects: source files + resource definitions (jobs, pipelines, dashboards, serving endpoints, models) + tests, declared in databricks.yml and deployed as a unit. Targets (dev/staging/prod), variables, and resources describe the whole project; the CLI lifecycle is bundle init → validate → deploy → run → destroy.
Which week teaches it: Week 24, 15-dabs-ci-cd.md.
17. System tables
System tables are read-only operational data in the system catalog (free; query compute is billed). Key schemas: system.access.audit (audit events), system.access.table_lineage/column_lineage, system.billing.usage/list_prices, system.compute.*, system.lakeflow.jobs/pipelines, system.query.history, system.serving.*, and system.information_schema.*. Most retain 365 days. They power lineage, FinOps, and observability dashboards.
Which week teaches it: Week 24 (16-governance-security.md, 17-finopps-cost.md).
18. Certification landscape
Databricks certifications are role-based, at Associate and Professional levels, plus free accreditations:
| Certification | Assesses |
|---|---|
| Data Analyst Associate | Data analysis with Databricks SQL |
| Data Engineer Associate / Professional | Data engineering on the Data + AI Platform |
| Machine Learning Associate / Professional | ML tasks on Databricks |
| Generative AI Engineer Associate | Design, build, deploy GenAI solutions |
| Context Engineer Associate | Design and govern context for AI agent systems |
| Associate Developer for Apache Spark | The Spark DataFrame API |
Accreditations: Databricks Fundamentals (Lakehouse) and Generative AI Fundamentals. Official training is on Databricks Academy with role-based learning pathways.
Which week teaches it: Week 24, 18-certification-path.md.
19. How it all hangs together (the one-paragraph arc)
ZoroLogistics lands raw shipment files into a Unity Catalog volume, loads them into Delta Lake bronze tables, cleans and dedupes them through a Lakeflow Pipeline with data-quality expectations into silver, and aggregates on-time KPIs into gold. Analysts query gold with DBSQL and build AI/BI dashboards; data scientists log MLflow experiments, assemble point-in-time features from the feature store, train an ETA model, register it to Models in Unity Catalog, and serve it behind the Unity AI Gateway. A RAG assistant over shipping documents uses AI Search + AI Functions, exposed via Genie to ops analysts. The whole thing ships as a DAB through CI/CD, is governed end-to-end by Unity Catalog, and is monitored for cost with system tables. That single arc: governed data → reliable pipelines → governed models → served AI, is Weeks 21 to 24, and it is exactly the modernization journey Zorost's Databricks practice runs for its clients.
20. Architecture diagram
§1 describes two planes in prose; here is the picture to hold in your head for the whole module.
%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart TB
subgraph CP[Control plane, Databricks-managed]
UI[Web UI · Genie One · notebooks metadata]
ID["Identity: SSO / SCIM / service principals"]
SCHED[Job scheduler · workflow metadata]
end
subgraph XP[Compute plane, runs the actual work]
SV[Serverless compute<br/>Databricks-managed]
CL[Classic compute<br/>your cloud VPC]
SQ[SQL warehouses<br/>Serverless / Pro / Classic]
end
UI --> SCHED
SCHED --> SV & CL & SQ
subgraph UC[Unity Catalog, governance]
CAT[catalog → schema → table / view / volume / function / model]
GOV[GRANT/REVOKE · row filters · column masks · lineage · audit]
end
SV & CL & SQ --> UC
UC --> DL[Delta Lake tables on object storage]
DL --> MED["Medallion: bronze → silver → gold"]
MED --> OUT[BI · ML · GenAI consumers]
subgraph AI[AI surfaces]
SERV[Model Serving + Unity AI Gateway]
AIS[AI Search · AI Functions]
GEN[Genie · Agent Framework]
end
OUT --> SERV & AIS & GENThe one sentence that goes with it: control plane decides, compute plane executes, Unity Catalog governs, Delta Lake stores, and the AI surfaces all consume the same governed tables one copy of data from raw file to served model.
21. Zero-to-hero learning sequence (detail)
The research fact base §10.3 gives the four-week arc; here it is expanded into the day-level detail a motivated engineer actually follows, with the ZoroLogistics artifact produced at each step.
| Day | Focus | ZoroLogistics artifact |
|---|---|---|
| 0 | Setup: workspace, UC metastore, CLI + OAuth U2M auth, Git repo, DAB scaffold | databricks.yml with dev/prod targets committed |
| 1 | UC: catalog/schema/table/volume; GRANT/REVOKE; managed vs external | zrl_bronze / zrl_silver / zrl_gold catalogs created |
| 2 | Delta Lake: ACID, time travel, schema evolution, VACUUM, liquid clustering | Load shipments.csv to a Delta table; roll back one version |
| 3 | Auto Loader + COPY INTO; volumes as landing zone | zrl_bronze.shipments_raw streaming from /Volumes/.../landing/ |
| 4 | Lakeflow Pipelines: streaming tables, expectations, AUTO CDC | Bronze→silver pipeline with weight_kg IS NOT NULL expectation |
| 5 | Lakeflow Jobs: DAG, depends_on, Run-if, retries | Orchestrate pipeline + a validation notebook as a job |
| 6 | DBSQL: SQL warehouse, editor, query history, alerts | zrl_gold.ontime_kpis query + an alert on low on-time % |
| 7 | AI/BI dashboards + Genie Agents | An on-time dashboard + a Genie Agent over gold tables |
| 8 | AI Functions | ai_classify on ticket text; ai_extract on bills of lading |
| 9 | MLflow: experiments, autologging, Models in UC (aliases) | Log the ETA model run; register zrl_ml.eta_model@champion |
| 10 | Feature Engineering: feature tables, point-in-time joins | carrier_features time-series table; leak-free training set |
| 11 | Model Training + batch inference | Train ETA model; score_batch against feature store |
| 12 | Model Serving + Unity AI Gateway | Serve ETA model behind the gateway with rate limits |
| 13 | AI Search + RAG | Index shipping policies; a RAG assistant with citations |
| 14 | Agent Framework + Agent Bricks | Knowledge Assistant over policies; Supervisor for triage |
| 15 | DABs + CI/CD: bundle init → validate → deploy | Deploy the whole project to prod via GitHub Actions |
| 16 | Governance + FinOps: system tables, budgets | A cost dashboard off system.billing.usage |
The sequence is deliberately incremental: every later day builds on an artifact from an earlier one, and the whole thing is the single arc §19 describes, governed data → reliable pipelines → governed models → served AI.
22. Feature → week map
Where each feature is taught, so you can jump straight to it.
| Feature (current name) | Formerly | Week | Platform file |
|---|---|---|---|
| Platform planes, accounts/workspaces | n/a | 21 | 00-day-zero-setup.md |
| Unity Catalog (namespace, GRANT/REVOKE, row filters/masks) | n/a | 21 | 01-unity-catalog.md |
| Compute (clusters, policies, Photon, DBU) | n/a | 21 | 02-compute.md |
| Delta Lake (ACID, time travel, liquid clustering) | n/a | 21 | 03-delta-lake.md |
| Databricks SQL + AI/BI | DBSQL / Lakeview dashboards | 21 | 04-dbsql.md |
| PySpark DataFrames + Structured Streaming | n/a | 22 | 05-pyspark.md |
| Lakeflow Pipelines + Jobs + Connect | DLT / Workflows / Ingest | 22 | 06-pipelines-jobs.md |
| MLflow experiments + Models in UC | MLflow Model Registry | 23 | 07-mlflow-experiments.md |
| Feature Engineering (point-in-time joins) | Feature Store | 23 | 08-feature-engineering.md |
| Model Training (Optuna, AI Runtime) | Mosaic AI Model Training | 23 | 09-model-training.md |
| Model Serving + Unity AI Gateway | Mosaic AI Serving / AI Gateway | 23 to 24 | 10-model-serving.md |
| AI Search + RAG | Vector Search | 23 | 11-vector-search-rag.md |
| AI Functions + Genie | n/a | 23 | 12-ai-functions-genie.md |
| Agent Framework + Agent Bricks | n/a | 23 to 24 | 13-agents.md |
| Databricks Apps + AI/BI dashboards | n/a | 24 | 14-apps-dashboards.md |
| DABs + CI/CD | Databricks Asset Bundles | 24 | 15-dabs-ci-cd.md |
| Governance + security (system tables) | n/a | 24 | 16-governance-security.md |
| FinOps + cost | n/a | 24 | 17-finopps-cost.md |
| Certification path | n/a | 24 | 18-certification-path.md |
23. Worked medallion example (concrete ZoroLogistics numbers)
The medallion pattern (§5, §19) on real ZoroLogistics data, the numbers come from the seeded zoro/data.py generators, so you can reproduce every figure.
Bronze: raw, append-only, source-faithful:
-- 100,000 shipments land as CSV; Auto Loader appends them raw, with all their flaws preserved.
CREATE STREAMING TABLE zrl_bronze.shipments_raw AS
SELECT _metadata.file_name, * FROM STREAM read_files('/Volumes/zrl_landing/landing/shipments');The generator plants two flaws for Week 2 to find: ~200 duplicate rows (0.2% concat) and ~300 rows with null weight_kg (0.3%). Bronze keeps both, that's its job: fidelity over cleanliness, so audit can always see what actually arrived.
Silver: cleaned, deduped, typed, joined:
CREATE OR REFRESH STREAMING TABLE zrl_silver.shipments
(CONSTRAINT weight_not_null EXPECT (weight_kg IS NOT NULL) ON VIOLATION DROP ROW) AS
SELECT DISTINCT s.shipment_id, s.carrier_id, c.carrier_name, s.lane_id,
l.origin, l.destination, l.distance_km, s.commodity, s.weight_kg,
s.value_usd, s.planned_arrival, s.actual_arrival, s.delay_hours, s.is_on_time
FROM STREAM(zrl_bronze.shipments_raw) s
LEFT JOIN zrl_bronze.carriers c ON s.carrier_id = c.carrier_id
LEFT JOIN zrl_bronze.lanes l ON s.lane_id = l.lane_id;Result: 100,000 clean rows (200 dupes removed by DISTINCT, 300 null-weight rows dropped by the expectation, and recorded as a metric so you know exactly how many). Types are enforced, and the carrier/lane dimensions are joined in.
Gold: business-aligned KPIs:
CREATE OR REFRESH MATERIALIZED VIEW zrl_gold.ontime_kpis AS
SELECT carrier_id, carrier_name, commodity,
COUNT(*) AS shipments,
ROUND(AVG(CASE WHEN is_on_time THEN 1 ELSE 0 END), 3) AS on_time_pct,
ROUND(AVG(delay_hours), 1) AS avg_delay_hours,
ROUND(SUM(value_usd), 0) AS total_value_usd
FROM zrl_silver.shipments
GROUP BY ALL;Concrete gold numbers you can sanity-check against the generators: ~82% overall on-time, with the worst carrier (C014, a low-reliability draw) near ~71%, and severe-weather rows running ~35% late (the weather_p_late table drives it). avg_delay_hours is dominated by the long-tail lognormal late rows, so the median (a better summary) is far below the mean, the kind of nuance a Week-3 histogram teaches.
Why the medallion earns its keep: a data-quality bug in bronze is contained (you can replay silver from bronze), a schema change is absorbed (silver evolves without touching gold), and an auditor can always point to the exact raw row behind any gold KPI. That traceability, from the dashboard cell back to the source file, is the whole reason the pattern exists.
24. Certification landscape (deepened)
§18 lists the certs; here they are with level, the natural "when to take it" in the program, and what it's for.
| Certification | Level | Take after | What it proves |
|---|---|---|---|
| Databricks Fundamentals (accreditation) | Free badge | Week 21 | You can navigate the platform and its vocabulary |
| Generative AI Fundamentals (accreditation) | Free badge | Week 23 | You know the GenAI surface (Serving, AI Search, Genie, Agents) |
| Data Analyst Associate | Associate | Week 21 | DBSQL + AI/BI dashboard competence |
| Data Engineer Associate | Associate | Week 22 | Delta, PySpark, Lakeflow Pipelines/Jobs fundamentals |
| Data Engineer Professional | Professional | Week 22+ | Production data engineering at scale |
| Machine Learning Associate | Associate | Week 23 | MLflow, Feature Engineering, Model Training basics |
| Machine Learning Professional | Professional | Week 23+ | Production ML: serving, monitoring, MLOps |
| Generative AI Engineer Associate | Associate | Week 23 | Build/deploy GenAI (RAG, Agents, Serving) |
| Context Engineer Associate | Associate | Week 23 | Design + govern context for agent systems |
| Associate Developer for Apache Spark | Associate | Week 22 | The Spark DataFrame API itself |
Path guidance:
- Data-engineer path: Data Analyst Associate → Data Engineer Associate → Professional.
- ML/GenAI path: ML Associate → GenAI Engineer Associate → Context Engineer Associate.
- The two free accreditations first: they're the cheapest confidence win and map one-to-one onto Weeks 21 and 23. Training lives on Databricks Academy with role-based pathways (§18).
25. How it breaks / common mistakes
| Mistake | Symptom | Fix |
|---|---|---|
| Used DBFS root / mounts instead of UC | "Works" but ungoverned, deprecated | All data through Unity Catalog; DBFS is legacy |
| Wrote to the wrong medallion layer | Analysts query raw; models read dirty data | Bronze=raw, silver=cleaned, gold=business, one job per layer |
| Forgot point-in-time joins | ETA model "predicts" the future (label leakage) | Time-series feature tables + timestamp_lookup_key AS OF |
| Skipped expectations in pipelines | Bad rows silently reach silver/gold | EXPECT ... ON VIOLATION DROP/FAIL with metrics |
| Used registry "stages" | Deprecated lifecycle model | Move to movable aliases (@prod, Champion/Challenger) |
| Trusted an old product name | "Delta Live Tables" gone from the docs | Teach/use current names (§3), Lakeflow Pipelines, AI Search, etc. |
| Left an all-purpose cluster running | Idle DBU burn | Auto-termination + serverless; job clusters for scheduled work |
The through-line: Databricks failures are usually a governance or lifecycle slip, a legacy API, a skipped quality check, a stale name, or a leaky join, not a Spark bug. The renames (§3) and the medallion discipline (§23) are what catch most of them.
26. Self-check questions
What runs in the control plane vs. the compute plane, and where does your data actually live?A: The control plane (Databricks-managed) runs UI/identity/scheduling metadata; the compute plane runs Spark/SQL. Data lives in Delta tables on your object storage, governed by Unity Catalog.
Why is "all tables are Delta by default" the load-bearing fact of the lakehouse?A: Because it means one copy of data serves batch, streaming, BI, and ML with ACID + time travel, no separate warehouse silo (§5).
What does the bronze layer preserve that silver deliberately removes?A: Bronze preserves source fidelity (raw rows, all flaws, the 200 dupes and 300 null weights stay); silver dedupes, types, and enforces quality via expectations (§23).
Why do point-in-time joins prevent label leakage in the ETA model?A: A time-series feature table joins features "AS OF" the label timestamp, so the model never sees a feature computed after the event it's predicting (§10).
Name the three stages the model lifecycle uses instead of "Staging/Production."A: Movable aliases, e.g.
@prod, or theChampion/Challengerconvention, which replace the deprecated registry stages (§9).
Passing bar: 5/5, these five (planes, Delta-default, medallion, point-in-time joins, and aliases) are the mental map you need before opening a notebook in Week 21.
Sources
Platform & compute
- https://docs.databricks.com/introduction/
- https://docs.databricks.com/getting-started/concepts/
- https://docs.databricks.com/getting-started/high-level-architecture/
- https://docs.databricks.com/lakehouse/
- https://docs.databricks.com/data-governance/unity-catalog/
- https://docs.databricks.com/compute/
- https://docs.databricks.com/compute/serverless/
- https://docs.databricks.com/compute/sql-warehouse/
- https://www.databricks.com/product/pricing
Data, SQL & AI/BI
- https://docs.databricks.com/delta/
- https://docs.databricks.com/lakehouse/medallion
- https://docs.databricks.com/sql/
- https://docs.databricks.com/ai-bi/
- https://docs.databricks.com/dashboards/
- https://docs.databricks.com/genie/
- https://docs.databricks.com/large-language-models/ai-functions
Pipelines, ML & GenAI
- https://docs.databricks.com/ldp/
- https://docs.databricks.com/jobs/
- https://docs.databricks.com/mlflow/
- https://docs.databricks.com/machine-learning/feature-store/
- https://docs.databricks.com/machine-learning/model-serving/
- https://docs.databricks.com/ai-gateway/
- https://docs.databricks.com/ai-search/ai-search/
- https://docs.databricks.com/agents/
Engineering, governance & certification
- https://docs.databricks.com/dev-tools/bundles/
- https://docs.databricks.com/admin/system-tables/
- https://docs.databricks.com/data-governance/unity-catalog/access-control
- https://www.databricks.com/learn/certification
- https://docs.databricks.com/resources/glossary
Docs index (source of truth for further lookup)
Renames move fast: verify against docs.databricks.com/llms.txt. This file teaches current names with the former name in parentheses. Product names, APIs, version numbers, and limits in this file were checked against the official documentation at the time of writing (Aug 2026) and should be re-verified against the live pages above before you publish or build on any specific value.
© 2026 Zorost Intelligence LLC · zorost.com