AI 原生开放教材

外观

AI Engineering Lab（24 周自学课程）系统课程·英文原文 ↗原件 ↓仓库 ↗

Databricks for AI Engineering: Zero to Hero Deep Dive ​

Purpose: This is the backbone reference for the AI Engineering Lab Databricks zero-to-hero module. It spans platform fundamentals → compute → data → SQL → Spark/pipelines → MLOps → GenAI → governance → cost → certification → how Zorost's Databricks Modernization Practice maps to it all.

Scope & provenance: Facts are synthesized in original wording from official documentation (docs.databricks.com) and databricks.com, verified against the live docs as of August 2026. Databricks has renamed many products recently; every rename is flagged inline and consolidated in the quick-reference below. Always teach the current name and note the legacy name.

Source of truth for further lookup: the machine-readable docs index at https://docs.databricks.com/llms.txt.

0. Naming quick-reference (current → formerly) ​

Databricks shipped a wave of renames through 2024 to 2026. The most important ones for a zero-to-hero learner:

Current nameFormerly known asNotes
Lakeflow PipelinesDelta Live Tables (DLT)Built on the open-source Apache Spark Declarative Pipelines (SDP); Python module dlt → pyspark.pipelines (import as dp)
Lakeflow JobsWorkflows / JobsSidebar entry is now "Jobs & Pipelines"
Lakeflow ConnectDatabricks Ingest / IngestionManaged + standard connectors for SaaS/databases/files/streams
Declarative Automation Bundles (DABs)Databricks Asset BundlesStill abbreviated "DABs"
OpenSharingDelta SharingOpen project now at opensharing.io
AI SearchVector SearchManaged vector retrieval
Unity AI GatewayAI GatewayGovernance control plane for LLM/agent/MCP traffic
Genie AgentsGenie SpacesNatural-language data Q&A configured by analysts
Genie OneDatabricks OneSimplified business-user UI
AI/BI dashboardsLakeview dashboardsLegacy "DBSQL dashboards" are archived
Data quality monitoringLakehouse MonitoringAnomaly detection + data profiling
SQL warehousesSQL endpointsRenamed 2023
Models in Unity CatalogMLflow Model RegistryRegistry "stages" → movable aliases (@prod, Champion/Challenger)
Git foldersReposBuilt-in Git client
Databricks AI assistive featuresDatabricksIQThe AI-engine brand was folded into the Genie family
Standard / Dedicated access modeShared / Single-user access modeCompute access modes
APPLY CHANGES (CDC)n/aSuperseded by AUTO CDC (same syntax)
Trigger.Oncen/aSuperseded by Trigger.AvailableNow

The "Mosaic AI" prefix has largely been retired across the docs (e.g., "Mosaic AI Model Serving" → "Model Serving", "Mosaic AI Model Training" → "Databricks Model Training"), though it still appears in some marketing material.

1. Platform fundamentals ​

1.1 The Data Intelligence Platform ​

Databricks is a unified, open analytics platform for building, deploying, and governing data, analytics, and AI. It runs inside your cloud account (integrating with your cloud storage and security) while Databricks manages the infrastructure on your behalf. Databricks layers AI on top of the lakehouse to understand data semantics and automatically optimize performance and infrastructure.

Two conceptual planes define the architecture:

PlaneWhat runs thereWhere it lives
Control planeBackend services, web UI, identity, job scheduling, workspace metadataDatabricks-managed (in the Databricks account)
Compute planeActual data processing (Spark executors, SQL engines)Two variants: classic (your AWS/Azure/GCP account) and serverless (Databricks-managed)
  • Classic compute runs in your virtual network → natural network isolation and control (customer-managed VPCs).
  • Serverless compute runs in a Databricks-managed serverless compute plane, created in the same region, inside a network boundary that isolates customer workspaces.
  • Workspaces have workspace storage (notebooks/queries/dashboards + internal "system data"). DBFS root / DBFS mounts are a deprecated legacy pattern, Databricks recommends Unity Catalog for all data access.

1.2 Lakehouse ​

The data lakehouse combines data-lake economics (cheap open-format storage of any data) with warehouse governance/performance, on standard cloud object storage. Three pillars plus Spark:

  • Apache Spark: massively scalable engine with decoupled compute and storage.
  • Delta Lake: optimized storage layer (ACID, schema enforcement/evolution, time travel).
  • Unity Catalog: unified fine-grained governance (access control, lineage, audit, discovery).

A common implementation pattern is the medallion architecture (bronze → silver → gold), covered in §3.

1.3 Accounts vs. workspaces ​

  • A workspace is a single Databricks deployment, where your team works with notebooks, jobs, clusters, SQL warehouses, and data.
  • A Databricks account is the top-level entity containing multiple workspaces. At account level you manage identity (users/groups/service principals, SSO/SCIM), workspace creation across regions, Unity Catalog metastores, and billing.
  • The account console (accounts.cloud.databricks.com) is where account admins configure account settings, subscription/billing, serverless quotas, and workspaces.

Two workspace deployment types:

TypeDescription
Serverless workspacePre-configured serverless compute + Databricks-managed default storage; can still attach your own cloud storage. Best for Genie One, AI/BI dashboards, Apps, serverless pipelines, most AI features.
Classic workspaceStorage + compute in your own cloud account. Needed for legacy Spark RDD code, Scala/R-first teams, direct on-prem/private connectivity. Serverless is still available inside classic workspaces.

Metastore ↔ region: an account can hold multiple Unity Catalog metastores; a single metastore can attach to multiple workspaces in the same region, giving them the same governed view.

1.4 Regions & clouds ​

Workspaces run on AWS, Azure, and Google Cloud, each with its own supported-region list. Not every feature is available in every region (see "Features with limited regional availability"). Some features ("Designated Services") use Databricks Geos, groups of regions giving predictable data-residency guarantees.

1.5 Unity Catalog core object model ​

Unity Catalog (UC) is the unified governance layer for data and AI, automatically enabled for all workspaces created after Nov 8, 2023, and also available as an open-source project. Objects use a three-level namespace:

metastore
 └── catalog
      └── schema (database)
           ├── table | view | volume | function | model
ObjectPurpose
MetastoreAccount-level metadata registry; one can attach to many workspaces in a region
CatalogTop-level container for isolating/organizing data
Schema (database)Holds tables, views, volumes, functions, models
TableGoverns tabular data (Delta by default)
ViewSaved query (incl. materialized views, dynamic views)
VolumeGoverns non-tabular data (files)
Function / ModelUDFs and MLflow-packaged models, also governed securables

Other UC securables sit directly under the metastore: storage credentials, external locations, connections, and shares (OpenSharing).

Managed vs. external vs. foreign tables:

ManagedExternalForeign
Data lifecycleUC managesYou manageExternal system manages
StorageUC-managedYou specifyExternal system
DROP TABLE deletes data?YesNo (metadata only)No
FormatsDelta, Apache IcebergDelta (rec.), CSV/JSON/Avro/Parquet/ORC/TextSystem-dependent
Best forProduction (default, recommended)Existing data / external clientsMigration / temporary federation

Legacy Hive metastore (and Hive tables, DBFS root) remains only for backward compatibility, migrate to UC.

1.6 OpenSharing (formerly Delta Sharing) ​

OpenSharing securely shares live data and AI assets outside your organization, whether or not recipients use Databricks. Core concepts: a share (read-only collection of tables, and for Databricks recipients, views/volumes/models/notebooks), a provider, and a recipient. Two protocols:

  1. Databricks-to-Databricks: between UC-enabled workspaces (no bearer token; full governance/audit).
  2. Databricks-to-Open: to any platform (Spark, pandas, Power BI…) via bearer tokens or OIDC federation.

Adjacent sharing features: Databricks Marketplace (data-product exchange) and Clean Rooms (privacy-preserving collaboration).

1.7 Personas ​

PersonaGoalTools
Data engineerReliable, governed ETL/ELTLakeflow Pipelines, Lakeflow Jobs, Auto Loader, Lakeflow Connect, Spark/Structured Streaming, DABs
Data analyst / BIQuery & visualize curated dataSQL warehouses, SQL editor, AI/BI dashboards, Genie, Catalog Explorer, alerts
Data scientist / ML engineerTrain/track/deploy modelsDBR ML, MLflow (experiments + Models in UC), Feature Store, Model Serving, AI Playground, AutoML
AdministratorGovern platform, identity, compute, costAccount console, workspace settings, Unity Catalog, compute policies, instance pools, budgets, system tables

Admin roles split into account admin (whole account), workspace admin (single workspace), and feature-specific roles (Metastore admin, Billing admin, Marketplace admin).

2. Compute ​

Three broad families: serverless (on-demand, auto-managed), classic (provisioned all-purpose/job clusters), and SQL warehouses (analytics-optimized).

2.1 All-purpose vs. job clusters ​

All-purpose (interactive)Job cluster
Created byUser (UI/CLI/API)Job scheduler, per run
LifecycleManual terminate/restart; sharedCreated when job runs, terminated when done; cannot restart
UseInteractive notebooks, ad-hoc analysisScheduled/automated production
Billing"Interactive" workload"Automated/job" workload, cheaper per DBU

2.2 Cluster config essentials ​

  • Databricks Runtime (DBR): Spark + Databricks optimizations. Variants: standard, DBR for ML (ML/DL libraries + GPU), and Long-Term Support (LTS) (recommended for production).
  • Driver + worker nodes: one driver, zero or more workers (one executor each). Pick instance types per node, or fleet types that auto-resolve. Single-node runs Spark locally on the driver.
  • Photon: Databricks' native C++ vectorized query engine (default on SQL warehouses and serverless; default-on classic DBR 9.1 LTS+). Billed at a different DBU rate.
  • Access modes: Standard (shared, recommended) vs. Dedicated (single user/group).

2.3 Cluster policies ​

A compute policy is a set of rules admins use to limit what users can configure: fix/hide settings, limit cluster count, cap cost (max DBU/hour), enforce library installs and tags for cost attribution. Policies inherit from policy families; admins can enforce compliance on existing clusters. Without the "unrestricted cluster creation" entitlement, users can only create compute via policies.

2.4 Autoscaling ​

  • Optimized autoscaling (Premium+): scales up in ≤2 events; down based on shuffle state + utilization windows.
  • Standard autoscaling (Standard plan): adds 8 nodes then grows exponentially; down after 10 min low activity.
  • Do not enable Spark Dynamic Allocation alongside Databricks autoscaling (conflicts cause churn/NODES_LOST).
  • Serverless scales automatically and transparently.

2.5 Serverless compute ​

Databricks-managed on-demand compute for notebooks, jobs, and Lakeflow Pipelines. No compute in your cloud account; "versionless" (always latest runtime, auto-upgraded); billed under serverless SKUs. Separate serverless infrastructure powers SQL warehouses, Model Serving, data quality monitoring, and predictive optimization. Serverless notebooks have a default 2.5 h execution timeout (overspend protection).

2.6 SQL warehouses ​

TypeComputePhotonPredictive I/OIWMStartup
ServerlessDatabricks✓✓✓ (AI autoscaling)~2 to 6 s
ProYour cloud✓✓n/a~4 min
ClassicYour cloud✓n/an/a~4 min
Lakehouse Real-Time (Beta)Serverless✓n/an/asub-second reads
  • Intelligent Workload Management (IWM) predicts resource needs and dynamically scales serverless warehouses.
  • Classic/pro scale ~1 cluster per 10 concurrent queries; serverless autoscales clusters of a chosen size.
  • Auto Stop default 10 min idle (min 5 via UI, 1 via API); idle still accrues DBUs.
  • Channels: Current (production) and Preview (test upcoming versions).

2.7 DBU pricing ​

A DBU (Databricks Unit) is a unit of processing capability per hour, based on the VM instance type. Billing is pay-as-you-go at per-second granularity; Committed Use Contracts give discounts. Rates vary by SKU (workload), cloud, and region; Photon instances bill differently.

SKU tierApplies to
Jobs ComputeAutomated/job workloads (also a "Jobs Light" variant)
All-Purpose ComputeInteractive/notebook workloads (higher rate)
SQLDatabricks SQL warehouses (CLASSIC/PRO; serverless billed separately)
ServerlessServerless jobs, notebooks, Lakeflow Pipelines

Other billed products (tracked in system.billing.usage by billing_origin_product): Lakeflow Pipelines (DLT CORE/PRO/ADVANCED), Model Serving, AI Search, Unity AI Gateway, Lakeflow Connect, AI Functions, Lakebase, etc.

2.8 Cost controls ​

  • Budgets & alerts (account-wide or filtered by team/project/workspace).
  • Cluster auto-termination (10 to 10,000 min inactivity) and SQL warehouse auto-stop.
  • Spot instances for latency-tolerant workers (driver always on-demand).
  • Instance pools: pre-warmed idle instances (pay cloud cost, no DBUs while idle) for faster start/scale.
  • Compute policies + tags for cost caps and chargeback.

3. Data & Delta Lake ​

3.1 Delta Lake fundamentals ​

Delta Lake is the default storage layer; unless configured otherwise, all tables on Databricks are Delta tables. It extends Parquet with a file-based transaction log providing ACID transactions and scalable metadata, fully compatible with Spark (incl. Structured Streaming, one copy of data for batch + streaming).

ACID: each SQL statement is its own atomic transaction; multi-statement transactions via BEGIN ATOMIC … END. Concurrency uses optimistic concurrency control; reads get snapshot isolation, writes get write-serializable isolation. PK/FK constraints are informational only (not enforced).

Schema enforcement & evolution: writes are validated against the schema (safe casts attempted); schemas can evolve via ALTER TABLE (add/reorder/rename/type-widening) or implicitly (mergeSchema / spark.databricks.delta.schema.autoMerge.enabled).

Time travel: every write creates a new table version. Query a prior state with VERSION AS OF / TIMESTAMP AS OF or @-syntax; RESTORE TABLE rolls back. Retention: logRetentionDuration (default 30 days), deletedFileRetentionDuration (default 7 days).

VACUUM: permanently removes data files older than the retention threshold (default 7 days). DRY RUN previews; VACUUM … LITE (DBR 16.4 LTS+) is a faster log-based variant. After VACUUM, time travel past the retention window is impossible. Soft-deletes (deletion vectors / dropped columns) are physically applied with REORG TABLE … APPLY (PURGE) first.

OPTIMIZE + Z-ORDER: OPTIMIZE compacts/bin-packs files; ZORDER BY (col) colocates related data for data-skipping on high-cardinality filter columns. Z-order and Hive partitioning are now superseded by liquid clustering.

Liquid clustering: CLUSTER BY (cols) automatically organizes data and lets you change keys without rewriting. GA for Delta on DBR 15.4 LTS+; CLUSTER BY AUTO (automatic) lets Databricks adapt keys from workload. Recommended for all new tables.

Deletion vectors: mark rows deleted in metadata instead of rewriting the whole Parquet file (accelerates DELETE/UPDATE/MERGE). Enable with delta.enableDeletionVectors = true; reads DBR 12.2 LTS+; all-optimized writes DBR 14.3 LTS+. Iceberg v3 tables include them by default.

Change data feed (CDC): tracks row-level changes between versions (table_changes() / readChangeFeed; metadata _change_type, _commit_version, _commit_timestamp). Two approaches: Automatic CDF (Public Preview, DBR 18 LTS+; computed at query time) and legacy CDF (materialized at write; delta.enableChangeDataFeed = true).

3.2 Medallion architecture ​

A progressive data-quality pattern (recommended best practice, not a requirement):

LayerPurposeConsumers
Bronze (raw)Ingest raw, unvalidated data; append-only; preserve source fidelity (store as STRING/VARIANT/BINARY)Data engineers, compliance/audit
Silver (validated)Clean, dedupe, normalize, enforce schema, handle nulls/late data, joinsData engineers, analysts, data scientists
Gold (enriched)Aggregated, dimensional-modeled, business-alignedBI, analysts, ML, executives

3.3 Lakehouse Federation ​

Governed, read-only access to external data through UC foreign catalogs, with automatic query pushdown and table-level access control. Two variants: query federation (JDBC pushdown to databases like MySQL/PostgreSQL/Oracle/Snowflake/BigQuery…) and catalog federation (query external catalogs, Hive, AWS Glue, Snowflake Horizon, directly in object storage). Setup for query federation: UC connection (credentials + URL) → foreign catalog → grant privileges → query.

3.4 Lakeflow Connect (managed ingestion) ​

Connectors to ingest from enterprise apps, databases, cloud storage, message buses, and files, powered by serverless Lakeflow Pipelines and governed by UC. Layered from custom → managed:

  • Managed connectors (fully managed): SaaS (Salesforce, Workday, ServiceNow, GA4, HubSpot, Confluence, NetSuite, Dynamics 365…), database CDC (SQL Server; PostgreSQL/MySQL CDC in preview; query-based Oracle/Teradata/Snowflake/Redshift/Synapse/BigQuery), file sources (SharePoint, Google Drive, SFTP…), streaming (Kafka, Kinesis, Pub/Sub, Pulsar).
  • Standard connectors and community/custom connectors for more control.

Rule of thumb: Lakeflow Connect copies data into governed Delta tables; Lakehouse Federation queries in place without copying.

3.5 Volumes, Auto Loader, COPY INTO ​

  • Volumes govern non-tabular data at catalog.schema.volume; access via /Volumes/<catalog>/<schema>/<volume>/<path>. Managed (UC-owned storage, 7-day delete retention) vs. external (existing object storage). Use for raw landing, staging, unstructured data, and library/checkpoint files.
  • Auto Loader (cloudFiles source) incrementally processes new files with exactly-once semantics (RocksDB progress store in the checkpoint), schema inference/evolution, and two detection modes: directory listing (default) vs. file notification (recommended).
  • COPY INTO idempotently/incrementally loads files into Delta via SQL. Use COPY INTO for thousands of files; Auto Loader for millions+ or evolving schemas.

4. Databricks SQL & AI/BI ​

4.1 Databricks SQL (DBSQL) ​

A cloud data warehouse on lakehouse architecture (ANSI SQL + Delta extensions): SQL warehouses (Serverless recommended / Pro / Classic), the SQL editor (Genie Code-assisted), saved queries (with "Run as viewer/owner" credential modes), query history (backed by system.query.history), and alerts (scheduled queries with conditions → email/Slack/webhooks; statuses OK/TRIGGERED/ERROR).

4.2 AI/BI dashboards & Genie ​

AI/BI is Databricks' compound-AI BI product: dashboards, Genie Agents, and Unity Catalog semantics (metric views).

  • AI/BI dashboards (formerly Lakeview dashboards): low-code AI-assisted dashboards with datasets, cross-filtering, custom calculations, scheduling/subscriptions, embedding, Git/source-control.
  • Legacy "DBSQL dashboards" are archived, no new ones can be created; migrate via "Clone a legacy dashboard to an AI/BI dashboard."
  • Genie family: Genie Agents (analyst-configured NL→SQL chat), Genie One (simplified business-user UI), Genie Code (developer AI assistant).

4.3 AI Functions (SQL) ​

Built-in SQL/PySpark functions that apply LLMs or Databricks research models to data in place, no endpoint/API key to manage. Requirements (Aug 2026): serverless compute required (not on Pro/Classic warehouses), Databricks Runtime 18.2+.

FunctionWhat it does
ai_queryGeneral-purpose: run a prompt against any supported Foundation Model API endpoint
ai_classifyClassify input text into labels you provide
ai_extractExtract structured fields from documents/text using a schema you define
ai_genFree-form generation from a prompt
ai_summarizeGenerate a concise summary (optional max_words)
ai_maskMask/redact specified entity types (PII)
ai_similaritySemantic similarity score between two strings (0.0 to 1.0)
ai_parse_documentParse unstructured documents (PDF/Office/images) into text, tables, figures, layout
ai_prep_search (Beta)Transform parsed output into search-ready chunks for AI Search/RAG
ai_fix_grammarCorrect grammatical errors
ai_analyze_sentimentSentiment (positive/negative/neutral/mixed)
ai_translateTranslate to a target language
ai_forecastTable-valued function forecasting time-series to a horizon (with bounds)

Newer Beta additions beyond the core list: ai_search (ranked, deduplicated retrieval + grounded answer), ai_top_drivers (rank dimensions driving a metric change), and vector_search (query an AI Search index).

5. PySpark & pipelines ​

5.1 Spark DataFrame API essentials ​

PySpark's primary abstraction is the DataFrame (named columns, immutable). Key mental model, lazy evaluation:

  • Transformations (select, filter, join, groupBy/agg, withColumn, orderBy, union) are lazy, they build a logical plan.
  • Actions (show/display, count, take, collect, saveAsTable) trigger execution.
  • In production, the write is usually the only action; extra actions interrupt optimization.

Reading (spark.read): .format(), .option(), .schema(), then .load(path) / .table(name). Writing (df.write): .mode("append"|"overwrite"|"error"|"ignore"), .saveAsTable() / .save(). Formats: Delta, Parquet, CSV, JSON, Avro, ORC, XML, text, Excel, streaming connectors.

Common transformations:

OpExample
select / filterdf.select("id").filter(col("amount") > 0)
withColumndf.withColumn("total", col("qty") * col("price"))
joindf.join(dim, "id", "inner") (inner/left/right/full/semi/anti)
groupBy/aggdf.groupBy("state").agg(sum("amount").alias("sales"))
distinctdf.dropDuplicates(["id"])
orderBydf.orderBy(col("amount").desc())

Window functions (pyspark.sql.Window with partitionBy()/orderBy()) compute across related rows without collapsing (e.g., row_number() for "latest per customer").

Caching: the automatic disk cache (Databricks-managed Parquet cache, no code needed) is distinct from manual Spark cache()/persist().

Broadcast joins: copy the small side to all executors. AQE auto-converts joins to broadcast; hint with df.join(F.broadcast(small_df), "key").

Partitioning: spark.sql.shuffle.partitions sets post-shuffle partitions (tune to 1 to 2× executor cores); repartition(n) reshuffles, coalesce(n) reduces without full shuffle; AQE coalesces automatically.

Spark SQL: df.createOrReplaceTempView("t") then spark.sql("SELECT …").

5.2 Structured Streaming ​

Near-real-time engine with end-to-end fault tolerance and exactly-once guarantees, using the same DataFrame API as batch.

  • Sources: Auto Loader (cloudFiles), Kafka, Delta change streams, Kinesis/Pub/Sub/Pulsar.
  • Sinks: Delta (toTable), Kafka, files, console/memory, foreachBatch (arbitrary per-micro-batch logic, e.g. MERGE upserts).
  • Output modes (stateful queries only): Append (default, emit only final rows), Update (rows that changed), Complete (full result). Delta sinks support append/complete, not update.
  • Triggers: unspecified (~3 to 5 s), processingTime, availableNow (one-shot incremental; replaces Trigger.Once), real-time mode (sub-second, Public Preview). Serverless supports AvailableNow/Once only.
  • Watermarks (withWatermark("ts", "10 minutes")) bound state and drop late data; windows: tumbling, sliding, session. Watermarks are mandatory for stream-stream outer joins; recommended for inner joins and dedup.
  • Checkpoints store offsets, commits, state, and metadata for exactly-once resume, each query needs its own checkpoint location (use a UC volume path).
  • Stateful ops: streaming aggregations, distinct/dropDuplicates, stream-stream joins, custom state (mapGroupsWithState, transformWithState). Databricks recommends the RocksDB state store with changelog checkpointing.

5.3 Lakeflow Pipelines (formerly Delta Live Tables) ​

Declarative batch + streaming pipelines in SQL or Python, with automatic orchestration (correct order, max parallelism, progressive retry).

Dataset types:

TypeSemanticsUse when
Streaming tableEach record processed once; incremental; append-only sourceStreaming ingestion
Materialized viewRecomputed to reflect current state ("always correct")Aggregations/joins; dimension changes
ViewEvaluated on demand, not persistedIntermediate checks

Streaming tables don't recompute on late dimension changes; materialized views do. All pipeline tables are Delta tables with ACID/time travel.

SQL (STREAM/read_files for streaming; CREATE OR REFRESH … MATERIALIZED VIEW for batch; CREATE PRIVATE … for pipeline-private views). Python uses from pyspark import pipelines as dp with @dp.table, @dp.materialized_view, @dp.temporary_view decorators (dataset functions must return a DataFrame and must not call actions like collect()/count()/toPandas()/save()).

Expectations (data quality): boolean constraints with three actions, warn (default; keep rows, record metrics), drop (ON VIOLATION DROP ROW), fail (ON VIOLATION FAIL UPDATE).

CDC: AUTO CDC (replaces APPLY CHANGES) computes SCD Type 1 (latest only) or Type 2 (versioned history with __START_AT/__END_AT) from a CDC feed.

Medallion in pipelines: bronze (Auto Loader → streaming table) → silver (expectations + joins) → gold (materialized views), with the DAG resolved automatically.

5.4 Lakeflow Jobs (formerly Workflows) ​

Workflow orchestration: a job schedules/coordinates tasks as a DAG.

Task types: Notebook, Python script, Python wheel, SQL (query/file/dashboard/alert), dbt, Pipeline (Lakeflow Pipeline), JAR, Run Job, If/else (conditional), For each (loop), and more.

DAG orchestration: depends_on relationships with Run if gating, ALL_SUCCESS (default), AT_LEAST_ONE_SUCCESS, NONE_FAILED, ALL_DONE, AT_LEAST_ONE_FAILED, ALL_FAILED.

Parameters: job-level parameters pushed to tasks via <code v-pre>{{job.parameters.<name>}}</code>; notebooks read dbutils.widgets.get().

Scheduling: triggers include Scheduled (cron), Periodic, File arrival, Table update, Continuous. Min 10 s between runs.

Resilience: per-task retries, repair (re-run failed tasks), per-task timeout.

Compute: job clusters (reused across tasks; recommended), existing/all-purpose clusters, or serverless (omit cluster config).

6. MLOps & machine learning ​

Databricks' ML stack: MLflow (tracking + registry + GenAI eval) → Models in Unity Catalog → Feature Engineering → Model Training → batch (Spark UDF) or real-time (Model Serving), all governed by UC.

6.1 MLflow ​

Experiments → runs → models. Two logging mechanisms: autologging (mlflow.<flavor>.autolog()) or the explicit logging API (log_param/log_metric/log_artifact/log_model). MLflow 3 adds GenAI observability/evaluation/prompt management, Logged Models, and Deployment Jobs; in MLflow 3 the default registry is databricks-uc (UC by default).

6.2 Models in Unity Catalog (formerly Model Registry) ​

  • Registered models use <catalog>.<schema>.<model>; require a model signature.
  • Stages are deprecated → use aliases (mutable named references): models:/<catalog>.<schema>.<model>@<alias>. Common convention: Champion/Challenger or @prod.
  • Lineage via mlflow.log_input(); sharing across workspaces in the same metastore; cross-account via OpenSharing.

6.3 Unity Catalog Feature Engineering (Feature Store) ​

A central registry for features with governance, lineage, point-in-time joins, and cross-workspace sharing: using features at train time eliminates training/serving skew.

  • Feature tables (GA): a Delta table in UC with a primary key; you own the pipeline.
  • Feature Views (Public Preview, recommended): declarative Feature objects; Databricks computes/materializes the pipeline.

Core API: FeatureEngineeringClient + FeatureLookup → create_training_set() → fe.log_model() → fe.score_batch() for batch inference with automatic feature lookup.

Point-in-time joins: a time-series feature table declares a timestamp key (timeseries_columns); timestamp_lookup_key drives an AS OF join at the label timestamp, preventing data leakage.

Online store (Lakebase): fe.create_online_store() provisions a Lakebase instance; fe.publish_table() syncs offline features. Publish modes: TRIGGERED (incremental), CONTINUOUS (streaming), SNAPSHOT (one-time).

Feature serving: FeatureSpec + FeatureFunction → create_feature_serving_endpoint() exposes features to external apps.

6.4 Databricks Model Training (formerly Mosaic AI Model Training) ​

  • Distributed training: TorchDistributor (PyTorch as Spark jobs), DeepSpeed (memory/pipeline optimization), Ray (parallel compute), Spark ML (pyspark.ml.connect, DBR 17.0+).
  • Hyperparameter tuning: Optuna (recommended single-node), Ray Tune (distributed), Hyperopt (deprecated, absent after DBR 16.4 LTS ML).
  • AutoML: classification/regression/forecasting; generates per-trial source notebooks + SHAP. Note: not built-in after DBR 18.0 ML+ (install databricks-automl-runtime).
  • Foundation Model Fine-tuning is deprecated (removal Aug 2026) → migrate to AI Runtime.

6.5 Classic ML on Spark ​

  • scikit-learn (single-node): train in pandas, tune with Optuna/Ray, score at scale via pandas UDF or mlflow.pyfunc.spark_udf.
  • XGBoost: single-node xgboost package; distributed via xgboost.spark (DBR 12.0 ML+).
  • Spark MLlib (pyspark.ml): distributed Pipelines API for classification/regression/clustering.

6.6 Deep learning ​

  • PyTorch / TensorFlow pre-installed in DBR ML; multi-node via TorchDistributor, DeepSpeed, or Ray; track with mlflow.pytorch.autolog().
  • GPU compute: GPU instances with GPU-aware scheduling + CUDA/cuDNN/NCCL.
  • AI Runtime (serverless GPU): no cluster config; 1×A10, 1×H100, or 8×H100 (@distributed decorator); DDP/FSDP/DeepSpeed; Ray support; recommended for LLM fine-tuning (LoRA/QLoRA/full), CV, recommenders.
  • Ray on Databricks: Ray 2.3.0+ on Spark clusters (Tune/Train/Data); serverless GPU via "Ray on AI Runtime".

7. GenAI & serving ​

Everything is a governed UC object, and every LLM call can route through one control plane (Unity AI Gateway).

7.1 Model Serving ​

Unified real-time/batch serving as a REST API on serverless, auto-scaling (>25k queries/sec, <50 ms overhead). Three model types:

TypeWhat it is
Custom modelsMLflow PyFunc (scikit-learn/XGBoost/PyTorch/HF); agents are custom models
Databricks-hosted foundation modelsFoundation Model APIs (e.g. Llama), pay-per-token or provisioned throughput
External modelsThird-party (OpenAI, Anthropic, Bedrock, Vertex AI, Cohere…) with centralized credentials

Foundation Model APIs are OpenAI-compatible. Pay-per-token (incl. "priority mode" for latency-sensitive traffic) for experimentation; provisioned throughput (dedicated capacity + performance guarantees) for production/fine-tuned models. Agents (LangChain/LangGraph, OpenAI Agents SDK) are served by wrapping them in MLflow's ResponsesAgent interface.

7.2 Unity AI Gateway (formerly AI Gateway) ​

Centralized governance for AI traffic, built on UC. Governs three dimensions:

DimensionWhat it governs
AssetModels, MCP servers, functions, connections as UC securables
TrafficCentral routing, rate limits, budgets, usage tracking, cost observability
BehaviorService policies (guardrails): ALLOW / DENY / ASK per request/response

It also governs external coding agents (Claude Code, Cursor, Codex, Gemini CLI) and external MCP servers.

Service policies (guardrails, Beta): SQL-UDF "content" policies evaluated ON CALL (request) and ON RESULT (response); built-in checks (block_unsafe_content, block_jailbreak, block_hallucination) are LLM-as-a-judge; evaluation is fail-closed.

7.3 AI Search (formerly Vector Search) ​

Managed vector search in UC. Create an index from a Delta table, query via REST/SDK.

  • Algorithms: ANN via HNSW (L2; cosine via normalized embeddings), keyword BM25, hybrid search (Reciprocal Rank Fusion).
  • Index types: Delta Sync (auto-syncs; managed or self-managed embeddings) vs. Direct Vector (manual upsert).
  • Endpoints: Standard (up to ~320M vectors) vs. Storage-optimized (1B+ vectors, cheaper).
  • RAG pattern: embed query → similarity search → retrieve docs → append to prompt → generate.

7.4 Agent Framework & Agent Bricks ​

  • Agent Framework (code-first): wrap any framework in ResponsesAgent (streaming, multi-agent, tracing).
  • Tools: managed MCP servers (Genie, AI Search, DBSQL, UC functions), external MCP servers, custom MCP servers (Apps), UC function tools, structured/unstructured retrieval.
  • Agent Bricks: Knowledge Assistant (document Q&A with citations, "Instructed Retriever") and Supervisor Agent / Supervisor API (multi-agent orchestration with built-in access controls).
  • Evaluation: MLflow 3 mlflow.genai.evaluate() with built-in/custom judges, the Review App (human feedback), and production monitoring.
  • Deploy: Model Serving (agent serving, deploy()), or Databricks Apps, now the recommended path for new use cases.

7.5 Genie, Databricks Apps, AI/BI ​

  • Genie Agents: analyst-configured NL→SQL chat (datasets, example SQL, instructions, business semantics); Genie One, business-user UI; Genie Code, developer assistant.
  • Databricks Apps: custom data/GenAI apps on serverless (Python: Streamlit/Dash/Gradio/FastAPI; Node.js: React/…), deployed via DABs/CLI with CI/CD.
  • AI/BI dashboards: AI-assisted dashboards + Genie for self-service analytics.

8. Engineering & governance ​

8.1 Declarative Automation Bundles (DABs) + databricks.yml ​

Infrastructure-as-code for data/AI projects: source files + resource definitions (jobs, pipelines, dashboards, serving endpoints, MLflow models) + tests, deployed as a unit. databricks.yml declares the bundle name, targets (dev/staging/prod), variables, and resources (or include:).

Lifecycle: databricks bundle init | validate | deploy -t <target> | run <resource> | destroy.

8.2 Databricks CLI & Python SDK ​

  • CLI wraps the REST API. Auth: OAuth U2M (interactive user), OAuth M2M (service principal client/secret), OIDC/token federation (CI/CD), PATs (legacy). Profiles in .databrickscfg switch workspaces/credentials. Command groups: workspace, compute, jobs, pipelines, ml, serving, catalogs/schemas/tables/grants, bundle, apps, etc.
  • Python SDK (databricks-sdk): typed WorkspaceClient/AccountClient; same unified auth.

8.3 Git folders (Repos) & workspace files ​

Git folders = built-in Git client (GitHub/GitLab/Bitbucket/Azure DevOps). Workspace files = arbitrary files in the workspace (.py, .sql, .yml, dashboards, etc.). Source-control options from lightest → fullest: Git-with-jobs → Git folders → Declarative Automation Bundles (recommended).

8.4 Unity Catalog security ​

  • GRANT/REVOKE privilege model (additive only, no DENY): need traversal (USE CATALOG/USE SCHEMA) + action (SELECT, MODIFY, READ VOLUME, EXECUTE) privileges; privileges inherit down the hierarchy. One owner per object (prefer groups); MANAGE delegates grant admin; BROWSE = metadata-only.
  • Fine-grained: row filters (BOOLEAN UDF, SET ROW FILTER), column masks (ALTER COLUMN … SET MASK), dynamic views (use current_user()/is_account_group_member() in the view's WHERE/CASE).
  • ABAC + governed tags: attribute-driven access from centralized policies (row-filter, column-mask, GRANT policies).
  • Data lineage: automatic table/column lineage in Catalog Explorer and system.access.table_lineage/column_lineage.
  • Service policies for AI securables (Beta): guardrails via Unity AI Gateway (see §7.2).

8.5 System tables ​

Read-only operational data in the system catalog (free; query compute billed). Key schemas: system.access.audit (audit events), system.access.table_lineage/column_lineage, system.billing.usage/list_prices, system.compute.*, system.lakeflow.jobs/pipelines, system.query.history, system.serving.*, system.ai_gateway.*, system.information_schema.*. Most retain 365 days.

8.6 Monitoring & data quality ​

  • Query history (UI + system.query.history), alerts, AI/BI dashboards on system tables.
  • Lakeflow Pipelines expectations = in-pipeline data quality (warn/drop/fail).
  • Data quality monitoring (formerly Lakehouse Monitoring): anomaly detection (freshness/completeness) + data profiling (drift detection) on serverless.

8.7 CI/CD ​

Reference flow: Version (Git) → Build → Deploy (bundles via GitHub Actions/Azure DevOps) → Test → Run → Monitor. Environment promotion = bundle targets (dev → staging → prod). Workload identity federation (OIDC) exchanges CI platform tokens for Databricks OAuth (no stored secrets).

9. Cost & operations ​

9.1 DBU & serverless vs. classic ​

  • DBU = normalized unit of compute/hour. Cost ≈ (DBU rate × workload SKU) × instances × duration.
  • Classic compute: predictable VMs, spot/pools/VPC/policies, but you pay for idle time and manage patching/scaling.
  • Serverless: near-instant start, auto-scaling, no idle cost, minimal ops; premium DBU rate but often lower TCO by eliminating idle/over-provisioning. Limitations: no custom data sources (Federation only), no cluster/spot policies, customer-managed VPC/keys don't apply.

9.2 Instance pools & spot ​

Instance pools = pre-warmed idle instances (no DBUs while idle) to cut start/scale time; Databricks now recommends serverless instead of pools where supported. Spot instances cut cost for latency-tolerant workers (driver always on-demand; use fleet types).

9.3 Performance tuning ​

  • Photon: vectorized query engine; default on SQL warehouses/serverless; accelerates SQL/DataFrame/ETL/stateless streaming. No UDFs, no RDD/Dataset, no stateful streaming.
  • Liquid clustering: replaces partitioning + Z-order (CLUSTER BY, CLUSTER BY AUTO).
  • Other: disk caching, dynamic file pruning, low-shuffle merge, AQE, cost-based optimizer (fresh stats), predictive I/O. Prefer built-ins over UDFs; tune spark.sql.shuffle.partitions.

9.4 Cost monitoring ​

  • system.billing.usage: granular DBU records (sku_name, usage_quantity, usage_metadata, custom_tags, billing_origin_product); join system.billing.list_prices for cost.
  • Budgets (account-wide/filtered) with alerts; tags for chargeback; Governance Hub (cost page); usage dashboards (import official AI/BI cost dashboards).

10. Certifications & learning paths ​

10.1 Certifications (current) ​

Role-based certifications (Associate/Professional), plus accreditations and a Spark-specific exam:

CertificationAssesses
Data Analyst AssociateData analysis with Databricks SQL (queries, visualizations, dashboards)
Data Engineer AssociateIntroductory data engineering on the Data + AI Platform
Data Engineer ProfessionalAdvanced data engineering tasks
Machine Learning AssociateBasic ML tasks on Databricks
Machine Learning ProfessionalAdvanced ML in production
Generative AI Engineer AssociateDesign, build, deploy GenAI solutions with Databricks
Context Engineer AssociateDesign, assemble, govern context for AI agent systems
Associate Developer for Apache SparkThe Spark DataFrame API

Plus accreditations (free badges): Databricks Fundamentals (Lakehouse) and Generative AI Fundamentals. The Hadoop Migration Architect exam was retired Aug 1, 2024.

10.2 Academy & learning paths ​

Official training is on the Databricks Academy (customer-academy.databricks.com). Role-based learning pathways exist for Data Analyst, Data Engineer, ML Engineer, Generative AI Engineer, Context Engineer, and a Platform Administrator specialty pathway. Each pathway culminates in the corresponding certification exam.

10.3 Zero-to-hero 3 to 4 week sequence ​

A practical sequencing for a motivated engineer (mirrors Zorost's "zero to hero" framing):

WeekFocusKey artifacts/skills
Day 0SetupFree trial/workspace, Unity Catalog metastore, CLI + SDK auth (OAuth U2M), a Git repo, a DAB scaffold
Week 1Data engineeringDelta Lake, medallion (bronze→silver→gold), Auto Loader, Lakeflow Pipelines (expectations, AUTO CDC), Lakeflow Jobs orchestration
Week 2DBSQL & analyticsSQL warehouses, SQL editor, AI/BI dashboards, Genie Agents, AI Functions, alerts
Week 3MLMLflow experiments, Models in UC (aliases), Feature Engineering (point-in-time joins), AutoML / scikit-learn / XGBoost, batch inference
Week 4GenAI & productionModel Serving (foundation + custom/agents), AI Search + RAG, Agent Framework + evaluation, Unity AI Gateway guardrails, CI/CD with DABs, cost monitoring

11. Zorost relevance: mapping the Modernization Practice to this curriculum ​

Zorost Intelligence is a Trusted Databricks Partner with a Databricks Modernization Practice. Each practice offering maps directly onto this curriculum:

Zorost offeringCurriculum sectionWhat it maps to
Legacy BI Migration§4 (DBSQL/AI/BI), §3.3 (Federation)Migrate legacy warehouses/BI → Databricks SQL + AI/BI dashboards + Genie
ETL Conversion → Lakeflow/Spark/DLT§5 (Spark, Lakeflow Pipelines)Re-platform ETL to Spark DataFrames, Structured Streaming, Lakeflow Pipelines
Dimensional Modeling on Delta Lake§3 (Delta, medallion), §3.2Star/snowflake schemas + gold-layer modeling on governed Delta tables
Streaming Pipelines§5.2 (Structured Streaming), §3.4 (Connect)Auto Loader/Kafka → streaming tables, watermarks, checkpoints, CDC
Unity Catalog Governance§1.5, §8.4 to 8.5Metastores, catalogs/schemas/tables/volumes, GRANT/REVOKE, row filters/masks, lineage, audit
Feature Engineering & MLOps§6 (MLflow, Feature Store, Model Training)Feature tables/views, point-in-time joins, model lifecycle + aliases
Mosaic AI Vector Search / Serving / Gateway§7.1 to 7.3(Now) AI Search + Model Serving + Unity AI Gateway
Agentic Workflows on Databricks§7.4 to 7.5Agent Framework, Agent Bricks (Knowledge Assistant/Supervisor), Apps
Power BI / Tableau Modernization§4, §1.6 (OpenSharing), §3.3BI tools over SQL warehouses / OpenSharing / Federation
Cost Optimization & FinOps§9DBU/SKU analysis, system.billing dashboards, budgets, serverless migration, pools/spot

This document is the teaching backbone that lets a Zorost engineer move from a legacy stack to a governed, serverless lakehouse AI platform in roughly four weeks, the same arc the Modernization Practice runs for clients.

Sources ​

Platform & compute

Data & Delta Lake

SQL & AI/BI

PySpark & pipelines

MLOps & ML

GenAI & serving

Engineering, governance & cost

Certification & learning

Docs index