AI 原生开放教材

外观

AI Engineering Lab（24 周自学课程）系统课程·英文原文 ↗原件 ↓仓库 ↗

14: AI/BI Dashboards & Databricks Apps ​

Part of AI Engineering Lab · Developed by Zorost Intelligence AI Lab · zorost.com

Two ways to put a surface on your lakehouse: AI/BI dashboards (low-code, AI-assisted analytics) and Databricks Apps (full custom-code web apps). Between them sits Genie One (the no-code business-user assistant). This file is the "how do I show this to a human" layer.

Week 24 · Production & Capstone. You have gold tables and a served model; this file turns them into a dashboard, a Genie space, or an app.

1. AI/BI dashboards (current: formerly Lakeview) ​

AI/BI is Databricks' compound-AI BI product: dashboards + Genie + Unity Catalog metric views. An AI/BI dashboard is a low-code, AI-assisted canvas over one or more datasets (SQL queries):

  • Datasets are the query layer, each is a SQL query saved as a named dataset, reused across visuals.
  • AI-assisted authoring: describe the chart you want; the assistant writes the SQL.
  • Cross-filtering: selecting a bar filters every other visual.
  • Custom calculations, scheduling, subscriptions, embedding, Git/source-control.
  • Built on metric views when you want governed, shared KPI definitions.

The surface decision, at a glance:

mermaid
%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart TD
    Q{What is the question's shape?}
    Q -->|recurring KPI| DASH[AI/BI dashboard\nlow-code, scheduled]
    Q -->|ad-hoc question| GEN[Genie One / Genie Agent\nchat → SQL]
    Q -->|bespoke tool / action| APP[Databricks App\ncustom code]
    DASH --> MV[metric views\ngoverned KPI definitions]
    GEN --> MV
    APP --> SRV[serving endpoint\n+ SQL warehouse]

1.1 Datasets, metric views, and cross-filtering ​

Three concepts make a dashboard governed rather than just pretty:

  • Dataset: a saved SQL query with a schema; multiple visuals can reference one dataset, so a metric change happens in one place.
  • Metric view: a Unity Catalog securable that names a KPI once (e.g. "on-time %" = that exact formula), shared across dashboards and Genie, so two teams cannot silently define the same KPI differently.
  • Cross-filtering: dashboard-level interactivity: click a carrier, and every visual re-filters to that carrier without a second query authored by hand.

Legacy "DBSQL dashboards" are archived, you cannot create new ones. Migrate with "Clone a legacy dashboard to an AI/BI dashboard."

Dashboard files are .lvdash.json (Lakeview format) and deploy via DABs as a resource with file_path, warehouse_id, and (CLI 0.281.0+) dataset_catalog/dataset_schema so the SQL is not hardcoded to one catalog, the pattern in 15-dabs-ci-cd.md.

1.2 Building an AI/BI dashboard (the workflow) ​

The order matters: test the SQL first, then render. A dashboard is only as good as the queries behind it.

  1. Know the data: list warehouses (databricks warehouses list), discover schema, and probe the aggregations you plan to show. Confirm the story is not flat, empty, or uninteresting.
  2. Test every query via CLI before deploying: a dashboard query that fails at render time is a broken dashboard. Use a SQL warehouse to run each dataset's SQL and check the result shape.
  3. Plan the layout: map each visual to a dataset; decide filters and confirm every dataset you want filtered contains the filter column.
  4. Write the .lvdash.json: datasets + pages, bare table names only (no catalog/schema prefix), widget fieldNames matching dataset columns exactly.
  5. Deploy + publish with databricks lakeview create … publish.
StepCLINotes
List warehousesdatabricks warehouses listgrab the warehouse id
Probe datadatabricks experimental aitools tools query --warehouse WH "SELECT …"validate the story
Createdatabricks lakeview create --display-name "X" --warehouse-id WH --dataset-catalog CAT --dataset-schema SCH --serialized-dashboard "$(cat d.json)"returns a dashboard id
List / inspectdatabricks lakeview list / get ID
Updatedatabricks lakeview update ID --serialized-dashboard "$(cat d.json)"update only changes the draft
Publishdatabricks lakeview publish ID --warehouse-id WHviewers see the published snapshot
Trashdatabricks lakeview trash ID

--dataset-catalog / --dataset-schema are flag-only (not JSON fields) and required they fill in the catalog/schema that your dashboard queries intentionally omit. Queries inside the JSON must use bare table names (FROM gold_on_time_kpis, never FROM zrl_.zorologistics.gold_on_time_kpis) or the flags are ignored and the dashboard is not portable across environments.

A minimal dashboard JSON (one counter + one bar chart over the gold KPIs):

json
{
  "datasets": [
    {
      "name": "ds_kpis",
      "displayName": "On-Time KPIs",
      "queryLines": [
        "SELECT carrier_id, month, total_shipments, on_time_shipments, ",
        "       on_time_pct, avg_delay_hours, late_48h_shipments ",
        "FROM gold_on_time_kpis "
      ]
    }
  ],
  "pages": [
    {
      "name": "main",
      "displayName": "Main",
      "pageType": "PAGE_TYPE_CANVAS",
      "layoutVersion": "GRID_V1",
      "layout": [
        {"widget": {"widgetType": "text", "spec": {"text": "# On-Time Performance"}},
         "position": {"x": 0, "y": 0, "width": 12, "height": 1}},
        {"widget": {"widgetType": "counter",
                    "dataset": "ds_kpis",
                    "query": {"fields": [{"name": "sum(total_shipments)", "expression": "SUM(`total_shipments`)"}]},
                    "encodings": {"value": {"fieldName": "sum(total_shipments)", "displayName": "Total Shipments"}}},
         "position": {"x": 0, "y": 1, "width": 4, "height": 3}},
        {"widget": {"widgetType": "bar",
                    "dataset": "ds_kpis",
                    "query": {"fields": [{"name": "carrier_id", "expression": "`carrier_id`"},
                                         {"name": "avg(avg_delay_hours)", "expression": "AVG(`avg_delay_hours`)"}]},
                    "encodings": {"x": {"fieldName": "carrier_id", "displayName": "Carrier"},
                                  "y": {"fieldName": "avg(avg_delay_hours)", "displayName": "Avg Delay (h)"}}},
         "position": {"x": 4, "y": 1, "width": 8, "height": 5}}
      ]
    }
  ]
}

The rule that breaks dashboards more than any other: the field name in query.fields must exactly match the fieldName in encodings ("sum(total_shipments)" both places). A mismatch renders "no selected fields to visualize."

1.3 Dashboard governance & lifecycle ​

  • Permissions follow the underlying data, a viewer needs SELECT on the source tables; the dashboard cannot show what the caller cannot read.
  • Source control: export the .lvdash.json into Git; promote dev → prod via the bundle, not by hand-editing in the UI.
  • Scheduling & subscriptions: a dashboard can be emailed on a schedule; alerts (system.query.history-backed) fire when a threshold trips.
  • Embedding: embed a live dashboard in an external app while keeping Databricks auth.

ZoroLogistics dashboard: an "On-Time Performance" dashboard over gold_on_time_kpis, on-time % by carrier, avg delay by lane, and a weather-severity breakdown, cross-filtered. This is the artifact the capstone ships (see capstone/).

2. Genie One ​

Genie One (formerly Databricks One) is the simplified business-user UI for asking questions in plain English. Where Genie Agents are analyst-configured per-domain (see 12-ai-functions-genie.md), Genie One is the whole-workspace assistant: one unified chat that answers across the tables the user can read, with generated SQL shown for trust.

When Genie One: you want non-technical users asking "what was on-time % by carrier last month" without an analyst building a dashboard first. It composes with dashboards, the dashboard answers the recurring questions; Genie answers the ad-hoc ones.

3. Databricks Apps ​

Databricks Apps hosts custom web apps on serverless compute, deployed as code. Two language tracks:

TrackStackUse when
AppKit (React/Node)databricks-apps AppKit, prebuilt data/AI components (charts, tables, Genie/chat)Data screens + Genie/chat assistants with a polished UI, fast
Python backendFastAPI (default), Flask, Dash, Streamlit, GradioPython-only teams, quick internal tools

AppKit is the default for a new app: it scaffolds a Node/TypeScript/React project with first-class components for the common data-app screen genres (KPI cards, tables, charts) and a built-in Genie/chat widget, plus a serving plugin to wire a Model Serving endpoint directly into the app. Reach for the Python backend track only when the team is Python-only or the app is a quick internal tool (Streamlit is the fastest "show a human something" path).

Both tracks give you: built-in auth (app access + single sign-on), resources (declare a SQL warehouse or serving endpoint the app can call), and deploy via DABs/CLI with CI/CD.

3.1 Auth ​

Apps authenticate the user automatically and, via the SDK, can act on the user's behalf or as a service principal. Data access stays governed: the app calls the SQL warehouse / serving endpoint under the app's declared permissions, so a user without SELECT still sees nothing the warehouse would not return.

The key facts to remember:

  • SSO is built in: no auth code to write; the app knows who the caller is.
  • Resource permissions are explicit: an app can only call a serving endpoint or warehouse it declares as a resource (with CAN_QUERY), so a compromised app cannot reach arbitrary endpoints.
  • Endpoints stay governed: the app's calls to a model still pass through the Unity AI Gateway rate limits/guardrails from 10-model-serving.md.

3.2 Resources ​

An app declares the Databricks resources it needs; DABs wires them:

yaml
# resources/my_app.app.yml
resources:
  apps:
    on_time_app:
      name: on-time-app-${bundle.target}
      description: 'ZoroLogistics on-time performance app'
      source_code_path: ../src/app
      resources:
        - name: eta_endpoint
          serving_endpoint:
            name: zrl-eta-model
            permission: CAN_QUERY

The app's Python backend then calls the endpoint through the SDK without managing a key.

3.3 Python backend (Streamlit example) ​

python
# src/app/app.py
import streamlit as st
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

st.title("ZoroLogistics ETA Lookup")
shipment_id = st.text_input("Shipment ID")

if shipment_id:
    resp = w.serving_endpoints.query(
        name="zrl-eta-model",
        dataframe_records=[{"shipment_id": shipment_id}],
    )
    st.metric("Predicted delay (hours)", round(resp.predictions[0], 2))

A slightly richer worked app, query the gold KPIs through the SQL warehouse and render a table, then call the served ETA model on demand:

python
# src/app/app.py (extended)
import streamlit as st
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

st.title("ZoroLogistics Ops Console")
st.caption("Gold on-time KPIs + live ETA lookup, all behind Databricks auth.")

@st.cache_data(ttl=300)
def on_time_kpis() -> list:
    resp = w.statement_execution.execute_statement(
        warehouse_id=st.secrets["WAREHOUSE_ID"],
        statement="SELECT carrier_id, month, on_time_pct, avg_delay_hours "
                  "FROM zrl_.zorologistics.gold_on_time_kpis ORDER BY month DESC LIMIT 20",
    )
    return [row.as_dict() for row in resp.result]

tab1, tab2 = st.tabs(["On-time KPIs", "ETA lookup"])
with tab1:
    st.dataframe(on_time_kpis())
with tab2:
    dist = st.number_input("Distance (km)", 120.0, 4200.0, 1800.0)
    weight = st.number_input("Weight (kg)", 5.0, 20000.0, 850.0)
    if st.button("Predict delay"):
        resp = w.serving_endpoints.query(
            name="zrl-eta-model",
            dataframe_records=[{"distance_km": dist, "weight_kg": weight,
                                "carrier_on_time_rate": 0.88, "weather_severity": "moderate"}],
        )
        st.metric("Predicted delay (hours)", round(resp.predictions[0], 2))

Note the two access patterns: the SQL warehouse for governed reads (KPI table) and the serving endpoint for real-time inference (ETA), both declared as app resources, neither using a hardcoded key.

3.4 Deploying ​

bash
databricks bundle validate --strict
databricks bundle deploy -t dev
databricks bundle run on_time_app -t dev     # deploy AND start (bare bundle deploy leaves apps stopped)
databricks apps logs on-time-app              # debug

Environment variables live in app.yaml inside the app source directory, not in databricks.yml (a common gotcha).

3.5 The apps dev loop ​

bash
databricks apps init --name on_time_app --features serving \
  --set "serving.serving-endpoint.name=zrl-eta-model" --run none   # AppKit scaffold
# …edit source…
databricks apps deploy on_time_app    # deploy + start
databricks apps logs on_time_app      # read stdout/stderr

AppKit's serving plugin wires the ETA endpoint in one flag; a Python app reaches the same endpoint through databricks-sdk. Either way, the endpoint remains a governed resource the app declares, never a hardcoded key.

3.6 AppKit vs Python backend (the 30-second rule) ​

  • AppKit (React/Node): you want a polished data screen with KPI cards, charts, and a Genie/chat widget, and the team is comfortable in TypeScript/React.
  • Python backend (Streamlit/FastAPI/Flask/Dash/Gradio): you want a quick internal tool and the team is Python-only; Streamlit is the fastest "show a human something" path.

Both deploy through the same DABs apps resource and get the same SSO + resource governance. The choice is a team/language and UI-polish decision, not a platform-capability one, an app that calls a serving endpoint and a SQL warehouse can be built either way.

4. Dashboard vs app vs Genie ​

AI/BI dashboardDatabricks AppGenie One
Code requiredNo (SQL + AI assist)Yes (React/Python)No
Custom interactivityCross-filter, paramsAnything you can codeNatural language only
AudienceAnalysts, opsOperators needing a custom toolBusiness users
Best forRecurring KPIs, scheduled viewsA product-like tool (ETA lookup, triage console)Ad-hoc questions
DeployDABs (.lvdash.json)DABs (apps resource)Workspace config

Rule of thumb: recurring metric → dashboard; a bespoke workflow or custom UX → app; a non-technical person with a question → Genie One. They coexist in one bundle, the capstone ships the on-time dashboard and an ETA app and a Genie space, all from one databricks.yml.

A worked decision (ZoroLogistics ops):

Ops needSurfaceWhy
"How did we do on-time last month?"AI/BI dashboardRecurring KPI, scheduled, cross-filterable
"What's the ETA for this specific shipment right now?"Databricks App (ETA lookup)Bespoke real-time tool calling the served model
"Which carriers keep breaching their lane SLA?"Genie OneUnpredictable, ad-hoc question over gold tables
"Show me the refund policy for a 3-day-late shipment."RAG endpoint / App chatRetrieval + generation, not a chart

The instinct to cultivate: match the surface to the shape of the question, a chart for a recurring number, a tool for a repeatable action, a chat for an unpredictable question.

5. Try it ​

Task 1: Deploy a two-widget dashboard. Write the §1.2 JSON, then databricks lakeview create … --dataset-catalog zrl_ --dataset-schema zorologistics, and publish. Acceptance check: databricks lakeview list shows the dashboard; the published URL renders the counter and the bar chart without a "no selected fields" error.

Task 2: Prove cross-filtering shares one dataset. Click a carrier bar in the bar chart. Acceptance check: the counter (same ds_kpis dataset) re-filters to that carrier, because both widgets read one dataset, not because you wrote a second query.

Task 3: Declare the ETA endpoint as an app resource. Add the §3.2 resources.apps.on_time_app block with serving_endpoint: zrl-eta-model, permission: CAN_QUERY, deploy, and call it from the app backend. Acceptance check: the app returns a prediction with no API key in the code; removing the resource declaration makes the call fail (the endpoint is not reachable undeclared).

6. Common mistakes ​

  1. Hardcoding catalog/schema in a dashboard query. FROM zrl_.zorologistics.gold_on_time_kpis breaks portability; the --dataset-catalog/--dataset-schema flags only fill in what you omit. Fix: use bare table names in queryLines.
  2. Mismatching field names. The name in query.fields must equal fieldName in encodings (both "sum(total_shipments)"). Fix: keep them byte-identical.
  3. update without publish. lakeview update only changes the draft; viewers keep the old snapshot. Fix: always publish after update.
  4. Re-create-ing to update. That mints a new dashboard id + URL and breaks shared links. Fix: update + publish on the same id.
  5. Putting env vars in databricks.yml. App environment variables belong in app.yaml inside the app source. Fix: keep app config in the app directory.
  6. Skipping SQL testing before deploying a dashboard. A query that fails at render time is a broken dashboard. Fix: run every dataset's SQL via CLI first and gate on the result.

Sources ​

Original AI Engineering Lab writing; dashboard JSON structure, AppKit, and app deployment details change frequently, verify against the live links above.

© 2026 Zorost Intelligence LLC · zorost.com