17: FinOps & Cost

Part of AI Engineering Lab · Developed by Zorost Intelligence AI Lab · zorost.com

FinOps on Databricks is a measurement discipline: know what a DBU is, read the billing system tables, and hunt the leaks. This file is the cost-control pass that turns a "serverless-first" platform decision into a defensible number, the last habit before the capstone ships.

Week 24 · Production & Capstone. Pairs with 02-compute.md (the SKUs) and 16-governance-security.md (who can see the bill).

The FinOps loop this file teaches:

mermaid

%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart LR
    TAG[tag workloads\ncompute policies] --> USE[usage lands in\nsystem.billing.usage]
    USE --> JOIN[join list_prices\nDBU → dollars]
    JOIN --> DASH[cost dashboard\n7-day moving avg]
    DASH --> ALERT[budget alert\nfires early]
    ALERT --> FIX[fix a leak\nidle / oversized / naive]
    FIX --> TAG

1. The DBU model

A DBU (Databricks Unit) is a unit of processing capability per hour, based on the VM instance type. Cost ≈ (DBU rate × workload SKU) × instances × duration, at per-second granularity.

SKU tier	Applies to	Relative cost
Jobs Compute	Automated/job workloads	Lowest
All-Purpose Compute	Interactive notebooks	Higher than jobs
SQL	SQL warehouses (Classic/Pro)	Per-warehouse rate
Serverless	Serverless jobs/notebooks/pipelines	Premium rate, no idle

Other billed products are tracked by billing_origin_product: Lakeflow Pipelines (DLT CORE/PRO/ADVANCED), Model Serving, AI Search, Unity AI Gateway, Lakeflow Connect, AI Functions, Lakebase, and more.

1.1 DBU cost math, worked

The formula is cost = usage_quantity (DBUs) × list_price. Worked examples (rates are illustrative: real DBU rates vary by cloud, region, and SKU; confirm against system.billing.list_prices):

Workload	DBUs/hour	Hours/day	DBUs/day	Rate (illustrative)	$/day
All-purpose cluster (2 × m5d.2xlarge), left on 24 h	2.4	24	57.6	$0.55	$31.68
Same cluster, auto-terminated after a 2 h job	2.4	2	4.8	$0.55	$2.64
Serverless job (same 2 h of work)	2.4	2	4.8	$0.70	$3.36
Serverless SQL warehouse, 30 min of queries	1.0	0.5	0.5	$0.70	$0.35
Pay-per-token serving (10k tokens)	n/a	n/a	n/a	per-1M-token	usage-only

The first row is the whole story of Databricks FinOps: the same job costs ~12× more because an all-purpose cluster idles 22 h/day. Auto-termination (row 2) or serverless (row 3) removes the idle; serverless charges a premium per DBU but only for the hours you actually use.

1.2 The one-line cost model

Every cost question reduces to three numbers pulled from the billing tables:

cost = SUM(usage_quantity × price)  over [sku, date, tags]

usage_quantity → system.billing.usage
price → system.billing.list_prices (join on sku_name + cloud)
tags → custom_tags (attribution) or usage_metadata.cluster_id (per-cluster)

1.3 Worked cost math: SQL warehouse + serving

Two more worked examples (rates illustrative, confirm against list_prices):

SQL warehouse. A Serverless warehouse runs ~0.5 h of queries a day at ~1 DBU/h. Cost ≈ 0.5 DBU × $0.70 ≈ $0.35/day. The same workload on a Pro warehouse that sits up 24 h at a lower rate but accrues DBUs while idle is usually more expensive, idle is the leak, not the rate.

Model Serving (scale-to-zero). A scale-to-zero endpoint bills $0 while idle; a warm replica floor (min_provisioned_throughput > 0) bills continuously. If ops queries the ETA endpoint ~2,000×/day in a 2-hour window, scale-to-zero costs ~2 h of compute + cold starts; a 24/7 warm floor costs 24 h. For spiky daytime traffic, scale-to-zero wins; for a latency SLA on a hot path, the warm floor is the price of the guarantee.

The pattern to internalize: every cost question is usage_quantity × price, summed, and the biggest lever is almost always eliminating hours you pay for but don't use (idle compute), not shaving the per-DBU rate.

2. Serverless vs classic: the tradeoff

	Classic	Serverless
Start	~minutes (cluster spin-up)	~seconds
Idle	You pay for idle VMs	No idle cost (scale-to-zero)
Scaling	You configure autoscale	Automatic
Ops	You patch/upgrade	Always latest, managed
DBU rate	Lower per-DBU	Premium per-DBU
Limits	Your VPC, spot/pools, policies	Federation-only, no custom data sources

The FinOps takeaway: serverless charges more per DBU but often wins on TCO because you eliminate idle clusters, over-provisioning, and the ops tax. The classic trap is comparing DBU rates instead of total monthly spend. Migrate predictable, spiky, or infrequently-run workloads to serverless; keep classic only where you need your own VPC or spot economics.

A worked comparison (the medallion pipeline): an all-purpose cluster idles ~22 h/day between nightly runs → you pay 24 h of interactive DBUs for 2 h of work. On serverless you pay for ~2 h of serverless DBUs plus a cold-start, at a higher rate but with no idle. The serverless bill is usually lower even at a premium rate, and the real saving is that nobody manages autoscaling, patches, or teardown.

The TCO comparison, in one table:

Line	Classic (all-purpose)	Serverless
DBUs (2 h of work)	4.8	4.8
DBUs (idle)	52.8	0
Total DBUs/day	57.6	~4.8
Rate	$0.55	$0.70
Compute $/day	$31.68	~$3.36
Ops (patching, autoscale tuning)	someone's hours	$0

3. Instance pools, spot, and warehouse scaling

Instance pools: pre-warmed idle instances (you pay cloud cost, no DBUs while idle) for faster cluster start/scale. Databricks now recommends serverless instead of pools where supported.
Spot instances: cut cost for latency-tolerant workers (driver stays on-demand); use fleet types for resilient spot.
SQL warehouse scaling: serverless warehouses autoscale (Intelligent Workload Management); classic/pro scale ~1 cluster per 10 concurrent queries. Auto Stop defaults to 10 min idle (idle still accrues DBUs), tune it down.

4. Billing system tables

system.billing.usage holds granular DBU records; join system.billing.list_prices for cost.

Column	Meaning
`usage_date`	Date of usage (partition, always filter)
`sku_name`	Product SKU
`usage_quantity`	DBUs (or other units) consumed
`usage_metadata`	MAP, cluster id/name, tags, etc.
`billing_origin_product`	Product (pipeline, serving, search…)
`custom_tags`	Your cost-attribution tags

sql

-- Daily DBU by SKU, last 30 days
SELECT usage_date, sku_name, SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY usage_date, sku_name
ORDER BY usage_date DESC, total_dbus DESC;

-- Compute vs SQL vs Serverless
SELECT
  CASE
    WHEN sku_name LIKE '%ALL_PURPOSE%' THEN 'All-Purpose'
    WHEN sku_name LIKE '%JOBS%'         THEN 'Jobs'
    WHEN sku_name LIKE '%SQL%'          THEN 'SQL Warehouse'
    WHEN sku_name LIKE '%SERVERLESS%'   THEN 'Serverless'
    ELSE 'Other'
  END AS compute_type,
  SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY 1
ORDER BY total_dbus DESC;

-- Estimated cost by SKU (join list prices)
SELECT u.sku_name,
       SUM(u.usage_quantity) AS total_dbus,
       SUM(u.usage_quantity * p.pricing.default) AS estimated_cost
FROM system.billing.usage u
LEFT JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name AND u.cloud = p.cloud
WHERE u.usage_date >= current_date() - 30
  AND p.price_end_time IS NULL
GROUP BY u.sku_name
ORDER BY estimated_cost DESC;

-- Chargeback by tag (requires tags enforced via compute policy)
SELECT usage_metadata.cluster_id,
       MAX(usage_metadata.cluster_name) AS cluster_name,
       SUM(usage_quantity)               AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
  AND usage_metadata.cluster_id IS NOT NULL
GROUP BY usage_metadata.cluster_id
ORDER BY total_dbus DESC
LIMIT 20;

-- Daily trend with 7-day moving average (catch a spike early)
SELECT usage_date,
       SUM(usage_quantity) AS daily_dbus,
       AVG(SUM(usage_quantity)) OVER (
         ORDER BY usage_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
       ) AS moving_avg_7d
FROM system.billing.usage
WHERE usage_date >= current_date() - 60
GROUP BY usage_date
ORDER BY usage_date;

Grant access explicitly: GRANT USE CATALOG ON CATALOG system, then USE SCHEMA + SELECT on system.billing.

4.1 More billing queries (the operational set)

sql

-- Spend by workspace (multi-workspace account view)
SELECT workspace_id, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY workspace_id
ORDER BY dbus DESC;

-- Month-over-month delta (catch a creep before it compounds)
SELECT date_trunc('MONTH', usage_date) AS month,
       SUM(usage_quantity) AS dbus,
       LAG(SUM(usage_quantity)) OVER (ORDER BY date_trunc('MONTH', usage_date)) AS prev_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 90
GROUP BY 1
ORDER BY 1;

-- Spend by product (pipeline vs serving vs search vs compute)
SELECT COALESCE(billing_origin_product, 'compute') AS product,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY 1
ORDER BY dbus DESC;

-- All-purpose DBUs that could be job/serverless (the migration candidate)
SELECT SUM(usage_quantity) AS all_purpose_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
  AND sku_name LIKE '%ALL_PURPOSE%';

5. Cost dashboard, tagging, budgets & alerts

Cost dashboard: build an AI/BI dashboard on system.billing.usage (daily DBU, top SKUs, per-cluster, 7-day moving average). Import the official usage dashboards to start faster.
Tagging: enforce tags via compute policies; custom_tags then powers chargeback ("this team's pipelines cost X"). The bundle already tags the ETL job; extend tags to every cluster and warehouse.
Budgets & alerts: set account-wide or filtered (team/project/workspace) budgets with alerts. Do this on day one, a $5 budget alert beats a surprise invoice.
Governance Hub: the account-level cost page for trends and anomalies.

A cost dashboard spec (what to build)

One dataset on system.billing.usage + one on list_prices, then:

Widget	Dataset query (shape)	Why
Total spend (KPI)	`SUM(usage_quantity × price)` last 30d	the headline number
7-day moving average (line)	daily DBU + `AVG() OVER (...)`	catches spikes same-day
Top SKUs (table)	`GROUP BY sku_name`	where the money goes
Compute type split (bar)	the `CASE` buckets	classic vs serverless
Per-cluster (table)	`GROUP BY usage_metadata.cluster_id`	find the idle culprit
Per-team (table)	`GROUP BY custom_tags.team`	chargeback

5.1 Budgets & alerts, concretely

A budget is account-wide or filtered (by team/project/workspace/tag). Set it to fire early:

Create a budget scoped to the project = zorologistics-capstone tag.
Set the threshold at ~120% of the expected spend, not your pain point. The alert is an early-warning tripwire, not a kill switch.
Route the alert to the team's Slack/email; name one person the responder.
Review it at the weekly FinOps cadence and tune the threshold down as you learn real spend.

The discipline: a $5 budget alert on day one beats a surprise invoice at month-end. The budget doesn't stop spend, it makes spend visible the day it happens, which is what turns FinOps from a monthly post-mortem into a daily habit.

6. Top-10 cost leaks (and the fixes)

#	Leak	Symptom	Fix
1	Idle clusters	All-purpose cluster runs 24/7	Auto-termination (10 min) or serverless
2	Over-provisioned warehouses	Warehouse at 2× needed size	Downsize; rely on autoscale + auto-stop
3	Naive streaming	Streaming pipeline runs continuously at full tilt	`availableNow`/`Trigger.AvailableNow` for one-shot; rightsize workers
4	Pay-per-token in prod	Token bill grows linearly with traffic	Provisioned throughput once latency/volume is known
5	Every AI function re-invoked per query	`ai_classify` in a view scanned on each read	Materialize once into Delta
6	No tags	Can't attribute spend	Enforce tags via compute policy
7	Untracked Model Serving / AI Search	Endpoints left idle or over-provisioned	`scale_to_zero_enabled`; storage-optimized endpoints
8	Idle SQL warehouse	Warehouse accrues DBUs while idle	Tune Auto Stop down to the minimum (5 min via UI, 1 via API)
9	All-purpose for scheduled jobs	Interactive rate on an automated workload	Use a job cluster (cheaper per DBU) or serverless
10	Ignored moving average	A spike is found at month-end, not day-of	Watch the 7-day moving average daily

6.1 Three leaks, deep-dived

The idle cluster (leak #1). An all-purpose cluster left on between runs is the #1 line item in almost every new workspace. The fix is mechanical, auto-termination at 10 min, and the payoff is immediate: 24 h of interactive DBUs becomes 2 h.

The live-view AI function (leak #5). ai_classify in a view re-bills tokens on every scan, and scans happen more than you think (dashboards, joins, exports). Materialize once; the Delta result costs cents to read.

Pay-per-token at scale (leak #4). Pay-per-token is usage-only and perfect for experiments, but it scales linearly with traffic. Once you know the volume and have a latency SLA, provisioned throughput buys a ceiling and a guarantee, the break-even is a number you should compute, not assume.

6.2 The serverless-migration checklist

The highest-leverage FinOps move is migrating the right workloads to serverless. Work down this list:

Find the idle: the per-cluster query (§4) names the all-purpose clusters running 24/7.
Move scheduled jobs to job clusters or serverless: interactive rate → job/serverless rate, and no idle.
Rightsize the SQL warehouse: serverless autoscale beats a Pro warehouse sized for peak.
Enable scale-to-zero on every low-traffic endpoint: $0 while idle.
Keep classic only where you must: your VPC, spot economics, a custom data source.

Measure each move as a before/after usage_quantity × price delta, and record it, the capstone's G10 gate is exactly this, done once and documented.

7.1 A worked FinOps note (copy the shape)

The one-paragraph note the capstone gate wants, as a fill-in-the-blank:

Change: moved the medallion pipeline from an all-purpose cluster (24/7) to a serverless job. Before: ~57.6 DBU/day (2 h of work + 22 h idle). After: ~4.8 DBU/day + cold start. Delta: −52.8 DBU/day ≈ −$29/day ≈ −$870/month (illustrative rates). Verification:system.billing.usage, billing_origin_product = 'pipelines', last 30 days.

Every cost claim in this module should reduce to that shape: the change, the number, the dollar delta, and the query that proved it.

Rates are illustrative everywhere in this file. Before you publish a dollar figure, pull the live rate from system.billing.list_prices for your cloud/region, the shape of the math never changes, only the constants do.

7. ZoroLogistics: cost exercise

A cost-tracking query (saved as the capstone's FinOps view):

sql

SELECT
  usage_date,
  CASE
    WHEN billing_origin_product IS NOT NULL THEN billing_origin_product
    ELSE 'Compute'
  END AS product,
  SUM(usage_quantity) AS dbus,
  COALESCE(SUM(usage_quantity * p.pricing.default), 0) AS est_cost_usd
FROM system.billing.usage u
LEFT JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name AND u.cloud = p.cloud AND p.price_end_time IS NULL
WHERE usage_date >= current_date() - 30
GROUP BY usage_date, 2
ORDER BY usage_date DESC, dbus DESC;

One concrete cost win (do it, don't just read it):

Find the pipeline's Jobs vs All-Purpose split in the query above.
Confirm the medallion pipeline runs on serverless (not an all-purpose cluster) and that the ETA serving endpoint has scale_to_zero_enabled: true.
Measure before/after in system.billing.usage and write a one-paragraph FinOps note: the change, the DBU delta, and the dollar delta.

Acceptance: the cost dashboard is live, tags are on the bundle resources, a budget alert fires at a threshold, and the one cost-win exercise is documented with a number, the same "every artifact ships with a metric" habit applied to spend.

A FinOps cadence (make it somebody's job)

Cadence	Do this
Daily	Glance at the 7-day moving average; a spike is caught same-day, not at month-end
Weekly	Top-5 SKUs and top-5 clusters; confirm each is still needed
Monthly	Chargeback by tag; compare against the budget; close one cost-win (idle cluster, oversized warehouse, naive stream)
Quarterly	Re-evaluate classic vs serverless; re-check provisioned throughput vs pay-per-token

8. Try it

Task 1: Reproduce the cost math. Run the "Estimated cost by SKU" query (§4) and pick the top SKU. Acceptance check: estimated_cost ≈ total_dbus × pricing.default for that SKU, you can state the exact dollar figure and the DBUs behind it, not just "spend is up."

Task 2: Find the idle cluster. Run the per-cluster chargeback query and identify the cluster with the most DBUs that has no recent job runs. Acceptance check: you name one cluster, its daily DBU, and the auto-termination (or serverless migration) that removes it, a specific fix, not a general "optimize compute."

Task 3: Set a budget alert and trip it. Create a filtered budget on the zorologistics-capstone tag with a low threshold, then run the pipeline. Acceptance check: the alert fires (or is scheduled to) at the threshold, and the moving average in your cost dashboard shows the pipeline's contribution the same day.

9. Common mistakes

Comparing DBU rates instead of total spend. The premium serverless rate looks worse on paper but wins on TCO once idle and ops are counted. Fix: always reduce to usage_quantity × price, summed, never per-DBU rate alone.
No budget alert on day one. The most expensive mistake is the one you find at month-end. Fix: a small, filtered budget alert before the first pipeline run.
Re-invoking AI functions per query. An ai_classify in a live view bills tokens on every scan. Fix: materialize once into a Delta table.
A streaming pipeline at full tilt for batch data. Fix: use availableNow/scheduled triggers, and rightsize the workers.
Scale-to-zero disabled on low-traffic endpoints. You pay for idle replicas you never use. Fix: scale_to_zero_enabled: true as the default.
Tags as an afterthought. Untagged spend is unattributable spend. Fix: enforce tags via compute policy from the start, and join custom_tags in every chargeback query.

Sources

Billing system tables: https://docs.databricks.com/admin/system-tables/billing
Usage & cost monitoring: https://docs.databricks.com/admin/usage
Instance pools: https://docs.databricks.com/compute/pool-index
SQL warehouses: https://docs.databricks.com/compute/sql-warehouse/
Pricing: https://www.databricks.com/product/pricing

Original AI Engineering Lab writing; SKU names, DBU rates, and system-table columns change, verify against the live links above.

17: FinOps & Cost ​

1. The DBU model ​

1.1 DBU cost math, worked ​

1.2 The one-line cost model ​

1.3 Worked cost math: SQL warehouse + serving ​

2. Serverless vs classic: the tradeoff ​

3. Instance pools, spot, and warehouse scaling ​

4. Billing system tables ​

4.1 More billing queries (the operational set) ​

5. Cost dashboard, tagging, budgets & alerts ​

A cost dashboard spec (what to build) ​

5.1 Budgets & alerts, concretely ​

6. Top-10 cost leaks (and the fixes) ​

6.1 Three leaks, deep-dived ​

6.2 The serverless-migration checklist ​

7.1 A worked FinOps note (copy the shape) ​

7. ZoroLogistics: cost exercise ​

A FinOps cadence (make it somebody's job) ​

8. Try it ​

9. Common mistakes ​

Sources ​