AI 原生开放教材

外观

AI Engineering Lab（24 周自学课程）系统课程·英文原文 ↗原件 ↓仓库 ↗

Week 19: Google Vertex AI & Gemini ​

Part of AI Engineering Lab · Week 19 of 24 · Section: Cloud AI Platforms · Category: Google 🎯 Use case: A multimodal bill-of-lading pipeline, Gemini OCR → structured JSON → BigQuery, plus the support agent rebuilt with ADK.

The problem ​

Last week the support agent learned to live behind Microsoft's governance machinery. This week the same agent moves to Google, and the problem inverts. Google gives you two doors to the same Gemini models: a free, key-in-your-browser sandbox (AI Studio) and a governed enterprise platform (Vertex AI). One is instant gratification; the other is production. The mistake most people make is treating them as interchangeable, and the cost of that mistake is real, because the free tier may use your data to improve Google's products by default.

But Google also hands ZoroLogistics a genuinely different job. Freight runs on paper: bills of lading, customs declarations, commercial invoices. Reading those documents is the bottleneck, and Gemini's native multimodal input, an image and a schema in one call, structured JSON back, turns "scan and key in the BoL" from a computer-vision project into a single API call. That is Google's honest edge over the other two clouds for document work, and it is why Week 19's use case is OCR, not another chat agent.

So the week has two threads. First, prove the pipeline: render synthetic bills of lading, extract ten fields with Gemini vision, and measure per-field accuracy and cost per document against ground truth. Second, rebuild the support agent code-first with ADK and evaluate it on the golden set, then compare that to the Foundry version, because that comparison is half the deliverable. Before the week, "Gemini OCR" is a demo you saw on Twitter; after, it is a measured pipeline with a per-document price and a one-line verdict on free tier vs Vertex. A pipeline without a per-field number is just a screenshot of a JSON blob.

Objectives ​

  • [ ] By Friday you can call Gemini from both Google AI Studio (API key) and Vertex AI (project + ADC) with the unified google-genai SDK.
  • [ ] By Friday you can build a multimodal BoL pipeline: render synthetic BoL images locally, extract fields to structured JSON with Gemini vision, and measure per-field accuracy against ground truth.
  • [ ] By Friday you can rebuild the support agent code-first with ADK and run a golden-set evaluation.
  • [ ] By Friday you can estimate cost per document and explain why the free AI Studio tier is off-limits for sensitive data.

Day-by-day plan ​

DayStudyRun / buildShipTime
MonAI Studio vs Vertex; the free-tier data ruleGet an API key; check quotas; gcloud auth application-default loginWorking Gemini call2 to 3 h
TueGemini multimodal + structured outputRender BoL PNGs; extract to JSON; measure per-field accuracyAccuracy table2 to 3 h
WedADK (code-first) vs Agent Builder vs Agent EngineRebuild the support agent with ADK; compare to the Foundry versionADK agent2 to 3 h
ThuVertex evaluation; BigQuery MLRun a golden-set eval; log results to BigQueryEval score + table2 to 3 h
FriPricing tiers & quotasEnd-to-end pipeline (images → clean table); publish cost per documentCost-per-document number3 to 4 h
SatReviewTake the quiz (8/10)Quiz score in Notes1 h

Concepts ​

Read the shared mental model in reference/knowledge-base/12-cloud-platforms.md and the runbook in reference/platforms/google-vertex/README.md. Google splits its generative-AI surface into two entry points over the same Gemini models, and that split is the week's first lesson.

Google AI Studio (aistudio.google.com) is the experiment sandbox: a Google account and an API key get a working Gemini call in minutes with zero infrastructure. Vertex AI (Google Cloud Console) is the enterprise platform, the same models, but with cloud-project billing, IAM, VPC Service Controls, quotas, and MLOps. The migration path is deliberate: you start in AI Studio and later point the same code at Vertex with different credentials (vertexai=True + project/location). The one rule to internalize early and never break: AI Studio's free tier may use your data to improve Google's products by default, so it is not for sensitive data. ZoroLogistics data is synthetic, so it is safe, but the segregation habit is the real deliverable.

Google AI StudioVertex AI
URLaistudio.google.comGoogle Cloud Console
AuthAPI key (Google account)GCP project + IAM / ADC
BillingFree tier + paid Gemini APICloud project billing
Governance~none (data may train Google by default)IAM, VPC-SC, CMEK, quotas, audit
Best forPrototyping, learningProductionizing

Gemini's nativity for multimodal input ​

The second concept is why the use case is OCR. Gemini accepts an image and text in one call and can return structured JSON against a response schema, which collapses "read a scanned bill of lading" into a single API call with a typed output. The model ladder is a cost/quality dial, not a single choice:

Model rungRoleTrade-off
gemini-2.5-flash-liteCheapest; high-volume/simple extractionLowest cost, lowest reasoning
gemini-2.5-flashWorkhorse; the lab defaultBest cost/latency balance
gemini-2.5-proFrontier reasoning; hard layoutsHighest cost, best quality
"Thinking" variantsInternal reasoning with a budgetMore tokens, deeper reasoning

The selection habit from Week 8 transfers directly: pick the cheapest model that clears your accuracy bar, and prove it with a number. Model Garden is Vertex's catalog layer around Gemini, it surfaces Imagen (images), Veo (video), Chirp (speech), and open models (Llama, Mistral, Gemma) when Gemini isn't the right tool.

The agent stack in three layers ​

Google gives you three ways to build the agent, from low-code to code-first:

LayerStyleWhen
Agent BuilderNo-code/low-code, grounded RAGFast prototypes, non-engineers
Agent EngineManaged runtime hosting agents as endpointsProduction deployment
ADKCode-first, open-source frameworkEngineering control; the lab default

ADK (Agent Development Kit) is the code-first, open-source framework, Agent(model, name, instruction, tools) with FunctionTool-wrapped functions, LangGraph-style orchestration, and multi-agent support. You rebuild the Week 14 to 16 support agent with ADK, deploy it to Agent Engine, and compare against the Week 18 Foundry version. ADK and Agent Engine both support MCP, so the Week 16 server ports across.

BigQuery ML: AI without leaving the warehouse ​

The distinctive Google move is BigQuery ML, run ML inside the warehouse with SQL, including remote calls to Gemini for generation and embedding. For ZoroLogistics this means: load the OCR output into BigQuery, then classify rows with ML.GENERATE_TEXT without leaving SQL. A worked example for the stretch exercise, classifying each extracted shipment's freight-terms risk:

sql
SELECT
  ml_generate_text_result['candidates'][0]['content']['parts'][0]['text'] AS risk_label
FROM ML.GENERATE_TEXT(
  MODEL `zorost.gemini_model`,
  (SELECT CONCAT('Classify the freight risk of this shipment: ', commodity, ' from ',
                 port_of_loading, ' to ', port_of_discharge) AS prompt
   FROM `zorost.bol_extractions`)
);

This is why data/analytics teams often lean GCP: the pipeline never leaves the warehouse. The end-to-end Week 19 pipeline, drawn as a flow:

mermaid
%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart LR
    B["Synthetic BoL text<br/>zoro.data.bol_samples()"] --> R["render_bol_png<br/>Pillow"]
    R --> IMG["BoL PNG image"]
    IMG --> G["Gemini vision<br/>gemini-2.5-flash"]
    G -->|"response_mime_type: json"| J["Structured JSON<br/>10 fields"]
    J --> V["field_matches<br/>vs ground truth"]
    V --> A["Per-field accuracy"]
    J --> BQ["BigQuery"]
    BQ --> ML["ML.GENERATE_TEXT<br/>classification"]
    A --> C["Cost per document"]

The left half is the OCR pipeline; the right half is the optional in-warehouse step. Both feed a measured number, per-field accuracy and cost per document, that lands in the Week 20 matrix.

Cost per document, and the free-tier boundary ​

Vertex and the paid Gemini API bill per token and per image; AI Studio's free tier is a daily request quota (historically tens to ~1,000 requests/day depending on model and tier), not a dollar budget. Worked example, one BoL document on gemini-2.5-flash: one image plus ~150 input tokens and ~120 output tokens. With placeholder prices of $0.000315/image, $0.30 per 1M input, and $2.50 per 1M output:

  • Image: $0.000315
  • Input: 150 × $0.30 / 1,000,000 = $0.000045
  • Output: 120 × $2.50 / 1,000,000 = $0.00030
  • Total ≈ $0.00066 per document.

That is why the free tier feels free at lab scale, but the binding constraint is the request quota, and the real cost is governance, not dollars: the same 100 documents a production day sends to Gemini would violate the data rule on the free tier even though the invoice is near zero. Watch for 429 errors (a quota problem, not a code bug), check quotas before the batch, and set a billing budget before any Vertex run.

How it breaks ​

Gemini pipelines break in predictable ways. Sending sensitive data through AI Studio: the free tier may train on it by default, the segregation habit is the deliverable, not a footnote. Hitting 429s mid-batch: that is a quota ceiling, raised per-region/per-model in the console, not something you fix in code. Region mismatch: Gemini availability is per-region; a model that 404s is usually disabled in that region (us-central1 is the safe default). Auth confusion: the same google-genai client changes behavior on the vertexai=True flag, an ADC error means you are in Vertex mode without gcloud auth application-default login. Unparsed JSON from the model: Gemini returns markdown fences or a trailing comma; strip fences and validate before scoring, exactly as the notebook does. Finally, measuring "it worked" instead of per-field accuracy: a correct looking JSON blob can still have the wrong consignee, the per-field number is the only honest metric.

Notebook walkthrough ​

Two notebooks; the local parts (image rendering, accuracy, cost, the dry-run agent loop) run without any key, and only the Gemini/Vertex call needs credentials.

notebooks/01-gemini-multimodal-ocr.ipynb bootstraps the repo, then render_bol_png draws each synthetic BoL's text onto a white PNG with Pillow (trying DejaVuSansMono.ttf, falling back to a default font) into a bol_images/ directory, this makes Gemini exercise vision, not read text from the prompt. Six samples load via data.bol_samples(n=6, seed=5). EXTRACT_PROMPT_OCR asks for exactly the ten JSON keys, and gemini_extract calls genai.Client(api_key=...) with config={"response_mime_type": "application/json"} and the image as inline_data. The loop runs over img_paths[:4], strips markdown fences, and scores with field_matches (the same ten-field, type-aware comparator as Week 18). The final cells print OVERALL_FIELD_ACCURACY and COST_PER_DOCUMENT_USD from placeholder prices ($0.000315/image, $0.30/M in, $2.50/M out). Correct output: accuracy in [0, 1], cost a sub-cent figure.

notebooks/02-vertex-agent-and-eval.ipynb loads 5,000 shipments and the four policy docs, defines track_shipment and check_refund_policy (refunds over $500 → "Requires human approval"), and wraps them as FunctionTools in an ADK Agent(model="gemini-2.5-flash", ...). The vertex_client() helper shows the enterprise path: genai.Client(vertexai=True, project=..., location="us-central1"). The golden set (Q1 to Q4) is scored by the local loop (which does real tool use: a tracking question calls track_shipment), printing EVAL_SCORE. The optional BigQuery cell logs the eval rows via bigquery.Client().load_table_from_dataframe when GOOGLE_CLOUD_PROJECT and BIGQUERY_EVAL_TABLE are set, and the final cell re-prints EVAL_SCORE. In dry-run the score is 1.000 (all four golden questions pass); the compare-with-Foundry note is the bridge to the Week 20 matrix.

The use case (Friday) ​

Deliverable: the end-to-end multimodal pipeline, BoL images → Gemini structured JSON → (optionally) BigQuery, with (a) per-field accuracy vs your Week 6 golden set, (b) a cost per document, and (c) the support agent rebuilt with ADK and evaluated on the golden set.

Zorost gate: a stranger can re-run your notebook (same seed, same rendered images) and reproduce your per-field accuracy and cost-per-document numbers, and you can show them what Gemini got wrong, the specific fields and documents that failed extraction, plus a one-line verdict on whether the free tier or Vertex is right for the job.

Stretch: load the extracted JSON into BigQuery and run one ML.GENERATE_TEXT classification over the rows (e.g. a freight-terms consistency or commodity-risk label), documenting the schema and one query. Fast learners can also re-run extraction on a second model rung (flash-lite vs flash) and add a two-row accuracy-vs-cost table.

Common pitfalls ​

PitfallSymptomFix
Sensitive data through AI StudioData may train Google by defaultPrototype on synthetic data; final pipeline on Vertex
429 mid-batchQuota, not a code bugCheck/raise per-region quotas before the batch
Region mismatchModel 404sConfirm the model is enabled in us-central1 (or your region)
vertexai=True auth confusionADC errorsgcloud auth application-default login first
Unparsed JSONjson.loads throws on fences/commasStrip fences; validate before scoring
Scoring "it worked"No per-field signalMeasure per-field accuracy against ground truth
Free-tier quota exhaustionBatch dies halfwayKeep the cloud run to 4 images until cost/quota confirmed
No billing budgetSurprise Vertex invoiceSet a budget + alert before any Vertex batch

Glossary ​

  • Google AI Studio: the free, key-based Gemini sandbox (aistudio.google.com).
  • Vertex AI: Google's governed enterprise platform for the same Gemini models.
  • ADC (application default credentials): the local credential chain (gcloud auth application-default login) for Vertex code.
  • google-genai: the unified SDK that targets AI Studio (api_key) or Vertex (vertexai=True).
  • Multimodal: accepting image + text in one call; Gemini's edge for document OCR.
  • Response schema: requesting structured JSON output constrained to a typed schema.
  • Model Garden: Vertex's catalog of first-party, open, and partner models.
  • ADK (Agent Development Kit): Google's code-first, open-source agent framework.
  • Agent Engine: the managed runtime that hosts ADK/Agent Builder agents as endpoints.
  • BigQuery ML: running ML (including ML.GENERATE_TEXT to Gemini) inside BigQuery with SQL.
  • Golden set: the Week 11 question/ground-truth pairs reused to score the agent.
  • Per-field accuracy: the fraction of the ten BoL fields extracted correctly, measured field-by-field.

Self-check (quiz) ​

Take quiz.md, 10 questions tied to these concepts and the notebook code. The passing bar is 8/10; record the score in your Notes.

Exercises ​

Four graded exercises (easy / standard / stretch / portfolio) in exercises.md, from recording per-field accuracy and cost, to a two-model rung comparison, to a BigQuery ML.GENERATE_TEXT query, to the Google column of the three-cloud matrix. Hints are in the same file.

Sources ​