AI Engineering from Scratch（英文原版）
Given a voice-first product spec (target TTFAB, mic sample rate, vision in yes/no, bilingual, full-duplex) and a compute constraint (GPU class, budget), size the Thinker-Talker pipeline.
Produce:
- Model family pick. Moshi (best latency), Qwen2.5-Omni (best open features), Qwen3-Omni (frontier quality), Mini-Omni (simplest).
- Thinker and Talker sizes. 7B Thinker + 200-300M Talker for <400ms TTFAB. 70B+ Thinker for quality, accept higher TTFAB.
- TTFAB breakdown. Component-by-component latency estimate.
- Duplex mode. Half-duplex with VAD turn-taking as default; full-duplex if product requires backchannel.
- Vision integration. TMRoPE with absolute timestamps for interleaved video frames.
- Deployment shape. Single-GPU vs split (Thinker on A, Talker on B) based on throughput needs.
Hard rejects:
- Proposing 70B Talker. Talker must be small to keep up with speech token rate.
- Using non-streaming speech decoder. TTFAB explodes.
- Claiming full-duplex is plug-and-play. It requires specialized training data.
Refusal rules:
- If target TTFAB <200ms, refuse anything larger than Moshi-class (7B fused) on a single A100.
- If product requires music generation in-stream, refuse this architecture and recommend a separate music pipeline.
- If mic sample rate is 48kHz with strict quality, flag the need for stronger speech encoder; don't downsample blindly.
Output: one-page streaming plan with model pick, sizes, TTFAB breakdown, duplex mode, vision strategy, deployment. End with arXiv 2503.20215 (Qwen2.5-Omni), 2410.00037 (Moshi).