T

Tail Latency

• Category: Reliability & operations
• What it actually means: The latency experienced by the slowest portion of requests, commonly summarized with a high percentile under a stated workload and time window.
• Why it matters: Averages can look healthy while a meaningful group of users waits much longer because of queueing, contention, retries, or variable request cost.
• In practice: Report several percentiles by route and workload, retain timeouts as censored or failed observations according to a documented rule, and trace slow requests across dependencies.
• Common confusion: Tail latency is not the single slowest request and has no meaning without the percentile, population, and measurement boundary.
• Learn it: Inference Metrics and Goodput
• Related terms: Time to First Token (TTFT), Time per Output Token (TPOT), Saturation, Goodput
• Sources: The Tail at Scale

Temperature

• Category: Models & inference
• What people say: A creativity setting.
• What it actually means: A decoding parameter that rescales logits before a probability distribution is formed. Higher positive values usually flatten the distribution; lower positive values sharpen it.
• Why it matters: Temperature changes sampling behavior, not the model's knowledge or factuality.
• Common confusion: A zero setting is often implemented as greedy decoding, but exact behavior and determinism depend on the provider, sampler, seed support, and serving system.
• Related terms: Softmax, Autoregressive, Token

Tensor

• Category: Data & representations
• What people say: A multidimensional array used for numerical computation.
• What it actually means: A typed array with a shape, data type, and device placement that frameworks use to represent inputs, parameters, activations, and gradients. Automatic-differentiation metadata is framework- and operation-dependent, not an inherent property of every tensor.
• Related terms: Autograd, Parameter, Mixed Precision

Tensor Parallelism

• Category: Infrastructure & serving
• What it actually means: Partitioning tensor operations within a model layer across devices, with collective communication combining partial results during the layer computation.
• Why it matters: It lets one layer use memory and compute from several devices, but frequent communication can dominate when the interconnect or partition is unsuitable.
• In practice: Match partition dimensions to model shapes, benchmark collective traffic, keep ranks on a fast interconnect, and record the sharding layout with checkpoints and serving configuration.
• Common confusion: Tensor parallelism splits work inside layers. Pipeline parallelism places different layer groups on different devices.
• Learn it: Scaling and Distributed Training
• Related terms: Tensor, Pipeline Parallelism, Expert Parallelism, Parameter
• Sources: Megatron-LM

Termination Condition

• Category: Agents & tools
• What it actually means: An explicit rule that ends or pauses an agent run when it succeeds, fails, exhausts a budget, reaches a safe boundary, or requires escalation.
• Why it matters: Without a termination condition, an agent can loop, repeat side effects, waste budget, or claim completion without satisfying the goal.
• In practice: Define success evidence, maximum steps and cost, non-retryable errors, and escalation states before starting the loop.
• Common confusion: A stop sequence ends text generation; a termination condition decides whether the task or workflow should stop.
• Related terms: Agent Harness, Token Budget, Verification Gate, Stop Sequence
• Sources: AutoGen

Test Oracle

• Category: AI-native development
• What it actually means: The mechanism, specification, reference, invariant, or human judgment used to decide whether observed program behavior is correct.
• Why it matters: Generating test inputs is not enough; automated verification requires an independent basis for classifying each result.
• In practice: Prefer executable invariants, reference implementations, schemas, and deterministic expected outputs, then document where human judgment remains necessary.
• Common confusion: The model that wrote the code should not be treated as an independent oracle merely because you ask it whether its own output is correct.
• Related terms: Regression Test, Verification Gate, Eval Set, Human-in-the-Loop (HITL)
• Sources: The Oracle Problem in Software Testing

Threat Model

• Category: Security & governance
• What it actually means: A documented account of protected assets, trust boundaries, potential adversaries, assumed capabilities, attack paths, impacts, and planned controls.
• Why it matters: Security controls cannot be judged without stating what they defend, against whom, and under which assumptions.
• In practice: Map data and authority across model, retrieval, tools, users, and external services, then turn credible abuse paths into red-team cases and mitigations.
• Common confusion: A threat model prioritizes plausible risks; it is not a checklist that proves the system secure or predicts every future attack.
• Related terms: Least Privilege, Prompt Injection, Sandbox, Red Teaming
• Sources: NIST SP 800-154; NIST Generative AI Profile

Time per Output Token (TPOT)

• Category: Infrastructure & serving
• What it actually means: For one request with N > 1 output tokens, the average post-first-token interval: (t_N - t_1) / (N - 1). System distributions then aggregate those per-request averages.
• Why it matters: Users can receive the first token quickly while the rest of the answer streams slowly, so startup latency alone does not describe generation responsiveness.
• In practice: Compute TPOT separately for each request, report percentiles across requests by output length and concurrency, and avoid pooling all token intervals or comparing systems with different tokenizers and measurement boundaries.
• Common confusion: TPOT is a per-request average. An individual inter-token latency is one gap between consecutive tokens, while time to first token includes the wait before output starts.
• Learn it: Inference Metrics and Goodput
• Related terms: Decode Phase, Time to First Token (TTFT), Streaming, Goodput
• Sources: DistServe

Time to First Token (TTFT)

• Category: Models & inference
• Aliases: TTFT
• What it actually means: The elapsed time from submitting a generation request until the client receives the first output token or content event under a defined measurement boundary.
• Why it matters: TTFT strongly affects perceived responsiveness and can reveal queueing, prompt processing, cache, or network delays.
• In practice: Record client-side TTFT by model, prompt length, region, and cache status, then separate it from total completion time.
• Common confusion: TTFT is not tokens per second. One measures startup latency; the other measures generation throughput after output begins.
• Related terms: Streaming, Prompt Cache, Observability, Token Budget

Token

• Category: Data & representations
• What people say: A word-sized piece of model input or output.
• What it actually means: An integer identifier produced by a model-specific tokenizer from text, bytes, images, audio, or another input representation. A token can be a whole word, part of a word, punctuation, whitespace, a byte sequence, or a special control symbol.
• Common confusion: Character-to-token ratios vary by language, content, and tokenizer, so count with the target model's tokenizer or provider tools.
• Learn it: Tokenizers
• Related terms: Token Budget, Context Window, Autoregressive

Token Budget

• Category: Prompting & context
• What it actually means: An explicit allocation of token capacity across instructions, evidence, history, tool results, reasoning or working space, and output.
• Why it matters: Every included token competes for context capacity, latency, and cost. Budgeting forces you to preserve high-value evidence first.
• In practice: Reserve output capacity, cap retrieved chunks, summarize old tool results into state, and stop or compact before the model limit is reached.
• Common confusion: A token budget is a planning constraint. It is not the same as the model's maximum context window.
• Learn it: Context Engineering
• Related terms: Context Window, Context Engineering, Progressive Disclosure, Cost per Successful Task

Tokenization

• Category: Data & representations
• What it actually means: Converting an input representation into the ordered token identifiers a specific model or tokenizer accepts.
• Why it matters: Tokenization determines sequence length, vocabulary boundaries, cost accounting, truncation behavior, and how text or code is represented before embedding.
• In practice: Use the exact tokenizer for the target model, version it with artifacts, and test multilingual text, code, whitespace, and special tokens.
• Common confusion: Tokenization is not always word splitting, and two models can assign different token counts and IDs to the same input.
• Related terms: Token, Vocabulary, Byte Pair Encoding (BPE), Embedding
• Sources: Neural Machine Translation of Rare Words with Subword Units

Tokens per Second (TPS)

• Category: Infrastructure & serving
• Aliases: TPS, output token throughput
• What it actually means: A throughput measure reporting how many output tokens a serving system produces per unit time under a stated scope and workload.
• Why it matters: It complements startup latency by showing how quickly generation proceeds after output begins and how serving behaves under load.
• In practice: State whether TPS is per request or aggregate, exclude or identify prefill, and report batch, concurrency, sequence lengths, hardware, and percentile latency.
• Common confusion: TPS is not directly comparable across different tokenizers, workloads, quality settings, or measurement boundaries.
• Related terms: Time to First Token (TTFT), Streaming, Prefill, Observability
• Sources: Sarathi-Serve

Tool Contract

• Category: Agents & tools
• What it actually means: The complete agreement for a tool boundary: purpose, typed inputs, outputs, validation, permissions, side effects, errors, timeouts, idempotency, and evidence returned to the caller.
• Why it matters: A schema tells a model what fields exist; a contract tells the system when the tool is safe and how failures must be handled.
• In practice: Define a file-write tool with an allowed root, expected base revision, maximum size, dry-run mode, explicit conflict errors, and a returned patch hash.
• Common confusion: A JSON Schema is part of a tool contract, not the whole contract.
• Learn it: Tool Use and Function Calling
• Related terms: Function Calling, Structured Output, Least Privilege, Idempotency

Top-k Sampling

• Category: Models & inference
• What it actually means: A decoding method that restricts the next-token distribution to the k highest-scoring candidates, renormalizes their probabilities, and samples from that set.
• Why it matters: It removes the long low-probability tail from sampling while keeping a fixed maximum candidate count.
• In practice: Evaluate k together with temperature, top-p, and stop settings, and record the complete sampler configuration with generated results.
• Common confusion: Top-k uses a fixed candidate count, while top-p uses a probability-mass threshold whose candidate count changes by step.
• Related terms: Nucleus Sampling (Top-p), Temperature, Decoding Strategy, Logits
• Sources: The Curious Case of Neural Text Degeneration

Trace

• Category: AI-native development
• What it actually means: A correlated record of one request or task across model calls, retrieval, tools, state transitions, retries, approvals, and evaluations.
• Why it matters: It lets you reconstruct where time, cost, and failure entered a multi-step workflow.
• In practice: Propagate one trace identifier through the agent harness and attach redacted spans for each model and tool operation.
• Common confusion: A trace should record operational evidence, not expose hidden model reasoning, secrets, or unredacted sensitive content.
• Learn it: OpenTelemetry GenAI Conventions
• Sources: OpenTelemetry traces
• Related terms: Observability, Agent State, Time to First Token (TTFT), Evaluation (Eval)

Transfer Learning

• Category: Math & training
• What people say: Reusing a pretrained model for a new task.
• What it actually means: Starting from representations or parameters learned on one data distribution or objective and adapting them for another. The transferable components and update strategy depend on architecture and task.
• Common confusion: Transfer is not limited to later layers, and successful transfer is not guaranteed when source and target tasks differ sharply.
• Related terms: Fine-tuning, Feature, SFT (Supervised Fine-Tuning)

Transformer

• Category: Models & inference
• What people say: The architecture behind many modern language models.
• What it actually means: A neural-network architecture built from attention, position information, feed-forward sublayers, residual connections, and normalization. Encoder, decoder, and encoder-decoder variants use different masks and information flows.
• Why it matters: Training can process many sequence positions in parallel, while autoregressive generation still produces outputs step by step.
• Common confusion: Self-attention does not imply unrestricted all-to-all attention in every transformer.
• Learn it: Build a Full Transformer
• Sources: Attention Is All You Need
• Related terms: Attention, Self-Attention, Encoder, Decoder

Trust Boundary

• Category: Security & governance
• What it actually means: An interface where data, instructions, identity, or authority crosses between components or principals that operate under different trust assumptions.
• Why it matters: A boundary crossing is where the system must authenticate actors, validate data, constrain permissions, and decide which claims can influence action.
• In practice: Draw boundaries around users, model context, retrieval sources, tools, networks, and data stores, then specify validation and authorization for every crossing.
• Common confusion: A network boundary is only one kind of trust boundary. Untrusted document text entering a privileged agent context also crosses one.
• Learn it: Jailbreak Taxonomy
• Related terms: Threat Model, Least Privilege, Sandbox, Indirect Prompt Injection
• Sources: Microsoft Learn: Trust Boundary, the Trust Zone Change Element; OWASP Threat Modeling