N

NaN (Not a Number)

• Category: Math & training
• What people say: A sign that numerical computation failed.
• What it actually means: A floating-point value representing an undefined or unrepresentable numerical result. In training, NaNs can come from invalid operations, overflow, unstable normalization, excessive updates, or earlier corrupted values.
• In practice: Find the first non-finite tensor, inspect its inputs, and add assertions or anomaly detection near that operation.
• Related terms: Mixed Precision, Learning Rate, Gradient

Normalization

• Category: Math & training
• What people say: Scaling data to a standard range.
• What it actually means: A family of transformations that rescale or recenter inputs, activations, or features using defined statistics. Batch normalization and layer normalization use different axes and behave differently across training and inference.
• Common confusion: Normalization can improve optimization stability, but it does not always permit a larger learning rate or improve every architecture.
• Related terms: Tensor, Activation Function, Mixed Precision

Nucleus Sampling (Top-p)

• Category: Models & inference
• Aliases: Top-p sampling
• What it actually means: A decoding method that samples from the smallest set of next-token candidates whose cumulative probability reaches a chosen threshold.
• Why it matters: The candidate-set size adapts to the distribution, retaining more options when uncertainty is broad and fewer when probability is concentrated.
• In practice: Evaluate the threshold with temperature and stop settings held constant, and record the complete decoding configuration with every result.
• Common confusion: Top-p is a probability-mass threshold, while top-k always keeps a fixed maximum number of candidates.
• Related terms: Top-k Sampling, Temperature, Decoding Strategy, Softmax
• Sources: The Curious Case of Neural Text Degeneration