Autolab Notes

Repo layout

• Seed hub snapshots live in research/reference/.
• Live hub refreshes should be written to research/live/.
• Archived local experiment diffs live in research/diffs/.

Local environment

• GPU: NVIDIA H100 80GB HBM3 (GPU 0)
• Local host GLIBC is 2.31, so the Hopper-only varunneal/flash-attention-3 wheel does not import here.
• Local runs use sitecustomize.py plus run-local.sh to redirect that kernel lookup to kernels-community/flash-attn3 without editing train.py.

Baseline

• Master hash: 765a36b0700b3a20d552f48b8ca2b75636aa3e69
• Hub master val_bpb: 0.962777
• Local baseline val_bpb under compatibility redirect: 0.962846
• Local baseline peak_vram_mb: 33609.3
• Local baseline mfu_percent: 43.84
• Local baseline total_tokens_M: 304.5
• Local baseline num_steps: 2323

Experiment: batch96

Hypothesis:

• Increase microbatch and total batch together to exploit large VRAM headroom while keeping grad_accum_steps=1.

Change:

• DEVICE_BATCH_SIZE = 96
• TOTAL_BATCH_SIZE = 96 * 2048

Observed early behavior:

• Peak memory around 50 GB, so it fits.
• Steady-state tok/sec stayed roughly flat around 1.02M to 1.04M instead of increasing materially.
• Training steps over the 300-second budget dropped substantially relative to baseline.

Conclusion:

• This looks weak. It spent much more memory for little or no throughput gain and likely sacrifices update count.
• Keep the diff in research/diffs/batch96.diff for reference, but do not treat it as a leading candidate.

Experiment: flr021

Hypothesis:

• Lower the final LR floor from 0.025 to 0.021.
• Post-master hub history showed FINAL_LR_FRAC=0.021 as the strongest near-miss after current master.

Change:

• FINAL_LR_FRAC = 0.021

Result:

• Local val_bpb: 0.962717
• Local training_seconds: 300.1
• Local total_seconds: 383.7
• Local peak_vram_mb: 33612.7
• Local mfu_percent: 43.88
• Local total_tokens_M: 304.7
• Local num_steps: 2325

Conclusion:

• This beat both the local baseline (0.962846) and the current hub master (0.962777).
• Submitted as patch 14808 against master 765a36b0700b3a20d552f48b8ca2b75636aa3e69.
• Workspace train.py was reset back to master after submission so the next experiment starts clean.

Experiment: au-41x lm_head_wd_003

Hypothesis:

• Increase only the lm_head AdamW weight decay from 0.002 to 0.003.
• Slightly stronger output-layer regularization may improve validation bpb without perturbing the other optimizer groups.

Change:

• lm_head optimizer group weight_decay = 0.003

Managed H200 result:

• Master hash: 765a36b0700b3a20d552f48b8ca2b75636aa3e69
• HF Job: 69c5a34525abd6f920b4fea3
• val_bpb: 0.961175
• training_seconds: 300.0
• total_seconds: 430.2
• peak_vram_mb: 33609.6
• mfu_percent: 45.18
• total_tokens_M: 313.7
• num_steps: 2393

Conclusion:

• This beats the current hub master (0.962777) by 0.001602.
• Submitted as patch 14810.

Experiment: scalar-lr-08

Hypothesis:

• Increase SCALAR_LR from 0.7 to 0.8 to test whether faster learning of per-layer residual and initial scaling parameters improves validation bpb.

Change:

• SCALAR_LR = 0.8 (was 0.7)

Managed H200 result:

• Master hash: 765a36b0700b3a20d552f48b8ca2b75636aa3e69
• HF Job: 69cbe00334fa24114ddf47a6
• Campaign: optimizer-tuning
• val_bpb: 0.963049
• training_seconds: 300.1
• total_seconds: 433.0
• peak_vram_mb: 33609.6
• mfu_percent: 44.82
• total_tokens_M: 311.3
• num_steps: 2375

Conclusion:

• FAILED. Increasing SCALAR_LR degraded validation bpb by +0.000272 compared to master (0.962777).
• The faster learning rate for scalar parameters hurt generalization.

Experiment: embedding-wd-001

Hypothesis:

• Increase embedding weight decay from 0.0005 to 0.001 to test whether stronger regularization on token embeddings improves validation bpb.

Change:

• Embedding optimizer group weight_decay = 0.001 (was 0.0005)

Managed H200 result:

• Master hash: 765a36b0700b3a20d552f48b8ca2b75636aa3e69
• HF Job: 69cbdff7942f980bf425a3d0
• Campaign: regularization-tuning
• Status: CANCELLED (STUCK)
• Job got stuck at step 02348 (97.6%) during final evaluation phase for 10+ minutes.

Conclusion:

• CANCELLED due to job hanging during evaluation. No val_bpb result obtained.
• The hypothesis remains untested; may need to retry with different infrastructure or debugging.

Experiment: unembedding-lr-015

Hypothesis:

• Increase UNEMBEDDING_LR from 0.01 to 0.015 to test whether a higher learning rate for the output layer improves validation bpb.

Change:

• UNEMBEDDING_LR = 0.015 (was 0.01)

Managed H200 result:

• Master hash: 935fdbf9f4ae8a5ef5bcb76552acea2bc5801965
• HF Job: 69cc18d8942f980bf425a641
• Campaign: optimizer-tuning
• val_bpb: 0.970805
• training_seconds: 300.1
• total_seconds: 431.5
• peak_vram_mb: 33609.6
• mfu_percent: 44.14
• total_tokens_M: 306.6
• num_steps: 2339

Conclusion:

• FAILED. Increasing UNEMBEDDING_LR degraded validation bpb by +0.008028 compared to master (0.962777).
• Higher learning rate for the output layer significantly hurt generalization.

Experiment: warmdown-075

Hypothesis:

• Decrease WARMDOWN_RATIO from 0.825 to 0.75 to allow more time at higher learning rates before the final LR floor.

Change:

• WARMDOWN_RATIO = 0.75 (was 0.825)

Managed H200 result:

• Master hash: 935fdbf9f4ae8a5ef5bcb76552acea2bc5801965
• HF Job: 69cc194534fa24114ddf48bd
• Campaign: schedule-tuning
• val_bpb: 0.962980
• training_seconds: 300.1
• total_seconds: 431.5
• peak_vram_mb: 33609.6
• mfu_percent: 43.88
• total_tokens_M: 304.7
• num_steps: 2325

Conclusion:

• FAILED. Decreasing WARMDOWN_RATIO degraded validation bpb by +0.000203 compared to master (0.962777).
• Shorter warmdown phase slightly hurt final convergence.