Hugging Face Paper Ideas

Use this file to turn literature scans into Autolab-ready hypotheses.

Rules:

• Start from the current local promoted master and refreshed DAG state.
• Propose only ideas that fit a single disciplined change in train.py.
• Reject ideas that are already present in the current code or already tested in research/notes.md.
• Promote only one hypothesis per experiment.

2026-03-27 shortlist for au-cyy

Context:

• target local promoted master for the next direct-master runs is 935fdbf9f4ae8a5ef5bcb76552acea2bc5801965 (val_bpb = 0.962777)
• explicitly exclude the already-low-confidence lines from recent history: attention temperature scaling, SLSL, GQA half-KV, and has_ve() every second layer
• also exclude direct repeats that already have fresh swarm evidence: power-law cooldown exponent 2.0 regressed on au-fpz, and constant Muon beta2 = 0.9 already has an archived win plus a live direct-master confirm experiment (au-08h)

1. WSD-style short cooldown length

• Papers: 2405.18392, 2508.01483
• Why this is next:
  • current WARMDOWN_RATIO = 0.825 means the run is in cooldown for 82.5% of the five-minute budget
  • the scaling/cooldown papers both point toward much shorter decay tails, with most of the gain saturating around a 10-20% cooldown rather than a nearly full-run anneal
• Exact train.py edit surface:
  • change only WARMDOWN_RATIO
  • leave get_lr_multiplier() and FINAL_LR_FRAC unchanged so attribution stays clean
• Recommended experiment title:
  • schedule: shorten warmdown to a 20% tail on 935fdbf

2. 1-sqrt cooldown shape

• Papers: 2405.18392, 2508.01483
• Why this is next:
  • 2405.18392 reports that a 1-sqrt cooldown can outperform linear decay once the cooldown is long enough
  • this is meaningfully different from the already-losing exponent-2 power-law trial because it changes the late-phase curvature without introducing an extra exponent hyperparameter
• Exact train.py edit surface:
  • change only the cooldown branch inside get_lr_multiplier()
  • keep WARMDOWN_RATIO and FINAL_LR_FRAC fixed for the first experiment
• Recommended experiment title:
  • schedule: replace linear warmdown with 1-sqrt cooldown on 935fdbf

3. Cooldown-only Muon beta2 ramp

• Papers: 2508.01483, 2601.14603
• Why this is next:
  • 2508.01483 reports consistent gains from higher beta2 during cooldown
  • 2601.14603 motivates variance-adaptive Muon extensions rather than treating beta2 as a fixed always-on toggle
  • this avoids spending another experiment on the already-known constant beta2 = 0.9 rerun and instead isolates whether the gain is specifically a late-phase variance-control effect
• Exact train.py edit surface:
  • add a get_muon_beta2(progress) helper next to get_lr_multiplier()/get_muon_momentum()/get_weight_decay()
  • in the training-loop param-group update, set group["beta2"] only for Muon groups while leaving matrix LR, momentum, and weight decay schedules unchanged
• Recommended experiment title:
  • optimizer: ramp Muon beta2 during cooldown on 935fdbf

Current Paper-Derived Candidates

Selective attention temperature

• Paper: 2411.12892 "Selective Attention: Enhancing Transformer through Principled Context Control"
• Why it matches this repo:
  • train.py already normalizes q and k, so the clean insertion point is the attention logit scale after QK normalization.
  • The model already uses lightweight value gating, so query-temperature or value-temperature extensions are architecturally compatible.
• Smallest credible experiments:
  • add one scalar attention-temperature multiplier to the attention logits
  • add one learned per-layer or per-head inverse-temperature initialized at 1.0
• Keep it single-change:
  • do not combine query-temperature and value-temperature in the same run
  • do not change the window pattern in the same experiment
• Main risk:
  • over-sharpening attention may hurt sliding-window layers

Power-law cooldown schedule

• Paper: 2408.13359 "Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler"
• Related cooldown analysis: 2508.01483 "Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler"
• Why it matches this repo:
  • the current schedule is a simple linear warmdown controlled by WARMDOWN_RATIO and FINAL_LR_FRAC
  • scheduler shape is isolated in get_lr_multiplier, so this is a clean one-function change
• Smallest credible experiments:
  • replace linear cooldown with a power-law cooldown using one new exponent hyperparameter
  • keep the same FINAL_LR_FRAC and only vary the cooldown curve shape
• Main risk:
  • changing both the cooldown shape and floor at once would make attribution poor

Cooldown-stage variants

• Papers:
  • 2502.15938 "Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs"
  • 2508.01483 "Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler"
  • 2410.05192 "Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective"
• Why they match this repo:
  • the scheduler is still concentrated in get_lr_multiplier() plus two scalar constants, so each cooldown-stage test can remain a one-function or one-constant change in train.py
  • these papers distinguish the floor choice, cooldown duration, and cooldown curve, which maps directly onto FINAL_LR_FRAC, WARMDOWN_RATIO, and the interpolation logic here
• Smallest credible experiments:
  • set only FINAL_LR_FRAC = 0.0 to test a straight-to-zero tail
  • set only WARMDOWN_RATIO = 0.90 to lengthen the cooldown phase
  • replace only the linear cooldown interpolation with a cosine interpolation while keeping the same floor and cooldown duration
• Keep it single-change:
  • do not combine a zero floor with a new cooldown shape in the same experiment
  • do not move both WARMDOWN_RATIO and FINAL_LR_FRAC together
• Main risk:
  • scheduler-only gains can be easy to overfit with multiple adjacent sweeps, so stagger these across separate experiments instead of treating them as a bundle

Gradient-preserving activation scaling

• Paper: 2506.22049 "GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling"
• Why it matches this repo:
  • the model is Pre-LN/RMSNorm and already has explicit residual mixing parameters, so residual-path balance is clearly part of the design space
  • GPAS can be tested as a local block-output scaling change without touching prepare.py
• Smallest credible experiments:
  • add one fixed or learned GPAS-style scale on block outputs while preserving gradients with a stop-gradient trick
  • apply it to either the attention output or the MLP output, not both
• Main risk:
  • this is a larger code change than an LR tweak, so isolate it aggressively

Muon variance adaptation follow-ups

• Papers:
  • 2510.05491 "NorMuon: Making Muon more efficient and scalable"
  • 2601.14603 "Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum"
• Why it matches this repo:
  • train.py already contains NorMuon-style row/column variance normalization in muon_step_fused
  • the current Muon groups still default beta2=0.0, so the second-moment path is underused by default
• Smallest credible experiments:
  • enable a non-zero Muon beta2
  • change only the variance statistic or normalization axis, not both
• Main risk:
  • treat this as a follow-up to the current optimizer, not a brand-new direction

Low-Priority Ideas

• 2311.00684 attention alignment for long-context extrapolation: interesting but less aligned with the fixed short-budget Autolab benchmark.
• Structured FFN or architecture-replacement papers: too invasive for disciplined one-change comparisons here.

Additional Ranked Candidates

Local-global window interleave

• Paper: 2408.00118
• Why it matches this repo:
  • WINDOW_PATTERN is already an explicit layerwise attention-pattern knob
  • switching from SSSL to SLSL is a one-string architectural test, not a subsystem rewrite
• Smallest credible experiment:
  • change only WINDOW_PATTERN from "SSSL" to "SLSL"
• Main risk:
  • a denser global pattern may buy quality at the cost of throughput or memory

Grouped-query attention

• Paper: 2305.13245
• Why it matches this repo:
  • attention already supports n_kv_head <= n_head
  • the current builder still ties n_kv_head = num_heads, so halving KV heads is a clean existing-capability toggle
• Smallest credible experiment:
  • set n_kv_head = num_heads // 2
• Main risk:
  • this changes parameter count and attention cache structure, so compare it carefully against optimizer-only wins

Denser value-embedding cadence

• Paper: 2410.17897
• Why it matches this repo:
  • the model already has explicit value-embedding placement and gating logic
  • recent wins suggest this subsystem is live rather than decorative
• Smallest credible experiment:
  • change has_ve() cadence from every third layer to every second layer
• Main risk:
  • this is adjacent to the current value-embedding regularization line, so keep it isolated from any weight-decay changes

Attention branch scale reset

• Paper: 2305.02790
• Why it matches this repo:
  • the block already hardcodes a non-residual attention amplification
  • testing 1.0 versus 1.3 is a scalar-only branch-balance experiment
• Smallest credible experiment:
  • change the attention branch scale from 1.3 to 1.0
• Main risk:
  • the current scale may already be compensating for another architectural choice, so this can regress sharply