Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean)

[abaybektursun]

Record: LeakyReLU² + Legal TTT + Parallel Muon — val_bpb 1.1194

val_bpb = 1.1194 (3-seed mean, std 0.0006) | ~15.95 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	Pre-TTT bpb	Post-TTT bpb	TTT gain	TTT time	Artifact
1337	83.3ms	7,179	1.1217	1.1192	-0.0025	410s	15,977,386
42	83.4ms	7,182	1.1227	1.1200	-0.0027	408s	15,876,510
2025	83.4ms	7,193	1.1212	1.1189	-0.0023	408s	15,990,006
Mean	83.4ms	7,185	1.1218	1.1194 (std 0.0006)	-0.0025	~409s

Key Innovation: LeakyReLU(0.5)²

One-line activation change delivering -0.003 BPB vs standard relu²:

# relu² (standard)
x = torch.relu(self.fc(x)).square()
# leaky relu² (this submission)
x = F.leaky_relu(self.fc(x), negative_slope=0.5).square()

Preserves negative gradient flow through the MLP. Source: PR #493 by @parinzee (ablated at -0.003), PR #518 by @sofiabod.

Legal TTT (Score-First, PR #461 Framework)

Every token scored BEFORE any weight update, enforced by torch.inference_mode():

for each 32K-token chunk:
    Phase 1 — SCORE: sliding window eval (inference_mode)
    Phase 2 — TRAIN: SGD(lr=0.002, mom=0.9), 3 epochs, all blocks unfrozen

Adapted from PR #461 by @Christopher-Lee-McClendon (changed freeze=2 → freeze=0 based on our ablation showing unfreezing all blocks is optimal at 3 epochs).

Total eval: ~530s (120s standard + 409s TTT) — within 10 min limit.

Training Architecture

PR #414 stack + Parameter Banking + Parallel Muon (PR #399):

• 11L, 512d, 8H/4KV, LeakyReLU(0.5)² MLP 3×
• BigramHash(1536), XSA4, Partial RoPE, LN Scale, VE128
• EMA(0.997) + Tight SWA, GPTQ-lite int6 + lzma
• Parameter Banking + Parallel Muon (83.4ms/step)

Credits

• LeakyReLU²: PR #493 by @parinzee, PR #518 by @sofiabod
• Optimizer: PR #399 by @abaybektursun
• TTT recipe: PR #461 by @Christopher-Lee-McClendon (adapted: freeze=0)
• Base model: PR #414 by @signalrush

🤖 Generated with Claude Code