Project Status: In progress
Long-Context Speculative Decoding
Dates:
Stealth — details after publication
Projects
What I build, from kernels to clusters.
Efficient Long-Context LLMs
-
- Venue: Preprint Year: 2026
Language Models Can Control Their Own Attention
Authors: Namgyu Ho * (equal contribution) , Huzama Ahmad * (equal contribution) , Woosung Koh * (equal contribution) , Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
Models declare where to attend so the engine can skip other context. Across 15 long-context tasks in zero-shot evaluation, total attended tokens during decoding fell by 31.1–52.0%, with model-dependent accuracy drops of 1.27–2.75 percentage points.
- Venue: Preprint Year: 2026
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Authors: Huzama Ahmad, Se-Young Yun
A lightweight learned selector focuses a frozen pretrained transformer’s attention. SpotAttention matches dense accuracy on the tested Qwen3 and Qwen3.5 models with contexts up to 128K tokens. At 128K, decoding runs 3.9× faster than FlashAttention.
Efficient-LLM Architecture & Training Systems
- Venue: Preprint Year: 2025
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
Authors: Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun
CascadeFormer narrows Transformer layers with depth, motivated by Gradient Fan-in Asymmetry. At the same training budget, it achieved comparable perplexity to a uniform baseline, with 8.6% lower latency and 9.4% higher throughput.
- Project Status: Completed
Foundations of Efficient LLMs
Dates:
Two years of experiments on TPU pods: running 1B–8B language models, training the smaller ones from scratch, compressing context into learned vectors, estimating uncertainty, and tapering width with depth.
Systems & Infrastructure
- System Status: Maintained
GPU Cluster
Dates:
I built and run the lab’s 67-GPU Slurm cluster for 30+ researchers. Nine heterogeneous nodes share identity, storage, and scheduling, with strict per-job GPU isolation and reservations around submission deadlines.
- System Status: Maintained
HomeLab
Dates:
A personal cloud for notes, photos, media, and home automation, running on machines in my apartment and a few rented hosts. I build the whole stack, from storage and networking to the services on top.
Earlier Work
- Project Status: Completed
Undergraduate Research Mentorship
Dates:
I mentored eight undergraduate teams studying multilingual model performance and four students evaluating cultural representation. Their work led to four published benchmarks: BEnQA, CLIcK, CULTDIFF, and MIXCUBE.
- Project Status: Completed
StePer: Self-Guided Reinforcement Learning for Arithmetic Reasoning
Dates:
StePer trains arithmetic reasoning with rewards drawn from the model’s own answers, comparing greedy responses with diverse samples. Without a human grader in the reward loop, accuracy improved by up to 5% across four math benchmarks.