A startup claims it broke a bottleneck holding back large language models
Subquadratic, a Miami startup that emerged from stealth in May, says it has eased one of the core constraints on large language models: the quadratic cost of dense attention, in which every token is compared against every other, so doubling the input roughly quadruples the work. Its model, SubQ, replaces that with dynamic sparse attention, selectively computing only the token pairs it judges important. The company reports third-party results from Appen showing SubQ running far faster than FlashAttention-based models, matching strong coding benchmarks, and handling 12-million-token contexts with high retrieval accuracy — at a fraction of competitors’ cost on long-context tests.
The caveats are substantial. SubQ reportedly reuses weights from the open-source Qwen model rather than training from scratch, public access remains limited despite thousands of sign-ups, and there is little real-world testing. As independent researcher Will Depue put it, “the public evidence does not yet justify the stronger claim that they have solved the quadratic attention bottleneck.” Efficient-attention schemes have been proposed for years and often trade accuracy for speed in ways that benchmarks don’t fully capture, so the meaningful test is whether SubQ holds up under open, independent use rather than vendor-supplied numbers.