Contents
1. Introduction
The hardest enterprise AI problems share a common shape. They require reasoning over complete artifacts: entire codebases, document collections, contracts spanning hundreds of pages, and financial filings that run into the tens of thousands of lines. For years, the industry worked around this problem by building retrieval pipelines, chunking strategies, and agentic scaffolding. These are useful tools, but they are ultimately workarounds for the context limitations of the underlying model architecture.
The root constraint has always been attention. Standard transformer attention scales quadratically with context length, which means that doubling your input quadruples the compute required. This math makes direct reasoning over large documents prohibitively expensive, forcing practitioners to split, summarise, and approximate rather than reason directly. SubQ was built to remove that constraint.
SubQ 1.1 Small introduces Subquadratic Sparse Attention (SSA), a novel attention mechanism that replaces the quadratic scaling of standard attention with a learned sparse formulation that scales linearly. The result is a model that achieves near-perfect long-context retrieval up to 12 million tokens while using up to 1,000x less attention compute than a standard transformer. This article breaks down how SSA works, how SubQ 1.1 Small performs on key benchmarks, and what this means for the future of enterprise AI.
Note on evaluation: The benchmark results discussed in this article are primarily published by Subquadratic Inc. themselves. While the company engaged Appen for independent verification of select benchmarks, the broader evaluation landscape remains limited. Community-driven testing and third-party replication have not yet been conducted at scale, as the model is currently only available to select design partners. Readers should interpret the results with this context in mind.
2. The Quadratic Problem (and Why It Matters)
To understand why SubQ 1.1 Small is significant, you first need to understand the problem it solves. In a standard transformer model, the attention mechanism computes relationships between every pair of tokens in the input sequence. For a sequence of length n, that means n x n comparisons. This is what computer scientists call O(n2) scaling, or quadratic complexity.
Here is an analogy: imagine you are at a party with 10 people. Saying hello to everyone is manageable. But if the party grows to 100 people, greeting each person individually becomes exhausting. At 1,000 people, it is no longer feasible. The greeting time grows quadratically because you need to acknowledge everyone twice. That is exactly how standard attention works.
Quadratic scaling creates a hard ceiling on context length. A model with 128K tokens of context requires about 16 billion attention computations per layer. Pushing that to 1 million tokens would require nearly 1 trillion computations per layer. The compute and memory costs become astronomical, which is why most models are limited to 128K or 256K tokens of effective context.
This limitation has real-world consequences. When a legal team needs to analyse a contract portfolio, they cannot simply feed all the documents into a model and ask questions. They must build retrieval pipelines, chunk documents into small pieces, and hope that the most relevant information is not lost in the process. When a software developer wants a model to understand an entire codebase, they cannot simply upload the repository. They must use agentic frameworks that navigate through files one at a time. These are not features; they are coping mechanisms for an architectural limitation.
Subquadratic Sparse Attention eliminates this limitation at the architectural level, changing the scaling from O(n2) to O(n). This is not an incremental improvement. It is a fundamental change in what is computationally feasible.
3. Subquadratic Sparse Attention (SSA) — How It Works
Subquadratic Sparse Attention replaces the dense attention matrix of standard transformers with a learned sparse formulation. Instead of computing attention scores between every pair of tokens, SSA learns which token relationships matter and prunes away the rest. The result is an attention mechanism that scales linearly with sequence length rather than quadratically.
Think of it as the difference between reading every single word in a library versus using a well-trained librarian who instinctively knows which books and pages are relevant to your question. The librarian does not need to scan everything; they have learned a map of what connects to what.
Concretely, SubQ 1.1 Small achieves the following efficiency gains:
- At 1 million tokens, SSA requires 64.5x less compute than standard dense attention.
- At 1 million tokens, SSA runs 56x faster than FlashAttention-2, a widely adopted optimised attention implementation.
- In NIAH (Needle In A Haystack) testing, the model achieves 100% retrieval accuracy at 1M and 2M tokens, 98% at 6M tokens, and 98% at 12M tokens.
The practical implication is that SubQ 1.1 Small can reason over extremely long contexts with near-perfect accuracy while using a fraction of the computational budget that a standard model would require. This changes the economics of both training and inference for long-context applications.
4. Benchmark Results
SubQ 1.1 Small was evaluated across multiple benchmarks covering reasoning, coding, long-context retrieval, and enterprise automation.
Caveat: Unless otherwise noted, these results are self-reported by Subquadratic Inc. The company has engaged Appen, a third-party data services company, to independently verify the published NIAH, RULER, GPQA Diamond, LiveCodeBench, and AutomationBench results (see the full Appen report linked in the original announcement). However, broader community validation and independent replication studies are not yet available. The model is currently deployed with select design partners, and production-grade public access is expected later in the year.
| Benchmark | SubQ 1.1 Small | Context / Notes |
|---|---|---|
| NIAH (1M) | 100% | Perfect needle retrieval at 1M tokens |
| NIAH (2M) | 100% | Perfect retrieval at 2M tokens |
| NIAH (6M) | 98% | Near-perfect at massive scale |
| NIAH (12M) | 98% | Strong at extreme context length |
| RULER (128K) | 99.12% | Long-context aggregation |
| GPQA Diamond | 85.4% | Graduate-level reasoning |
| LiveCodeBench v6 | 89.7% | Competitive coding (pass@4) |
| AutomationBench Finance | 13% | Financial workflow automation |
Compared to frontier models from major labs, SubQ 1.1 Small holds its ground in reasoning and coding while offering dramatically better long-context capabilities:
| Model | GPQA Diamond | LiveCodeBench v6 | AutomationBench Finance |
|---|---|---|---|
| GPT-5.5 | 93.2% | 92.0% | 18% |
| Opus 4.8 | 92.0% | 92.2% | 16% |
| SubQ 1.1 Small | 85.4% | 89.7% | 13% |
| Sonnet 4.6 | 87.5% | 88.9% | 8% |
| GPT-5.4-mini | 87.5% | 78.6% | 0% |
| GPT-5.4-nano | 81.7% | 78.2% | N/R |
| Haiku 4.5 | 67.2% | 69.7% | 3% |
The standout result is the LiveCodeBench score. At 89.7%, SubQ 1.1 Small is competitive with Sonnet 4.6 (88.9%) and not far behind GPT-5.5 (92.0%). On AutomationBench Finance, it outperforms every model except GPT-5.5 and Opus 4.8. This is impressive for a model whose primary design focus was long-context efficiency rather than peak reasoning performance.
It is worth noting that the comparison data for GPT, Claude, and other models in the table is sourced from their respective public technical reports and third-party evaluations, and may have been produced under different testing conditions. Direct head-to-head comparisons under identical conditions would require a standardised evaluation framework.
5. How They Built It
SubQ 1.1 Small starts from an open-weight frontier model and replaces its standard dense attention with Subquadratic Sparse Attention (SSA). The training process involved three major stages:
Stage 1: Architecture Surgery. The team took an existing open-weight model and surgically replaced the attention mechanism. The dense attention layers were swapped with SSA layers, which use a learned sparse formulation to compute attention over only the most relevant token pairs. Everything else about the model architecture remained intact.
Stage 2: Staged Context Extension. Training a model to handle very long contexts is not something you do all at once. SubQ was trained in stages, progressively extending the context length: 262K tokens, then 512K, then 1 million, then 2 million. Each stage taught the model to handle longer dependencies without losing coherence or retrieval accuracy.
Stage 3: Continued Pretraining on Long Artifacts. The model underwent approximately 1 trillion tokens of continued pretraining on long-form documents. This is not the kind of data used in typical pretraining, which is heavily skewed toward short sequences. Instead, the team curated datasets of entire codebases, full-length legal contracts, complete financial filings, and other long artifacts to teach the model how to reason over extended contexts.
In total, the team ran over 100 experiments across 6 to 7 model generations to arrive at the final SubQ 1.1 Small configuration. This iterative approach reflects the difficulty of balancing long-context capability with general reasoning performance.
6. Real-World Use Cases
SubQ 1.1 Small unlocks use cases that were previously impractical or prohibitively expensive with standard transformer models:
Financial Analysis and Due Diligence. Financial analysts regularly work with documents that run hundreds of pages: 10-K filings, prospectuses, M&A contracts, and regulatory submissions. With SubQ 1.1 Small, these documents can be processed in their entirety without chunking or retrieval pipelines. An analyst can ask questions about a complete filing and get answers that draw on information from any part of the document, including cross-references between sections that might be hundreds of pages apart.
Legal and Contract Work. Legal teams deal with contract portfolios that can span thousands of pages. Standard approaches require splitting these into chunks, which often breaks cross-references and makes it difficult to understand the full scope of obligations. SubQ 1.1 Small can ingest full contract portfolios and answer questions that require understanding the interplay between multiple agreements.
Software Engineering. Whole-repository reasoning is perhaps the most exciting use case. A developer can feed an entire codebase into SubQ 1.1 Small and ask questions that require understanding the full architecture: how authentication flows from the frontend through the API layer to the database, where a specific business rule is implemented, or whether a change in one module will break another. The 12M token context window is large enough to accommodate most production codebases in a single pass.
7. What This Means for AI
Subquadratic Sparse Attention represents a shift in how the industry thinks about context length. For years, the prevailing wisdom was that context windows would grow incrementally as hardware improved and attention mechanisms were optimised. SSA breaks that incremental trajectory by changing the fundamental scaling law from O(n2) to O(n).
The implications are far-reaching. First, it makes long-context inference economically viable. At 1 million tokens, SSA uses 64.5x less compute than dense attention. That is not a minor efficiency gain; it is the difference between a task being too expensive to run in production and being cheap enough to run at scale.
Second, it changes the architecture of AI applications. Many of the most complex systems being built today, from agentic coding assistants to document analysis platforms, are essentially elaborate workarounds for context limitations. If the model can see everything at once, the retrieval pipeline, the chunking strategy, and the agentic navigation framework can be dramatically simplified or eliminated entirely.
Third, it opens up new categories of applications. Continuous auditing of financial filings, real-time analysis of legislative documents, whole-codebase security reviews, and full-document contract negotiations become feasible when the model can hold the entire artifact in its context window at once.
The third-party verification by Appen is also significant. In an industry where benchmark results are often published with selective reporting, independent validation adds credibility to the claims. Appen confirmed that SubQ 1.1 Small achieves the published results on the NIAH, RULER, GPQA Diamond, LiveCodeBench, and AutomationBench evaluations. However, it is important to note that Appen’s verification is one data point, not a comprehensive independent audit. The broader research community has yet to weigh in through replication studies or adversarial testing.
8. Conclusion
SubQ 1.1 Small is not just another model release. It is a proof point that Subquadratic Sparse Attention works at scale. The model achieves near-perfect retrieval at 12 million tokens, competitive reasoning and coding scores against frontier models, and does so with dramatically less compute than any standard transformer.
The team behind SubQ is now working with a first cohort of design partners to field-test the model in production environments. A broader rollout is planned, with general model releases expected by the end of the year. If the current results hold up in real-world deployments and independent verification, Subquadratic Sparse Attention could fundamentally change what is possible with large language models.
For now, SubQ 1.1 Small is the most compelling demonstration yet that attention does not need to be the bottleneck. The quadratic constraint has been the silent tax on every long-context AI application built to date. SubQ has just made that tax optional. The next step is for the community to validate these claims independently.
