Compressing Documents into LoRA Adapters

This is the first post in a series on whether we can extract everything knowable from a document and store it inside a language model - well enough that the model can answer questions about the document from memory. Training a model from scratch to do this would be expensive, slow, and might not even be necessary, so we are starting with a LoRA adapter: a simple post-training approach that fine-tunes a small set of extra weights on top of an existing base model. The claim we are testing is that, if this works, loading a document becomes loading a few megabytes of weights. The experiments are about finding the limits and practicalities of that approach: what the adapter can learn, what kind of supervision works, where it fails, and what would still be needed for a useful system.

The longer-term vision is a document-memory system built from these adapters. A small base model would answer factual subquestions with a per-document LoRA adapter applied on top, while a larger model would handle the user-facing workflow: decide what facts to ask for, call the relevant document memories, and synthesize the final response. The corpus would be a directory of adapter files: adding a document means training a new adapter, updating one means retraining, removing one means removing the adapter.

This is admittedly a wild idea. It is not obvious that a LoRA adapter can carry a document well enough to answer questions about it with correct citations, that the approach scales past a handful of documents, or that it is ever competitive with a well-tuned retrieval system. We do not know yet if any of this is practical - and that is exactly why it is worth investigating.

Before worrying about many documents or many adapters, we have to show the idea works on a single page. So we picked one page of one real document, trained a LoRA adapter on a small base model, and held it to a strict output contract: every answer is a tiny JSON object with an answer and a citation page. The rest of this post explains the experiments we ran to understand what kind of training signal actually builds that memory.

Compressing a Document into an Adapter

Compression is what this project is actually about. One useful way to think about the idea is that it sits at the intersection of three small literatures: memorization capacity, synthetic supervision, and parameter-efficient storage. The pieces from each are what make "fit a document into an adapter" a well-posed question rather than vibes.

The paper that makes this quantitatively discussable is Michael Morris et al.'s How Much Do Language Models Memorize?. They train hundreds of small transformers on controlled data, separate what a model has memorized from what it has generalized, and measure the capacity directly: a trained parameter carries on the order of 3.6 bits of its training data. So an N-parameter model has roughly 3.6N bits of storage to work with. The number is not a law of nature, but it gives a concrete unit for reasoning about whether a document might fit into a small trainable parameter budget at all - and it is what lets this project be framed as compression instead of just "fine-tuning on some questions."

The immediate follow-up literature mostly complicates that picture rather than overturning it. Generalizing over Memorized Data in LLMs argues that memorization and generalization are stages rather than alternatives: models first memorize their training data and, with continued training, generalize over what they have memorized. For document-memory experiments the implication is direct. The goal is not verbatim recall; it is answering many differently phrased questions about the same underlying material, which needs the generalization step on top of the memorization step. Memorization Dynamics in Knowledge Distillation for Language Models matters for a different reason: it studies how memorization behaves in teacher-student pipelines, which is structurally close to the setup here, where a larger model generates the supervision and a smaller model or adapter is the one actually doing the memorizing. Together those papers suggest the interesting question is less whether parameter memory exists and more what kind of training signal turns that memory into something queryable.

There is also a smaller set of papers that qualify the headline number itself. Benchmarking Unlearning in LLMs Using Fully Synthetic Data trains models on fully synthetic documents - where the full content of the training set is known up front - and in that setting reports an effective memorization density below Morris's 3.6 bits per parameter. The gap is a reminder that the headline figure depends on what the training data looks like and how the model is trained, not just on parameter count. Training Data Pruning Improves Memorization of Facts pushes on the same question from a different direction: instead of asking how much a model can memorize, it asks what changes when the training set is curated, and reports that pruning low-value examples substantially improves factual memorization. And SoK: The Landscape of Memorization in LLMs is the broad survey that places all of this in the wider memorization literature: privacy leakage, extraction attacks, duplication, fine-tuning effects, and the basic problem that different papers often mean different things by "memorization" in the first place. The current literature does not yet give a clean recipe for "documents as adapters," but it gives a plausible framing: parameter memory is real, its usable density is highly regime-dependent, and a big part of the open question is whether synthetic QA-style supervision can make that stored information reliable enough to substitute for retrieval in a fixed corpus.

That synthetic-supervision question is what the experiments below actually push on, and the framing there leans on Chan et al.'s Scaling Synthetic Data Creation with 1,000,000,000 Personas. They treat personas as "distributed carriers of world knowledge": if you ask a model enough questions from enough different perspectives, you can surface a surprising amount of what it already knows. This project takes that idea literally at the document level. Instead of training the adapter on the document text alone, we train it on many different ways a reader might ask about that text. If world knowledge can be surfaced from a model by asking it many different questions, the content of a document can be surfaced the same way.

The Setup

Before we get into the experiments, some basic setup. This section introduces the document, base model, adapter design, document key, and output contract used for the first set of experiments. Later sections discuss the training data and training process in more detail.

One page, one document

All experiments below focus on one page from one document. The document is the New York City Council's Response to the Fiscal 2026 Preliminary Budget and Fiscal 2025 Preliminary Mayor's Management Report, and the page is its executive summary, reproduced in full below. The plan was to use a single page as a proof of concept before training on the full document, which kept the scope and complexity manageable while we worked through the basic training approach.

Base model

The base model used was Qwen 2.5 1.5B Instruct. It was picked at the start of the project because it was one of the stronger small instruction-tuned models available at the time and was reported to follow structured-output prompts reliably. In keeping with the project vision, it is a small model, which keeps the adapter small as well.

The goal is to understand whether per-document memory can work on a small base model. If it only works on, say, a 70B model, the "swap a small adapter per document" framing loses most of its point: you are already loading a large model into memory, and the marginal cost of a few more megabytes of retrieval index is not obviously worse. The interesting investigation is whether a 1–2B model that already handles JSON output and instruction-following reliably can also act as a reliable per-document reader once an adapter is loaded. More recent Qwen releases (Qwen 2.5 3B, Qwen 3 1.7B, and their successors) might be worth trying as well, though none of the experiments in this post use them. The eventual dream is to push this down to an even smaller base model.

Adapter

On the parameter-efficient-storage side, the size of an adapter on disk is set by five independent knobs, each of which moves the adapter's size multiplicatively:

1. Base model width and depth. Hidden dim, intermediate dim, and number of transformer layers fix the per-layer projection sizes that any adapter has to match.
2. Number of layers adapted. You can train LoRA on every layer, the top half, attention-only layers, and so on.
3. Modules per layer. q alone, q + v, or all seven attention and MLP projections (q, k, v, o, gate, up, down).
4. LoRA rank r (or r² in the LoRA-XS family). Sets the bottleneck dimension inside each adapted projection.
5. Storage precision. Bits per parameter on disk - bf16, int8, int4, int2 - set by quantization.

The chart below shows how rank and module set play out across two representative base model sizes at bf16, holding the other knobs fixed.

Combining both sides of the ledger gives two separate requirements. An adapter with N parameters stored at B bits per parameter occupies N·B bits on disk. If the 3.6 bits per parameter estimate above applies here, those same N parameters could carry at most 3.6N bits of learned information. Under that assumption, compression imposes two conditions:

1. The learned representation of the document has to fit inside the adapter's capacity.
2. The adapter file has to be smaller than the source document; otherwise it is not compression in the usual storage sense.

The second condition is why storage precision matters. If the adapter is stored at more than 3.6 bits per parameter, the file is larger than the Morris-style capacity estimate for the same parameters. Under that estimate, compression only becomes arithmetically possible once quantization brings the stored bits per parameter below 3.6, and even then the learned document representation still has to fit.

There are two caveats. First, this assumes the 3.6 bits per parameter capacity reported by Morris is correct and applies to this use case. Second, compression depends on knowledge representation. The ideas and concepts in a document overlap in ways that are difficult to map out ahead of time; the more overlap there is, the more compression may be possible. Testing that gap between raw text, learned representation, and adapter size is part of the experiment.

Rather than walk through that abstractly, it is easier to fix a base model and look at two concrete configurations at opposite extremes of those knobs. For the Qwen 2.5 1.5B Instruct base we use in the experiments below, a maximal configuration maxes every knob for this model size - rank 64, all seven projections, bf16 - while a minimal configuration picks the parsimonious end of each - rank 8, only q and v, int4.

Both numbers come from plain LoRA arithmetic. A rank-r LoRA adapter on a linear layer with input dimension d_in and output dimension d_out adds r × (d_in + d_out) parameters - two low-rank factors stacked in series (Hu et al., 2021). Plugging in Qwen 2.5 1.5B's actual projection dimensions - 28 transformer layers, hidden dim 1536, intermediate dim 8960, and grouped-query attention with KV dim 256 (config.json) - gives the breakdown below.

per-layer LoRA params at rank r, all 7 projections:

  q_proj       r x (1536 + 1536)  =   3,072 r
  k_proj       r x (1536 +  256)  =   1,792 r
  v_proj       r x (1536 +  256)  =   1,792 r
  o_proj       r x (1536 + 1536)  =   3,072 r
  gate_proj    r x (1536 + 8960)  =  10,496 r
  up_proj      r x (1536 + 8960)  =  10,496 r
  down_proj    r x (8960 + 1536)  =  10,496 r
                                     ---------
  per layer                          41,216 r
  x 28 layers                     1,154,048 r

MAXIMAL   r = 64, all 7 projections, bf16

  params       64 x 1,154,048            =  73,859,072
  on disk      73,859,072 x 16 bits / 8  =  147.7 MB
  Morris cap   73,859,072 x  3.6 bits /8 =   33.2 MB

MINIMAL   r = 8, q + v only, int4

  per layer (q + v only)                 =       4,864 r
  x 28 layers                            =     136,192 r
  params        8 x 136,192              =   1,089,536
  on disk      1,089,536 x  4 bits / 8   =   0.54 MB
  Morris cap   1,089,536 x  3.6 bits / 8 =   0.49 MB

Taken on its own, this picture is not a compression win. The maximal adapter is on the order of 500x the size of a typical few-hundred-KB document, and even its 3.6 bits per parameter capacity is about 100x more storage than such a document would need. The minimal adapter lands within a factor of two on both disk and capacity, but that is also the configuration with the least room for an optimizer to fit a document and a strict JSON contract.

Those numbers are the ceilings the arithmetic implies, not what is known to hold for a trained per-document adapter. The Morris figure is itself regime-dependent, and the gap between an adapter file and the information it actually carries for a specific document has not been measured end-to-end. Part of the reason to run this at all is to test the compression claim empirically: the mechanism might fall short, or it might close that gap in a way that adds something small to the memorization literature. We are moving forward anyway because the question on the table right now is not whether this is efficient, it is whether the mechanism works at all.

We deliberately start at the maximal end. The priority is finding out whether a LoRA adapter can learn enough document knowledge to answer questions with a strict citation contract - not minimizing bytes. Maxing rank, adapting all seven projections, and training in bf16 gives the optimizer as much room as possible to fit the document and the JSON contract, so if the idea fails here it likely fails everywhere. Only after that is established does it make sense to test smaller adapter configurations. That size-reduction work is for a later post.

In concrete terms, that maximal configuration means LoRA rank 64, α = 128, no dropout, applied to all seven projections in every transformer layer (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), trained and stored in bf16. That lands at the 147.7 MB / 33.2 MB Morris-capacity point on the scenarios chart – roughly 500x the raw page on disk.

Doc key and output contract

Every training and evaluation example carries the same system prompt:

The tag [MEM:NYC-Prelim-2026] is the document key, and every single training and evaluation example is tagged with it. The adapter never sees a question without that tag attached, so it learns the association between the key and the facts from page 4 directly: this string points at this body of knowledge. In the larger vision from the opening, the orchestrator uses a key like this to decide which adapter to hot-swap onto the base model before asking a question, and the key then tells the loaded adapter which document the user is actually asking about. Here it is always the same key because there is only one adapter, but that association still has to be learned during training.

The model is required to answer with a JSON object with two fields:

a is the answer to the question. c is the citation: the page number backing the answer. For every example in this post c is 4 because there is only one page in play. Even with a single page the citation matters: it tells the reader exactly which page to go to to verify an answer, and it makes accuracy measurement and debugging much easier once training spans more than a single page (and by extension more than one document). The strict JSON-and-citation format is what makes behavior measurable downstream: an answer that is not parseable JSON or that cites the wrong page is wrong, regardless of how reasonable the prose looks.

Sentence, paragraph, and page levels

Training data was generated by GPT-5.2 in a two-stage pipeline, run once per level. In the first stage the model was given a chunk of source text from page 4 – a single sentence, a single paragraph, or the entire page – and asked to extract atomic facts with answers normalized to the provided canonical format. For paragraph- and page-level extraction the prompt asked for at least 60 distinct facts when the chunk supported that level of coverage; sentence-level extraction had no minimum and simply asked for every fact genuinely present in the sentence.

In the second stage the model was given those facts and asked to write at least three distinct question phrasings per fact, each of which had to resolve to the same canonical answer.

The three levels correspond to different kinds of knowledge in a document. A dollar amount is a sentence-level fact. A comparison between two numbers stated in different sentences of the same paragraph is a paragraph-level fact. A trend that only surfaces when you look at the entire page is a page-level fact. Compressing the page into an adapter means teaching it all three. The natural next level is cross-page facts – knowledge that spans more than one page of the document – but the experiments in this post are single-page, and those are reserved for a follow-up.

Sentence level

Facts that live inside a single sentence on page 4. Atomic values, named entities, dates, and simple relations between terms that co-occur in the same sentence.

Paragraph level

Facts that require combining information across more than one sentence in the same paragraph. This includes resolved references, where a phrase like "this increase" has to be tied back to the earlier sentence that describes what actually increased, together with causal chains between events, comparisons and differences between quantities, rankings, and trends across the paragraph. Single-sentence facts were explicitly excluded at this level in the extraction prompt, since they were already covered by the sentence-level pass.

Page level

Facts that require evidence from multiple paragraphs within the page. Summary quantities, cross-paragraph trends, and relationships between different sections of the page that no single paragraph states on its own.

Question types

Within each level, questions are written four ways:

• Standard QA — a direct question with a short factual answer drawn from the page. This is the headline metric: it is the closest match to how an orchestrator model would actually query a per-document adapter at serving time, and to how a user would ask a question if the model were used directly.
• Cloze — a sentence from the page with a span blanked out; the model fills in the blank.
• Fact correction — a plausible but wrong claim; the model has to reject it and supply the correct version.
• Multi-choice — a question with a small set of alternatives where the model must pick the one backed by the page.

Standard QA is the only type we actually expect to serve. The other three are in the training mix because we had a theory that training on several question formats over the same underlying facts would build a more complete internal picture of the page and, by doing so, improve Standard QA itself. That theory is tested directly in the atomic QA ablation further down, and the answer turns out to be yes. One real example of each, all at sentence level:

Canonical answer forms

The generated answers were not raw quotes from the page. Each answer was normalized to a specific format: fiscal years as "FY25", money as "$116.5 billion" with exactly one decimal place and a lowercase unit, percentages as "3.7%" with no space before the percent sign, named plans capitalized exactly ("Preliminary Plan", "November Plan"), and OCR artifacts like "re -estimates" corrected. The goal was to compress each fact into a single canonical answer string so that many different question phrasings could resolve to the exact same answer. A standardized answer format also allowed for more accurate exact-match grading.

This made the first training target narrow and measurable. The initial question-answer data focused on atomic facts with short, standardized answers. That let us test whether the adapter could learn facts from the page and return them in a consistent format before moving on to more natural questions whose answers are sentences or short paragraphs.

Training loop

Every full training run discussed in the rest of this article trains the adapter from the same base-model checkpoint for up to 12 epochs at a constant learning rate of 9.5e-6 with 10% warmup. Effective batch size is 4 (per-device 2 with gradient accumulation 2), max sequence length 512, gradient checkpointing on, grad-norm clipped at 0.3, evaluation at the end of every epoch, and the best checkpoint by validation loss is kept.

Those numbers came out of two small Weights & Biases sweeps run early in the project. Both sweeps scored runs on validation accuracy – the fraction of a few hundred validation questions answered verbatim. They used sentence- and paragraph-level data only; page-level supervision was added in the first full run below.

The first sweep was a Bayesian search over the full space – learning rate (log-uniform, 2e-6 to 1e-5), LoRA rank, α, dropout, warmup ratio, epochs, and gradient accumulation – with Hyperband early termination killing badly tracking runs after two epochs. It converged on rank 64, α 128, no dropout, warmup 0.10, and gradient accumulation 2.

The second sweep froze those winners and ran a grid over the remaining knobs – learning rate (7e-6 vs 9.5e-6), epochs (12, 14, 16), and a training-data fraction (50%, 75%, 100%) to see how sensitive accuracy was to data volume. The best operating point was 9.5e-6 at 12 epochs on 100% of the training data, which is what every experiment below uses.

Detailed sweep accuracy figures

The table below breaks the sweep out by the levels and question types just introduced. For each metric, Worst and Best are the lowest and highest values any single configuration in the top ten reached at its best checkpoint. Configurations in sweep 2 varied in learning rate, number of training epochs, and fraction of training data, so each row is a comparison across different configurations, not across training steps of one run.

A few things were obvious even at this stage. Sentence-level accuracy sat 20–25 points above paragraph-level across the entire top of the sweep – the best run reached 77.8% on sentence-level questions and only 55.6% on paragraph-level, and no configuration closed that gap.

Standard QA – the most important of the four question types, as discussed above – topped out at 84.7%. That is high enough to take the approach seriously, but well short of a production system; an 85% answerer is wrong on roughly one question in seven. Closing that gap is most of what the rest of this post is about.

Cloze scored 65–83% across sweep configurations. Multi-choice sat above 90% on every configuration – it is also the narrowest task, selecting the best answer from a few given options, so its high accuracy says less about whether the page has really been memorized than Standard QA or Cloze do. Fact correction never cleared 17%, and no combination of learning rate, epochs, or training-data fraction moved it much.

With the training configuration fixed, the runs below focus on different dataset sizes and mixtures.

The First Full Run

With the sweeps done and a settled training recipe, the next step was a full training run evaluated on a held-out test set.

Training, validation, and testing all used a mix of sentence-, paragraph-, and page-level question-answer pairs, with the adapter trained using the settings from the winning sweep configuration described above.

The split

Across all four question types and three levels (page, paragraph, sentence) page 4 yielded 3,506 question-answer pairs. As a reminder, these questions were generated by passing a sentence, paragraph, or the full page to GPT-5.2 using the two-stage approach described above, which produced questions answerable from the associated passage.

We chose a validation, training, testing split of 80/10/10 which resulted in 2,804 train / 351 validation / 351 test question-answer pairs. Every combination of level and question type appears in the test set in roughly the same proportion as in the training set, with one exception: there is no paragraph- or page-level multi-choice. That was an oversight – the paragraph- and page-level multi-choice generators were added just as the page-4 data for this run was being produced, and the data generation job was not re-run against them afterwards. The effect on this experiment is minimal, and later experiments in the project do include multi-choice at all three levels.

The table below shows the number of question-answer pairs in each split for each combination of level and question type.

Results

The maximal adapter, trained on all 2,804 training examples with the fixed loop, answers 58.1% of them correctly (204 / 351) under exact-match grading. Broken out by question type and level:

Four things come out of this first run.

Training works. Training a small model on question-answer pairs about one page is enough to teach it that page well enough to answer held-out questions about it later. Across the full held-out test set the adapter reaches 58.1% exact match, which is enough to validate the basic premise. On Standard QA — the form that matters most because it is closest to how this system would actually be used in practice — it reaches 82.1%, which is very good for a first experiment. From here the work is about refining the approach and pushing that performance higher.

Standard QA is good, but not good enough. Although 82.1% exact match on Standard QA is a strong result for a first pass, it is still not accurate enough for a production rollout. A document QA system needs near 100% reliablity. We rolled out numerous experiments to improve this number, and the next section will explore some of the key findings.

Sentence-level question-answering performed best. Every sentence-level type except fact correction cleared 90% exact match. Paragraph- and page-level accuracy dropped substantially on every type: getting the adapter to answer a single-sentence fact was very different from getting it to answer a question that requires cross-sentence integration.

Fact correction was the most challenging question type. 28.6% at sentence, 8.8% at paragraph, 5.0% at page — every row's worst type by a wide margin. The model is much better at producing a memorized span on request than at rejecting a plausible near-miss and replacing it with the correct fact.

Training on the document text before question answering

The first full run trained on question-answer pairs. One question we had was whether it helped to add a text-training step first. This involved training the model on ordinary next-token prediction using the document as the training source. The question was whether this standard training method would improve the later question-answer training.

We tested four setups: page 4 text or full-document text, each presented either as plain source text or inside the same chat format used by the question-answer pairs.

The exact-match accuracy score was zero across all four setups. This was expected. The model had not been trained on the JSON output format or the canonical answer forms that exact match expects. For this reason, we used an LLM as a judge to evaluate each response. We passed in the model-generated answer along with the gold answer and asked the judge to return true or false based on whether there was a semantic match. The judge was instructed not to return true if the gold answer merely appeared somewhere in the model output but the response did not directly answer the question. In this way, next-token prediction outputs that happened to contain the answer were still penalized.

Page-4 accuracy was higher than full-document accuracy in both text presentations. This was somewhat surprising because page 4 is the executive summary of the document. All facts summarized on page 4 appear in more detail elsewhere in the document. The initial hypothesis was that training on the whole document might give the model more supporting context and improve accuracy. It did not in this run. One possible explanation is that full-document training spread the adapter's capacity across far more text, while the page-4 run concentrated its updates on the page the validation questions were about. That is worth testing again once the question-answer training moves beyond page 4, but for this experiment the page-only text run was better.

We also ran one additional setup with the document tag [MEM:NYC-Prelim-2026] in the page-4, chat-formatted setup. As described above, question-answer pair training uses this tag as the trigger for the model to retrieve facts about this particular document from memory. This resulted in 9 / 351 = 2.6% judged validation accuracy. When the tag was present during next-token prediction training, the recall trigger backfired. The model often treated it as a cue to repeat verbatim parts of the document instead of giving a targeted answer.

From there, we took the best-performing text setup — page 4 source-text training, without the document tag in the text-training data — and trained question-answer pairs on top of it. That gave us the practical comparison: does a model trained first on page text and then on question-answer pairs outperform a model trained only on the question-answer pairs?

It did not. The model first trained on page-4 text and then trained on question-answer pairs was slightly worse on both exact match and LLM-judged accuracy than the model trained only on question-answer pairs. Since text training added time and compute without improving the final accuracy, the remaining experiments stayed focused on question-answer pair training only.

Question-answer ablations

To improve question-answer accuracy, we ran three ablations on the original page-4 split. The goal was to understand which training conditions mattered before scaling the dataset further.

The three ablations changed one thing at a time: the number of question-answer pairs, the question types included in training, and the sentence/paragraph/page level of the questions.

Count ablation

The count ablation trained the same model on increasingly large subsets of the page-4 question-answer dataset. Training on more questions continued to improve accuracy, with no flattening in this range. Moving from 100 examples to the full 2,804-example training split changed validation accuracy from 12.5% to 57.8%.

On Standard QA, the question format closest to how the system would actually be used, validation accuracy rose from 12.5% to 82.1%. That result motivated the larger dataset extension later in the project: if the curve was still moving, it was worth asking whether it would keep moving with many more question-answer pairs.

Question-type ablation

The question-type ablation changed which kinds of question-answer pairs were included during training. The matrix below shows exact match on the held-out test rows for each available question type. The n/a cells mark question types that were not included in that run's test split. Because Standard QA is the main use case, it was not removed in the upper half of the ablation; there is no "All except Standard QA" run.

Focusing again on Standard QA, training only on that question type reached 65% exact match. Training on the full mix reached 82%, and the best mix reached 87%. That is roughly a 20 percentage-point gain, which made the case for training on a fuller mix of question types.

The highest Standard QA accuracy came from the "All except multi-choice" run at 87%, compared with 82% for the full mix. Both are measured on 112 Standard QA test questions: 97 correct versus 92 correct. Since the absolute gap in correctly answered questions was small, the later runs continued to use all question types. The broad, directional signal from this ablation was that training on more question types supported higher Standard QA accuracy.

Sentence, paragraph, and page ablation

The sentence, paragraph, and page ablation asked how training on one question level affected accuracy at each evaluation level. Does training only on sentence-level questions help paragraph-level accuracy? Does page-level training help sentence-level accuracy?

Sentence-only training performed best on sentence-test questions, but much worse on paragraph and page tests. Paragraph-only training performed best on paragraph questions, but was weaker on sentence and page questions. Page-only training actually performed better on sentence and paragraph tests than on page tests.

The main takeaway is straightforward: training across all levels produced the highest performance across the board, so the remaining page-4 experiments used sentence, paragraph, and page examples together.

Best training setup so far

At this stage the best default was clear: do not train on the document text first, use all available question-answer pairs, keep all question types, and train on sentence-, paragraph-, and page-level questions together. That setup became the basis for extending the dataset and testing whether the count curve would keep improving with more question-answer pairs.

Extending the Question Dataset

The first count ablation did not flatten, so the next step was to extend the page-4 dataset and push the same setup much harder. The recipe stayed the same: no text-only pre-training first, all question types, all question levels, and as many question-answer pairs as were available. What changed was scale.

The extension split grew to 15,043 training examples, with 1,880 validation examples and 1,880 held-out test examples. The page-4 source text is only 710 words long, so the full training split works out to about 21 question-answer pairs per source word. The full count sweep took one week to run locally on an Nvidia RTX 3090.

Performance continued to improve as the training split grew, with smaller gains toward the end. Standard QA exact-match validation accuracy rose from 12.0% at 100 training examples to 84.1% at the full 15,043-example training split.

The final full-count model was then evaluated on the held-out test split with both exact match and LLM judging. It reached 86.6% exact match on Standard QA. With LLM judging, which credits answers that are semantically correct but not exact string matches, Standard QA accuracy reached 93.8%.

The First User-Style Questions

The simple question-answer runs showed that the adapter could learn to answer short factual questions about the page. These questions are useful when there is a need to know a single fact that has a short, direct answer. For example: "What is the proposed Fiscal 2026 budget?" "$114.5 billion."

But some questions are more complex. So the next step was to test whether simple-question training generalized to complex questions: would training on short factual questions let the model answer broader questions about the same document? To check, we asked GPT-5.2 to produce questions about the page as if it were a user. Those questions often asked for several related facts, an explanation, or a comparison rather than a single short answer.

Here is one example. GPT-5.2 asked:

What are the main reasons the Council says the City still faces fiscal risk even though the Preliminary Plan looks more optimistic?

When the simpler adapter was asked the factual pieces of that question, its answers came back as thin fragments rather than the evidence needed for a complete response.

The simple-question adapter had seen the underlying facts many times. The question was whether that training would generalize to broader questions that required combining those facts into longer answers. It did not. On the 444-question complex test set, the simple-question adapter scored 0% exact match and only 2.7% with LLM judging.

The important point was that the GPT-5.2 generated questions were useful: the kind of questions someone might reasonably have about the NYC Preliminary Budget Plan. The new challenge was to make the adapter answer questions that required a more complex and nuanced response. The next experiment therefore shifted the training approach toward training directly on complex questions.

Training on Complex Questions

That result set the next training target. Simple question-answer training was enough to teach many page-4 facts, but it did not transfer reliably to complex questions. To train that behavior directly, we generated a new dataset of complex question-answer pairs and trained the adapter on questions that required synthesis, comparison, explanation, and multi-fact answers.

The complex-question prompt was intentionally different from the earlier simple question-answer prompts. Instead of asking for atomic facts as in the simple-question generation prompt, it asked GPT-5.2 to write useful, user-facing questions that often required combining multiple facts from the page. It also asked for a category tag for each question, one of: comparison, cause/effect, tradeoff, timeline, trend, negation, summary, or multi-fact explanation.

Here are three examples from the generated complex-question dataset. Each includes the category tag, the question, and the generated answer used for training.

The first complex-question dataset produced 4,445 raw question-answer pairs. After deduplication, 4,440 remained and were split into 3,552 training examples, 444 validation examples, and 444 held-out test examples. Every example was page-level.

Because of the complexity and length of the answers in this training run, it was unlikely that the model response would match the gold label answer word for word. For this reason, LLM judging was used so semantically correct answers could be credited even when the wording differed.

The result was encouraging: complex-question training worked, and starting from the simple-question adapter caused a small improvement in accuracy. This showed that the adapter could learn a more complex answer style when the training examples elicited that style.

Complex Training Caused Forgetting on Simple Questions

This first complex run also exposed a problem. The model that started from the simple-question adapter and then trained on the 3,552 complex examples was better at complex answers, but it no longer preserved the earlier Standard QA behavior. To check that, we reran the 112 held-out Standard QA questions from the first ablation split.

Compared with the earlier simple-question adapter on the same set of 112 Standard QA questions, exact match dropped 66 percentage points, from 82.1% to 16.1%; LLM-judged accuracy dropped 29 percentage points, from 93.8% to 65.2%. Complex training made the model better at answering complex questions, but it did not reliably preserve the simpler question-answer behavior the earlier adapter had learned.

This drop in accuracy meant that for optimal performance on both simple and complex questions we could not just train on simple questions and then switch to complex ones. The model needed a training mixture which preserved both kinds of examples deeper into the training. However, before testing that mixed setup, we first aimed to increase the accuracy of complex question answering to similar levels we had reached with simple question answering. To do so we applied the lesson learned earlier with simple question training: many more training examples continued to improve performance, so we expanded the complex-question dataset.

Extending the Complex Question Set

The simple-question extension had already shown that more questions improved performance, so we applied the same idea to the complex questions. The first complex run was a proof of concept: it showed that the model could learn the task, but it was still much smaller than the extended simple-question dataset. The next step was to preserve the original generated dataset, add many new GPT-5.2 generations, deduplicate the combined dataset, and create a larger train, validation, and test split.

The extension generated 17,317 raw complex question-answer pairs. After deduplication, 17,228 remained. The split contained 13,782 training examples, 1,723 validation examples, and 1,723 held-out test examples. Like the first complex dataset, every example was page-level, but the distribution was much larger across each question type.

We did not train any models on the extended complex question dataset as a standalone experiment. Having seen that training on the smaller complex question dataset caused forgetting of simple questions in our existing dataset mix, we instead moved directly to improving the training mixture of simple and complex questions. The goal was to train a model that had strong performance across both types of questions.

Training on Simple and Complex Questions Together

To train the model on both simple and complex questions we used a 50/50 mix. We took all 13,782 complex training rows from the extension, sampled 13,782 simple rows from the extended simple-question training set, shuffled each side with a fixed seed, and then alternated them through training: one simple example, one complex example, repeating across the full dataset. This resulted in 27,564 total training examples.

We trained two versions with this same mixed dataset. The first version continued from the best available adapter that had already been trained on the full simple-question dataset and then continued on the first complex-question dataset. The second version trained from the base model.

For the simple-question side, the final evaluation used the Standard QA-only slice of the extended simple test split: 515 questions out of the full 1,880-example simple test set. The complex evaluation used the full 1,723-question complex test split.

The two versions were close, but the tradeoff moved in opposite directions. Continuing from the already-trained adapter preserved more simple Standard QA accuracy. Training the same mix from the base model gave the higher complex-question accuracy. In both cases, the mixed dataset solved the main problem: the model could answer complex questions without collapsing on the simpler Standard QA format.

Dataset Sizes Across the Runs

The experiments used several related datasets as the work moved from simple questions to complex questions and then to a mixed training set. The rows below separate full train / validation / test splits from the smaller held-out subsets used for specific Standard QA evaluations.

Ablation sweep

Simple questions

Complex questions

Mixed questions

Testing the New Model

We ran two short checks after training on the mixed dataset. First, we asked a few ad hoc human-written questions to the model with the mixed adapter loaded to make sure simple and moderately complex answers were working. Then we used a second set of GPT-5.2-generated user questions.

The second check used five GPT-5.2-generated user questions. The examples below show each question, the answer from the model with the mixed adapter loaded, and our assessment.

This concludes the first post. Starting from one page, we trained a small-model adapter that can answer a wide variety of questions about the executive summary with high accuracy. That is a strong proof of concept, but it is still an expensive way to train on a single page: the final mixed run used 27,564 training examples. The next step is to push accuracy higher while making the training process more efficient.