What a model learns from your documents, and what it does not

A language model acquires its base knowledge during pre-training on very large volumes of text. Subsequent fine-tuning on a company's documents transfers two things: form, that is, how answers are structured, technical terms, in-house naming, typical work steps and tone, and the company's stable expert knowledge, provided it appears in the training material often enough and in enough variations. That condition is the crux.

Gekhman et al. (EMNLP 2024) show that examples containing facts new to the model are learned considerably more slowly than examples consistent with what it already knows, and that the tendency to produce false statements rises with the share of such examples once the model finally takes them on. Ovadia et al. (2024) report that models do absorb new facts when they meet the same information in many different phrasings. In practice this means: a fact that appears once in the material leaves a faint trace; a training run has to generate many grounded examples from each document instead of copying it once. Whatever changes continually, such as prices or stock levels, belongs in document retrieval, not in the weights.

Memorisation is real and measurable. Carlini et al. (ICLR 2023) showed for a public model with 6 billion parameters that 33 per cent of the training passages selected for measurement could be elicited verbatim given 50 tokens of context, and 65 per cent given 450 tokens. Three factors drive the effect: model size (a tenfold increase raises memorisation by 19 percentage points), how often a text appears in the material, and the length of the prompt supplied in the query.

The frequency effect matters most in practice. Kandpal et al. (ICML 2022) measure that a sequence appearing ten times in the material is emitted roughly a thousand times more often than one appearing once. Building blocks that occur identically across dozens of company documents, meaning contract clauses, boilerplate and standard letters, are therefore the critical category, not the isolated special case.

Two further findings belong in any sober assessment. Nasr et al. (2023) show that aligning a model afterwards does not remove memorisation but merely conceals it, and that training data can be extracted from production systems at scale. Ippolito et al. (INLG 2023) built a filter that blocks verbatim reproduction completely and showed that it can be circumvented with slightly reworded prompts. On the current state of the art there is no guarantee against reproduction of training content.

This applies to cleaning before fine-tuning as well. Lukas et al. (IEEE S&P 2023) record that automated removal of personal data lowers the risk but does not eliminate it, because no detection method works without error.

There is a technical lever. Instead of recomputing all of a model's weights, only a small additional layer can be adapted (LoRA: fine-tuning that learns a few million additional parameters alongside the unchanged model). Wang and Li (2025) compared both routes: under full retraining, verbatim and paraphrased reproduction of training content rose above 50 per cent once texts appeared twenty times in the material, whereas with the narrow adaptation it stayed close to zero even at that level of repetition.

This evidence supports the direction but carries no guarantee. The measurements come from models of the GPT-2 family, well below present production sizes, and the authors state openly that a theoretical explanation is missing and that stronger extraction attacks could shift the result. Hou et al. (2025) measure on a model with 7 billion parameters that membership in the training set remains detectable after LoRA fine-tuning, with scores between 0.72 and 0.85 where 0.5 would correspond to guessing. The narrow adaptation therefore lowers the risk substantially; it does not remove it. Consistent with this, Biderman et al. (TMLR 2024) show that this route absorbs less of the new material than full retraining and in exchange preserves the base model's capabilities better.

Language models generate text that is probable, not text that has been verified. Huang et al. (ACM TOIS 2025) summarise the state of research: outputs are regularly plausible and nonetheless factually wrong. Kalai et al. (2025) trace this to the incentives in training and evaluation: guessing under uncertainty scores better on common benchmarks than stating the uncertainty, so models learn to guess. Xu et al. (2024) argue formally that this cannot be eliminated entirely. False statements are thus not an operating fault a supplier could switch off, but a property of the technology to be managed by organisational means.

What a model knows ends with the material it was computed on. Cheng et al. (2024) show in addition that the effective state of knowledge is often older than the stated cut-off date, because web collections carry along substantial amounts of older content. Vu et al. (2023) measure that models of any size fail on questions whose answer changes within a year, and that access to current sources at answer time closes this gap. A new price, a new contract status, a figure from yesterday are not in the weights.

Four measures are either evidenced or legally required. First, cleaning before fine-tuning: remove personal data and reduce duplicates, since repetition is the strongest driver of memorisation (Kandpal et al.). Second, document retrieval for facts: Shuster et al. (EMNLP 2021) demonstrate that looking up stored sources markedly reduces false statements, and Vu et al. show the same for questions about current events. Niu et al. (ACL 2024) note at the same time that even with retrieval, statements arise that contradict the stored sources; retrieval improves the position, it does not replace review. Third, human review wherever decisions concern people; Article 22 GDPR gives data subjects the right not to be subject to a solely automated decision with significant effect. Fourth, purpose limitation: such a system belongs in service as an internal assistance system, with a clear statement to staff that outputs are drafts.

IonKon fine-tunes two base models by LoRA, Qwen3.5-9B and Qwen3.8-27B, that is, by the route for which the measurements cited above show lower memorisation. Personal data is removed from the material automatically before fine-tuning. The documents yield training examples in many variations; the grounded facts among them can be traced back to their source passage. That is how the model learns the company's stable expert knowledge without copying documents. For current individual facts, document retrieval is the intended path, supplied with the AnythingLLM interface that ships with the model: you add and remove documents yourself, without new fine-tuning. Outputs carry a signed, machine-readable marking under Article 50(2) of the AI Act, so that machine-generated material remains identifiable. The system is intended as an internal assistance system, not as a basis for decisions about people. What is not stated here is a guarantee against verbatim reproduction or against false statements. The state of the art does not provide one, and anyone promising it is promising too much.