Search documents or fine-tune a model: what each is good for

The first is document retrieval (retrieval augmented generation, or RAG). Here the model itself is left untouched. For every question, matching passages are pulled from a stored document collection and handed to the model along with the question, and it answers on that basis. The term goes back to a 2020 paper that combined knowledge held inside a pre-trained model with a searchable body of text alongside it (Lewis et al., 2020).

The second is training a model on your own material (fine-tuning). Here a finished model is trained further on company documents and changes its internal values, known as weights (Microsoft Learn).

The usual misconception is that you have to pick one of the two. In fact they answer different questions. Microsoft draws the dividing line in its developer documentation as follows: training on your own material suits stable content and a clearly defined task, while document retrieval suits changing content and broad topic coverage (Microsoft Learn).

Training on your own material shifts how the model phrases things, how it structures them and how it weighs them. It picks up the technical vocabulary of a company, the customary structure of a written assessment, the abbreviations that are taken for granted internally, and the way questions are answered in that organisation. Microsoft lists as a benefit that this approach helps the model understand the language, style and terminology of a new domain (Microsoft Learn). OpenAI describes the same effect from the practitioner's side: you train a model to produce output reliably in a given format and to correct recurring instruction-following failures (OpenAI, Model optimization).

What matters is what does not happen: the model does not memorise sentences, it learns patterns. Afterwards it writes an assessment the way your company writes assessments, including for a case that appeared in none of the training documents.

State current individual facts. A new price, an amended contract status, yesterday's figure: knowledge of that kind is not held in the weights if it came into existence after training.

Older knowledge comes with a condition too: a fact that appears only once in the material leaves a faint trace. A 2024 comparative study concludes that language models reliably absorb new facts through training only when they meet the same information in many different phrasings (Ovadia et al., 2024). Copying a document once is therefore not enough. Each document has to yield many grounded examples, and that is exactly how IonKon's training run is built.

Then there is upkeep. Whatever changes continually belongs in retrieval, not in the weights; that is why Microsoft assigns fine-tuning to stable content (Microsoft Learn). When the stable body of knowledge grows, the addition comes in through a further training run; for that we ship the training adapter with every model.

Look things up. Documents are split into sections, converted into a searchable numerical form and stored in a database. For each question the matching sections are selected and passed on to the model (Microsoft Learn).

The practical advantage is maintainability. AWS puts it briefly: document retrieval extends a model to an organisation's internal knowledge base without the model having to be retrained (AWS). New documents in, outdated ones out, and the answers change with them. The same source names two further points: the approach costs less than retraining, and the output can carry source citations, so a statement can be traced back to the originating document.

It does not know your organisation. It searches text without having learned it. The use case, the requirements, the working methods and the language of your company remain foreign to it. It supplies passages that match the question and leaves everything else to the model it is feeding. If that model is a general off-the-shelf one, the answer will sound like it.

Second, it only helps when the answer is written down somewhere. For questions about judgement, procedure or phrasing there is no passage to retrieve.

An objection suggests itself here: if the model is supposed to master a procedure, why not simply hand it the written procedure, through retrieval or in the prompt? The objection is half right. The description can be handed over. The practice cannot.

An example: a company has worked with the same in-house interface for twenty years. Retrieval can deliver the documentation for it. What it does not deliver is the experience with it. Which call makes sense in which order, how a special case is handled, what an error usually comes down to even though the message points somewhere else: none of that is written down anywhere, because nobody ever wrote it down. It comes out of many cases, and many cases are also what the training examples are made of.

On top of that, retrieval needs a question. Procedural knowledge is not asked about, it is applied. Nobody types “how do we handle a complaint from abroad”, they put the complaint on the table and expect the usual handling. For retrieval to help here, somebody would have to know in advance which document is needed and remember to attach it. That is exactly the step that gets skipped in daily work.

And the route through the prompt is not free. Anything handed over again in every conversation takes up context, adds to the response time and is gone once the conversation ends. A long context is also not used evenly: a 2023 study shows that language models make far better use of information at the beginning and the end of a long input than of information in the middle (Liu et al., 2023). If you want a way of working to hold permanently, it belongs in the weights rather than in every single request.

The dividing line therefore does not run between two kinds of knowledge, but between two kinds of task. Retrieval answers questions. The model acts in situations. Factual knowledge is often the smaller part of daily work; what matters more is that the result is right without anyone having to explain first how it should come about.

These questions point to document retrieval:

• "What does contract 4711 say about the notice period?"
• "What is the price of item X on the current list?"
• "What did the supplier commit to in Tuesday's minutes?"
• "Which test standard applies since the June update?"

Common feature: the answer is a single fact, it sits in exactly one document, and it may change next week.

These questions point to the trained model:

• "Draft the rejection letter to an applicant the way we do it here."
• "Summarise this inspection report in our usual structure."
• "Explain to a new colleague how a complaint is handled here."
• "Assess this enquiry against our criteria and give a short justification."

Common feature: there is no passage to retrieve. What is being asked for is working method, tone or technical language, and the answer has to hold up in cases that have never come up in that form.

These questions need both:

• "Write customer Meier a statement about the delay on order 4711."
• "Prepare the quotation enquiry from company Y the way we usually do."

Structure, tone and evaluation criteria come from the model; the order status and the figures come from retrieval. That is the normal case in daily work, which is why the combination is the usual answer: stable long-term knowledge in the weights, maintainable factual knowledge in retrieval.

You do not need to buy a second product for retrieval. AnythingLLM, the interface through which your team works with the model in the browser, is a self-hosted application built to run entirely offline (AnythingLLM). Document retrieval there is not an add-on but the core: you create workspaces, and each workspace has its own document collection and its own user assignment (All features).

A workspace for sales therefore sees different material from one for the legal department. A single file can also be attached to one conversation without adding it permanently; whether the model draws on documents at all is controlled per workspace through the chat mode (Chat modes).

No separate search infrastructure is required for this. By default the vector store is LanceDB on the same machine; the documentation states explicitly that document text and embeddings never leave the application (LanceDB). The embedding model that turns text into searchable vectors is built in as well and runs without further setup (Embedding models). Both components can be swapped for external services where an installation calls for it. For the use case at hand, the opposite is the point: no service reads along, because none is involved.

IonKon supplies the part that does not come as an add-on: the trained model. A company's approved documents become its own language model through LoRA fine-tuning. The method freezes the pre-trained weights and trains only small additional matrices, which sharply reduces the number of trained values without loss of quality (Hu et al., 2021).

There are two sizes: M with 9 billion parameters based on Qwen3.5-9B at 25,000 euros net, one-off, and L with 27 billion parameters based on Qwen3.8-27B at 60,000 euros net, one-off (IonKon pricing). You upload your material as a ZIP file, up to 8 GB on tier M and up to 16 GB on tier L. The model then runs locally on your own hardware. Processing takes place in the EU and Switzerland, and uploaded documents are deleted no later than 30 days after delivery (IonKon statement of services).

Document retrieval comes with the supplied interface at no extra purchase and runs on the same hardware. So you are not choosing between two products: you get the building block that cannot be retrofitted, and you retrofit the other one yourself, whenever and as often as you like.