“Where was that again?” Employees searching for internal documents often ask this question. Every day, companies generate vast amounts of data, including contracts, minutes, internal policies, presentations, wiki pages and much more. These are often stored in different locations and can only be found after a considerable amount of searching. Even when the document you’re looking for is available, it can be a challenge to quickly grasp the content it describes.
Searches take time and can quickly lead to frustration.
With the help of LLMs (Large Language Models), it is possible to take search to a whole new level. LLMs fall within the field of Generative Artificial Intelligence (GenAI) and focus specifically on the processing of language and text. These language models offer an innovative approach to optimising internal knowledge management within organisations.
In this article, we want to focus in particular on Retrieval Augmented Generation (RAG). RAG is an increasingly popular method in which LLMs are fed with in-house knowledge to produce accurate results. Below, we discuss the advantages and disadvantages of RAG and compare it with an alternative method: fine-tuning.
Contents
- Fine-tuning – a labour-intensive standard?
- What is RAG and how does it work?
- RAG vs. fine-tuning
- Summary
Finetuning – a labour-intensive standard?
Let’s start with the better-known method: in fine-tuning, existing models are adapted for specific use cases and equipped with domain-specific knowledge. The model is retrained using specially prepared data and learns existing patterns and information. The user can then send queries to the ‘fine-tuned LLM’ and receive responses generated using knowledge from the domain-specific dataset.
Figure 1: Fine-tuning
Until now, fine-tuning has been the standard method for training an LLM so that it can, for example, respond to company-specific questions. However, the method harbours a number of unpleasant pitfalls. It can, in fact, result in an unreasonably high expenditure of costs, time and resources. This is primarily due to the data preparation required for training, as additional data must be aggregated, cleaned and labelled. Only then can further training take place, which in turn consumes significant resources.
During fine-tuning, the model can only access the knowledge that was available at the time of training. With this approach, more recent data cannot be taken into account without retraining the model. A further disadvantage arises from a lack of good training data: whilst the model learns patterns from the data and can derive information from them, it applies this information stochastically – which can lead to imprecise results.
An alternative with a more reliable mode of operation is required, and this brings us to the RAG approach.
What is RAG and how does it work?
In Retrieval Augmented Generation (RAG), LLMs are fed information from external knowledge databases. The difference compared to fine-tuning is that the underlying LLM is not retrained. When generating a response, RAG draws on existing knowledge in the form of documents and incorporates this knowledge into the existing context. This enables the generation of precise, up-to-date and contextually relevant text.
Figure 2: Retrieval Augmented Generation (RAG)
To better understand how RAG works, let’s look at the method using a typical question as an example:
“How do I complete a travel expense claim?”
There are two key steps involved in RAG:
- Semantic search for relevant documents in response to a query to the LLM
- Generating a response
Both the query and the response are in natural language.
Step 1: Semantic search
The fundamental tool of RAG, and of semantic search in particular, is what is known as a vector database. A vector database specialises in indexing and efficiently querying vector embeddings of words, sentences and documents in natural language (also referred to as embeddings). Embeddings are simply the representation of natural language as vectors that can be arithmetically manipulated. These vectors are generated by pre-trained embedding models and can capture the semantic meaning of language in numerical form. The advantage of this is that it allows texts to be compared with one another to identify which texts are semantically similar.
To better understand how this works, let’s look at the example of a travel expense claim. The prerequisites for semantic search are, firstly, that documents containing the relevant information on the matter exist, and secondly, that they are stored in vectorised form in a database and can be retrieved. In a semantic search, the search query in natural language is converted into vectors, and the documents relevant to the query are retrieved from the vector database. This is the case, for example, when the content of the user’s query and the document being searched for are similar – in our example, therefore, when both deal with travel expenses. Thanks to embeddings, this similarity can be calculated mathematically and the search process automated.
Step 2: Generating the response
Once one or more relevant documents relating to the user’s query have been found using semantic search, these are appended to the query as context. The LLM can then generate a response based on the user’s query and the context, drawing on the knowledge contained in the company’s internal wiki. So, in response to the question“How do I actually complete a travel expense claim?”, the LLM has all the relevant information on travel expense claims at its disposal and can, where appropriate, paraphrase this information and refer to the relevant documents.
RAG vs. fine-tuning
The main advantage of RAG is that it is more lightweight than fine-tuning.
- The LLM does not need to be trained, but can be used in its original form. One simply makes use of its ability to process natural language. This alone saves a great deal of time and money that would otherwise be spent on fine-tuning the LLM.
- It is much easier to keep a RAG system up to date with the latest knowledge. Instead of having to train an LLM with the new knowledge, one simply needs to incorporate the new knowledge into the vector space. The RAG system will then automatically access the latest information when the next user query is made.
However, a major disadvantage of RAG compared to fine-tuned LLMs is the size of the context passed to the LLM. For every query, the relevant knowledge must be appended from the vector database. Depending on the document size, the context can thus grow to a considerable size. This is particularly significant when accessing an LLM via a third-party API. Billing is usually based on a ‘pay-per-use’ principle and is proportional to the token length, i.e. the number of text fragments transmitted.
Summary
Compared to model fine-tuning, RAG offers a lightweight solution that can be used to efficiently and cost-effectively augment an LLM with up-to-date, domain-specific knowledge. RAG can be updated with new information without the need to retrain the LLM, and is therefore more resource-efficient. However, compared to model fine-tuning, it should be noted that, due to the way RAG works, the number of tokens per query is comparatively high. After all, the relevant knowledge must always be appended to the user’s query. With frequent and long-term use involving large contexts, this can lead to considerable additional costs per query.
RAG is the method of choice when it comes to generating fact-based responses. Not only does RAG require fewer resources, but it also delivers more reliable results than a fine-tuned LLM. Fine-tuning should only be considered if the LLM’s behaviour needs to be adapted to specific requirements: for example, if the response behaviour needs to be tailored and/or specific linguistic codes are to be used.
Do you have any questions or need assistance?
You can find out more about the possibilities of generative AI for your business on our website.
