The integration of LLMs (Large Language Models) such as ChatGPT into modern business applications opens up new possibilities for human-centred interactions and automated workflows. By incorporating MCP (Model Context Protocols), LLMs can be specifically connected to internal data sources and APIs – for example, to access business data in real time or to trigger processes directly.
In this article, we demonstrate how ChatGPT’s capabilities can be seamlessly integrated into the Java/Spring ecosystem using Tool Calling and Spring AI, including secure connectivity via MCP.
Many companies are still faced with a patchwork of siloed solutions that make end-to-end automation and an intuitive user experience difficult to achieve. For the first time,generative AI makes it possible to holistically link processes and design them in a consistent manner. Examples of such projects can be found in customer service or in internal workflows such as holiday requests. Any organisation that fails to create an AI-compatible system landscape now risks growing integration problems and increasing customisation costs in the future. At the same time, users increasingly expect seamless support from intelligent assistants, the sort they have long been familiar with in their private lives.
Tool Calling describes the ability of a large language model (LLM) to control external tools and APIs using natural language.
For example, the sentence ‘Show me open orders’ is all it takes for the AI system to independently call the relevant endpoint.
Loose coupling of the individual ‘tools’ is crucial for successful business integration. This ensures flexibility and ease of extension.
The simplest way to implement tool calling is via MCP. MCP is an open standard developed by Anthropic that abstracts and standardises the connection between LLMs and internal data sources and services. Just as HTTP does for web applications, MCP provides a consistent communication framework and can be integrated flexibly, scalably and easily into existing systems.
In short: from a business perspective, standardisation using MCP and Spring AI enables:
These benefits make Tool Calling with MCP a key building block for the next generation of intelligent enterprise applications.
The technical integration follows a clearly structured architecture:
Spring AI abstracts the connection to various AI providers and provides a unified interface. This makes it easier to swap models and significantly reduces the integration effort. Azure OpenAI brings the power of OpenAI models into a scalable and secure cloud environment.
This section is aimed at those with a technical interest and developers. If you are primarily interested in the concept, you can skip this section – or refer back to it later as required.
You canfind a working example project at: https://gitlab.com/Daniel.stahr/spring-ai-azure-mcp. This will allow you to get started straight away, carry out initial tests and try out the technology in your own environment.
The example contains two Kotlin projects:
The setup is based on an Azure OpenAI instance with GPT-4.1 and utilises Spring AI. The README describes how to start the application with your own parameters.
BusinessServiceMCP
To enable the functionality of an MCP server, we need the following package:
Functions are annotated using @Tool and are thus automatically registered in the MCP context, similar to Swagger or OpenAPI.
In the following example, a function is provided to search for a dog by name:
Registration:
Configuration (abridged):
ChatClientMCP
On the client side, a connection is established to the Spring AI MCP Client WebFluxbusiness instance and the tool functionality is activated. All configuration is declarative; new services can be integrated by adding an entry to the YAML file.
Dependencies:
Configuration (abridged):
To make debugging easier, it is advisable to enablelogging for some packets:
All that remains now is to extend the ChatClient so that it automatically takes the tool specifications into account. After that, everything happens automatically:
The project includes another example in which a local tool definition is also registered. In the early stages, new functionalities can be tested quickly in this way.
The registration then looks as follows:
The example project demonstrates how communication between the user, the LLM, ClientChat and the business backend takes place via MCP, in a fully decoupled and standardised manner. The individual steps can be outlined as follows:
Important to note:
The entire process is not ‘hard-coded’. The LLM dynamically decides which API to call based on the tool descriptions. The actual execution takes place locally and is not carried out by the LLM itself, but by ClientChat via MCP. This enhances both security and control.
Extensibility through configuration
Another advantage: new backend applications can be seamlessly integrated without any code changes on the client side. All it takes is another entry in the YAML configuration:
This ensures the application remains flexible even as the system landscape grows or business requirements change.
The integration of generative AI into existing enterprise architectures brings not only opportunities but also a number of challenges. To ensure that the solution remains stable, secure and scalable, the following aspects should be taken into account:
Security
A key principle is that the chatbot should never communicate directly with the business backend. Instead, the MCP acts as an intermediary layer. This allows the existing security infrastructure of the business applications to be utilised. It is important that MCP requests are made on behalf of the user –for example, by passing the access token from the existing security context. This is the only way to ensure secure authentication and authorisation throughout the entire processing chain.
Logging and Monitoring
A robust logging and monitoring framework is essential. Every call initiated by the chatbot should be documented in a traceable manner – ideally including user context, timestamps, the request and the response. This not only facilitates fault analysis but also helps ensure compliance with regulatory requirements.
Error handling
The underlying AI model cannot interpret every query correctly. It is therefore important to handle error cases – such as unclear or ambiguous inputs – appropriately and provide the user with clear feedback. Ideally, this should include a request for further clarification. Even in the event of backend errors, responses should be designed to be user-friendly.
Scalability and maintainability
Particularly as the system landscape grows, it is advisable to version APIs and provide modular services. This allows new functionalities to be easily integrated and existing components to be updated independently of one another.
Key features of Spring AI and Azure OpenAI
When using Spring AI and Azure OpenAI, aspects such as API limits, token lengths and response formats should be taken into account. Spring AI helps to establish a consistent connection and abstract different LLM providers – a clear advantage in the event of future extensions or a change of provider.
Determinism – an underestimated factor for success
For productive tool calls via MCP, the LLM’s responseshould be as deterministicas possible crucial. Only if the model consistently generates the same function call under identical conditions can the workflow be reliably tested and operated. In practice, this means:
The combination of deterministic behaviour and flexible model response is a balancing act – but it is crucial for stable AI-supported process chains.
The successful integration of generative AI into existing business systems improvesprocesses, saves time and delivers a noticeably better user experience. Companies that embrace AI now will secure long-term competitive advantages.
The journey continues: in future, we can expect even more far-reaching automation and the integration of further business systems. With Spring AI, Azure OpenAI and MCP, taking this step will be easier than ever before.
Get started today and build your own AI assistant!
You can find out more about the possibilities of generative AI for your business on our website.