Insights into AI Security and the OWASP Top Ten for Machine Learning

In a world where artificial intelligence (AI) is becoming increasingly important, it is crucial to ensure the security and reliability of AI applications. The OWASP Top Ten for machine learning is a key resource for security in AI development. They highlight the key threats and countermeasures for AI systems. To illustrate these risks, we’ll delve into the importance of security aspects in automotive systems, inspired by the work of the Deutsches Museum in Bonn.

AI security: Why it matters now

The rapid development of AI in sectors such as healthcare, finance and the automotive industry is making security issues increasingly important. AI systems influence decisions that have a direct impact on our lives. This is not just about protecting internal and personal data, but also about ensuring that these systems are fair and free from bias. As AI becomes increasingly autonomous, the risk of errors that could have serious consequences rises. It is therefore important that companies actively ensure the security of their AI systems.

Modern AI models are often complex and difficult to understand, which can lead to unexpected errors. Legislation such as the GDPR or the proposed AI Act requires AI systems to be secure and compliant with regulations. AI security is therefore not only a technical issue, but also an ethical and legal one. Companies must plan their AI systems carefully and monitor them continuously in order to meet these requirements.

Avoiding pitfalls: The top ten security risks

Avoiding security risks in AI systems requires the consistent identification and management of potential threats. The OWASP Top Ten [1] for machine learning can help to identify common pitfalls and tackle them in a targeted manner. OWASP is a non-profit organisation dedicated to improving software security. The OWASP Top Ten is a regularly updated list of the ten most common security risks. It serves as a guide for prioritising the securing of applications against the most widespread threats. The following security risks are listed:

1. Input Manipulation Attack: In this attack, it is possible to cause the system to behave undesirably by deliberately manipulating the information processed by an AI system.
2. Data Poisoning Attack: In this attack, the training data of an AI model is deliberately tainted with incorrect information, leading to inaccurate or distorted outputs from the system.
3. Model Inversion Attack: This attack attempts to extract sensitive information about the original training data from the outputs of an AI model.
4. Membership Inference Attack: This attack aims to determine whether specific data was used in the training of an AI model, which can raise data privacy concerns.
5. Model Theft: This involves an attempt to steal the AI model itself, often to use the technology without permission or a licence.
6. AI Supply Chain Attacks: Attacks on the AI supply chain involve manipulating various components of AI development, such as algorithms or programme libraries, in order to cause harm.
7. Transfer Learning Attack: This attack exploits the commonly used technique known as transfer learning. In this process, base models designed for a specific use case are simply fine-tuned. Security vulnerabilities in the base models are then also present in the fine-tuned models and can be exploited.
8. Model Skewing: This involves manipulating the frequency of certain training examples within the training data – for example, including many positive examples and few negative ones – so that the model makes unrealistic predictions.
9. Output Integrity Attack: This involves attempting to manipulate the output of AI systems.
10. Model Poisoning: By deliberately manipulating a model’s parameters, the system’s behaviour can be altered.

The wide range of attack vectors described in the OWASP Top Ten makes the assessment of AI systems in practice comparatively complex. One way to make this complexity manageable is through early-stage threat analyses or comprehensive penetration tests of AI applications. This should be tackled jointly by a multidisciplinary team comprising security, AI and software engineering experts.

Security in automotive systems: lessons from the Deutsches Museum Bonn

A clear example is provided by Steffen Jendrny’s final-year dissertation, which was produced in collaboration with the Federal Office for Information Security (BSI) and which resulted in the creation of an exhibit for the Deutsches Museum Bonn. [2]

The study investigated how the security of driver assistance systems can be evaluated using so-called hardware-in-the-loop systems. By repeatedly presenting virtually manipulated road signs to a driver assistance system, it was possible to iteratively calculate the most successful attack possible.

The exhibit at the Deutsches Museum Bonn now demonstrates in an easily understandable way how attackers can manipulate AI systems by deliberately altering the input (see OWASP Top Ten – security risk ‘Input Manipulation Attack’ [3]). In this specific case, museum visitors can step into the role of the attacker, manipulate road signs and thus attempt to outwit the system. Using a sticker, a ‘Stop’ sign, for example, can be altered so that the system recognises it as a ‘Give Way’ sign. If adverse weather conditions or attackers were to succeed in altering the system’s classification in this way in reality, there would be a risk of serious accidents.

It is easy to see why, in practice, it is important for systems to be built securely and robustly from a technical perspective, and why we, as users, also need to understand the decision-making processes of AI systems.

Practical steps to secure your AI systems

The OWASP Top Ten for machine learning should be taken into account right from the ideation stage. It is important to integrate application protection into development and operational processes and to continuously improve security.

Software developers should be supported by AI and security experts throughout the entire process, from the start of development right through to operation. The OWASP Top Ten enables the most common security vulnerabilities to be identified and countermeasures to be put in place. Threat analyses and penetration tests can also help to identify risks and security vulnerabilities.

The integration of AI into business processes opens up incredible opportunities, but also entails new risks. By gaining a better understanding of the OWASP Top Ten for machine learning and the importance of security in system-critical sectors such as the automotive industry, organisations can deploy these technologies safely and responsibly.

Sources:

[1] https://owasp.org/www-project-machine-learning-security-top-10/

[2] https://www.bsi.bund.de/DE/Service-Navi/Presse/Alle-Meldungen-News/Meldungen/KI-Sicherheit-im-Auto_230515.html

[3] https://owasp.org/www-project-machine-learning-security-top-10/docs/ML01_2023-Input_Manipulation_Attack.html

Do you have any questions or need assistance?

You can find out more about the possibilities of artificial intelligence and IT security for your business on our website.

Feel free to get in touch with us.