Apply for class 03 now!

A radiation meter in your hand or a nuclear reactor under control?

  • From the projects
  • ulysse.gks & FARI / betterimagesofai.org / CC-BY-4.0

(Generative) AI is part of the software world that surrounds us. It is being directly integrated into an ever-increasing number of applications, often without any awareness of the associated risks. Mark Wernsdorfer from the project LLMSecTest discusses how to deal with this in his guest article:

Generative AI and large language models (LLMs) are the major technological developments of recent years, and they will have a decisive impact on many areas of life over the coming decades. More and more people are convinced that large language models are not merely a technological gimmick: in medicine, education, the arts and industry, artificial intelligence is already being used to improve, accelerate or enhance processes. But as with any new technology, LLMs also carry risks. The question is: how can we harness the enormous power of artificial intelligence, rather than simply observing where it takes us?

AI as the splitting of the atom

Over the course of my career, I have already contributed to the use of AI in education and medicine. However, AI can cause great harm in these high-risk areas if the systems deployed are not designed, developed and monitored with foresight and in accordance with strict guidelines.

This serious potential for harm forms the basis of a helpful analogy: AI resembles nuclear power in that it has unimaginable potential. However, without a reliable framework for containment, it can also be immensely destructive. Two strategies for countermeasures can be derived from this analogy: on the one hand, empowering individuals to recognise these dangers and avoid them – like a radiation meter. On the other hand, protection through containment: secure protective casings that have undergone rigorous radiation testing.

The strategy behind my last project, “SpotTheBot”, was to equip people with radiation meters for AI-generated content. The aim was for them to learn to recognise and avoid dangerous and misleading content themselves. What I have learnt since is that this strategy forces people into a defensive position. It is of little use once the reactor is already leaking. Safety must be built into the reactor’s design.

The root of the problem

With SpotTheBot from Prototype Fund Round 14, I wanted to teach end-users media literacy in recognising AI-generated content. After all, self-empowerment is the only way out of a state of dependency – and the first step on this path is awareness. But of course, it is problematic to shift the responsibility onto those who are affected.

This has become clear, amongst other things, due to rapid technological progress. AI texts can now be generated in a way that reliably prevents human detection. There is therefore little prospect of people empowering themselves in this regard today without technological assistance. Technological detectors such as Pangram, on the other hand, identify AI content with unprecedented precision. The task of self-empowerment requires tools that scale with technological development.

This realisation has led me to rethink my target audience. To help people truly effectively, we must address the developers of AI-integrated applications, not their users. This allows protection to scale by orders of magnitude more effectively than by targeting individuals. So what is the greatest danger that developers unwittingly and unknowingly open the door to when they integrate LLMs into their applications?

When the recipe can no longer be distinguished from the ingredients

Large language models understand text and follow instructions. There is a distinction between these two that people are often completely unaware of in everyday conversations: intuitively, we know whether something is an instruction to take action or merely the conveyance of knowledge. Language models struggle to make this distinction, as LLMs are presented with instructions and information within the same text stream. They must decide for themselves whether a text segment is a piece of information or a command, thereby blurring a line - between the process and the data on which it operates - that is traditionally drawn very clearly in computer science. If language models make the wrong decision or are misled, information intended for processing can become an instruction to take action.

This is where LLMSecTest comes in. Based on the OWASP TOP 10 LLM attack vectors, this very problem is divided into ten threat categories, each of which can be exploited in different ways and must be tested in different ways.

Prompt Injection is just one of the ten categories. There is also System Prompt Leakage, which queries the system instructions – in the hope that they contain sensitive data. Sensitive Information Disclosure, on the other hand, asks directly for a value, which is correctly refused by almost all applications. In one application, however, it worked to output it in Base64-encoded form. This bypasses filters that check for the value. Improper Output Handling does not check the model itself, but rather whether potentially harmful output (e.g. a `javascript:` link in the generated Markdown, which might be executed by readers) is being produced.

Among the daily tests I run on a cohort of around 50 applications connected to an LLM via LangChain, LlamaIndex or Haystack (e.g. AnythingLLM), all were vulnerable to prompt injection: the smuggling in of process instructions disguised as data. For example, around 29 applications disclosed secret system instructions simply by being politely asked to do so.

LLMSecTest calls applications via their own HTTP interface, exactly as users would do. This brings system prompts, guardrails and the application’s own tools into play. A specific number of requests is made per category. To ensure that the output values are not ‘hallucinated secrets’, a known value is first stored in the system prompt. If the response contains exactly this value, the test is failed. A request with no response counts neither as a pass nor a fail.

A particularly serious example from my test runs: an application attempted to ensure that an internal password did not leak to the outside world. It did this by comparing the output text against a hash of the password. If any of the words had the same hash value as the stored password, the output was withheld. In my automated tests, I have small, open-ended language models carry out attacks against the applications. They formulate prompts specifically designed to slip instructions into the systems, disguised as passive data content. In this case, the attack model simply asked the application to spell out the password, with spaces between each character. The application spelled out the password without flinching.

The current situation

I run LLMSecTest daily on this test fleet of 50 open applications. The idea behind this was to fulfil several functions:

  1. If today’s results differ from yesterday’s, then I know that I have introduced unintended changes since the last run.
  2. I can see whether my automatically generated security reports remain consistent, even though LLMs are essentially non-deterministic. I can see whether my automatically generated security reports remain consistent, even though the results always vary slightly from run to run due to the use of LLMs.
  3. I get an overview of the current security status of applications with LLM integration.

The case involving the spelled-out password highlighted something important. To prove that LLMSecTest really does what it promises, both sides must be taken into account. On the one hand, the small language model must carry out attacks that are sufficiently sophisticated and actually capable of circumventing security measures. On the other hand, the applications being tested must actually have security vulnerabilities in order to demonstrate that LLMSecTest does indeed deliver on its promises.

At the moment, therefore, my daily results on the test cohort say less about the security status of applications with LLM integration and more about the reliability and reproducibility of my automatically generated reports. Only once this has been sufficiently established can we assume that the generated reports actually say something meaningful about the tested applications.

The rule of thumb I have derived from this experience is this: unless it has been established that a security test can actually fail, passing it means nothing. Or, to stick with the analogy: a reactor’s containment vessel is useless if one does not know when it will fail.

Give it a go!

LLMSecTest is my attempt to make reactor development as secure as possible: it is an MIT-licensed, pytest-native test suite with a CLI that tests applications with LLM integration against the OWASP LLM Top 10 threat categories. It’s easy to install via `pip install`, can be integrated into a CI/CD pipeline, and generates machine-readable (SARIF) and human-readable (HTML and PDF) reports detailing the severity, specific attack vectors and possible countermeasures. Developers can run it directly on a live application or on a repository.

The system is completely open-source, it is currently being developed on a daily basis and presented in detail here. The documentation is an integral part of the development process and is kept up to date alongside the code. If you’re running your own application with an integrated language model: install LLMSecTest, test your application and open a GitHub issue with your feedback. Right now, I need to know whether you’re finding this useful.

markwernsdorfer.com

Mark Wernsdorfer earned his PhD in Bamberg, where he researched how artificial systems acquire meaning. Since then, he has been building machine learning systems in areas where errors are costly: a clinical decision support system at Leipzig University Hospital, tools to combat disinformation at Media Lab Bayern, and educational software for primary school children. At the Prototype Fund, he is developing LLMSecTest, a test suite for applications using language models.