Tech

LLM Security Testing: A Practical Guide to Finding Vulnerabilities in Production AI

admin13 min read7 views
Production

Why Production AI Needs LLM Security Testing

Large language models now sit inside support desks, internal search, code assistants, claims workflows and customer-facing agents. Each deployment accepts untrusted text from users, documents and web pages, and turns it into outputs and, increasingly, actions. That is a different AI application security problem from the one most security programmes were built to handle.

Traditional vulnerabilities are deterministic. A SQL injection flaw either exists in a query or it does not, and a scanner can check for it. LLM-specific risks are probabilistic and context-dependent: the same prompt can be refused on Monday and answered on Tuesday after a model update, a prompt change or a new document in the retrieval index. The attack surface is not only the code. It includes the system prompt, the model, the retrieved data, the tools the model can call and everyone who can type into it.

That is why testing an AI application once before launch is not enough. Production AI changes constantly: models are upgraded, prompts are edited, knowledge bases grow, new tools are connected. Each change can reopen a risk that was closed last quarter. LLM security testing therefore has to be an ongoing practice rather than a release gate. This guide covers what it involves, the vulnerabilities it should find, a nine-step workflow, and how it fits with AI evaluation and AI code scanning.

What Is LLM Security Testing?

LLM security testing is the practice of deliberately probing an LLM-powered application with adversarial inputs to find out whether it can be made to behave in ways its owners did not intend: leaking data, ignoring instructions, producing harmful output or taking unauthorised actions. It is also called LLM vulnerability testing or AI red teaming, and it covers the whole application rather than the model alone.

How it differs from traditional application security testing

Conventional testing such as SAST, DAST and penetration testing looks for defects in code and configuration. It assumes the system’s logic is fixed and that malicious inputs follow recognisable patterns, like SQL fragments or script tags. An LLM application breaks both assumptions. Its behaviour is shaped by natural language, so a malicious input can read like an ordinary sentence, and its “logic” is a model whose responses vary from run to run. A security test for an LLM application typically examines:

  • Adversarial prompts: inputs crafted to override instructions or bypass safety behaviour, including multi-turn and encoded variants.
  • Malicious inputs from indirect sources: instructions hidden in documents, emails, web pages or tool outputs that the model later reads.
  • Model manipulation: persona framing, role-play or gradual escalation across a conversation to shift the model’s behaviour.
  • Sensitive data exposure: whether the application reveals personal data, credentials, system prompts or proprietary content.

Why it has to be continuous

Because behaviour is probabilistic, one clean test run proves little. A prompt that fails nine times may succeed on the tenth. Attack techniques also evolve, and so does your own application. Continuous testing, meaning a stable attack suite re-run on every material change and extended over time, turns security from a snapshot into a trend you can track.

The Most Common Security Vulnerabilities in Production LLM Applications

The OWASP Top 10 for LLM Applications is the most widely used reference for these risks, and the items below map to it. Where the 2025 edition names a risk differently, that is noted. The examples are illustrative scenarios, not reported incidents.

Prompt injection attacks

Crafted input changes what the model does (OWASP LLM01). Direct injection comes from the user; indirect injection arrives through content the model reads. In one plausible scenario, a support assistant summarises inbound emails, and one email contains hidden text instructing it to include the customer’s account notes in its reply.

Jailbreak attempts

A jailbreak aims to make the model ignore its safety policy, often through role-play, hypothetical framing or escalation over several turns. Single-prompt tests miss many of these, because the bypass only works after the conversation has been set up.

Sensitive information disclosure

The application reveals personal data, credentials, internal documents or its own system prompt (LLM02, with system prompt leakage listed separately as LLM07 in 2025). For example, a user asks the assistant to repeat everything above their message and receives internal routing rules and an API endpoint.

Insecure output handling

Model output is passed to another component without validation (called improper output handling, LLM05, in 2025). If a response is rendered as HTML, run as SQL or passed to a shell, the model becomes an injection path into the rest of the stack. Model output should be treated as untrusted input.

Excessive agency and unauthorised actions

The model has more permissions, tools or autonomy than its task needs (LLM06). A scheduling agent that can read calendars but can also delete events and send email turns one successful injection into actions well beyond scheduling.

Insecure plugin and tool integrations

Tools often accept parameters generated by the model. Without strict schemas and authorisation enforced on the tool side, a manipulated model can pass arguments the developer never anticipated. The 2023 OWASP list had a dedicated “insecure plugin design” entry; later editions cover much of this under excessive agency and supply chain risk.

Retrieval-Augmented Generation (RAG) data leakage

Retrieval pipelines can return documents the requesting user is not authorised to see, because the index flattens permissions that existed in the source system (vector and embedding weaknesses, LLM08). An HR assistant that retrieves a salary review for someone who should only see policy documents is the typical case.

Model denial-of-service attacks

Very long prompts, recursive tool loops or requests that force maximal output can degrade availability or inflate cost. The 2025 OWASP list calls this broader category unbounded consumption (LLM10).

Insecure handling of AI-generated code

Generated code may contain hardcoded secrets, outdated dependencies or unsafe patterns, and may reach production with little review. This overlaps with improper output handling and is where code-level scanning matters most.

How to Perform LLM Security Testing: A Practical Step-by-Step Approach

The workflow below works for a single chatbot or a portfolio of agents. The order matters, because each step supplies the scope for the next.

  1. Identify all deployed LLM applications and integrations. Include internal prototypes and team-built tools, not only flagship products. For each, record the model, prompts, data sources, connected tools and owner.
  2. Map the attack surface. List every input channel (users, documents, email, web pages, tool outputs), every output destination (UI, databases, downstream APIs) and every privilege the application holds. Mark the trust boundaries between them.
  3. Test prompt injection and jailbreak vulnerabilities. Prompt injection testing should cover direct, indirect, multi-turn and encoded variants (Base64 or ROT13, for instance). Run each case several times and record a success rate, not a single pass or fail.
  4. Evaluate sensitive data exposure. Try to extract the system prompt, personal data and credentials present in context. Seed test records with unique marker strings so any leak is easy to detect.
  5. Test tool permissions and agent behaviour. For each tool, ask what the worst action would be if the model were fully manipulated. Attempt out-of-scope calls and tampered parameters, and confirm that authorisation is enforced by the tool, not by the prompt.
  6. Assess RAG pipelines and external integrations. Query as users with different roles and verify that retrieved documents respect source permissions. Plant a document containing hidden instructions to test indirect injection.
  7. Analyse AI-generated outputs. Check what downstream components do with responses. Feed outputs containing markup, SQL or shell syntax through the real pipeline, and scan any generated code.
  8. Document vulnerabilities and prioritise remediation. Record the prompt, the response, the success rate, the affected asset and the business impact. In an AI security assessment, severity depends on what the model can reach: an occasional injection against an agent with payment permissions outranks a frequent jailbreak on a read-only chatbot.
  9. Retest after security fixes. Re-run the same cases, then paraphrased variants, because prompt changes and filters often fail under rewording. Keep the results as evidence.

Why AI Evaluation Is Essential for LLM Security

AI evaluation is the systematic measurement of how an AI system behaves against defined criteria such as accuracy, groundedness, consistency, policy compliance and safety. In enterprise settings it usually runs on curated datasets and repeats whenever the model, prompt or data changes.

Functional AI evaluation asks whether the system does its job: correct answers, low hallucination, the right format. Security-focused AI evaluation asks whether it can be pushed off that job: does it hold its policy under adversarial input, and how often does it fail? Adversarial test cases are the bridge. Every successful attack found during testing becomes a permanent evaluation case, so a regression is caught the next time a prompt or model changes.

The same logic applies after deployment. Production failures, flagged conversations and real attack attempts are the best source of new evaluation data, because they reflect how users and attackers actually behave. AI evaluation complements LLM security testing rather than replacing it: evaluation provides repeatable measurement, while security testing supplies the creative, adversarial exploration that shows what needs measuring.

The Role of AI Code Scanning in Securing LLM Applications

LLM applications are still software, and much of their risk sits in ordinary-looking code: API keys committed to a repository, model output passed into a database query, user data sent to a model provider without redaction, an agent framework granted broad filesystem access. Conventional static analysis can catch some of this, but its rules were not written with LLM patterns in mind. It may not recognise, for instance, that a variable holding a model response should be treated as untrusted input.

AI code scanning applies rules designed for these patterns during development. Run in the IDE, on pull requests and in CI/CD pipelines, it flags issues while the fix is still cheap and can fail a build on critical findings, so vulnerable code is caught before it reaches production.

Code scanning and runtime testing answer different questions. Scanning shows where the code is unsafe by construction. Runtime LLM security testing shows how the deployed system behaves when attacked. A hardcoded credential is found by the first; a multi-turn jailbreak only by the second.

LLM Security Testing vs AI Evaluation vs AI Code Scanning

Approach Primary Focus When It Helps Typical Finding
LLM Security Testing Identifying AI-specific attack vulnerabilities During security assessments and production testing A multi-turn jailbreak that exposes the system prompt
AI Evaluation Measuring AI behaviour, reliability and safety During development and continuous monitoring A drop in policy adherence after a model upgrade
AI Code Scanning Identifying vulnerabilities in application code During development and CI/CD A hardcoded API key or unvalidated model output in a query

These are complementary layers, not alternatives. Code scanning reduces the number of flaws that ship. AI evaluation measures whether behaviour stays within acceptable bounds as things change. LLM security testing attacks the running system to find what the other two cannot see. Skipping any one leaves a category of risk unexamined.

Best Practices for Securing Production LLM Applications

Production AI security comes down to a small set of habits applied consistently:

  • Adopt continuous LLM security testing. Re-run the attack suite on every model, prompt or tool change.
  • Apply least privilege to AI agents. Give each agent only the tools and data its task requires.
  • Validate and sanitise inputs and outputs. Treat model output as untrusted before it reaches any other system.
  • Protect sensitive information. Keep secrets and unnecessary personal data out of prompts and context.
  • Secure API keys and model integrations. Store credentials in a secrets manager and rotate them.
  • Test RAG permissions and data boundaries. Verify that retrieval enforces the same access rules as the source systems.
  • Integrate AI code scanning into development workflows. Scan in the IDE, on pull requests and in CI.
  • Establish repeatable AI evaluation processes. Version datasets and track results over time.
  • Maintain evidence of testing and remediation. Keep test cases, results and fix verification for audits and incident reviews.

How Trusys Helps Organisations Strengthen AI Security

Trusys is an AI assurance platform, and two of its products map to the layers described above. TruScout automates red-team testing. According to its product page, teams choose from predefined attack strategies, including multi-turn, tree-based and iterative jailbreaks and encoding-based attacks, select from 50+ vulnerability categories, and run the evaluation across them in a single run. The page says it is built on the OWASP Top 10 for LLMs, MITRE ATLAS and NIST AI RMF, and that it covers both pre-deployment testing and post-deployment monitoring.

TruScan is an open-source scanner for AI applications. It runs in VS Code and Visual Studio, on pull requests, and in CI/CD through GitHub Actions, Jenkins and GitLab CI. Output is available as console text, JSON or SARIF, and custom Semgrep rules can extend it. The page lists 50+ prebuilt rules and support for 10+ frameworks, including OpenAI, Anthropic, LangChain, FastMCP and LlamaIndex. Installation is a single pip command (trusys-llm-scan).

Used together, the two cover the build and run stages: code scanning catches unsafe patterns before merge, and red-team testing checks how the running application responds to attack. Teams can keep the output of both as the evidence trail described in the previous section.

Conclusion: Make LLM Security Testing a Continuous Process

Production AI changes faster than a quarterly review cycle, and its risks live in prompts, retrieved data and tool permissions as much as in code. Securing it takes more than traditional application security. LLM security testing, AI evaluation and AI code scanning each examine a different layer, and together they give a defensible view of risk across development and production.

Frequently Asked Questions

What is LLM security testing and how is it different from penetration testing?

LLM security testing probes an LLM application with adversarial prompts and malicious inputs to find AI-specific weaknesses such as prompt injection, data leakage and excessive agency. Penetration testing targets code, infrastructure and configuration. Because LLM behaviour varies between runs, LLM testing repeats cases and measures success rates rather than relying on a single result.

How often should production LLM applications be security tested?

Test on every material change, such as a model upgrade, prompt edit, new data source or new tool, and on a regular schedule between changes. Attack techniques evolve, so a fixed test suite should be extended over time. Continuous testing is more useful than a one-off assessment before launch.

How do you test an LLM application for prompt injection?

Build a suite of direct, indirect, multi-turn and encoded injection cases. Include instructions hidden in documents, emails and tool outputs, not only user messages. Run each case multiple times, record how often it succeeds, fix the cause, then retest with paraphrased variants.

What is the difference between AI evaluation and LLM security testing?

AI evaluation measures behaviour against defined criteria such as accuracy, consistency and policy compliance, usually on repeatable datasets. LLM security testing actively tries to break the application. They work together: attacks found in testing become evaluation cases that catch regressions later.

Can AI code scanning replace runtime LLM security testing?

No. Code scanning finds unsafe patterns in source code, such as exposed credentials or unvalidated model output, before deployment. It cannot show how a running model responds to a multi-turn jailbreak or a poisoned document. The two address different layers and are best used together.