Episode 51 · Module 10 · Infrastructure

Trusting the AI Developer — Verifying Agent-Generated Code

19 May 2026 · 7:59 · Security for Legal SaaS

7:59 7:59

AI coding agents — tools like GitHub Copilot, Cursor, Claude Code, and Amazon Q Developer — have fundamentally changed how software is written. Developers describe what they want in natural language, and an AI generates the code. For legal tech teams building contract review tools, case management systems, and e-filing integrations, these tools accelerate development dramatically. But they introduce a threat model that traditional security practices never anticipated: the code writer itself is a source of risk.

Today’s Lesson

Security for Legal SaaS — Episode 51: Trusting the AI Developer — Verifying Agent-Generated Code

When Your Developer Is a Probabilistic Model

Unlike a human developer who might introduce bugs through carelessness or ignorance, an AI agent can generate code that looks syntactically perfect, passes superficial review, and contains subtle security flaws — because the model is optimising for plausible-looking code, not provably correct code.

Research from Endor Labs analysing vulnerabilities in AI-generated code found recurring patterns: missing input validation, improper error handling that leaks internal state, hardcoded credentials in example code that gets shipped to production, and insecure default configurations.¹

Hallucinated Dependencies: The Slopsquatting Threat

The most novel risk from AI-generated code is "slopsquatting" — a term coined to describe what happens when AI models hallucinate package names that don't exist, and attackers register those names with malicious code.

A University of Texas at San Antonio study analysing 576,000 code samples generated by 16 different large language models (LLMs — the AI systems behind tools like ChatGPT and Copilot) found that nearly 20% of package dependencies referenced by AI don't actually exist.² That's 205,474 unique hallucinated package names across the study — each one a potential attack vector.

The attack pattern: An AI tells a developer to install flask-auth-helper. The package doesn't exist on PyPI (Python's package registry — the central repository where Python libraries are published and downloaded). An attacker registers flask-auth-helper on PyPI with malicious code. The next developer who follows the AI's suggestion installs malware. This is slopsquatting — typosquatting powered by AI hallucination.³

What makes this worse: the hallucinations aren't random. The study found that 43% of hallucinated package names appeared repeatedly across multiple queries.⁴ Attackers can predict which fake names the AI will suggest and pre-register them.

A security researcher demonstrated this in practice by registering `huggingface-cli` — a package name commonly hallucinated by AI tools. Within three months, it had accumulated over 30,000 downloads.⁵

Open-source models are worse. The study found open-source models hallucinated dependencies at a rate of nearly 22%, compared to just over 5% for commercial models.² If your team uses locally-hosted models for cost savings (as we discussed in the context of local LLM deployments), the hallucination rate for package names may be significantly higher.

Beyond Hallucinations: Vulnerable Versions and Insecure Patterns

Hallucinated packages aren't the only risk. A separate academic study analysing 117,062 dependency changes found that AI agents select vulnerable versions of real packages at a rate of 2.46%, compared to 1.64% for human developers.⁶ Agent-driven development produced a net increase of 98 new vulnerabilities, while human-authored changes produced a net reduction of 1,316.

Common insecure patterns in AI-generated code include:

Pattern	Risk	Legal SaaS Impact
Missing input validation on API endpoints	Injection attacks (SQL injection, as covered in Episode 8)	Attacker modifies case records or extracts client data
Hardcoded API keys in generated code	Credential exposure if code is committed to version control	Third-party service compromise; billing abuse
Overly permissive CORS headers (as covered in Episode 12)	Cross-origin data theft	Browser-based attacks against logged-in lawyers
Default `admin/admin` credentials in scaffolded code	Trivial unauthorised access	Full platform compromise
Disabled TLS certificate verification	Man-in-the-middle attacks (as covered in Episode 13)	Intercepted client communications

Full-Auto Deployment: The Unreviewed Pipeline

The most dangerous configuration is an AI agent that can write code, run tests, and deploy — all without human review. This "full-auto" mode is increasingly common in development workflows. The agent creates a feature branch, writes the code, generates tests, the tests pass, and the code ships.

The problem: the agent wrote both the code and the tests. An AI that generates an authentication bypass will also generate tests that don't check for authentication bypasses. The tests pass because they were designed by the same model that created the flaw.

CSO Online documented cases where supply chain attacks specifically targeted AI coding agents, exploiting their tendency to follow instructions embedded in repository files, package descriptions, and even code comments.⁷

The legal obligation: If your contract analysis tool processes privileged attorney-client communications, deploying AI-generated code without human security review creates a reasonable-efforts argument under ABA Model Rule 1.6(c) that you are not making adequate efforts to protect client data.⁸

Verification Strategies That Work

1. Lock Your Dependency Sources

Maintain a private registry or mirror of approved packages. Configure your package manager to install only from this trusted source. When the AI suggests a dependency, it must exist in your registry before it can be installed. This eliminates slopsquatting entirely.

# npm example: .npmrc restricting to internal registry
registry=https://registry.internal.yourfirm.com/

2. Require Human Review of Security-Critical Code

Define security-critical paths in your codebase: authentication, authorisation, encryption, data access layers, and anything that handles client PII (Personally Identifiable Information). Require manual code review for changes to these paths, regardless of who (or what) wrote the code.

3. Force the Agent to Write Security Tests — Then Write Your Own

Have the AI generate security tests for its own code: tests for SQL injection, tests for authentication bypass, tests for authorisation failures. Then have a human write additional security tests independently. If the human-written tests catch what the AI-written tests missed, you've found a gap in the AI's security reasoning.

4. Sandbox the Agent's Workspace

The AI coding agent should never have access to production credentials, SSH keys, `.env` files, or deployment secrets during development. ActiveState's analysis of AI supply chain risks recommends restricting agents to allow-listed tools and requiring human approval before any high-impact actions like dependency installation or deployment.⁹

5. Run Static Analysis on Everything

SAST tools (Static Application Security Testing — automated scanners that examine source code for vulnerabilities without running it) like Semgrep and CodeQL should run on every pull request, whether the code was written by a human or an AI. AI-generated code should receive the same — or stricter — automated scrutiny as human-written code.

6. Pin and Audit Dependencies

Lock every dependency to a specific, verified version. Use tools like `npm audit`, `pip audit`, or Snyk to continuously scan for known vulnerabilities. When the AI suggests adding a new package, verify: Does this package exist? Is it actively maintained? Does it have known vulnerabilities? Is there a more established alternative?

The Trust Calibration

AI coding agents are tools, not developers. They produce output that must be verified with the same rigour you'd apply to code from an untrusted contractor — arguably more, because the AI cannot explain its reasoning, has no accountability, and will confidently generate insecure code without hesitation.

For legal SaaS teams, the standard is not "does the code work?" but "can we demonstrate that we verified the code meets security requirements before it touched client data?" The AI writes the first draft. Humans own the final version.

Next episode: Day Zero — bootstrapping security for a brand new project, from the very first commit.

Sources & references

Endor Labs, The Most Common Security Vulnerabilities in AI-Generated Code.
Trax Tech / University of Texas at San Antonio, 20% of AI-Generated Code Dependencies Don't Exist.
Infosecurity Magazine, AI Hallucinations Create "Slopsquatting" Supply Chain Threat.
BleepingComputer, AI-hallucinated code dependencies become new supply chain risk.
DevOps.com, AI-Generated Code Packages Can Lead to 'Slopsquatting' Threat.
Simon Roses Femerling, The Dependency Trap: Supply Chain Risks in AI-Generated Code (Part 4).
CSO Online, Supply-chain attacks take aim at your AI coding agents.
ABA, When Should Law Firms Notify Clients About Data Breaches?.
ActiveState, Is AI-Generated Code Poisoning Your Software Supply Chain?.
Capitol Technology University, AI-Driven Hallucinations in Cyber Supply Chain Lead to New Threat: Slopsquatting.
TechDebt.best, AI Code Security Risks — Hallucinated Dependencies & Beyond.

Alice: Welcome back to Security for Legal SaaS. I'm Alice.

Dan: And I'm Dan. Episode 51 — and this one feels very current. AI agents writing code. Copilot, Cursor, Claude Code, Amazon Q. Developers describe what they want in plain English, and an AI writes the code. Alice, we've talked about threats from attackers, from misconfiguration, from human error. But what happens when the thing writing your code is itself a risk?

Alice: It's a fundamentally different threat model. A human developer might introduce a bug because they're tired or they don't know better. An AI agent can generate code that looks perfect — clean syntax, good structure, reasonable variable names — and has a subtle security flaw baked in. Not because the AI is malicious, but because it's optimising for code that looks right, not code that is provably secure. It's a probabilistic model. It generates the most likely next token. And "most likely" is not the same as "most secure."

Dan: Mm. So what kinds of flaws are we talking about?

Alice: The most novel one is something called slopsquatting. Here's how it works. You ask an AI to write a Python script. The AI says: "install this library called flask-auth-helper." You go to install it — and it doesn't exist. The AI made it up. It hallucinated a package name. Now here's where it gets dangerous. Researchers at the University of Texas studied over half a million AI-generated code samples from sixteen different language models. Nearly twenty percent of the package dependencies the AI recommended didn't exist. Two hundred thousand fake package names.

Dan: Hmm. Twenty percent. But if the package doesn't exist, the install just fails, right? Where's the attack?

Alice: The attack is that someone registers that fake name before you try to install it. A security researcher proved this. They registered a package called "huggingface-cli" — a name that AI tools commonly hallucinate. Within three months, over thirty thousand people had downloaded it. If that package had contained malware instead of a proof-of-concept, thirty thousand development environments would have been compromised.

Dan: Right. And the AI keeps suggesting the same fake names?

Alice: Forty-three percent of the hallucinated names appeared repeatedly across different queries. They're not random — they're predictable. An attacker can study which fake names the popular AI models suggest, register them on package registries like PyPI or npm — those are the central repositories where Python and JavaScript libraries are published — and wait for developers to follow the AI's instructions.

Dan: Yeah, that's clever. Disturbingly clever. What about beyond the fake packages — what about the code itself?

Alice: A separate study looked at over a hundred thousand dependency changes and found that AI agents choose vulnerable versions of real packages more often than human developers do. Humans reduced net vulnerabilities by over thirteen hundred. AI agents increased them by almost a hundred. And the code patterns are concerning too — missing input validation on API endpoints, which we covered back in Episodes 7 and 8. Hardcoded API keys in generated code. Overly permissive security headers. Default credentials left in scaffolded projects. The AI generates it, the developer glances at it, it looks reasonable, they ship it.

Dan: Mm-hmm. And what about the full-auto scenario — where the AI writes the code, writes the tests, and deploys it, all without a human looking at it?

Alice: <sigh> That's the configuration that keeps me up at night. Because the AI writes both the code and the tests. If the AI creates an authentication bypass — some subtle flaw where certain requests skip the login check — it's also going to write tests that don't test for that bypass. The tests pass because the same blind spot that caused the bug also shaped the tests. You get a green check mark on a pipeline that just deployed a security hole.

Dan: So the tests are only as good as the model's understanding of what could go wrong.

Alice: Exactly. And for legal SaaS — if your contract analysis tool handles privileged attorney-client communications, deploying code without human security review creates a real problem under ABA Model Rule 1.6(c). That's the rule requiring reasonable efforts to protect client information. "The AI wrote it and the AI's own tests passed" is not going to satisfy a regulator asking what you did to prevent a breach.

Dan: Mm. So what does a safe workflow look like when you're using AI to write code?

Alice: First — lock your dependency sources. Set up a private registry of approved packages. Configure your package manager so it can only install from that registry. When the AI suggests a package, it has to exist in your approved list. This kills slopsquatting completely. The AI can hallucinate all it wants — if the package isn't in your registry, it can't be installed.

Dan: Right. What about the code itself?

Alice: Define security-critical paths in your codebase. Authentication, authorisation, encryption, anything that touches client data. Require human review for changes to those paths, no matter who or what wrote the code. Then — and this is important — have the AI write security tests, but also have a human write independent security tests. If the human-written tests catch things the AI-written tests missed, you've found the gap. You've measured how much the AI doesn't know about security.

Dan: Yeah. Two sets of tests from two different sources.

Alice: And sandbox the agent's workspace. The AI should never have access to production credentials, SSH keys, environment files, or deployment secrets during development. If a malicious instruction is hidden in a repository file or a package description — and researchers have documented exactly this kind of attack — the agent shouldn't be able to reach anything valuable even if it's tricked into executing something.

Dan: Mm. And then static analysis on top of all of that?

Alice: Always. Run SAST tools — that's static application security testing, automated scanners that examine code for vulnerabilities without running it — on every pull request. Semgrep, CodeQL, whatever your team uses. AI-generated code should get the same automated scrutiny as human code. Arguably more, because the AI won't push back when you tell it to fix something.

Dan: So to put it plainly — treat AI-generated code like code from an untrusted contractor who can't explain their reasoning.

Alice: That's the right mental model. The AI is a tool, not a developer. It produces a first draft. Humans own the final version. For legal tech, the standard isn't "does the code work." It's "can we demonstrate we verified this code meets security requirements before it touched client data." The AI helps you build faster. Verification is what keeps your clients safe.

Dan: Next episode — Day Zero. Bootstrapping security for a brand new project, from the very first commit.

Alice: Until then, I'm Alice.

Dan: And I'm Dan.

Alice: Security for Legal SaaS is a series written with AI assistance. Alice and Dan are AI-generated voices — no professional advice here, just education.

Security for Legal SaaS is a series written with AI assistance. Alice and Dan are AI-generated voices — no professional advice here, just education.