AI security

AI-generated code is a security problem hiding in plain sight

, 4 min read

AI coding assistants write working code quickly. The research on whether that code is secure has been consistent for five years: a large share of it isn't, and the people using the tools tend to think it is.

What the studies found

  • NYU, 2021 ("Asleep at the Keyboard?"). Researchers gave GitHub Copilot 89 scenarios built around high-risk weakness types from MITRE's CWE Top 25, and generated 1,689 programs. About 40% contained a vulnerability. Published at IEEE Symposium on Security and Privacy 2022.
  • Stanford, 2023 ("Do Users Write More Insecure Code with AI Assistants?"). In a user study, participants with an AI assistant wrote significantly less secure code than those without one, and were more likely to believe their code was secure. Participants who questioned the assistant and reworked their prompts did better. Published at ACM CCS 2023.
  • Veracode, 2025 (GenAI Code Security Report). Veracode tested more than 100 language models on tasks in Java, Python, C# and JavaScript. 45% of the code samples failed security tests and introduced an OWASP Top 10 weakness. Java was worst, at 72%. The models failed to defend against cross-site scripting in 86% of relevant samples. Newer and larger models did no better than older ones.

The second finding matters most for how teams work. The risk isn't only that the code has flaws. The person accepting it is less likely to look for them.

What it looks like

This is the kind of handler an assistant will happily produce when asked for "a login endpoint":

app.post('/login', async (req, res) => {
  const { email, password } = req.body;
  const user = await db.query(
    `SELECT * FROM users WHERE email = '${email}' AND password = '${password}'`
  );
  if (user.rows.length > 0) {
    const token = jwt.sign({ id: user.rows[0].id }, 'secret-key');
    res.json({ token });
  }
});

It runs, and a quick test passes. It also has SQL injection through string interpolation, passwords stored and compared in plain text, a signing secret committed to source, no rate limit, and no response at all on failure. Each of those is common in the public code these models learned from.

Why models produce it

Language models learn from public code: tutorials, answers, prototypes, abandoned projects. Most of it was written to work, not to withstand an attacker. A model predicts code that looks like what it has seen, with no built-in sense of your threat model, your trust boundaries or which input is hostile. The patterns that show up again and again:

  • Injection: string-built SQL, shell commands and templates
  • Missing authorisation: endpoints that check who you are but not what you may touch
  • Hard-coded secrets: example keys that become real keys
  • Outdated or invented dependencies: versions from the training data, sometimes with known CVEs, and package names that don't exist at all. A USENIX Security 2025 study of 576,000 generated samples found hallucinated packages in at least 5.2% of commercial-model outputs and 21.7% of open-source-model outputs. Any of those names can be registered by an attacker.
  • Happy-path error handling: failures that leak detail or fail open

The answer is the pipeline, not a ban

Banning assistants is unenforceable, and the productivity gain is real. What works is making security checks independent of who, or what, wrote the code:

  • SAST on every pull request. Semgrep, CodeQL or similar, with the build failing on high-confidence, high-severity findings. This catches the injection and hard-coded-secret classes above.
  • Secret scanning before and after commit. Gitleaks or TruffleHog as a pre-commit hook and again in CI.
  • Dependency scanning and review. Flag new dependencies, known-vulnerable versions and packages that are brand new or barely downloaded. That last check matters with assistants that invent package names.
  • DAST against staging. Some flaws, such as missing authorisation checks and bad headers, only show when the app is running.
  • Review AI output like an outside contribution. The Stanford result says your instincts will be too trusting. Make "where does untrusted input go?" a standing review question.

A minimal SAST gate with Semgrep's open-source CLI and a public ruleset:

sast:
  runs-on: ubuntu-latest
  container: semgrep/semgrep
  steps:
    - uses: actions/checkout@<commit SHA>
    - run: semgrep scan --config p/owasp-top-ten --error

--error makes Semgrep exit non-zero when it finds something, which fails the job. Start with a narrow, high-confidence ruleset and widen it once the team trusts the signal.

Why it's worth the effort

IBM's 2026 Cost of a Data Breach Report puts the global average cost of a breach at $4.99 million. A pipeline gate costs some CI time per pull request. It's also the one control that scales with the volume of code assistants produce. Reviewers get tired. Scanners don't.

Sources

Related posts

Start with a $500 audit.

See exactly where you stand. Actionable findings in a week.

The full report and debrief call. Delivery in 5–7 business days.

Book a $500 Audit Hire on Upwork