Securing AI coding assistants in the firmware SDLC

Three distinct, separately documented risks, not one. First, the code an assistant originally generates can itself be insecure: a peer-reviewed study of GitHub Copilot found roughly 40% of generated programs vulnerable across scenarios drawn from MITRE's CWE Top 25. Second, even when an assistant is asked to *fix* a known vulnerability rather than write new code, GitHub's own current documentation warns that "the system may suggest fixes that fail to remediate the underlying vulnerability or introduce new vulnerabilities" and that it "has no understanding of the programmer's intent." Third, assistants can hallucinate dependencies that do not exist — a 2025 academic study found large language models invent non-existent package names at an average rate of 19.6%, and an attacker who registers one of those names under a firmware SDK's package manager gets a dependency pulled straight into a build. Firmware code carries extra exposure on all three: C/C++ memory-safety classes dominate the CWE Top 25 Copilot study, and vendor SDK package registries typically have far less namespace policing than npm or PyPI.

In "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions" (Pearce, Ahmad, Tan, Dolan-Gavitt and Karri, IEEE Symposium on Security and Privacy 2022), researchers prompted Copilot across 89 scenarios drawn from MITRE's CWE Top 25 and OWASP's Top 10, producing 1,689 generated programs. Their finding, stated in their own abstract: "we found approximately 40% to be vulnerable." That study is now several years old and covers one specific assistant generation; GitHub's own current responsible-use documentation for its newer Copilot Autofix feature still carries the same caveat today, stating a suggested fix "may suggest fixes that are syntactically valid but change the semantics of the program" and that "developers must evaluate each suggestion and verify it maintains the codebase's intended behavior" — the underlying unreliability has not been declared solved, it has been handed back to the reviewer.

Slopsquatting is the practice of registering a package name that large language models are known to hallucinate, so that a developer — or an autonomous coding agent — who blindly installs what the AI suggested pulls down the attacker's malicious package instead of an error. The scale is not theoretical: "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs" (Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala, USENIX Security 2025) tested 16 coding models and found "all 16 tested coding models exhibited notable rates of package hallucinations, averaging 19.6% across the board," with commercial models averaging around 5% against roughly 21% for open-source models, and recorded "205,474 unique non-existent packages" generated across their test runs. OWASP's own Top 10 for LLM Applications names the adjacent risk category directly: LLM03:2025 "Supply Chain" cites real incidents including attackers who "tricked model developers into downloading a compromised PyTorch dependency with malware" and the "Shadow Ray attack on the Ray AI framework," where OWASP states "five vulnerabilities are believed to have been exploited in the wild affecting many servers."

NIST SP 800-218A, "Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile" (July 2024), is explicit that AI-generated code gets no pass: "Practices and tasks in this Profile do not distinguish between human-written and AI-generated source code, because it is assumed that all source code should be evaluated for vulnerabilities and other issues before use." Its risk-modeling practice (PW.1.1, enhanced to a "High" priority task for generative-AI use) instructs teams to "incorporate relevant AI model-specific vulnerability and threat types in risk modeling," explicitly naming "poisoning of training data, malicious code or other unwanted content in inputs and outputs, denial-of-service conditions arising from adversarial prompts, supply chain attacks, unauthorized information disclosure, theft of AI model weights, and misconfiguration of data pipelines" as the categories to cover — a list that reaches well past "does the suggested code compile."

Treat every AI-authored or AI-modified line the same as a new contributor's pull request: it goes through the same static analysis, code review and test suite before merge, with no fast lane because a human didn't type it — directly consistent with NIST SP 800-218A's refusal to distinguish AI-generated from human-written code. Pin firmware dependencies to an exact version and hash rather than a bare package name an assistant suggested, so a hallucinated or squatted package name cannot silently resolve to something new. Keep a current software bill of materials built from what actually ships, not from a build manifest alone, since a manifest reflects what was *requested*, not what a binary actually contains. And budget review time specifically for AI-suggested dependency additions and AI-suggested "fixes" to existing vulnerabilities — both are exactly the two places the research above found the most AI-introduced risk concentrated.