NIST Special Publication 800-115, Technical Guide to Information Security Testing and Assessment (September 2008, authored by Karen Scarfone, Murugiah Souppaya, Amanda Cody and Angela Orebaugh), groups technical security assessment methods into three categories — Review, Target Identification and Analysis, and Target Vulnerability Validation — each carrying increasing risk to the systems being tested, in that order. It replaced the earlier SP 800-42, Guideline on Network Security Testing, and remains current on NIST's publication list as of November 2026, with no superseding revision published. Although developed under NIST's FISMA responsibilities for federal agencies, NIST's own text states it "may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright."
Key takeaways
- The structured, repeatable half of this work — enumerating components and matching them against public vulnerability data — is the Target Identification and Analysis layer.
- The document draws a sharp, often-blurred line between these two techniques.
- Planning. Rules are set, management approval finalized, goals agreed.
- SP 800-115 separates two independent axes of "viewpoint" that shape how a test is run.
What are the three types of assessment method, not just "testing"?
SP 800-115 opens by separating three distinct assessment methods that are often conflated under the single word "testing": Testing ("exercising one or more assessment objects under specified conditions to compare actual and expected behaviors"), Examination ("checking, inspecting, reviewing, observing, studying, or analyzing one or more assessment objects ... to obtain evidence"), and Interviewing ("conducting discussions with individuals or groups ... to identify the location of evidence"). The document's technical focus is on the first two, organized into named techniques.
What are the three categories of technique?
Section 2.2 groups the document's technical techniques into three categories, each carrying a different level of risk to the systems being assessed:
| Category | What it does | Risk level | Example techniques |
|---|---|---|---|
| Review | Passive examination of systems, policies, and configuration | Minimal — "generally conducted manually" | Documentation, log, ruleset and system configuration review; network sniffing; file integrity checking |
| Target Identification and Analysis | Finds systems, ports, services, and potential vulnerabilities, usually with automated tools | Moderate | Network discovery; network port/service identification; vulnerability scanning; wireless scanning; application security examination |
| Target Vulnerability Validation | Confirms a vulnerability is real by exercising it | Highest — "the greatest amount of risk in assessments" | Password cracking; penetration testing; social engineering; application security testing |
NIST is explicit that no single technique is sufficient on its own: "Since no one technique can provide a complete picture of the security of a system or network, organizations should combine appropriate techniques to ensure robust security assessments." A typical penetration test, for instance, relies on target identification and analysis techniques (port scanning, vulnerability scanning) to find candidates before the validation phase.
How do the three categories map onto a connected device?
SP 800-115 was written for enterprise IT networks, but its three-category structure maps cleanly onto a connected product, where the "system" is the device, its firmware image and the cloud or mobile services it talks to:
| SP 800-115 category | On the firmware image | On the running device | On the device's back end |
|---|---|---|---|
| Review | Inventory components (firmware SBOM); check file integrity and signing material | Documentation review of the update, boot and logging design; ruleset review of on-device firewall/ACLs | Configuration review of the cloud API, IAM policy and TLS settings |
| Target Identification and Analysis | Match the SBOM against known-vulnerability data; scan extracted binaries for weak settings | Port/service identification on the live device; vulnerability scanning of exposed network services; wireless scanning of Wi-Fi/BLE radios | Network discovery and vulnerability scanning of the API endpoints |
| Target Vulnerability Validation | — | Confirming a finding by exercising it on a bench unit under the Rules of Engagement | Confirming an API finding against an authorized test tenant |
The structured, repeatable half of this work — enumerating components and matching them against public vulnerability data — is the Target Identification and Analysis layer. The OWASP Firmware Security Testing Methodology gives the device-specific version of these stages, where SP 800-115 stays framework-level.
How is vulnerability scanning different from penetration testing?
The document draws a sharp, often-blurred line between these two techniques. Vulnerability scanning (Section 4.3) "identifies hosts and host attributes (e.g., operating systems, applications, open ports), but it also attempts to identify vulnerabilities rather than relying on human interpretation of the scanning results" by comparing a host's fingerprint against a signature database — NIST flags that this approach "can have a high false positive error rate" and only reliably finds "surface vulnerabilities," weaknesses that exist in isolation rather than in combination with others.
Penetration testing (Section 5.2) goes a step further: "While vulnerability scanners check only for the possible existence of a vulnerability, the attack phase of a penetration test exploits the vulnerability to confirm its existence." Most penetration tests specifically look for "combinations of vulnerabilities on one or more systems that can be used to gain more access than could be achieved through a single vulnerability" — exactly the kind of chained, interacting weakness a scanner's isolated signature matching tends to miss. On a device, a clean component scan means no known signature matched; it says nothing about whether a chain of lower-severity issues could be combined into a real compromise.
What are the four phases of a penetration test?
Section 5.2.1 lays out what it calls an example of how penetration testing can be divided into phases — NIST notes directly that "there are many acceptable ways of grouping the actions involved" — using four phases that run partly in parallel:
PlanningRules are set, management approval finalized, goals agreed. "No actual testing occurs in this phase." DiscoveryTwo parts: information gathering/scanning (host and service enumeration, OS fingerprinting, open-source reconnaissance), then vulnerability analysis — comparing findings against vulnerability databases such as NIST's own National Vulnerability Database (NVD). AttackThe validation step, with a loopback: per Figure 5-2, a confirmed weakness can lead to further access, and any new foothold can trigger another round of Discovery against newly reachable systems. ReportingRuns concurrently with the other three phases rather than only at the end — the assessment plan is itself a reporting artifact from the Planning phase, and periodic status reports continue through Discovery and Attack before a final report closes the engagement.
The common weakness categories the document lists for this phase — misconfigurations, kernel flaws, buffer overflows, insufficient input validation, race conditions and incorrect file/directory permissions — are as relevant on embedded Linux as on a server.
Which testing viewpoints apply: external/internal, overt/covert?
SP 800-115 separates two independent axes of "viewpoint" that shape how a test is run. External vs. internal: external testing is "conducted from outside the organization's security perimeter," typically starting with public reconnaissance before enumerating what's reachable through the firewall; internal testing places assessors "on the internal network" with some baseline access, focused on privilege escalation and system-level hardening. NIST recommends running external testing first if both are scheduled, "as it keeps [assessors] from acquiring insider information ... that would not be available to an adversary." For a device, "external" is the attacker on the same network segment or the internet-facing cloud API; "internal" is the assessor holding a bench unit with console access and the firmware image.
Overt vs. covert: overt ("white hat") testing happens "with the knowledge and consent of the organization's IT staff," which lets staff limit impact and learn from watching the test. Covert ("black hat") testing happens "without the knowledge of the organization's IT staff but with the full knowledge and permission of upper management." NIST is candid about the trade-off: covert testing's purpose "is to examine the damage or impact an adversary can cause — it does not focus on identifying vulnerabilities," and it is typically "time-consuming and costly," whereas overt testing "is less expensive, carries less risk ... and is more frequently used." Most product security work is overt: the goal is to find and fix weaknesses in the design, not to measure the vendor's own detection and response.
What goes into the assessment plan or Rules of Engagement?
Section 6 treats planning as the foundation of a safe assessment, and Appendix B gives a sample Rules of Engagement (ROE) template. NIST is explicit that the structure "is intended to be illustrative" — organizations should organize the ROE in whatever way suits them — but the sample sections are a useful checklist:
| ROE section (Appendix B) | What it pins down |
|---|---|
| 1. Introduction (purpose, scope, assumptions/limitations, risks) | Which device variants, firmware versions and interfaces are in scope; inherent risks and the mitigations for intrusive tests |
| 2. Logistics (personnel, schedule, test site, test equipment) | Named testers and client contacts; the schedule, including any off-hours windows; where testing happens; the bench and tooling used |
| 3. Communication strategy (general, incident handling and response) | Meeting cadence; and "critical" — the criteria for halting testing if a procedure harms the device or a real intruder appears, plus how testing is reinstated |
| 4. Target system/network | Authorized and unauthorized identifiers, and crucially the "exclude list" of systems not authorized for testing |
| 5. Testing execution | The allowable and unallowable activities for the specific test type |
The incident-handling clause matters most for devices in the field: NIST's worked concern is what happens if assessors discover a real intruder mid-test — should testing stop, by whose authority, and how does it resume?
How do you choose techniques without breaking a fragile device?
Section 2.3 warns that "any time that a test or tester directly interacts with a system or network, the potential exists for unexpected system halts and other denial of service conditions," and Section 6.5 explicitly names SCADA systems as having "a number of unique components with which a traditional security assessor may not be familiar." Embedded devices with small CPUs, limited memory and real-world actuators are exactly this fragile class. Section 6.3's technique-selection logic, read for a device, is:
Start from the objectiveA compliance baseline needs review and target-identification techniques; proving real-world exploitability needs validation, with its higher risk and cost. Prefer non-production targets for intrusive stepsNIST: "if a particular test technique is likely to cause a denial of service, it should probably be used against a non-production system" — for a device, a dedicated bench unit, never a unit in service. Schedule aggressive scans for low-traffic windowsand keep stress testing off anything safety-related. Bring a subject-matter expert(Section 6.5) when the assessor lacks device/protocol knowledge.
What should analysis and the report contain?
Section 7.3 is about turning raw findings into trustworthy ones. Automated tools "can produce a significant number of findings, but these findings often need to be validated to isolate false positives" — by manual examination or a second tool — and NIST pushes assessors past individual issues to root cause, listing recurring ones such as insufficient patch management, lack of security baselines, poor integration of security into the development life cycle, and weak or unenforced policy. For a device program, poor integration of security into the SDLC and insufficient patch management are usually the roots behind a long component-vulnerability list.
Sections 8.1–8.3 then require that each finding carry a mitigation recommendation (technical and process), that results feed a Plan of Action and Milestones (POA&M) with "specific, measurable, attainable, realistic, and tangible actions," and that remediation is verified by a retest — "the test team will be able to verify its implementation only if a mirror copy of the original test is performed."
Worked example (synthetic): scoping a test of a gateway
The following is a synthetic illustration, not a description of a real engagement. A manufacturer wants an overt assessment of a new industrial gateway before shipping firmware v2.3.
Plan (ROE)Scope: two bench units running v2.3, the firmware image, and the staging cloud tenant. Exclude list: production tenant and any customer device. Halting criterion: stop if a unit fails to boot or a safety output changes state; resume on the client lead's written approval. ReviewDocumentation review of the update and boot design; configuration review of the on-device services; build an SBOM from the image. Target identification and analysisPort/service identification on a bench unit; vulnerability scanning of the exposed services and the staging API; match the SBOM against known-vulnerability data (NVD, KEV, EPSS) to rank candidates. ValidationFor the two or three highest-ranked candidates, confirm on the bench unit under the ROE — off-hours, non-production only. Analysis and reportSeparate false positives, attach a root cause (here: an outdated component pinned in the build system), and record each finding with a mitigation and a POA&M entry. RetestAfter v2.3.1, re-run the exact scan and validation steps to verify the fix.
What are the common pitfalls?
- Treating a clean scan as a passed penetration test. A scan finds surface vulnerabilities only; it cannot speak to chained weaknesses.
- Writing no exclude list. Appendix B's "exclude list" is what keeps a tester off a production unit or a shared cloud tenant.
- Skipping the halting criteria. On a device with physical actuators, "when do we stop" is a safety control, not paperwork.
- Stopping at the finding, not the root cause (Section 7.3), which is where real improvement comes from.
- No retest. A fix that is not re-tested with a mirror of the original test is unverified (Section 8.3).
- Assuming the 2008 tool lists are current. The method is durable; the specific tool names in its appendices are not.
Where does SP 800-115 sit next to other references?
SP 800-115 is a technique-and-methodology reference, not a compliance checklist or a risk-scoring framework. For prioritizing which vulnerabilities matter once they're found, pair it with CVSS (see our CVSS v4 vs v3.1 guide) or CISA's KEV/EPSS signals (see our KEV + EPSS guide). For OT environments, where active scanning can disrupt control systems, read it alongside NIST SP 800-82 Rev. 3 (see our NIST SP 800-82 OT security guide); and NIST SP 800-53A picks up where SP 800-115's analysis phase leaves off.
A workable starting point
Pick the right category for the objective— a quick compliance check needs review and target-identification techniques; proving real-world exploitability needs target vulnerability validation, with its correspondingly higher risk and cost. Write the ROE before any tool runs, including an explicit exclude list and incident-handling/halting criteria, so a discovered real intrusion during testing doesn't get mistaken for test traffic or vice versa. Don't treat a vulnerability scan as a penetration testA clean scan result means no known signature matched — it says nothing about whether chained, lower-severity issues could still be combined into a real compromise. Decide overt vs. covert deliberatelyCovert testing measures detection and response readiness; overt testing measures the technical security posture itself. They answer different questions and cost different amounts.
SoftWSP's device and firmware work contributes to the Target Identification and Analysis layer of this method: its firmware scan extracts a CycloneDX software bill of materials from a binary image and matches the components against NVD, CISA KEV, FIRST EPSS and CISA's CSAF ICS advisories, with an AI-drafted report that cites the record behind each finding for an engineer to review — a structured input to scoping and analysis, not a substitute for the validation and reporting the standard describes.
This guide summarises NIST SP 800-115; it does not constitute legal or compliance advice. The document is from 2008 and has not been revised since — confirm current tool and technique recommendations against more recent sources before relying on them operationally.
Put this guide to work
Frequently asked questions
What is NIST SP 800-115?
NIST Special Publication 800-115, "Technical Guide to Information Security Testing and Assessment," published September 2008 by Karen Scarfone, Murugiah Souppaya, Amanda Cody and Angela Orebaugh. Its stated purpose is "to provide guidelines for organizations on planning and conducting technical information security testing and assessments, analyzing findings, and developing mitigation strategies." It replaced the earlier NIST SP 800-42, Guideline on Network Security Testing, and was developed under NIST's FISMA statutory responsibilities, though NIST's own text states nongovernmental organizations may use it voluntarily.
What are the three categories of technical assessment techniques in SP 800-115?
The document groups techniques into three categories, per its own Section 2.2: Review Techniques ("examination techniques used to evaluate systems, applications, networks, policies, and procedures to discover vulnerabilities, and are generally conducted manually" — documentation, log, ruleset and system configuration review, network sniffing, and file integrity checking); Target Identification and Analysis Techniques ("testing techniques" that "identify systems, ports, services, and potential vulnerabilities," generally with automated tools — network discovery, network port/service identification, vulnerability scanning, wireless scanning, and application security examination); and Target Vulnerability Validation Techniques (which "corroborate the existence of vulnerabilities" — password cracking, penetration testing, social engineering, and application security testing). The document states these carry increasing risk in that order: review techniques have minimal risk, target identification/analysis techniques moderate risk, and target vulnerability validation the highest risk to the target system.
What is the four-phase penetration testing methodology in SP 800-115?
Section 5.2.1 describes penetration testing as proceeding through Planning, Discovery, Attack, and Reporting phases (Figure 5-1), with Reporting running concurrently with the other three rather than only at the end. The document is explicit that "there are many acceptable ways of grouping the actions involved in performing penetration testing" — this is presented as one example, not the only valid model. The Discovery phase has two parts: information gathering/scanning, then vulnerability analysis against databases such as NIST's National Vulnerability Database (NVD). The Attack phase (Figure 5-2) has a feedback loop back to Discovery: Gaining Access leads to Escalating Privileges, System Browsing, and Installing Additional Tools, and any new access can trigger another discovery cycle on newly reachable systems.
What is the difference between overt and covert testing in SP 800-115?
Section 2.4.2 defines overt security testing ("white hat testing") as "performing external and/or internal testing with the knowledge and consent of the organization's IT staff," which allows the IT staff to limit impact and learn from observing the test. Covert security testing ("black hat testing") is performed "without the knowledge of the organization's IT staff but with the full knowledge and permission of upper management," often via a trusted third-party agent so the target's security staff doesn't prematurely confirm an exercise is underway. The document states covert testing's purpose "is to examine the damage or impact an adversary can cause — it does not focus on identifying vulnerabilities," and that it is "often time-consuming and costly" because the team must stay "under the radar," while overt testing is "less expensive, carries less risk ... and is more frequently used."
How does vulnerability scanning differ from penetration testing in this framework?
SP 800-115 places them in different technique categories with different purposes. Vulnerability scanning (a Target Identification and Analysis technique, Section 4.3) "identifies hosts and host attributes ... but it also attempts to identify vulnerabilities rather than relying on human interpretation of the scanning results," by matching discovered software against a signature database — the document notes it "can have a high false positive error rate" and only detects "surface" vulnerabilities that exist in isolation. Penetration testing (a Target Vulnerability Validation technique, Section 5.2) goes further: "While vulnerability scanners check only for the possible existence of a vulnerability, the attack phase of a penetration test exploits the vulnerability to confirm its existence," and penetration tests typically chain multiple vulnerabilities together to show real attacker reach rather than scoring each weakness independently.
Sources
- NIST: SP 800-115, Technical Guide to Information Security Testing and Assessment (September 2008, full text)
- NIST CSRC: SP 800-115 publication page
- OWASP Firmware Security Testing Methodology (FSTM), project repository
- NIST: SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security — publication page
About this guide
SoftWSP Firmware Security Desk
Covers the Firmware vulnerabilities & fixes cluster: 129 guides.

In this cluster
Questions about this page?
