10 security tool evaluation guidelines
Extending the DevOps pipeline with security automation won’t come easy. To avoid wasted effort and unnecessary frustration, proceed carefully as you integrate and automate security tooling. During tool evaluation, take care to follow these guidelines:
Tools used within the DevSecOps pipeline must have a command-line interface (CLI). If a tool doesn’t have a CLI, then it cannot be added to automation servers that manage the DevSecOps pipeline.
Tools must provide a machine-parsable output, such as XML or JSON, so that other tools can parse and use the results in the vulnerability management system.
Mark and resolve false positives. Since invalid detections are likely to arise, staff should review and reject such findings to report only relevant information.
Include thresholds to specify the point at which specific pipeline step should fail. It is unrealistic to expect that the pipeline should fail when a single minor issue is reported.
Consider the requirements imposed on domains and set realistic thresholds. An example would be to permit no more than five vulnerabilities in total and not a single critical severity vulnerability.
The pipeline shouldn't run for an excessive amount of time. Aim to find an optimal set of tools and remove any duplications.
Pipelines should be parameterized to allow the option of specifying the level of detail in a scan.
Optimize the pipeline to skip steps. An example is to perform software composition analysis only when software dependencies change.
Occasionally, run the entire non-optimized pipeline to possibly capture potentially obscure vulnerabilities.
Be aware of licensing limitations for each tool you evaluate. There may be restrictions on parallel scans or threads that increase costs when scaling. Economize when sensible by using high-quality free and open-source tools.