A match made in heaven? AI-driven matching of vulnerabilities and security unit tests

Software vulnerabilities are often detected via taint analysis, penetration testing, or fuzzing. They are also found via unit tests that exercise security-sensitive behavior with specific inputs, called vulnerability-witnessing tests. Generative AI models could help developers in writing them, but they require many examples to learn from, which are currently scarce. This paper introduces VuTeCo, an AI-driven framework for collecting examples of vulnerability-witnessing tests from Java repositories. VuTeCo carries out two tasks: (1) The "Finding"task to determine whether a unit test case is security-related, and (2) the "Matching"task to relate a test case to the vulnerability it witnesses. VuTeCo addresses the Finding task with UniXcoder, achieving an F₀.₅ score of 0.73 and a precision of 0.83 on a test set of unit tests from Vul4J. The Matching task is addressed using DeepSeek Coder, achieving an F₀.₅ score of 0.65 and a precision of 0.75 on a test set of pairs of unit tests and vulnerabilities from Vul4J. VuTeCo has been used in the wild on 427 Java projects and 1,238 vulnerabilities, obtaining 224 test cases confirmed to be security-related and 35 tests correctly matched to 29 vulnerabilities. The validated tests were collected in a new dataset called Test4Vul. VuTeCo lays the foundation for large-scale retrieval of vulnerability-witnessing tests, enabling future AI models to better understand and generate security unit tests.

Content

Subjects

Language Models

Mining Software Repositories

Security Testing

Unit Testing

Vulnerability-witnessing Tests

DDC Class

005.8: Computer Security

006.3: Artificial Intelligence

Funding References

Funding(s)

Cybersecurity for AI-Augmented Systems

Files

Lizenz

https://creativecommons.org/licenses/by/4.0/

Publication version

publishedVersion

Download

Name

3793302.3793345.pdf

Type

Main Article

Size

1.09 MB

Format

Adobe PDF