17 results
Now showing1 - 10 of 17
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Emerging challenges in threat modeling for GenAI-augmented systems: a view from the trenches(2026-10); ; ; 16 2 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , A match made in heaven? AI-driven matching of vulnerabilities and security unit tests(Association for Computing Machinery (ACM), 2026-04); ; Software vulnerabilities are often detected via taint analysis, penetration testing, or fuzzing. They are also found via unit tests that exercise security-sensitive behavior with specific inputs, called vulnerability-witnessing tests. Generative AI models could help developers in writing them, but they require many examples to learn from, which are currently scarce. This paper introduces VuTeCo, an AI-driven framework for collecting examples of vulnerability-witnessing tests from Java repositories. VuTeCo carries out two tasks: (1) The "Finding"task to determine whether a unit test case is security-related, and (2) the "Matching"task to relate a test case to the vulnerability it witnesses. VuTeCo addresses the Finding task with UniXcoder, achieving an F₀.₅ score of 0.73 and a precision of 0.83 on a test set of unit tests from Vul4J. The Matching task is addressed using DeepSeek Coder, achieving an F₀.₅ score of 0.65 and a precision of 0.75 on a test set of pairs of unit tests and vulnerabilities from Vul4J. VuTeCo has been used in the wild on 427 Java projects and 1,238 vulnerabilities, obtaining 224 test cases confirmed to be security-related and 35 tests correctly matched to 29 vulnerabilities. The validated tests were collected in a new dataset called Test4Vul. VuTeCo lays the foundation for large-scale retrieval of vulnerability-witnessing tests, enabling future AI models to better understand and generate security unit tests.Publicationtype: Conference Paper19 8 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Discrete prompt optimization using genetic algorithm for secure Python code generationLarge language models (LLMs) have become powerful tools that enable novice developers to generate production-level code. However, research has highlighted the security risks associated with such code generation, due to the high volume of generated software vulnerabilities. Recent studies have explored various techniques for automatically optimizing prompts to elicit desired responses from LLMs. Among these methods, Genetic Algorithms (GAs), which search for optimal solutions by evolving an initial population of candidates through iterative mutations, have gained attention as a lightweight and effective prompt optimization approach that does not require large datasets or access to model weights. However, their potential has not yet been examined in the context of secure code generation. In this paper, we use GA to develop a discrete prompt optimization pipeline specifically designed for secure code generation. We introduce two domain-specific prompt mutation techniques and assess how incorporating these security-focused mutations alongside general-purpose techniques, such as back translation and paraphrasing, affects the security of Python code generated by LLMs. Results demonstrate that our security-specific mutation techniques led to prompts with richer security context compared to the generic mutation techniques. Furthermore, combining these techniques with generic mutations substantially reduced the number of security weaknesses in the LLM-generated code. We also observed that prompts optimized for a particular LLM tend to perform best on that same model, highlighting the importance of model-specific prompt optimization.Publicationtype: Journal Article37 19 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Beyond prompting: the role of phrasing tasks in vulnerability prediction for JavaPredicting vulnerability in a code element, such as a function or method, often leverages machine or deep learning models to classify whether it is vulnerable or not. Recently, novel solutions exploiting conversational large language models (LLMs) have emerged, which allow the formulation of the task through a prompt containing natural language elements and the input code element, obtaining a natural language as a response. Although initial promising results, there is currently no broad exploration of (i) how the input prompt influences the prediction capabilities and (ii) what characteristics of the model response relate to correct predictions. In this paper, we conduct an empirical investigation into how accurately two popular conversational LLMs, i.e., GPT-3.5 and Llama-2, predict whether a Java method is vulnerable by employing a thorough prompting strategy by (i) adhering to the Zero-Shot and Zero-Shot Chain-of-Thought techniques and (ii) formulating the prediction task in alternative ways via rephrasing. After a manual inspection of the responses generated, we observed that GPT-3.5 displayed more variable F1 scores compared to Llama-2, which was steadier but often gave no direct classification. ZS prompts achieved F1 scores between 0.53 and 0.69, with a tendency of classifying methods positively (i.e., ‘vulnerable’); conversely, ZS-CoT presents a broader range of scores, varying from 0.35 to 0.72, with often inconsistencies in the results. Then, we phrased the task in their “inverted form”, i.e., asking the LLM to check for the absence of vulnerabilities, which led to worse results for GPT-3.5, while Llama-2 occasionally performed better. The study further suggests that textual metrics provide important information on LLM outputs. Despite this, these metrics are not correlated with actual outcomes, as the models respond consistently with uniform confidence, irrespective of whether the outcome is correct or not. This underscores the need for customized prompt engineering and response analysis strategies to improve the precision and reliability of LLM-based systems for vulnerability prediction. In addition, we applied our study to two state-of-the-art LLMs, validating the broader applicability of our methodology. Finally, we performed an analysis of various textual properties of the model responses, such as response length and readability scores, to further explore the characteristics of the responses given for vulnerability detection tasks.Publicationtype: Journal Article17 20 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Prompting techniques for secure code generation: a systematic investigation(Association for Computing Machinery (ACM), 2025-10-04); ; ; ; Large Language Models (LLMs) are gaining momentum in software development with prompt-driven programming enabling developers to create code from Natural Language (NL) instructions. However, studies have questioned their ability to produce secure code and, thereby, the quality of prompt-generated software. Alongside, various prompting techniques that carefully tailor prompts have emerged to elicit optimal responses from LLMs. Still, the interplay between such prompting strategies and secure code generation remains underexplored and calls for further investigations. Objective: In this study, we investigate the impact of different prompting techniques on the security of code generated from NL instructions by LLMs. Method: First, we perform a systematic literature review to identify the existing prompting techniques that can be used for code generation tasks. A subset of these techniques are evaluated on GPT-3, GPT-3.5, and GPT-4 models for secure code generation. For this, we used an existing dataset consisting of 150 NL security-relevant code generation prompts. Results: Our work (i) classifies potential prompting techniques for code generation (ii) adapts and evaluates a subset of the identified techniques for secure code generation tasks, and (iii) observes a reduction in security weaknesses across the tested LLMs, especially after using an existing technique called Recursive Criticism and Improvement (RCI), contributing valuable insights to the ongoing discourse on LLM-generated code security.Publicationtype: Journal Article40 50 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Comparison of static analysis architecture recovery tools for microservice applications(Springer, 2025-10-01); ; ; ; Architecture recovery tools help software engineers obtain an overview of the structure of their software systems during all phases of the software development life cycle. This is especially important for microservice applications because they consist of multiple interacting microservices, which makes it more challenging to oversee the architecture. Various tools and techniques for architecture recovery (also called architecture reconstruction) have been presented in academic and gray literature sources, but no overview and comparison of their accuracy exists. This paper presents the results of a multivocal literature review with the goal of identifying architecture recovery tools for microservice applications and a comparison of the identified tools’ architectural recovery accuracy. We focused on static tools since they can be integrated into fast-paced CI/CD pipelines. 13 such tools were identified from the literature and nine of them could be executed and compared on their capability of detecting different system characteristics. The best-performing tool exhibited an overall F1-score of 0.86. Additionally, the possibility of combining multiple tools to increase the recovery correctness was investigated, yielding a combination of four individual tools that achieves an F1-score of 0.91.Publicationtype: Journal Article17 22 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , A taxonomy of functional security features and how they can be located(Springer, 2025-05-28); ; ; ; Security must be considered in almost every software system. Unfortunately, selecting and implementing security features remains a challenge due to the wide variety of security threats and possible countermeasures. While security standards are intended to help developers, they are usually too abstract and vague to help implementing security features, or they merely help configuring such. A resource that describes security features at an abstraction level that lies between high-level (i.e., rather too general) and low-level (i.e., rather too specific) security standards could facilitate secure systems development. This resource should support the selection of appropriate security features to achieve high-level security goals, allow easy retrieval of relevant low-level details, and provide pointers to suitable ways to realize the security features. To realize security features, developers typically use external security libraries or frameworks, to minimize implementation mistakes. Even when using libraries, developers still make mistakes when writing code to integrate them, often resulting in security vulnerabilities. When security incidents occur or the system needs to be audited or maintained, it is essential to know what security features have been implemented and, more importantly, where they are located. This task, commonly referred to as feature location, is often tedious and error-prone. While dedicated feature location techniques exist, they require significant manual effort or adherence to strict development processes, preventing their use. Therefore, we have to support long-term tracking of implemented security features. We present a study of security features presented in the literature and their coverage in popular security frameworks. We contribute (1) a taxonomy of 68 functional implementation-level security features including a mapping to widely used security standards, (2) an examination of 21 popular security frameworks concerning which of these security features they provide, and (3) a discussion on the representation of security features in source code. Our taxonomy aims to aid developers in selecting appropriate security features and security frameworks, as well as relating them to security standards when they need to choose and implement security features for a software system.Publicationtype: Journal Article26 29 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , On the understandability of design-level security practices in infrastructure-as-code scripts and deployment architectures(Association for Computing Machinery (ACM), 2024-12-26); ; ; ; Infrastructure as Code (IaC) automates IT infrastructure deployment, which is particularly beneficial for continuous releases, for instance, in the context of microservices and cloud systems. Despite its flexibility in application architecture, neglecting security can lead to vulnerabilities. The lack of comprehensive architectural security guidelines for IaC poses challenges in adhering to best practices. We studied how developers interpret IaC scripts (source code) in two IaC technologies, Ansible and Terraform, compared to semi-formal IaC deployment architecture models and metrics regarding design-level security understanding. In a controlled experiment involving ninety-four participants, we assessed the understandability of IaC-based deployment architectures through source code inspection compared to semi-formal representations in models and metrics. We hypothesized that providing semi-formal IaC deployment architecture models and metrics as supplementary material would significantly improve the comprehension of IaC security-related practices, as measured by task correctness. Our findings suggest that semi-formal IaC deployment architecture models and metrics as supplementary material enhance the understandability of IaC security-related practices without significantly increasing duration. We also observed a significant correlation between task correctness and duration when models and metrics were provided.Publicationtype: Journal Article18 36 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Managing security evidence in safety-critical organizationsWith the increasing prevalence of open and connected products, cybersecurity has become a serious issue in safety-critical domains such as the automotive industry. As a result, regulatory bodies have become more stringent in their requirements for cybersecurity, necessitating security assurance for products developed in these domains. In response, companies have implemented new or modified processes to incorporate security into their product development lifecycle, resulting in a large amount of evidence being created to support claims about the achievement of a certain level of security. However, managing evidence is not a trivial task, particularly for complex products and systems. This paper presents a qualitative interview study conducted in six companies on the maturity of managing security evidence in safety-critical organizations. We find that the current maturity of managing security evidence is insufficient for the increasing requirements set by certification authorities and standardization bodies. Organizations currently fail to identify relevant artifacts as security evidence and manage this evidence on an organizational level. One part of the reason are educational gaps, the other a lack of processes. The impact of AI on the management of security evidence is still an open question.Publicationtype: Journal Article10 73 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , Detection strategies for microservice security tactics(2024-05-01); ; ; ; Microservice architectures are widely used today to implement distributed systems. Securing microservice architectures is challenging because of their polyglot nature, continuous evolution, and various security concerns relevant to such architectures. This article proposes a novel, model-based approach providing detection strategies to address the automated detection of security tactics (or patterns and best practices) in a given microservice architecture decomposition model. Our novel detection strategies are metrics-based rules that decide conformance to a security recommendation based on a statistical predictor. The proposed approach models this recommendation using Architectural Design Decisions (ADDs). We apply our approach for four different security-related ADDs on access management, traffic control, and avoiding plaintext sensitive data in the context of microservice systems. We then apply our approach to a model data set of 10 open-source microservice systems and 20 variants of those systems. Our results are detection strategies showing a very low bias, a very high correlation, and a low prediction error in our model data set.Publicationtype: Journal Article53 57