REFSQ 2023
Mon 17 - Thu 20 April 2023 Barcelona, Spain
Tue 18 Apr 2023 11:40 - 12:20 at Llívia - Session R2 - NLP and ML for RE I Chair(s): Sallam Abualhaija

[Context and Motivation] Requirements in tender documents are often mixed with other supporting information. Identifying requirements in large tender documents could aid the bidding process and help estimate the risk associated with the project. [Question/problem] Manual identification of requirements in large documents is a resource-intensive activity that is prone to human error and limits scalability. This study compares various state-of-the-art approaches for requirements identification in an industrial context. For generalizability, we also present an evaluation on a real-world public dataset. [Principal ideas/results] We formulate the requirement identification problem as a binary text classification problem. Various state-of-the-art classifiers based on traditional machine learning, deep learning, and few-shot learning are evaluated for requirements identification based on accuracy, precision, recall, and F1 score. Results from the evaluation show that the transformer-based BERT classifier performs the best, with an average F1 score of 0.82 and 0.87 on industrial and public datasets, respectively. Our results also confirm that few-shot classifiers can achieve comparable results with an average F1 score of 0.76 on significantly lower samples, i.e., only 20% of the data. [Contribution] There is little empirical evidence on the use of large language models and few-shots classifiers for requirements identification. This paper fills this gap by presenting an industrial empirical evaluation of the state-of-the-art approaches for requirements identification in large tender documents. We also provide a running tool and a replication package for further experimentation to support future research in this area.

Tue 18 Apr

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

11:00 - 12:30
Session R2 - NLP and ML for RE IResearch Papers at Llívia
Chair(s): Sallam Abualhaija University of Luxembourg
11:00
40m
Technical design
Using Language Models for Enhancing the Completeness of Natural-language Requirements
Research Papers
P: Dipeeka Luitel University of Ottawa, A: Shabnam Hassani University of Ottawa, A: Mehrdad Sabetzadeh University of Ottawa, D: Sarmad Bashir RISE Research Institutes of Sweden
Pre-print
11:40
40m
Scientific evaluation
Requirement or not, that is the question: A case from the railway industry
Research Papers
P: Sarmad Bashir RISE Research Institutes of Sweden, A: Muhammad Abbas RISE Research Institutes of Sweden AB, A: Mehrdad Saadatmand RISE Research Institutes of Sweden, A: Eduard Paul Enoiu Mälardalen University, A: Markus Bohlin Mälardalen University, A: Pernilla Lindberg Alstom, D: Dipeeka Luitel University of Ottawa
DOI Pre-print