[Context and motivation] Requirements assessment by means of the Kano model is common practice. As suggested by the original authors, these assessments are done by interviewing stakeholders and asking them about the level of satisfaction if a certain feature is well implemented and the level of dissatisfaction if a feature is not or not well implemented.
[Question/problem] Assessments via interviews are time-consuming, expensive, and can only capture the opinion of a limited set of stakeholders.
[Principal ideas/results] We investigate the possibility to extract Kano model factors (basic needs, performance factors, and delighters) from a large set of user feedback (i.e., app reviews). We implemented, trained, and tested several classifiers on a set of 2,592 reviews. In a 10-fold cross-validation, a BERT-based classifier performed best with an accuracy of 0.928. To assess the classifiers’ generalization, we additionally tested them on another independent set of 1,622 app reviews. The accuracy of the best classifier dropped to 0.725. We also show that misclassifications correlate with human disagreement on the labels.
[Contribution] Our approach is a lightweight and automated alternative for identifying Kano model factors from a large set of user feedback. The limited accuracy of the approach is an inherent problem of missing information about the context in app reviews compared to comprehensive interviews, which also makes it hard for humans to extract the factors correctly.
Tue 18 AprDisplayed time zone: Brussels, Copenhagen, Madrid, Paris change
14:00 - 15:30 | Session R4 - Requirements and App Review ClassificationResearch Papers at Llívia Chair(s): Maya Daneva University of Twente | ||
14:00 40mTechnical design | Requirements classification using fastText and BETO in Spanish documents Research Papers P: Maria Isabel Limaylla Lunarejo Universidade da Coruña, A: Nelly Condori-Fernández Universidad de Santiago de Compostela, A: Miguel Rodríguez Luaces Universidade da Coruña, D: Michelle Binder University of Cologne, D: Annika Vogt University of Cologne File Attached | ||
14:40 40mTechnical design | Automatically Classifying Kano Model Factors in App Reviews Research Papers P: Michelle Binder University of Cologne, P: Annika Vogt University of Cologne, A: Adrian Bajraktari University of Cologne, A: Andreas Vogelsang University of Cologne, D: Maria Isabel Limaylla Lunarejo Universidade da Coruña |