Automated Visual Understanding and Detection of User Interface Dark Patterns

Abstract

As user-facing applications become more prevalent in modern society, there has been an increase in interest related to the development of efficient UX and GUI design, yet a lack of focus on ethical design practices. Perhaps one of the most damaging ethical issues currently facing users is the use of dark patterns, or interfaces designed to maliciously persuade users into taking actions benefiting the service. The goal of this project is to analyze GUIs in mobile and web applications and detect dark patterns using machine learning tools and techniques. To do this, we began by preparing a dataset of over 1500 screenshots containing dark patterns from popular shopping websites. In order to provide our machine learning models with precise information from which to learn the visual semantics of these dark patterns, we placed labels on all the instances of dark patterns within the screenshots. We plan to use this data to train a Faster-RCNN neural object detection algorithm to automatically detect the presence of dark patterns present within software GUIs while exploring the possibility of extracting textual information through Optical Character Recognition. The ultimate goal of our work is to design a browser or mobile phone plugin that can run in the background and automatically detect the presence of dark patterns when users encounter them, and provide guidance on the potential actions that can be taken.