Don’t Do Your Experiments Double-blind: The Importance of Checking Your Data

Abstract

In this paper, we investigate what could happen if you run machine learning experiments on data found somewhere on the internet, without first examining this data. As an example, we did polarity recognition on a data set extracted from Booking.com. We found that a) the form of the data in the dataset sometimes made polarity judgements hard for humans and probably also for systems, b) naive use of the data results in a different task than polarity recognition as the content of the data fields does not always comply with the field descriptors, and c) the comments in the data set come in several, quite different, subtypes, so that recognition quality rather varies with the choice for training and test data. On the basis of these findings we conclude that our advice to inspect data before using it is indeed valuable.