Paper Reading #21

Human model evaluation in interactive supervised learning


Authors:

Rebecca Fiebrink Princeton University, Princeton, New Jersey, USA
Perry R. Cook Princeton University, Princeton, New Jersey, USA
Dan Trueman Princeton University, Princeton, New Jersey, USA


Proceeding
CHI '11 Proceedings of the 2011 annual conference on Human


Summary
Machine learning allows for algorithms to determine output from an input based on a trained model for the data. Not every input is specifically represented in the model, so the model needs to make generalizations. Interactive machine learning (IML) uses model evaluation so that users can assess how well a certain model is performing. Without IML, understanding these models can become a difficult debugging process.

Hypothesis /Motivation

The first goal of the paper is to learn more about how interactive machine learning based on user evaluation. The second goal is to determine how these evaluations effect what users do in utilizing IML, specifically what algorithms they use. The last goal is to describe the utility of supervised machine learning.

Methods
A tool called the Wekinator was designed to encompass several standard machine learning algorithms. It also allows for the training, modifying, and visualization of real time data used to create the model. Three studies were performed using the Wekinator.

• Study 'A' involved 6 PhD students and one faculty member at Princeton's Music Composition department. For 10 weeks, they met and discussed how they were using the tools in their work, and offered suggestions for the software. At the end, they completed a questionnaire about how they felt about the software.
• Study 'B' involved 21 undergraduate students. They were asked to design systems of input and train models using neural networks to create different sounds. The inputs needed to be continuously controlled.
• Study 'C' was a case study that involved a cellist for the purpose of building a gesture recognizing cello bow, called the "K-Bow." This had sensors for acceleration, tilt, and relative position. Also grip pressure is measured. The goal was to create a model to process the sensor data to create physically appropriate music.

Results

All 3 study groups used direct evaluation to determine the system is working properly in processing input data. However, in studies 'B' and 'C', the researchers noted that the participants used cross validation todetermine how well the models evaluated new data. Subjective evaluation was used in study C to find errors in the model that were not found using cross or direct evaluation.

Discussion
Machine learning is very interesting to me, primarily since I'm enrolled in CSCE 420. Creating good models is very good, but I personally would like to see a fully automated process for model creation and evaluation. This isn't to say that intelligence depends on having best model of the world, but it is a good start.