Gaussian Process Classification and Active Learning with Multiple Annotators



Learning from multiple annotators took a valuable step towards modeling data that does not fit the usual single annotator setting, since multiple annotators sometimes offer varying degrees of expertise. When disagreements occur, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we generalize GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework for building proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.


Machine Learning, crowdsourcing, gaussian processes, active learning


Efficient learning from crowdsourcing data

Related Project

Crowds - Understanding urban land use from digital footprints of crowds


International Conference on Machine Learning (ICML 2014), June 2014

Cited by

Year 2016 : 3 citations

 C Long, G Hua, A Kapoor, A joint gaussian process model for active visual recognition with expertise estimation in crowdsourcing, International Journal of Computer Vision, 2016

 D Padmanabhan, D Garg, S Shevadeâ?¦, A Robust UCB Scheme for Active Learning in Regression from Strategic Crowds, arXiv preprint arXiv: …, 2016

 E Weigl, W Heidl, E Lughofer, T Radauerâ?¦, On improving performance of surface inspection systems by online active learning and flexible classifier updates, Machine Vision and …, 2016

Year 2015 : 3 citations

 C Long, G Hua, Multi-class Multi-annotator Active Learning with Robust Gaussian Process for Visual Recognition, Proceedings of the IEEE International Conference …, 2015

 N Rohani, P Ruiz, E Besler, R Molinaâ?¦, Variational Gaussian process for sensor fusion, … 2015 23rd European, 2015

 M Venanzi, J Guiver, P Kohli, N Jennings, Time-Sensitive Bayesian Information Aggregation for Crowdsourcing Systems, arXiv preprint arXiv: …, 2015