Towards Fair Pay and Equal Work

Imposing View Time Limits in Crowdsourced Image Classification

Abstract

Crowdsourcing is a vital tool for rapid data annotation, yet flat-rate compensation often results in significant pay inequity due to worker speed variability. This paper investigates using task time limits to stabilize pay rates while maintaining data quality. Through a human study on an image classification task, we found that worker performance diminishes only slightly as view time decreases, and consensus algorithms remain effective at filtering complex images to preserve overall accuracy. Quantitatively, participants maintained consistent effort throughout the study and showed a psychometric trend favoring shorter time limits. These findings suggest that implementing task time limits is a practical approach to achieving more equitable compensation, mitigating the risks of overpayment and underpayment by creating a more predictable hourly rate. Our code and data are available at https://github.com/gordon-lim/sdogs-10h.