Diminished Control in Crowdsourcing: An Investigation of Crowdworker Multitasking Behavior
This sixth, and final, contribution in this jam-packed issue of TOCHI investigates the complexities of crowdworkers who (much like the rest of us, sadly enough) typically cannot spend even 5 minutes (and probably far less if they happen to be an Editor-in-Chief) on any task without getting distracted, or interrupted, by something else.
Yet, as the authors of this article point out, the presence of naturalistic interruptions in crowdwork, in completely uncontrolled settings outside of the laboratory environment, presents both challenges and opportunities.
The challenge, of course, is rather obvious: if crowdworkers are attending to other tasks, they are most pointedly not attending to their crowdsourced task, which may be a source of undesired variability (if not outright poor performance) in experimental results. The authors therefore set out to see if they can detect, and possibly mitigate, such interruptions.
The opportunity is less obvious, but arguably even more important. The effect of naturally occurring interruptions—as opposed to those artificially imposed by laboratory settings—has received surprisingly little attention in the literature, presumably because such naturally occurring interruptions are deviously difficult to elicit, and study.
The article makes great progress on both fronts, illustrating in detail how crowdworker inattention can be detected, and even limited through appropriate interventions, while also illustrating how the presence of interruptions and distractions in the real world allows for naturalistic study of these behaviors, turning a supposed “nuisance” into the object of study.
It’s a clever turnabout by the authors, and I learned a great deal about this space by reading the article.
But I must admit—
I do find it vaguely disturbing that such is the state of modernity, that it has proven necessary for the term crowdworker to creep into the English language in the first place … but then again, perhaps I am bit too familiar with science-fictional tropes for my own good (grin).
(http://dx.doi.org/10.1145/2928269).