Crowdsourcing, Collaboration, Feedback, and Grading
January 10, 2013 § Leave a comment
One of the challenges of crowdsourcing identified by the Transcribe Bentham project was a disconnect between a vision for crowdsourcing as driven by collaborative effort and community feeling and the reality that individual contributors seemed to be more driven by receiving feedback from editorial staff and not from fellow contributors. In “Building A Volunteer Community: Results and Findings from Transcribe Bentham” after reporting the relatively low amount of collaborative work on manuscripts, Tim Causer and Valerie Wallace conclude,
This all suggests that volunteers appeared to prefer starting transcripts from scratch, and to work alone (Table 6), with communication and acknowledgement from staff being of much greater importance than collaboration with other users. (Digital Humanities Quarterly. 2012. 6.2, paragraph 72)
This situation reminds me of the challenge of promoting collaborative work in the classroom, a challenge I often experienced when managing peer review of student writing. I found that I had to “sell” peer review to my students because they didn’t value the feedback of fellow students but rather wanted feedback from the instructor–essentially the authority figure in the classroom. I think we see the same thing with the Transcribe Bentham project. Like students, contributors want validation from the experts, i.e., those running the project.
This need for expert acknowledgment presents a challenge for workflow in crowdsourcing projects. On the one had, projects use crowdsourcing because they need help with staffing, but crowdsourcing, itself, may require significant staff time. In a session on crowdsourcing at THATCAmp Texas 2011 (google doc with notes is here), Ben Brumfield and others made the point that maintaining relationships with volunteers is crucial, especially when you have a handful of contributors and not so much of a crowd. Ben described how he had somehow inadvertently offended a major contributor to his project and had to figure out how to lure him back. Likewise, Transcribe Bentham had a handful of “Super Transcribers” (“Building a Volunteer Community,” paragraph 47) and needed to maintain those relationships.
Given limited staff time, crowdsourcing only pays off for projects when there is an automated interface that can manage as much of the volunteer contributions as possible. Being able to provide satisfaction to volunteers is an added benefit for such an interface. Some projects rely on gamification to provide that satisfaction. For example, Transcribe Bentham used the following elements of incentives and competition:
The “Benthamometer” tracked the progress of transcription, while the leaderboard recorded and publicly recognised the efforts of the most diligent transcribers (Figures 5 and 6). Volunteers received points for every edit made; as an incentive we devised a multi-tiered ranking system, a progress ladder stretching from “probationer” to “prodigy” for transcribers to climb. We also intended to utilise a gift function which allowed editors to award users with virtual gifts – an image of the Collected Works for example – whenever they reached a milestone. “Team-building” features like these have been found to be useful in stimulating participation by other projects like Solar Stormwatch and Old Weather: we hoped to facilitate interaction between users, to generate healthy competition, and to develop a sense of community. (“Building a Volunteer Community,” paragraph 24).
The idea is that the interface can provide feedback to contributors to demonstrate their progress, but as described above, these elements may not have been enough for the Bentham transcribers who wanted validation from the authorities running the project not from a scoreboard. So, how can projects offer that validation when they are limited by staff time. Or to put it back in the context of teaching and learning, how can faculty efficiently give feedback to their students?
Tuesday, an article in Inside Higher Ed called Crowdsourcing Comments caught my attention because it suggests a potential solution. In it, Alexandra Tilsley reports on a new tool called “Caesar” created by MIT computer science professor, Rob Miller to divide up chunks of student-created code in his “Elements of Software Construction” course and assign these code-chunks to other students, alumni and graders for review so that their are multiple reviewers for each piece of code. Reviewers are selected not randomly, but rather based on reputation score. Tilsley explains,
Each reviewer has a reputation score, based on the quality of his comments, as judged by how often the comments get a “thumbs up” or “thumbs down.” (“Crowdsourcing Comments“)
So, students get feedback on their feedback. The system also teaches them to be better reviewers by pairing more and less skilled reviewers; those with less skills will benefit from the model of the more skilled reviewers. This feature is essential for peer review, since one of the key benefits of peer review is learning to be a better critic first of the work of others and then for a students’ own work.
One final feature, seems especially relevant for crowdsourcing projects:
Though graders still review every student’s work, the process goes much faster, Miller said, because they are only looking at certain sections, chosen by Caesar, and because they can simply give a thumbs-up to a comment, rather than starting from scratch. (“Crowdsourcing Comments“)
This feature seems promising for projects where contributors want that validation from project staff because it will reduce the time that project staff must spend on validating contributors. And, such a system could both teach contributors to give feedback to each other and manage collaboration between contributors. I’m not sure if Miller is working with any crowdsourcing projects in digital humanities, but it would be worth investigating. And back in the writing classroom, the idea of reputation score may help students become better reviewers and participants in peer review.