NewRedBook – The Nick Garner Blog

I think we need some warm bodies

A quick post…Some time ago I wrote a post with a pdf link to the leaked Google human reviewer document written in 2007. As some of you may know there has been a little more ‘openess’ in the big G and following that theme, this article gives great insight into how they use humans to augment the algo.

A few nice quotes:

Q: Can you give me a sense for how you approach evaluation?

A: We use two main kinds of evaluation data. One kind is we have human evaluators all over the world for whom we have a workflow system. They come to it and are fed things to evaluate. A typical thing is: Here is a query, you’re speaking French in Switzerland, here’s a URL, tell us on some kind of scale or some set of flags and description how good of a URL is that for that query.

Q: Who are these human raters?

A: They’re not volunteers. They’re paid, through contractors from third parties. We look for a basic level of education and communication skills, and in particular our one requirement is that they need to be able to some level in English. Other than that, what we’re really looking for is a broad cross-section of folks. Not a technology background, just like to use the Internet. We have some screening around testing of their ability to do some of the tasks we want them to do and follow the instructions.

Q: How important are the human raters vs. the more automated methods? And do you get alerts if search results or search behavior is not what you expect?

A: The human evaluators are pretty important for us today. The more automated or user behavior/click-based things really give you complementary kinds of data. Both have noise in them: Human evaluators make mistakes. Clicks are hard to interpret; people click or don’t click for all kinds of reasons.

It’s well worth taking some time out to trawl through this lot since it will give you a much better idea of how Google uses humans to help it’s search engine.