How to avoid sourcing dross from the crowd

Research shows a questionnaire can weed out time-wasters

When Heinz in the US launched a user-generated advertising competition in 2007, asking members of the public to submit videos to help create a marketing campaign for its ketchup product, the company's marketing executives soon discovered that Sturgeon's Law – that "90% of everything is crap" – can be all too true.

They received hundreds more submissions than they expected. As a result, they had to spend enormous amounts of time and resources wading through the entries, which were of variable and often questionable quality, in order to discover anything of any value.

The entire experience reputedly cost the company at least as much as any professionally managed advertising campaign.

As an editorial piece by Louise Story in The New York Times in May, 2007, stated: "These companies have found that inviting consumers to create their advertising is often more stressful, costly and time-consuming than just rolling up their sleeves and doing the work themselves.

"Many entries are mediocre, if not downright bad, and sifting through them requires full-time attention. And even the most well-known brands often spend millions of dollars upfront to get the word out to consumers. Some people, meanwhile, have been using the contests as an opportunity to scrawl digital graffiti on the sponsor and its brand.

"Rejected Heinz submissions have been showing up on YouTube anyway, and visitors to Heinz's page on the site have written that the ketchup maker is clearly looking for 'cheap labour' and that Heinz is 'lazy' to ask consumers to do its marketing work."

The Heinz crowdsourcing campaign, from value to execution to participant experience and brand image, was a failure.

Avoiding the pain

Other brands have proven crowdsourcing can be a powerful platform for innovation, creative problem-solving and cheaper (or "greater value") results. The more minds at work on a project, logic tells us, the better the final result will be.

And there's the rub. Inviting so many people to share opinions and place submissions can also bring its own special type of pain for a business or brand.

But what if a company was able to weed out the dross? A recent research study led by Paul Patterson, a professor of marketing at UNSW Business School, reveals a key to the solution.

'The company can isolate the people they want in the contest and not worry about the 90% of terrible submissions'

- PAUL PATTERSON

Along with colleagues Bhuminan Piyathasanan from Thammasat Business School in Thailand, and Christine Mathies, a senior lecturer at UNSW Business School, Patterson partnered with Thailand's prominent MK Restaurants chain which, via its Facebook page, asked people to create a 30-second advertising campaign to promote the brand, its restaurants and its food. 

"The Heinz example showed the sometimes large, hidden cost to crowdsourcing. And crowdsourcing itself is a topic that hasn't been the focus of much empirical research," Patterson says.

"We wanted to work out how crowdsourcing for a creative campaign could create value to the company as well as how it can create value for the participants and, finally, how can companies work smarter by identifying the right people to make a submission," Patterson says.

"We wanted to find out how you could crowdsource 50 great ideas rather than 300 poor ones. Is it possible to identify the correct personality type to respond to a creative crowdsourced challenge? Is it possible to get the right people for the right job? In other words, work smarter instead of harder."

For the campaign there were just 10 prizes, including a couple of Apple computer products and several cash prizes of $100 each. The small relative value of the prizes meant they were likely a minor part of the motivation to enter.

The company received almost 300 submissions which were eventually whittled down to a finalist group of 10. Each person who made a submission also answered a series of questions that helped the academics understand their motivation, personality type, perceived level of personal creativity and level of engagement with the project.

 Innovative traits

"Studies that have been done on innovation indicate that three traits – innovative cognitive style, creative identity and creative process management – can drive people's ability to perform creatively," Patterson says.

He defines innovative cognitive style as how a person gathers, processes and organises information and generates original ideas or solutions; creative identity as how creative a person thinks they are; and creative process management as how engaged a person is with a project.

The results of the research indicated that creative process management had no influence at all on the quality of a submission. In fact, for those who were most engaged with the project – the more effort they put into that project – the worse the result tended to be. Creative identity was also not a significant driver of performance.

"The big driver of results was innovative cognitive style," Patterson says.

"When people go about gathering information and thinking about the project and problem, the way they gather, process and organise the information is actually a personality trait. People either have it or they don't. It is not something you can learn."

The ones you want

A high level of innovative cognitive style means a greater capacity for the generation of original, new and unusual perspectives. People in this group, Patterson discovered, are the ones you want to have responding to a crowdsourcing campaign around a creative problem. But what does this mean on a practical level?

There is a simple questionnaire, Patterson says, established in the field of social psychology that contains seven to nine questions or statements, each answered on a numbered scale from Strongly Agree to Strongly Disagree. This questionnaire can be used to identify people with high innovative cognitive style.

Sample statements include:

I am a person who has fresh perspectives on old problems; I am a person who copes with several new ideas and problems at the same time; I am a person who is methodical and systematic; I am a person who has original ideas.

"The company can isolate the people they want in the contest and not worry about the 90% of terrible submissions," Patterson says.

"Or you could let 300 people make a submission but ensure part of the submission is to answer the important personality questions. So you get 300 submissions but you only need to look at 30 of them. You can still thank everybody and give them recognition but you only really need to look at the submissions from people that scored high on the innovative cognitive style scale."