New Study Shows How Social Influence Can Significantly Manipulate Online Ratings

Digital ratings platforms proliferate our lives and influence the consumption of wide ranges of goods and services. Given their widespread use and relative transparency, most users have come to expect that the comments and ratings on  sites such as TripAdvisor, Angie’s List or Open Table accurately portray the wisdom of the crowds.

But recently published research should make you change your expectations.

According to the article just published in Science by professors Lev Muchnik, Sinan Aral and Sean Taylor, the aggregate ratings can be manipulated. With very little prodding, people are easily herded.  The randomized experiment they ran is extremely clever with more than 300,000 ratings that were scored over five months.  For the details of exactly how they did it, read the paper.  But in layman’s terms, they worked with an unnamed news sharing site, gave a positive first comment to a portion of comments, a negative comment to another portion of comments and had a control where they did nothing.

It turns out, the first rating  is extremely important.

• If the first manipulated comment was positive, this comment begat significantly more positive comments, increasing the likelihood of positive ratings by 32%, and a 25% increase in the mean rating over the five months of the experiment.
• If the first manipulated comment was negative, there was a higher probability that subsequent comments would be negative, relative to the control.  However, over the five months, this was offset by a larger “correction effect” so that the negative effect was neutralized.

Since this experimental research was for one specific kind of rating system for one web site,  how consistent is  the 25% kick up in mean positive ratings across different kinds of web sites? In a recent interview,  co-author and MIT Sloan School of Management  Professor Sinan Aral said, “I do believe that there will be some variation across different web sites, but I think the numbers that we specify, the parameter estimates  that are in the paper are relatively generalizable and you will find the same order of magnitude that you would find on other web sites."

Aral says that one practical application of this research for ratings platforms is that they should aggressively solicit any and all responses.  “Because the negative feedback doesn’t  snowball, but the positive feedback does,  increased turnout for providing UGC or ratings only seems to help  on net. So yes, soliciting feedback would prove to be on average universally good for these companies."

Can this experimental research encourage unethical behavior?  Aral says, “For businesses there is clearly an incentive to manipulate the ratings,  especially early on. Because you get this 25% increase in the mean rating of the item with a single positive up vote of the item. And that is a big marginal change in a final score from a single action at the beginning."

He continues, “And for the platforms that use ratings to try to inform consumers, I think it is a clear sign and indication that we need more science to try to understand this behavior and better design to try to inoculate ratings and rating systems from this type of bias and incentives for this type of manipulation. We clearly need to do more research. And we clearly need to test out better designs for collective intelligence systems like this. “

The recent paper was based on a ratings of opinions that have no commercial value; but the group is currently  conducting studies where the users are paying money for actual products. Those results are not yet in. “My intuition is that there will be variation in the impact of social influence as a function of price,” says Aral.  "Maybe people rely  more or less on the opinions of others when they are buying a high ticket item, versus a low priced item or a free item. But it is still not clear.  There hasn’t been research that shows that definitively yet.  We’re working on it, and others should work on it.”