Agenda > Tous les événements

Nicolas Carayol

(GREThA)

« Evaluating the Underlying Qualities of Items and Raters from a Series of Reviews »

avec Matthew O. Jackson (Stanford University)

Abstract:
Consumers and businesses rely on others’ ratings of many items when making choices. However, individual reviewers are very heterogeneous: they vary in the accuracy of their ratings, and some are biased — either systematically over- or under-rating items, or even distorting reviews to manipulate a rating. Thus, the prominent technique of providing an average of a product’s ratings is generally a biased estimate of an item’s true quality, and ends up being inefficiently noisy as it equally weights to the least and most accurate reviewers. We provide a new technique that processes a series of ratings by a group of reviewers and simultaneously provides: unbiased and efficient estimates of the items’ true qualities, together with estimates of each reviewer’s bias and accuracy. We show that our technique generates significant improvements over average rating in recovering true qualities on simulated data, showing this edge increases dramatically as the number of items increases.
We illustrate the technique with two applications to wine tastings in which multiple experts rate multiple wines. Revisiting the famous 1976 wine tasting that compared Californian and Bordeaux wines, our technique finds substantial variation in the reviewers’ accuracies and the new scores provide essentially a three-way tie for the top wine, with a narrow best point-score by a French wine, in contrast to the original average ratings which picked an American wine as the `winner’. We also apply the technique to more than forty thousand expert ratings of Bordeaux wines of vintages from 1998 to 2015, and provide estimates of prominent experts’ biases and accuracies. We show that our estimated wine quality significantly predicts prices (with a ten percent increase in estimated quality predicting a thirty five percents increase in wine price), even when controlling for prominent experts ratings which are insignificant when matched with our quality estimates, or with the average rating. We also predict experts’ later adjustments to their ratings and show that our estimates of experts’ accuracies correspond to the extent to which their ratings predict prices.