How Many Points is That Wine in the Window?

Quite a flurry o' writing and posts this week on wine rating systems. I started this as a comment over on Winecast, in response to Tim's proposed five star rating system. (And then it grew...)

Briefly, Tim surveys the various systems in use, from the WS/Parker 100 point lunacy to the 10 point system used over on vinography. Of course, vinography's scale is really a 20 point scale since Alder uses half points - and similarly, Winecast's five stars are really a 10 point scale since he proposes using half stars. (And what is a half star anyway? A planet?)

There is really nothing new about a five star scale. Like Amazon, WineQ has been using five stars, though they seem to have 1/3 stars too, and I still haven't understood how two 3.5 star ratings can average 3.0 stars....

Myself, I've been thinking of proposing an 11 point scale to take into account all of the over-ripe and over-oaked bomber wines out there. I think an 11 point scale would be a better way to go, since a "five and a half star" system just sounds funny, and the 11 points would match the volume control on my Spinal Tap brand amplifier (sorry).

Of course, then someone would want half point to differentiate between a 10.5 and 11.0 fruit bomb, and the madness starts again....

I realize we are a species of measurers. (Our ancestors probably knew how far it was in mackerels between the village and the river long before the distance from Cal Tech to Pasadena City Hall was known but I digress...) But with wine, I want to know if you have a strong opinion about this hooch, either way, and then most importantly WHY you feel that way - because your 2 star acidic shower cleaner could be my perfect 5 star crisp Sunday afternoon refreshment. But I won't know this unless you also post why you gave it those lousy 2 stars!

That brings up another point, which is the averaging of points from several ratings. I realize this probably feeds our innate desire to have a "leader" or "winner" - but why should one rating be able to pull all the others down? In the above example, those scores would be averaged on most sites to a 3.5 stars (3.0 stars on WineQ). As a result, we have a wine that actually generates strong feelings either way, being relegated to the anonymity of the pack.

So, if you are going to have numeric ratings, please back them up with meaningful descriptions. And please lose the average ratings - unless you are clever and devise some way to display standard deviations using half moons or something....

Comments

[Mike Duffy]

You may have noticed over on Amazon.com that they now show you how many people rated something with each star rating (1-5), so that you can see whether it's both high and low ratings averaging to the middle, or a bunch of average ratings.

"Life is too short to drink bad wine from cheap glasses."

And thanks for referring the Wines & Vines writer over my way.

[el jefe]

hi Mike - I had not noticed that! I actually like that, but it's one of those things where it works when there are enough ratings. Kind of like averages, means, and deviations...:) Sounds like a threshold is called for.

I told Tina you taught me everything I know about blogging - sorry!

[el jefe]

oh, and: "I would rather drink good wine from cheap glasses then drink bad wine from expensive glasses."

[Joshua Zader]

Jeff,

Since I actually coded the script that calculates our ratings on the WineQ web site (Marsh was busy that day doing real programming) I can answer your questions.

At WineQ, we calculate our ratings using what's called a True Bayesian Estimate.

This is what the Internet Movie Database -- and any other reputable large-volume rating site -- uses to calculate their average scores.

In essence, the way a True Bayesian Estimate works is that it compensates for the fact that a wine with two ratings that average 4.5 probably isn't as good as a wine with ten ratings that average 4.5.

To put it differently: It's a lot harder to get ten people to rate you a 4.5, than to get two people to rate you a 4.5 -- and when you calculate your averages, you should compensate for this fact.

As a nice by-product, this makes it harder for you and your wine-wench-of-the-night to plug a couple 5-star ratings into a new wine on our web site and -- presto! -- have a perfect 5-star rating on our site.

Instead you need something like ten wenches. But by then we'll have caught onto your sorry ass and put you in your place.

If that seems like it just might be confusing, you can confirm that it is indeed confusing by Googling "True Bayesian Estimate" to find out how the math works.

Hope that helps.

Now back to my Tempranillo....

Joshua

[ryan]

I use the 5star/10 with .5's, and actually it's more of a 9star (or grape in my case) since I've never thought to rate something a 1/2grape

Maybe thumbs up and down is a better idea, now that I think of it.
Down - Don't bother
Up - you might like it
But then we'd end up with a "neutral thumb, half up rating" - Maybe "I could care less"?

Ratings make me laugh in the end. I understand the need, but they really do so little for my appreciation of a wine, that personally I could care less.

Tell me a story, now that I care about!

[Philip James]

Josh

Good post, i was going to say we do the same at Snooth, but i couldn't remember the proper name for how it worked.

Anyway, yeah, the point was - you need a consensus amongst several people to generate a high (or low - it works both ways) rating.

Philip

[Tim]

Jeff,

Thanks for continuing and expanding the conversation over here.

Your point at the end is very well taken about the prose of the review being far more important than the score itself; my bad for not making the point in my post.

But I do think there is value in a community rating even with strong differences of opinion. As Joshua writes above, these sites are using some pretty fancy math to make sure the composites are a valid representation of all the reviews. I also think that if the majority of wine bloggers used one rating system, all these community sites would deliver be better results (assuming everyone used/fed these sites).

Now I've got to see if WineQ publishes an RSS feed of my reviews to put into my aggregator...

[Dr. Debs]

OK, much to say. First, I may be convinced by the star rating system, but not if you can give it half stars. I mean, the whole point is to keep it simple and straightforward. Now, once they're averaged, like at WineQ, I don't mind 1/2s, 1/3s, etc because I figure it is the only way to capture the difference between someone who gives a wine a 1 and someone who gives a wine a 5.

But I agree totally that a star rating, by itself, doesn't adequately do the job simply because the standards for assigning that star can vary hugely. My standards, not terribly surprising, are varietal characteristic and price. I can have a cab I adore, but if it tastes like a Syrah and is $$, then it isn't getting 5 stars. I would probably give it 3. But this may not be how others use the system, which makes it of less use to me if there are no explanatory notes.

[Michelle]

Well, as I've said on two other comments elsewhere, I'm sticking with the happy faces although I'm all for the visuals, but more for expressive words.
I'm really posting to tell you that your technorati tags made me laugh, a lot, and very loudly. I now feel like eating half a bowl of Lucky Charms.
Cheers!

[winehiker]

I prefer the Goldilocks approach:

100 points? Too complex to be a valuable learning tool; unwieldy for all but the geekiest of bloggers.

Five stars? Too simple; open to many interpretations, as noted above; inadequate as a learning tool.

20 points? Tried and true educational tool, simple to understand; along with supporting tasting notes, it's a happy medium, just right!

[el jefe]

Actually (if anyone is still reading here!) the scoring sheet over at winehiker is pretty comprehensive and easy to understand. I encourage everyone to check it out!

Michelle - I probably severely abuse my technorati tags but sometimes it is just too tempting...