Scientific Wine Test, Pt. 2
If you saw the length of last week’s post, you may have surmised that the fun I had at my birthday party exceeded my energy available for blogging. But have no fear; I bring to you now, today, the results of our Scientific Wine Test.
We actually did make a fair attempt at making this test as blind and controlled as possible. There are a lot of practical problems doing this in the context of a birthday party; some visible examples were everyone asking loudly if they were all drinking the same wine or different wine, speculating on who had what, and tasting each other’s wine.
My chief scientist was Gus Dunn, a molecular biologist at UC Irvine who is but a skip and a jump away from his Ph.D. Gus, labcoat and all, designed and administered the testing, and analyzed the results. This birthday-necessity limited us to single blinding. Another exigency was that Gus and I met only an hour before the party to begin exploring questions like “What the hell should we test for and how should we do it.” It is, of course, basic to any experiment to have a question you want to answer. We decided on testing whether people who consider themselves better wine experts can actually tell the difference between expensive wine and cheap wine, better than those who rate themselves ignorant of all things wine.
We had a small sample size, and the environment was fairly poorly controlled, but we did our best to assure that nobody knew what they were drinking and tasters were identified by arbitrary number only. There were 17 tasters (I was #6, if anyone cares). Each ranked himself on a scale of 1 to 5 as to his wine expertise. Nine different bottles of wine were used in the test, two reds and a white from each of three price categories (cheap, average, and expensive). Each taster was given four randomly selected glasses and asked to rank how well they liked it (1 to 5) and whether they thought it was cheap, average, or expensive.
(It should be stated, at this point, that I am making no effort to write this up in a scientific kind of way. It should also be stated that as I type this, I am finishing off the worst of the bottles from the test.)
Time constraints entered the equation about halfway through the evening, and it was decided to terminate the average price category, nullify any average guesses, and proceed with only six bottles. Gus did an admirable job of maintaining the standards. He had his own little room where he did all the pouring, random assignments, and tabulation. Every little while he’d come out with a tray of glasses, numbered for each subject. Considering the short time he had to prepare, his attention to professionalism was impressive throughout the evening, until people started leaving and he was able to ditch the labcoat and sit down with his own glass at long last. Kudos to Gus.
A preliminary review showed that our results were all over the map. There was no preference for cheap wine or expensive wine. Our tastes were all over the map, and to prove, Gus gave us an early p-value of .46. This means the results were completely useless. Random chance alone was 46% as likely to give us these same results. In scientific terms, this means we learned dick.
Click to see full size
Here is an example of one of the box plots from the test. This first one was to find out if experienced tasters (as self reported) preferred red or white. They did not. Although the boxes make it look like there was a preference for red, the p-value of .6 makes this finding completely invalid. We can proudly and safely state that, based on our test, we have no idea whether experienced wine tasters prefer red or white wine.
Anecdotally, almost all of us there preferred red wine. This was determined less scientifically, by examining who signed up to bring what kind of wine first on the Evite. Somebody quickly skewed these results by posting that red wine is better on the Evite comments, so this finding is suspect even beyond its basic nature as being suspect.
Click to see full size
However, we did end up with one finding that was solid. According to this second box plot, experienced tasters preferred more expensive wine to cheap wine. This ignores their assessment of whether they thought it was expensive or not, and simply goes by their preference of whether they liked it. And with an impressive p-value of .04, expensive wines were preferred to cheap wines by tasters who ranked themselves as more experienced. Take this to the bank, and feel free to cite it in your thesis papers.
There was another test we planned, but didn’t have time for, and that’s to see if people could tell red wine from white when it’s not visible. A bit of research told me that pinot noir and pinot grigio are the two most likely to be confused, when served at the same temperature. I still have the two bottles (not the most expensive variety), and the opaque styrofoam cups with lids and straws. We’ll do it another time.
Ah, one way to interpret the results is that with practice you learn how the more expensive wines do taste and as they are more expensive, you very easily convince yourself that just this type of taste is better.
Did the experienced tasters rate the average wines between the cheap and the expensive wines?
Like I said, the average wines were removed from the test for time constraints.
fun little study! not sure what stats you did on this, but you may be able to get some more traction when using something called “latent class analysis”. you may know about it already, but it’s good because it would allow you to compare people’s ratings to everybody else. Once you’ve corrected for that you may get better agreements.
the following site has been useful to me before:
http://www.john-uebersax.com/stat/agree.htm
Were the reds and whites matched by grape variety in each price category?
No, they were all totally random.
Yeah, that’s an important consideration, since some grape varieties are more expensive and others are more common as cheap kinds (e.g., lots of $2-3 merlots but few zinfandels or sangiovese).
When I go to a winery & taste a few wines I usually buy a dozen bottles of bad tasting wine due to the co-efficient of intoxication. Was this factored into your calculations?
All local wines? I hope you saved a bottle of tasty Coonawarra Shiraz!
Yeah, taste is subjective. I love wine, but that tends to be all wine. I’ve even been known to drink Boone’s Strawberry… I know. I’m not proud.
My wife and I recently got a fancy aerator. You pore the wine through it and it’s supposed to make it immediately taste like it tastes after it’s been decanted for a long time. It does make a loud slurping noise and seems like it’s mixing air with the wine, so I guess it does something. So far everyone who tries it swears that it instantly turns ordinary wine into ambrosia. I’m not sure I can tell the difference though. I thought it would be interesting to do a blind test with the aerated wine and wine just poured from the bottle.
Incidentally, we also got a device that’s used to seal the wine bottle. It sucks all the air out and maintains a vacuum. It’s supposed to keep the wine good indefinitely after it’s been opened.
The vacuum sealers are a great investment. They will keep your wine good for longer. Let me know about the airator, I’ve been curious about those for sometime now.
Most wines that have preservatives (sulfites) in them will keep fairly well on the kitchen counter after being opened if stoppered, for even a couple of weeks or more contrary to most lore.I have done this for years with no problems.If you refrigerate them they will last quite a bit longer.Even freezing wine doesn’t seem to hurt it much. I have amazed friends by taking a red wine out of the fridge, and zapping it in the microwave 10 seconds or so,and the wine is none the worse (just don’t get it too warm).
I suspect that the aerator thing is probably useless,but I would also like to hear the results from a blind taste test on it.
I am a believer in the aerator.
though it probably doesn’t make a difference in regards to all wines, I have a favorite zinfandel that tastes like dirt if you drink it right away. It really improves if left out to breath for 10 minutes (or run through the fancy aerator).
and @tmac-you are lucky then! my mom gave me wine that had only been open about a week and it tasted like sour grape juice, it had gone bad. I told her to invest in the nicer boxed wines if they were going to just drink one glass at a time (because despite people’s fears there are some fairly decent boxed wines out there)
Karen,no luck to it.I have done this with well over 200 bottles of wine,and never had a problem.The wine your mom gave you may have been contaminated,or possibly an organic wine with no added sulfites.Another possibility is that it was too warm where it was stored,but I leave opened bottles on my counter for a week or two in the summer when my kitchen gets up to 82F. I do always use a stopper though.They will keep for long periods of time in the refrigerator stoppered as well.
If you are game,please try the blinded taste test using the aerator/no aerator. I would love to hear the results,as I am still a bit skeptical about it.
could be had a foul bottle of wine to begin with and didn’t realize (my mom is not really a big wine drinker). It definitely wasn’t organic and I don’t know that it was in a warm spot necessarily.
I’ll try the aerator with my favorite zinfandel and see (we actually have two aerators and a couple decanters so maybe we’ll try it all that stuff and see what we get for taste. It would be a fun experiment! most of the time i drink wine more or less after we’ve opened the bottle and i can’t tell a difference in aeration or having it sit over a few hours. This bottle of zin is the only time i’ve noticed a marked difference (I think, we’ll see!) in taste when its had some air.
It is fun to test stuff like that.I have done blind beer tastings before,and the results are quite interesting and fun.The thing that strikes me the most,is how often people are just dead wrong about what they think is their favorite beer vs what they end up liking while blinded.I would like to try the red vs white wine switcheroo sometime with some of my friends who “only” like red wine.I have been trying for years to get people to open up to the merits of white wines,but once someone declares I only like (whatever) it is really hard to get them to expand their palates.Let us know how the experiment goes.Cheers!
Oh,by the way,if you do try leaving the wine out for a week or two after opening it,I would leave it in the bottle(stoppered),and not decant it. Exposure to the extra air in the decanter,along with the possibility of picking up some bacteria (which causes wine to turn vinegary) might hasten it’s demise.
I think your science is more or less correct Brian, my husband and I (who happen to be big wine-O’s stumbled across this article not too long ago explaining why wine ratings are basically bullshit…
http://online.wsj.com/article/SB10001424052748703683804574533840282653628.html
Hi Brian,
Great idea for a Birthday Party activity!
I love the WSJ article posted by Karen.
There is a series of books called “The Wine Trials” By Robin Goldstein and Alexis Herschkowitsch that might be intersting for you.
I guess very few scientific trials have been done on wine tasting, although this was an admirable attempt. I’ve always thought the industry is ripe for skewering by talented sceptics, and I highly suspect the wine people buy (and how much they pay for it) is based on peer pressure, the media etc rather than any real taste preference. Definitely a case of the Emperor’s New Clothes.
I am curious about the statistics you used to analyze the data. For example, the box plots show the ratings of experienced tasters. Did you just select people with ratings of 4 or 5? What kind of procedure did you use? Depending on how you did this, there may be a problem with dependence in the data. For example, if one person dislikes wine more generally and rates all of the wines they taste a 1 or 2, that might skew the results. There are statistical procedures to control for this type of person-specific variance that might increase some of your p-values.
(from somebody who likes wine, psychology, and statistics, so you’ve hit three of my favourite topics in one post!)
You forget to mention what a hot guy Gus is, and how chicks totally dig dudes in labcoats.