Tasting and Evaluating Wine – You got to be kidding me!

It’s been a while since I added a blog to this site. I feel compelled to do so because our last gathering was, by my way of thinking, was not only very informative, but quite surprising. Hmmm, okay, that’s a big lie. It was not at all surprising to me. But it absolutely was to most participants.

On October 6th our Chapter of the NM Vine and Wine Society held a wine tasting and “judging” event. For those of you that were unable to attend I will provide a semi comprehensive accounting of the day’s events. 

It was at least two years ago that Mike and I discussed doing this convoluted tasing and judging event. Frankly, I was quite apprehensive when the time came to do it. I felt for sure that I would irritate at least some of the folks that participated in the tasting. Wendy had warned me to rein in my normally contrary and surly approach to making a point. I knew that was going to be hard. Nonetheless, I believed that we would all benefit, to one extent or another, from participating in this wine judging event. With considerable effort, and lots of upfront thought and research, I was able to not make a mule of myself.

We tasted three wines. Of the approximately 45 folks that attended the event all, but a couple participated in the judging. I started the tasting by stating that we would be tasting three wines. All would be New Mexico red wines; the same type and vintage and that we had three bottles of each. During the first round several participants speculated on the type of red wine. The guesses ranged from Cabernet Sauvignon, Syrah, Merlot and Pitit Verdot. One person guessed the right type of wine. Between each wine I asked everyone to cleanse their pallet. The first wine received an informal “average” score of 7.5 out of 10. Based on prior tasting and judging within our group, that was a very high score. The second wine received an average score of 4 out of 10. I was more than surprised, and I made a big deal over the fact that it absolutely could not be so low. The third tasting received an informal average score of 6.5. Based on prior experience with this group I’ve learned that if you add two points to the points given and multiply the resulting score by 10, the scores will be close to how the wine is “professionally” rated. Not that that means much.

Mike revealed each wine. They were all Vivac 2022 Petit Verdot. The wine sold at Vivac’s tasting room for $28 per bottle. All nine bottles were randomly numbered.

As this gathering was intended to have an “educational” focus, I spent considerable effort researching how well educated and trained wine judges very often lack consistency in there judging scores. I believe that it is important to point out that swallowing the wine allows it to coat the back of your palate and release more aromas when you exhale. It can also give you a sense of what drinking a full glass of the wine would taste like. However, swallowing can make it harder to taste new wines. During wine judging competition, wine judges do not drink the wine. Nonetheless, their palette is influenced by previous wines tasted.

I found several articles online regarding the lack of consistency when it came to wine judging. I focused on judging at the international competition level. I found a couple of articles that covered this topic. I was particularly impressed by two articles based on a study conducted by wine judge researcher Robert Hodgson. The following is a very brief synopsis of what he found and is mostly a cut and paste.

In 2000 California winemaker and retired statistics professor Robert Hodgson became curious when some wines that he entered in multiple competitions earned very different evaluations. For example, his 1993 Zinfandel won the San Francisco International Wine Competition Best of Show yet earned no award in a subsequent event. Being curious, and I assume, a little bewildered, this experience prompted Hodgson to set out on a comprehensive and systematic study to determine whether the inconsistencies in wine judging that he encountered were infrequent or commonplace. As a retired professor of statistics, he used his background to develope an approach to prove or disprove his theory that there was no perfect approach to consistently evaluate the quality of a given wine. In 2005, as a member of the advisory board for the California State Fair Commercial Wine Competition, Hodgson asked and received permission to conduct tests to check the reliability of the judges. It was not his intent to discredit the judges or the judging process. In these tests, four-judge panels were presented with their usual flights of wine samples to sniff, taste, and examine. However, to measure the judges’ consistency, some of the wines were presented to the panel three times and poured from the same bottle each time. Hodgson conducted these tests at the state fair each year over an eight-year period. Several different wines were evaluated by multiple judge panels. I should point out that I have no idea what variety of wines were tasted and evaluated. I only point this out because I do not know if scores given were more consistent for one type of wine rather than another. 

The wine evaluation scale used by the judges ranged from a low score of 50 to a high score of 100; most of the rated wines were given scores in the 70s, 80s, and low 90s. By the way, in nearly all international wine competition almost all wines evaluated fall within a narrow band between 85 and 100 points. The scale doesn’t start at zero, it starts at 50. It’s important to realize that no wine maker is going to enter a competition unless he or she believes that their wine will be judged in the 85 or better range.

Who were these judges? Were they equivalent in competency to the judges that evaluate the New Mexico State Fair wine competition? No. They were recognized experts in the American wine industry, including professional winemakers, certified sommeliers, well-known wine critics and wine consultants, and even university professors who taught classes in winemaking and conducted research in winemaking. 

The approach they used was common in wine judging. In this study, the judges would first indicate the wine’s score independently, which was then recorded. Afterwards, the judges discussed the wine, and based on the discussion, some judges modified their initial score. A key point is only the first independent score was used by Hodgson to analyze an individual judge’s consistency in scoring wines. As it turns out, it is not unusual for judges to be influenced by other judges especially judges that have great influence.

Results from the first four years of Hodgson’s study were published in 2008 in the Journal of Wine Economics, and they indicated that, overall, these wine experts were not very consistent judges. Over the three blind tastings, a typical judge’s scores for the “planted” wine varied by about plus or minus four points. For example, a wine rated as a good 90 would be rated a few minutes later by the same judge as an acceptable 86 and then a bit later as an excellent 94. An eight-point spread is considerable when it comes to awarding medals. Only about 10% of the judges were consistent in rating the identical wines, meaning that they gave the same wine presented to them three different times scores that varied by just plus or minus two points. In other words, these were the only judges whose ratings of the same wine typically stayed within the range of one medal. Yet even those judges who were consistent in their evaluations at Year 1 often were not very consistent at Year 2, so inconsistency was a more accurate descriptor of the vast majority of the judges’ performances. At the opposite end, another 10% of the judges gave the same wine far different ratings, ranging from a wine receiving a gold medal to the same wine receiving no medal at all. Consider our own judging of the Pitit Verdot, our range was 4 to 7.5. Assuming that you are the average wine consumer and do not have considerable training in evaluating wine, what does this mean to you? Don’t pay a whole lot of attention to how wine is scored by the “professionals”.  Heck, based on our own tasting exercise, we can’t even trust our own ability to judge a wine.

In additional studies beyond California’s borders, Hodgson tracked wine that had been entered in at least two competitions in the United States and found that about 99% of the wines that earned gold medals in one event received no award in a different event. Several gold medal-winning wines were entered in five competitions. None of them received five gold medals, or even four gold medals. In commenting on his research findings as a whole and their implications for wine judging events generally, Hodgson stated, “Chance appears to have a great deal to do with the awards that wines achieve or miss out on.” He readily admitted that there are many individuals who are expert wine tasters with exceptional abilities to critically judge wines when a few samples are placed before them. Yet this is not what generally takes place at many wine-judging events. Hodgson explained, “When you sit 100 wines in front of a judge, the task is beyond anyone’s ability.”

In a different study I found that $100 bottle of wine is no more likely to get a medal than a $20 bottle of wine. Oh, one more thing, most people don’t take the time to taste their wine. They just drink it.

Raise your hand if you made it all the way through this blog.