Yandex GPT

 

  Hello! In today's article I'd like to talk about some research done by experts on whether YandexGPT understands culture-specific phenomena: references to movies and songs, quotes, traditions, jokes, memes, and more. This is a very important discussion, as YandexGPT is used in major products like Search and Alice, which millions of people interact with every day - the neural network must be able to understand cultural references of all kinds.

 

How it all started

Cultural code is a system of signs, traditions, norms, and concepts that distinguish one group of people from another. The cultural code can describe anything: generations, hobbies, professions, religion — any groups of people united by a significant common context. However, it is most often recalled when talking about cultural differences between nationalities and countries.

In December 2023, the team of experts took on a big task — to digitize the understanding of the modern Russian cultural code. Together with a team of AI trainers, they conducted thorough research and decided to break down this task into the following high-level categories, which, in turn, consist of smaller ones.

 

How to measure cultural understanding

They can be measured using the typical approach of academic benchmarks for factual knowledge.

Formulating tasks on the knowledge of quotes, idioms, and colloquial expressions is not difficult: we show the model an incomplete quote with a blank, which we ask it to fill in. However, these tasks usually do not pose a challenge for the model either.

 

- Fill in the missing word in the quote from The Caucasian Captive: "Cursed be the day when I sat behind the wheel of this...!"

- Finish the phrase: "I am hard to find, easy to lose, and impossible to..."

 

These tasks are more of a test of the model's ability to memorize facts. They do not test its understanding or ability to interpret. Therefore, experts introduced another type of question — open-ended ones. In such questions, they do not provide a specific quote, but rather describe it indirectly.

And finally, one more complication: formulated open-ended questions in a more intricate way. A person needs to think a bit longer to answer such a question, but it remains manageable for them. However, it’s much harder for the model.

 

- What does the truth do to our eyes when we find it unpleasant to hear?       or

- According to a saying, which animal demonstrates the worst handwriting (at least with its paw)?

 

Comparing a human and a neural network

In the end, they created a benchmark with 2000 diverse questions — it was time for the final check! Experts asked a larger number of AI trainers to answer the questions. 211 people participated in the markup, and each question was reviewed by five different trainers.

By the way, this check helped them find and fix the last typos and inaccuracies in the questions and answers of their benchmark. For example, if none of the five people answered a particular question, it was likely that something was wrong with the question itself. An important point: when answering the questions, no one was allowed to search for information on the internet — to keep it fair.

 

In the end, the average score of the AI trainers was 78% correct answers. This is the number they should use as a benchmark for future measurements. Ideally, of course, the model should have a result higher than that of humans, since it has seen the entire internet many times over and could certainly recognize and understand our cultural code.

 

Human based measures	YandexGPT Pro 4
78%	63%