U.S. tech giant Google LLC faces controversy surrounding the performance of its large language model (LLM) artificial intelligence (AI) Gemini launched last week.

According to Bloomberg and TechCrunch on Sunday, the demo of Gemini was created not from real-time usage but through still images and text.

The demo depicted human interaction with AI in a seemingly uninterrupted manner, transitioning seamlessly to different tasks.

However, this was a result not of real-time footage but carefully inputting text based on recognized still images. The responsiveness of Gemini also varies significantly from reality.

Parmy Olson, a Bloomberg tech columnist, pointed out that there is a substantial difference between Google’s explanation of Gemini observing and reacting to the surrounding world in real-time and its demo video, suggesting an exaggeration of performance for marketing purposes.

Criticism has emerged regarding the evaluation of Gemini’s abilities in one of the tests assessing massive multitask language understanding (MMLU), where it scored 90 percent, surpassing OpenAI’s GPT-4’s 86.4 percent.

However, it is noted that the criteria for evaluation differed.

While GPT-4’s results were obtained through five attempts, Gemini used a method called CoT@32.

CoT@32 stands for Chain of Thought with 32 Examples, showcasing superior reasoning abilities compared to conventional repetitive attempts.

When the actual score of Gemini, after five attempts like GPT-4, was compared, it yielded a lower 83.7 percent, trailing behind GPT-4.

The fact that Google’s highest version Gemini Ultra has not been unveiled serves as evidence of its impatience.

Unlike Gemini applied directly to Bard, Ultra is set to be unveiled early next year. Google cited ongoing safety and ethical evaluations of Gemini as reasons for the delay.

Meanwhile, the GPT-4 Turbo is already available to all paid users.

By Lee Duk-joo and Minu Kim

[ⓒ Pulse by Maeil Business News Korea & mk.co.kr, All rights reserved]