A visualisation of major large-language models (LLMs), ranked by performance, using MMLU (Massive Multitasks Language Understanding) a benchmark for evaluating the capabilities of large language models.

» main source: Life Architect data
» see our data
» chart rendered with VizSweet

notes

The MMLU rating consists of 16,000 multiple-choice questions across 57 academic subject (link)

MMLU has some critiques, mainly that LLM creators may be wise to the metric and be pre-training their models to answer MMLU questions. (New York Times article). And here’s a general cautionary article.

If you’re interested in ongoing metrics, LLMArena has a leaderboard ranked subjectively by tens of thousands of users. Note: it privileges English language LLMs (there are no Chinese models on there for example). Intel also run a leaderboard with many measurements.

Note: we excluded specialised maths and coding LLMs like DeepSeek Coder, Mathstral, Granite Code etc. Let us know if we’ve missed any LLMs.

 
 
An earlier graphic of ours

The rise and rise of AI-based Large Language Models (LLMs) like GPT4, LaMDA, LLaMa, PaLM and Jurassic-2.

Click the company names to filter the data.

» See the data

We’ll keep this graphic updated as new models emerge.

» Made with our visualisation tool VizSweet

Learn to Create Impactful Infographics

further notes & essential reading

» Why LLMs Can’t Play Wordle and Why That Means They Won’t Lead to AGI
» How ChatGPT and other LLMs Work – And Where They Could Go Next (good Wired article)

» More detailed (understandable) explanation of how LLMs work

» While we’ve plotted these LLMs by the size of each model in billion parameters, there is a growing sense of diminishing returns for simply increasing the model size (Wired article)

» Will A.I. become the new McKinsey? Author Ted Chiang argues that AI is likely to function like larger corporate consulting firms, acting as a “willing executioner”, accelerating job loss (New Yorker)

» The Mounting Environmental & Human Costs of Generative AI (Arstechnica article) &TLDR: larger models = more consumption of planetary resources (minerals, energy, water for cooling) + AI training needs large-scale human supervision so very real possibility of ‘AI sweatshops‘ + the serious issues of copyright infringement for artists and creators.

summarised here:

A Quick Data Story

Some simple filtering reveals an interesting story in our early LLM data.

Google drove a burst of innovation in the LLM space

Sharing their knowledge and research with the AI world. (Transformers, for example, the ‘T’ in GPT, originated from research at Google). But then the broader company was pipped to the practical-application post by OpenAI. Will their latest release PaLM2 overtake ChatGPT?

OpenAI, creators of ChatGPT, stole the LLM show

They made steady, solid progress over the last three years, driving the curve. Slow and steady won the race?

And here’s why Microsoft invested in OpenAI

You can see they weren’t directly active in the space with their own research. Instead they invested early and hard in OpenAI ($1bn in both 2019 and 2021) and that paid serious dividends.

Meta / Facebook also drove significant early innovation in the field…

What muted their breakthroughs? Maybe the models weren’t large enough (see how many are below the ‘magic’ 175 billion parameter line). Maybe, like Google, there’s was too much emphasis on internal applications & processes versus public tools? Maybe, also, their research was chastened by the poor reception of its science-specialised LLM Galactica.

Meanwhile, in the background, China is also making steady progress

In the advent of ChatGPT, release of Chinese-language LLM’s and chatbots have significantly accelerated.

What about Amazon?

Well, they have steamed in at the end – too late to the party? Time will tell… Though they have recently invested heavily in Anthropic, creators of impressive LLM Claude