The Transformer and large language models
Ashish VaswaniNoam ShazeerNiki ParmarJakob UszkoreitLlion JonesAidan GomezŁukasz KaiserIllia Polosukhin
This page delivers no verdict; it presents claims and evidence with their sources. No theory in science is absolute. Method
For everyone
The Transformer architecture, proposed in 2017, computes how the words in a text relate to one another with an "attention" mechanism. Large language models such as GPT, Claude and Gemini are built on it. They have shown great progress in writing, translation, programming and reasoning tasks. Whether they really "understand" is debated.
For specialists
Self-attention is computed as softmax(QKᵀ/√d)V with query (Q), key (K) and value (V) matrices, which permits parallel training compared with recurrent networks. Scaling laws show loss falling as a power law with parameters, data and compute (Kaplan et al. 2020). Critics stress that these models rely on statistical pattern matching, lack genuine understanding and can fabricate false information ("hallucination").
Claims
- An architecture based solely on attention outperforms recurrent and convolutional networks on language tasks.[1]
Evidence
Supporting
The Transformer achieved the then-best result on the WMT 2014 English–German translation task at lower training cost.[1]
Model loss was measured to fall according to a predictable power law as model size, data and compute increase.[2]
Challenging
It was argued that large language models are "stochastic parrots" imitating linguistic form without grasping meaning, and that this carries risks.[3]
Connections
Builds on
Related
This topic is a stop on these journeys
Sources
- Preprint (2017). Attention is all you needarXiv:1706.03762archive
- Preprint (2020). Scaling laws for neural language modelsarXiv:2001.08361archive
- Article (2021). On the dangers of stochastic parrots: can language models be too big?. Proceedings of FAccT 2021, 610–623DOI 10.1145/3442188.3445922archive
Search the literature
Look for scholarly work on this topic in other indexes:
Cite this page
Athena contributors. (2026). The Transformer and large language models. Athena: The tree of human knowledge. Retrieved November 1, 2026, from https://latifkedi.github.io/athena/en/d/transformer/
@misc{athena_transformer,
title = {The Transformer and large language models},
author = {{Athena contributors}},
year = {2026},
howpublished = {\url{https://latifkedi.github.io/athena/en/d/transformer/}},
note = {Athena: The tree of human knowledge. Accessed: 2026-11-01}
}Added: September 27, 2026 · Last changed: September 28, 2026 · Page history