transformer
Sign in to saveAlso known as transformer model, transformer architecture, transformers
machine-learning model architecture first developed by Google Brain
~40 min read
Encyclopedic overview
A standard transformer architecture. Many modern diagrams show the pre-layer normalization (pre-LN) convention, while the original 2017 paper used post-layer normalization (post-LN).
In deep learning, the transformer is a family of artificial neural network architectures built around the attention mechanism. Transformers were introduced to model sequential data without recurrence and without convolutions, allowing much more parallel computation during training. They are now a dominant architecture for natural language processing, computer vision, speech processing, multimodal learning, robotics, and many other sequence-modelling tasks.
Excerpted from Wikipedia’s “transformer” article, available under the CC BY-SA 4.0 licence.