Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.
via Wikipedia infobox
Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.
Like its predecessor, GPT-2, it is a decoder-only transformer model of deep neural network, which supersedes recurrence and convolution-based architectures with a technique known as "attention". This attention mechanism allows the model to focus selectively on segments of input text it predicts to be most relevant. GPT-3 has 175 billion parameters, each with 16-bit precision, requiring 350GB of storage since each parameter occupies 2 bytes. It has a context window size of 2,048 tokens, and has demonstrated strong "zero-shot" and "few-shot" learning abilities on many tasks.
Discovered by embedding cosine similarity (sentence-transformers MiniLM, 384-dim).