Arena

FastChat is an open platform for training, serving, and evaluating large language model based chatbots. FastChat powers Chatbot Arena (lmarena.ai), serving over 10 million chat requests for 70+ LLMs. Chatbot Arena has collected over 1.5M human votes from side-by-side LLM battles to compile an online LLM Elo leaderboard. FastChat's core features include: The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench). A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs. News [2024/03] 🔥 We released Chatbot Arena technical report. [2023/09] We released LMSYS-Chat-1M , a large-scale real-world LLM conversation dataset. Read the report. [2023/08] We released Vicuna v1.5 based on Llama 2 with 4K and 16K context lengths. Download weights. [2023/07] We released Chatbot Arena Conversations , a dataset containing 33k conversations with human preferences. Download it here. More [2023/08] We released LongChat v1.5 based on Llama 2 with 32K context lengths. Download weights. [2023/06] We introduced MT-bench , a challenging multi-turn question set for evaluating chatbots. Check out the blog post. [2023/06] We introduced LongChat , our long-context chatbots and evaluation tools. Check out the blog post. [2023/05] We introduced Chatbot Arena for battles among LLMs. Check out the blog post. [2023/03] We released Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality . Check out the blog post. Model Weights Vicuna Weights Vicuna is based on Llama 2 and should be used under Llama's model license. You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos. Downloaded weights are stored in a .cache folder in the user's home folder (e.g., ~/.cache/huggingface/hub/ ). See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below. Old weights : see docs/vicuna weights version.md for all versions of weights and their differences. Other Models Besides Vicuna, we also released two additional models: LongChat and FastChat-T5. You can use the commands below to chat with them. They will automatically download the weights from Hugging Face repos. (Experimental Feature: You can specify --style rich to enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.) Supported Models FastChat supports a wide range of models, including LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, xDAN-AI and more. Single GPU The command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B. See the "Not Enough Memory" section below if you do not have enough memory. --model-path can be a local folder or a Hugging Face repo name. Multiple GPUs You can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine. Tips: Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs. You can use --max-gpu-memory to specify the maximum memory per GPU for storing model weights. This allows it to allocate more memory for activations, so you can use longer context lengths or larger batch sizes. For example, CPU Only This runs on the CPU only and does not require GPU. It requires around 30GB of CPU memory for Vicuna-7B and around 60GB of CPU memory for Vicuna-13B. Metal Backend (Mac Computers with Apple Silicon or AMD GPUs) Use --device mps to enable GPU acceleration on Mac computers (requires torch = 2.0). Use --load-8bit to turn on 8-bit compression. Intel XPU (Intel Data Center and Arc A-Series GPUs) Install the Intel Extension for PyTorch. Set the OneAPI environment variables: Ascend NPU Install the Ascend PyTorch Adapter. Set the CANN environment variables: Not Enou