Protocol Buffers

Explains how Protocol Buffers encodes data to files or to the wire. This document describes the protocol buffer wire format , which defines the details of how your message is sent on the wire and how much space it consumes on disk. You probably don’t need to understand this to use protocol buffers in your application, but it’s useful information for doing optimizations. If you already know the concepts but want a reference, skip to the Condensed reference card section. Protoscope is a very simple language for describing snippets of the low-level wire format, which we’ll use to provide a visual reference for the encoding of various messages. Protoscope’s syntax consists of a sequence of tokens that each encode down to a specific byte sequence. For example, backticks denote a raw hex literal, like 70726f746f6275660a . This encodes into the exact bytes denoted as hex in the literal. Quotes denote UTF-8 strings, like "Hello, Protobuf!" . This literal is synonymous with 48656c6c6f2c2050726f746f62756621 (which, if you observe closely, is composed of ASCII bytes). We’ll introduce more of the Protoscope language as we discuss aspects of the wire format. The Protoscope tool can also dump encoded protocol buffers as text. See for examples. All examples in this topic assume that you are using Edition 2023 or later. Variable-width integers, or varints , are at the core of the wire format. They allow encoding unsigned 64-bit integers using anywhere between one and ten bytes, with small values using fewer bytes. Each byte in the varint has a continuation bit that indicates if the byte that follows it is part of the varint. This is the most significant bit (MSB) of the byte (sometimes also called the sign bit ). The lower 7 bits are a payload; the resulting integer is built by appending together the 7-bit payloads of its constituent bytes. How do you figure out that this is 150? First you drop the MSB from each byte, as this is just there to tell us whether we’ve reached the end of the number (as you can see, it’s set in the first byte as there is more than one byte in the varint). These 7-bit payloads are in little-endian order. Convert to big-endian order, concatenate, and interpret as an unsigned 64-bit integer: A protocol buffer message is a series of key-value pairs. The binary version of a message just uses the field’s number as the key – the name and declared type for each field can only be determined on the decoding end by referencing the message type’s definition (i.e. the .proto file). Protoscope does not have access to this information, so it can only provide the field numbers. When a message is encoded, each key-value pair is turned into a record consisting of the field number, a wire type and a payload. The wire type tells the parser how big the payload after it is. This allows old parsers to skip over new fields they don’t understand. This type of scheme is sometimes called Tag-Length-Value , or TLV. When the sint32 or sint64 is parsed, its value is decoded back to the original, signed version. Similarly float and fixed32 have wire type I32 , which tells it to expect four bytes instead. float values are encoded in IEEE 754 single-precision format. The syntax for these consists of adding an i32 suffix. 25.4i32 will emit four bytes, as will 200i32 . Tag types are inferred as I32 . Length prefixes are another major concept in the wire format. The LEN wire type has a dynamic length, specified by a varint immediately after the tag, which is followed by the payload as usual. Submessage fields also use the LEN wire type. Here’s a message definition with an embedded message of our original example message, Test1 : The last three bytes (in [] ) are exactly the same ones from our very first example . These bytes are preceded by a LEN -typed tag, and a length of 3, exactly the same way as strings are encoded. Missing fields are easy to encode: we just leave out the record if it’s not present. This means that “huge” protos