internal Package

api_implementation Module

This module is the central entity that determines which implementation of the API is used.


containers Module

Contains container classes to represent different protocol buffer types.

This file defines container classes which represent categories of protocol buffer field types which need extra maintenance. Currently these categories are:

  • Repeated scalar fields - These are all repeated fields which aren’t composite (e.g. they are of simple types like int32, string, etc).
  • Repeated composite fields - Repeated fields which are composite. This includes groups and nested messages.
class google.protobuf.internal.containers.BaseContainer(message_listener)[source]

Bases: object

Base container class.

sort(sort_function=<built-in function cmp>)[source]
class google.protobuf.internal.containers.RepeatedCompositeFieldContainer(message_listener, message_descriptor)[source]

Bases: google.protobuf.internal.containers.BaseContainer

Simple, list-like container for holding repeated composite fields.


Appends the contents of another repeated field of the same type to this one, copying each individual message.


Adds a new element at the end of the list and returns it. Keyword arguments may be used to initialize the element.


Extends by appending the given sequence of elements of the same type as this one, copying each individual message.

class google.protobuf.internal.containers.RepeatedScalarFieldContainer(message_listener, type_checker)[source]

Bases: google.protobuf.internal.containers.BaseContainer

Simple, type-checked, list-like container for holding repeated scalars.


Appends the contents of another repeated field of the same type to this one. We do not check the types of the individual fields.


Appends an item to the list. Similar to list.append().


Extends by appending the given sequence. Similar to list.extend().

insert(key, value)[source]

Inserts the item at the specified position. Similar to list.insert().


Removes an item from the list. Similar to list.remove().

cpp_message Module

decoder Module

Code for decoding protocol buffer primitives.

This code is very similar to – read the docs for that module first.

A “decoder” is a function with the signature:
Decode(buffer, pos, end, message, field_dict)
The arguments are:

buffer: The string containing the encoded message. pos: The current position in the string. end: The position in the string where the current message ends. May be

less than len(buffer) if we’re reading a sub-message.

message: The message object into which we’re parsing. field_dict: message._fields (avoids a hashtable lookup).

The decoder reads the field and stores it into field_dict, returning the new buffer position. A decoder for a repeated field may proactively decode all of the elements of that field, if they appear consecutively.

Note that decoders may throw any of the following:
IndexError: Indicates a truncated message. struct.error: Unpacking of a fixed-width field failed. message.DecodeError: Other errors.

Decoders are expected to raise an exception if they are called with pos > end. This allows callers to be lax about bounds checking: it’s fineto read past “end” as long as you are sure that someone else will notice and throw an exception later on.

Something up the call stack is expected to catch IndexError and struct.error and convert them to message.DecodeError.

Decoders are constructed using decoder constructors with the signature:
MakeDecoder(field_number, is_repeated, is_packed, key, new_default)
The arguments are:

field_number: The field number of the field we want to decode. is_repeated: Is the field a repeated field? (bool) is_packed: Is the field a packed field? (bool) key: The key to use when looking up the field within field_dict.

(This is actually the FieldDescriptor but nothing in this file should depend on that.)
new_default: A function which takes a message object as a parameter and
returns a new instance of the default value for this field. (This is called for repeated fields and sub-messages, when an instance does not already exist.)

As with encoders, we define a decoder constructor for every type of field. Then, for every field of every message class we construct an actual decoder. That decoder goes into a dict indexed by tag, so when we decode a message we repeatedly read a tag, look up the corresponding decoder, and invoke it.

google.protobuf.internal.decoder.BoolDecoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.BytesDecoder(field_number, is_repeated, is_packed, key, new_default)[source]

Returns a decoder for a bytes field.

google.protobuf.internal.decoder.DoubleDecoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.EnumDecoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.Fixed32Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.Fixed64Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.FloatDecoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.GroupDecoder(field_number, is_repeated, is_packed, key, new_default)[source]

Returns a decoder for a group field.

google.protobuf.internal.decoder.Int32Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.Int64Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.MessageDecoder(field_number, is_repeated, is_packed, key, new_default)[source]

Returns a decoder for a message field.


Returns a decoder for a MessageSet item.

The parameter is the _extensions_by_number map for the message class.

The message set message looks like this:
message MessageSet {
repeated group Item = 1 {
required int32 type_id = 2; required string message = 3;



google.protobuf.internal.decoder.ReadTag(buffer, pos)[source]

Read a tag from the buffer, and return a (tag_bytes, new_pos) tuple.

We return the raw bytes of the tag rather than decoding them. The raw bytes can then be used to look up the proper decoder. This effectively allows us to trade some work that would be done in pure-python (decoding a varint) for work that is done in C (searching for a byte string in a hash table). In a low-level language it would be much cheaper to decode the varint and use that, but not in Python.

google.protobuf.internal.decoder.SFixed32Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.SFixed64Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.SInt32Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.SInt64Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.SkipField(buffer, pos, end, tag_bytes)

Skips a field with the specified tag.

|pos| should point to the byte immediately after the tag.

The new position (after the tag value), or -1 if the tag is an end-group tag (in which case the calling loop should break).
google.protobuf.internal.decoder.StringDecoder(field_number, is_repeated, is_packed, key, new_default)[source]

Returns a decoder for a string field.

google.protobuf.internal.decoder.UInt32Decoder(field_number, is_repeated, is_packed, key, new_default)
google.protobuf.internal.decoder.UInt64Decoder(field_number, is_repeated, is_packed, key, new_default)

encoder Module

Code for encoding protocol message primitives.

Contains the logic for encoding every logical protocol field type into one of the 5 physical wire types.

This code is designed to push the Python interpreter’s performance to the limits.

The basic idea is that at startup time, for every field (i.e. every FieldDescriptor) we construct two functions: a “sizer” and an “encoder”. The sizer takes a value of this field’s type and computes its byte size. The encoder takes a writer function and a value. It encodes the value into byte strings and invokes the writer function to write those strings. Typically the writer function is the write() method of a cStringIO.

We try to do as much work as possible when constructing the writer and the sizer rather than when calling them. In particular: * We copy any needed global functions to local variables, so that we do not need

to do costly global table lookups at runtime.
  • Similarly, we try to do any attribute lookups at startup time if possible.
  • Every field’s tag is encoded to bytes at startup, since it can’t change at runtime.
  • Whatever component of the field size we can compute at startup, we do.
  • We avoid sharing code if doing so would make the code slower and not sharing does not burden us too much. For example, encoders for repeated fields do not just call the encoders for singular fields in a loop because this would add an extra function call overhead for every loop iteration; instead, we manually inline the single-value encoder into the loop.
  • If a Python function lacks a return statement, Python actually generates instructions to pop the result of the last statement off the stack, push None onto the stack, and then return that. If we really don’t care what value is returned, then we can save two instructions by returning the result of the last statement. It looks funny but it helps.
  • We assume that type and bounds checking has happened at a higher level.
google.protobuf.internal.encoder.BoolEncoder(field_number, is_repeated, is_packed)[source]

Returns an encoder for a boolean field.

google.protobuf.internal.encoder.BoolSizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.BytesEncoder(field_number, is_repeated, is_packed)[source]

Returns an encoder for a bytes field.

google.protobuf.internal.encoder.BytesSizer(field_number, is_repeated, is_packed)[source]

Returns a sizer for a bytes field.

google.protobuf.internal.encoder.DoubleEncoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.DoubleSizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.EnumEncoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.EnumSizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Fixed32Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Fixed32Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Fixed64Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Fixed64Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.FloatEncoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.FloatSizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.GroupEncoder(field_number, is_repeated, is_packed)[source]

Returns an encoder for a group field.

google.protobuf.internal.encoder.GroupSizer(field_number, is_repeated, is_packed)[source]

Returns a sizer for a group field.

google.protobuf.internal.encoder.Int32Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Int32Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Int64Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.Int64Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.MessageEncoder(field_number, is_repeated, is_packed)[source]

Returns an encoder for a message field.


Encoder for extensions of MessageSet.

The message set message looks like this:
message MessageSet {
repeated group Item = 1 {
required int32 type_id = 2; required string message = 3;




Returns a sizer for extensions of MessageSet.

The message set message looks like this:
message MessageSet {
repeated group Item = 1 {
required int32 type_id = 2; required string message = 3;



google.protobuf.internal.encoder.MessageSizer(field_number, is_repeated, is_packed)[source]

Returns a sizer for a message field.

google.protobuf.internal.encoder.SFixed32Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SFixed32Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SFixed64Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SFixed64Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SInt32Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SInt32Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SInt64Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.SInt64Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.StringEncoder(field_number, is_repeated, is_packed)[source]

Returns an encoder for a string field.

google.protobuf.internal.encoder.StringSizer(field_number, is_repeated, is_packed)[source]

Returns a sizer for a string field.

google.protobuf.internal.encoder.TagBytes(field_number, wire_type)[source]

Encode the given tag and return the bytes. Only called at startup.

google.protobuf.internal.encoder.UInt32Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.UInt32Sizer(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.UInt64Encoder(field_number, is_repeated, is_packed)
google.protobuf.internal.encoder.UInt64Sizer(field_number, is_repeated, is_packed)

message_listener Module

Defines a listener interface for observing certain state transitions on Message objects.

Also defines a null implementation of this interface.

class google.protobuf.internal.message_listener.MessageListener[source]

Bases: object

Listens for modifications made to a message. Meant to be registered via Message._SetListener().

dirty: If True, then calling Modified() would be a no-op. This can be
used to avoid these calls entirely in the common case.

Called every time the message is modified in such a way that the parent message may need to be updated. This currently means either: (a) The message was modified for the first time, so the parent message

should henceforth mark the message as present.
  1. The message’s cached byte size became dirty – i.e. the message was modified for the first time after a previous call to ByteSize(). Therefore the parent should also mark its byte size as dirty.

Note that (a) implies (b), since new objects start out with a client cached size (zero). However, we document (a) explicitly because it is important.

Modified() will only be called in response to one of these two events – not every time the sub-message is modified.

Note that if the listener’s |dirty| attribute is true, then calling Modified at the moment would be a no-op, so it can be skipped. Performance- sensitive callers should check this attribute directly before calling since it will be true most of the time.

class google.protobuf.internal.message_listener.NullMessageListener[source]

Bases: object

No-op MessageListener implementation.


python_message Module

Contains a metaclass and helper functions used to create protocol message classes from Descriptor objects at runtime.

Recall that a metaclass is the “type” of a class. (A class is to a metaclass what an instance is to a class.)

In this case, we use the GeneratedProtocolMessageType metaclass to inject all the useful functionality into the classes output by the protocol compiler at compile-time.

The upshot of all this is that the real implementation details for ALL pure-Python protocol buffers are here in this file.

google.protobuf.internal.python_message.InitMessage(descriptor, cls)[source]
google.protobuf.internal.python_message.NewMessage(descriptor, dictionary)[source]

type_checkers Module

Provides type checking routines.

This module defines type checking utilities in the forms of dictionaries:

VALUE_CHECKERS: A dictionary of field types and a value validation object. TYPE_TO_BYTE_SIZE_FN: A dictionary with field types and a size computing

TYPE_TO_SERIALIZE_METHOD: A dictionary with field types and serialization
FIELD_TYPE_TO_WIRE_TYPE: A dictionary with field typed and their
coresponding wire types.
TYPE_TO_DESERIALIZE_METHOD: A dictionary with field types and deserialization
google.protobuf.internal.type_checkers.GetTypeChecker(cpp_type, field_type)[source]

Returns a type checker for a message field of the specified types.

cpp_type: C++ type of the field (see field_type: Protocol message field type (see
An instance of TypeChecker which can be used to verify the types of values assigned to a field of the specified type.
class google.protobuf.internal.type_checkers.Int32ValueChecker[source]

Bases: google.protobuf.internal.type_checkers.IntValueChecker

class google.protobuf.internal.type_checkers.Int64ValueChecker[source]

Bases: google.protobuf.internal.type_checkers.IntValueChecker

class google.protobuf.internal.type_checkers.IntValueChecker[source]

Bases: object

Checker used for integer fields. Performs type-check and range check.

class google.protobuf.internal.type_checkers.TypeChecker(*acceptable_types)[source]

Bases: object

Type checker used to catch type errors as early as possible when the client is setting scalar fields in protocol messages.

class google.protobuf.internal.type_checkers.Uint32ValueChecker[source]

Bases: google.protobuf.internal.type_checkers.IntValueChecker

class google.protobuf.internal.type_checkers.Uint64ValueChecker[source]

Bases: google.protobuf.internal.type_checkers.IntValueChecker

class google.protobuf.internal.type_checkers.UnicodeValueChecker[source]

Bases: object

Checker used for string fields.


wire_format Module

Constants and static functions to support protocol buffer wire format.

google.protobuf.internal.wire_format.BoolByteSize(field_number, b)[source]
google.protobuf.internal.wire_format.BytesByteSize(field_number, b)[source]
google.protobuf.internal.wire_format.DoubleByteSize(field_number, double)[source]
google.protobuf.internal.wire_format.EnumByteSize(field_number, enum)[source]
google.protobuf.internal.wire_format.Fixed32ByteSize(field_number, fixed32)[source]
google.protobuf.internal.wire_format.Fixed64ByteSize(field_number, fixed64)[source]
google.protobuf.internal.wire_format.FloatByteSize(field_number, flt)[source]
google.protobuf.internal.wire_format.GroupByteSize(field_number, message)[source]
google.protobuf.internal.wire_format.Int32ByteSize(field_number, int32)[source]
google.protobuf.internal.wire_format.Int64ByteSize(field_number, int64)[source]

Return true iff packable = true is valid for fields of this type.

field_type: a FieldDescriptor::Type value.
True iff fields of this type are packable.
google.protobuf.internal.wire_format.MessageByteSize(field_number, message)[source]
google.protobuf.internal.wire_format.MessageSetItemByteSize(field_number, msg)[source]
google.protobuf.internal.wire_format.PackTag(field_number, wire_type)[source]

Returns an unsigned 32-bit integer that encodes the field number and wire type information in standard protocol message wire format.

field_number: Expected to be an integer in the range [1, 1 << 29) wire_type: One of the WIRETYPE_* constants.
google.protobuf.internal.wire_format.SFixed32ByteSize(field_number, sfixed32)[source]
google.protobuf.internal.wire_format.SFixed64ByteSize(field_number, sfixed64)[source]
google.protobuf.internal.wire_format.SInt32ByteSize(field_number, int32)[source]
google.protobuf.internal.wire_format.SInt64ByteSize(field_number, int64)[source]
google.protobuf.internal.wire_format.StringByteSize(field_number, string)[source]

Returns the bytes required to serialize a tag with this field number.

google.protobuf.internal.wire_format.UInt32ByteSize(field_number, uint32)[source]
google.protobuf.internal.wire_format.UInt64ByteSize(field_number, uint64)[source]

The inverse of PackTag(). Given an unsigned 32-bit number, returns a (field_number, wire_type) tuple.


Inverse of ZigZagEncode().


ZigZag Transform: Encodes signed integers so that they can be effectively used with varint encoding. See wire_format.h for more details.