▪️ 🧑🏾‍🤝‍🧑🏽 🌐 New Go API for Protocol Buffers 😴 🌩️ 🏑

On the eve of the start of the "Golang Developer" course , we have prepared for you a translation of an article from the official Golang blog.

Introduction

We're excited to announce the release of a major revision of the Go API for protocol buffers , Google's language-independent data exchange format.

Prerequisites for updating the API

The first protocol buffer bindings for Go were introduced by Rob Pike in March 2010. Go 1 won't be released for another two years.

In the ten years since the first release, the package has grown and developed along with Go. The requests of its users have also grown.

Many people want to write programs using reflection to work with protocol buffer messages. The package reflectallows you to view Go types and values, but omits the protocol buffer type system information. For example, we might need to write a function that scans the entire log and clears any field annotated as containing sensitive data. Annotations are not part of Go's type system.

Another common need is to use data structures other than those generated by the protocol buffer compiler, such as, for example, a dynamic message type capable of representing messages of type unknown at compile time.

We also noticed that a common source of problems was that the interfaceproto.Message, which identifies the values of the generated message types, skimps on describing the behavior of these types. When users create types that implement this interface (often inadvertently by embedding a message in another structure) and pass values of those types to functions that expect generated message values, programs crash or behave unpredictably.

All three problems have the same root and one solution: the interface Messagemust completely define the behavior of the message, and functions that operate on values Messagemust freely accept any type that correctly implements the interface.

Since it is impossible to change the existing definition of the Message type while maintaining the API compatibility of the package, we decided that it was time to start working on a new incompatible major revision of the protobuf module.

Today we are pleased to release this new module. We hope you enjoy it.

Reflection

Reflection is the flagship feature of the new implementation. Just like the package reflectprovides a view of Go's types and values, the google.golang.org/protobuf/reflect/protoreflect package provides a view of values according to the protocol buffer type system.

A complete package description protoreflectwould take too long for this post, but nevertheless, let's see how we could write the log cleanup function we mentioned earlier.

First we have to write a file .protodefining an extension like google.protobuf.FieldOptions so that we can annotate the fields as containing sensitive information or not.

syntax = "proto3";
import "google/protobuf/descriptor.proto";
package golang.example.policy;
extend google.protobuf.FieldOptions {
    bool non_sensitive = 50000;
}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {
    string public_name = 1 [(golang.example.policy.non_sensitive) = true];
}

Next, we need to write a Go function that takes an arbitrary message value and removes all sensitive fields.

// Redact      pb.
func Redact(pb proto.Message) {
   // ...
}

This function accepts proto.Message - an interface implemented by all generated message types. This type is an alias for the type defined in the package protoreflect:

type ProtoMessage interface{
    ProtoReflect() Message
}

To avoid filling the generated message namespace, the interface contains only one return method protoreflect.Messagethat provides access to the message content.

(Why alias? Because it protoreflect.Messagehas a corresponding method that returns the original proto.Message, and we need to avoid import cycle between the two packages.)

The method protoreflect.Message.Rangecalls a function for each filled field in the message.

m := pb.ProtoReflect()
m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
    // ...
    return true
})

The range function is called with protoreflect.FieldDescriptordescribing the type of the field's protocol buffer and protoreflect.Value containing the field's value.

The method protoreflect.FieldDescriptor.Optionsreturns the field options as a message google.protobuf.FieldOptions.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why type assertion? Since the generated package descriptorpbdepends on protoreflect, the protoreflect package cannot return a specific type of option without calling import cycle.)

Then we can check the options to see the value of our extension's boolean variable:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
    return true //   non-sensitive 
}

Note that here we are looking at the field descriptor , not the field value . The information we are interested in is about the protocol buffer type system, not Go.

This is also an example of an area where we've simplified the protopackage API . The original proto.GetExtensionreturned both a value and an error. The new proto.GetExtensiononly returns the value, returning the default value for the field if it is missing. Extension decoding errors are reported to Unmarshal.

Once we've identified the field that needs editing, it's easy enough to clear it:

m.Clear(fd)

Putting all of the above together, our editing function looks like this:

// Redact      pb.
func Redact(pb proto.Message) {
    m := pb.ProtoReflect()
    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
        opts := fd.Options().(*descriptorpb.FieldOptions)
        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
            return true
        }
        m.Clear(fd)
        return true
    })
}

A better version could recursively descend into message value fields. We hope this simple example provides an introduction to reflection in a protocol buffer and its use.

Versions

We call the original version protocol buffers Go APIv1, and the newer version APIv2. Since APIv2 is not backward compatible with APIv1, we need to use different module paths for each.

(This version of the API is not the same as the version of the protocol buffer Language: proto1, proto2, and proto3. APIv1 and APIv2 - this particular implementation in the Go, which support both the language version proto2and proto3)

in the module github.com/golang/protobuf - APIv1.

In the google.golang.org/protobuf module - APIv2. We took advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. (We consideredgoogle.golang.org/protobuf/v2to make it clearer that this is the second major version of the API, but settled on a shorter path as the best choice in the long term.)

We understand that not all users will migrate to the new major version of the package at the same speed. Some will switch quickly; others may remain on the old version indefinitely. Even within the same program, some parts may use one API and others may use another. Therefore, it is important that we continue to support programs that use APIv1.

github.com/golang/protobuf@v1.3.4 Is the most recent version of APIv1 prior to APIv2.
github.com/golang/protobuf@v1.4.0Is a version of APIv1 implemented based on APIv2. The API is the same, but the basic implementation is supported by the new API. This version contains functions for converting between proto.MessageAPIv1 and APIv2 to facilitate the transition between them.
google.golang.org/protobuf@v1.20.0 — APIv2. github.com/golang/protobuf@v1.4.0, , APIv2, APIv1, .

(Why did we start with a version v1.20.0? For clarity. We do not expect APIv1 to ever reach v1.20.0, so a single version number should be enough to uniquely distinguish between APIv1 and APIv2.)

We intend to continue to support APIv1 without setting any deadlines.

This arrangement ensures that any program will only use one protocol buffer implementation, no matter which version of the API it uses. This allows programs to implement the new API gradually or not at all, while maintaining the benefits of the new implementation. The principle of choosing the minimum version means that programs can remain in the old implementation until the maintainers decide to update it to the new one (directly or by updating the dependencies).

Additional features to look out for

The package google.golang.org/protobuf/encoding/protojsonconverts protocol buffer messages to and from JSON using canonical JSON mapping , and also fixes a number of issues with the old package jsonpbthat were difficult to change without causing new problems for existing users.

The package google.golang.org/protobuf/types/dynamicpbprovides an implementation proto.Messagefor messages whose protocol buffer type is determined at runtime.

The package google.golang.org/protobuf/testing/protocmpprovides functions for comparing the protocol buffer of messages to the package github.com/google/cmp.

This package google.golang.org/protobuf/compiler/protogenprovides support for writing protocol buffer compiler plugins.

Conclusion

The module google.golang.org/protobufis a major overhaul of the Go support for the protocol buffer, providing first-class support for reflection, custom messaging, and a cleaned-up API. We intend to continue to support the previous API as a wrapper for the new one, allowing users to gradually implement the new API at their own pace.

Our goal with this update is to strengthen the benefits of the old API and address its weaknesses. As we completed each component of the new implementation, we started using it in the Google codebase. This gradual deployment gave us confidence in both the usability of the new API and the better performance and correctness of the new implementation. We are confident that it is ready for production.

We are very excited about this release and hope that it will serve the Go ecosystem well for the next decade or more!

Learn more about the course.

New Go API for Protocol Buffers