#6 Protobuf and gRPC: buf, validate and connect-es

a tool for all aspects of Protocol Buffers, a validation plugin and an idiomatic gRPC-Web replacement.

May 16, 2023

Welcome to Behind the Mutex! Our weekly newsletter summarizes notable activity in open-source, new and growing projects and releases.

Open Source Landscape

Behind the Mutex picks a few categories and explores new and popular projects and features there.

Protobuf and gRPC

Protobuf and gRPC is an industry standard for establishing RPC between distributed services. Vast adoption, prominent maintainers and a rich ecosystem make it an obvious choice when an engineering team is looking for a solution to offer their customers a defined, versioned and backward- and forward-compatible API.

buf: https://github.com/bufbuild/buf

Have you ever struggled with setting up an environment for compiling your Protocol Buffers and gRPC definitions into one or multiple languages? How about keeping consistency across the definitions maintained by a team? Or timely identifying breaking changes? buf is all you need as it elegantly handles the mentioned challenges, as well as solves even more pain points. Let’s find out how.

buf is a CLI tool that drastically simplifies and streamlines operations with Protobuf:

Code Generation. buf completely solves any issues with setting up protoc plugins and running them against your .proto files. There is no need to install any prerequisites in order to generate code. Buf comes with a repository of plugins for many popular languages.
Linting. The tool is bundled with an excellent linter that enforces good standards and consistency for your .proto files.
Formatting. On top of linting, buf also offers a command to automatically format your definitions according to industry standards.
Detecting Breaking Changes. Usually it is not enough to have a properly formatted and well-designed Protobuf code. You also want to make sure that your changes do not break existing clients. Luckily buf handles this for us.

In addition to the key features above, the tool also takes care of dependencies to your Protobuf definitions, including resolving the dependency tree, as well as properly pinning the versions to avoid unexpected changes in the dependencies.

Here’s a quick example of how to generate Go code, including gRPC stubs. A set of required plugins is defined in buf.gen.yaml:

version: v1
plugins:
  - plugin: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc/go
    out: gen/go
    opt: paths=source_relative

Then,

buf generate

runs the specified plugins against your Protobuf modules and writes output files into the desired directories. Similarly, lint, breaking and format commands are available.

If you are looking for a way to integrate Protobuf operations into your CI/CD pipelines, buf provides easy-to-use GitHub Actions to further simplify the setup process.

Protoc-gen-validate: https://github.com/bufbuild/protoc-gen-validate

How do you validate Protobuf messages? One can come up with a number of solutions, including writing complicated imperative logic. The validate protoc plugin allows you to declaratively express your validation right in your Protobuf definitions. The generated code then can be used to enforce the validation rules in run time.

You may think that it might not be enough and you still might want to have some further validation based on your business logic. And you would be correct. But the plugin makes it easier to split concerns and introduce different levels of granularity in terms of testing messages for correctness.

protoc-gen-validate comes with an impressive set of built-in rules that cover many popular cases, spanning across:

primitive types, such as numerics, bools, strings, bytes, enums etc.
messages, repeated, one-of.
and well-known types, like timestamps, durations, any and more.

The plugin generates validate and validateAll methods for your messages that can be called to get the first error or all errors in the message correspondingly.

It is worth noting though that for Python no code generation is required as the plugin offers a Python package that relies on message descriptors to validate messages in run time.

The validate protoc plugin is also available for buf.

Connect-ES: https://github.com/bufbuild/connect-es

Looking for a more idiomatic version of gRPC-Web? Connect for ECMAScript implements Web clients for two RPC protocols: Connect and gRPC-Web. The library relies on Protobuf for ES that generates JavaScript / TypeScript code with no boilerplate inherent to Protobuf.JS.

As you can see, Protobuf for ES generates classes for messages and allows passing nested initializer objects. One-off fields, enums are supported. Message classes are instrumented with functionality for cloning, comparison, serialization and more. Well-known types are available.

Similarly to messages, Connect-ES generates service stubs with concise signatures:

Both callback- and promise-based clients are there to cover many use cases.

If you have any feedback or would like to see certain open-sources projects highlighted in our upcoming summaries and reviews, please feel free to comment, send an email or DM the author on Twitter @dalazx.

Behind The Mutex

REVIEW: Chroma - AI-native embedding database

#5: Containers and Kubernetes - Portainer, Telepresence and Clair

Discussion about this post