How to save money with technology: Protobuf

For several years now, we have reached a point where it makes financial sense to use the latest technologies in our products. Today’s article is about how to save costs by using protobuf instead of e.g. xml or json. But let’s first clarify what protobuf is.

But what is this Protobuf anyway?

Protobuf – short for Protocol Buffers – is a method for serializing structured data. An interface description language automatically generates the model code for several programming languages. This generated code includes the creation of one of the entities described in the model, but also offers the possibility to identify such an entity from an existing byte array.

The goal of the Protocol Buffers was simplicity and speed, in particular the goal was to be smaller and faster than XML. Google developed this technology for internal purposes and then released it under an open source license. Therefore Protobuf is also widely used by Google for storing and exchanging all kinds of data. This especially in combination with grpc – a high performance RPC implementation by Google.

The small size is achieved by serializing to a binary format, so it is compact, but not self-descriptive and therefore not human readable and can only be processed by computers.

And how can I save money with it?

Nowadays more and more well-known companies are migrating to an IT infrastructure that is totally in the cloud. According to statistics, global cloud computing revenue will increase by around 80% by 2022, which means that it is expected that it is likely to reach $331.2 billion in revenue per year by 2022.

But the cloud offerings are not always the cheapest and have costs you can easily overlook. During the evaluation for Relance it turned out that the egress – outgoing – network traffic is charged, which in turn led to high costs due to the large amount of data. Since network traffic is billed per gigabyte, it depends on the format in which the data is transferred. But why is only the outgoing traffic billed? This is due to the fact that ISPs – the Internet service providers – offer much higher download performance than upload performance. In order for the cloud provider to be able to offer this upload performance, it must pass on the additional costs to the customer.

Would you like more information about Protobuf? We would be delighted to support you in your digital transformation into a company with future-proof technologies! Call us, we don’t bite.

Baseline situation

Now we come to the most exciting part. Our calculations are based on a bookstore that manages data structured as in the diagram below. If you assume several thousand orders per month, the data adds up and it quickly pays to optimize the transfer and that’s where Protobuf comes into play.

Disclaimer: The lengths of the field sizes are only chosen as an example and do not correspond to any proposal for the design of a productive application.

Results

After a short implementation of a random data generator you will get the following results. In contrast to xml and json, Protobuf saves almost a quarter to half of the data in uncompressed state. With gzip-compressed data, things look a little different. Since the proto data is already available in binary format, it is not possible to compress it much further. Thus, the conventional data formats in the “compressed” category are back in very good form.

Now to the financial: If we now assume the following prices for network traffic from the major cloud providers, the technology alone can save up to 38% in costs. This is because Protobuf saves 38% of data compared to XML and 21.1% compared to JSON, i.e. based on the current costs for data transmission in cloud services, the following applies:

Another aspect you should not lose sight of is the processing time, i.e. the time a computer takes to convert your data to XML/JSON or Proto. Protobuf needs about 45ms for this processing of 100’000 data records, which is up to 7x faster compared to other data formats – which means that you have the possibility to reduce the size of your server environment and therefore the monthly cloud budget is again smaller by factors.

We have already successfully accelerated several projects with grpc and protobuf! If you also want to benefit from these cost savings, give us a call!

Summary

So, with Protobuf, you have the ability to better utilize your virtual machines, requiring less power for each machine. In addition, the transfer literally saves money. With an average price of 0.07572 Swiss Francs per Gigabyte you can save up to 38% and thus 0.02877 CHF.

Of course, Protobuf is not the cure-all for data transfer, but it does a good job as a small, high-performance format. And in combination with grpc you have the possibility to optimize the entire transmission path for performance and thus create a fast, reactive product while saving a lot of money on your own server infrastructure.