Before doing game development , The game server and front end adopt Protobuf For data transmission , In order to avoid being maliciously cracked , Also on the Protobuf The generated data is simply offset . It's going to be used again recently Protobuf 了 , So simply record the relevant content .
The most commonly used data communication format should be JSON, But in some applications with large requests ,JSON Yes 2 A question ：
1.JSON There are a lot of business irrelevant data in , Such as braces 、 Brackets, etc , When transmitting , This data wastes bandwidth .
2.JSON Slow decoding speed , When the quantity is large , Decoding takes up a lot of server resources .
Google Encountered these problems , And then put forward Protobuf, Its core purpose is to solve the above problems 2 A question .
Protobuf Choose the form of binary encoding , Code the business data into it , There will be no irrelevant data , Even field names are not encoded , This reduces bandwidth pressure , Besides Google I designed the encoding and decoding algorithm , To ensure that resources are occupied reasonably and as quickly as possible .
Besides ,Protobuf There is also a blessing effect ： It is difficult for humans to read the encoded data . This side effect achieves the effect of data protection to a certain extent , Some websites use this feature to implement anti - crawling .
For development reasons ,Protobuf It is divided into proto2 and proto3 Two versions , The two are incompatible , That is to use proto2 Apps cannot be used with proto3 Application communication , This article mainly discusses proto3, And will not involve proto2 And proto3 Comparison .
Use Protobuf The first step in communication , Is to define proto file , Let's start with a simple proto file , as follows ：
message Person{
required string name = 1;
message Info{
required int32 id = 1;
repeated string phonenumber = 2;
}
repeated Info info = 2;
}Think about it first , Why do you need proto file ？ Why not like JSON How about using it directly ？
Take a look back. Protobuf One of the advantages ： Only the service related data is encoded into the binary stream to be transmitted , To save bandwidth .
Protobuf The principle of this feature is ： Both sides of the communication have defined proto file .
The sender passes this proto The file defines what to send , This content doesn't need to be like JSON like that , There are field names 、 These are bracketed , After receiving the data , Then use what you have proto file , Analyze the data directly according to the algorithm .
I always stressed ,JSON When transmitting , There will be field names 、 There are brackets , Some people may be confused , Let's give you an example , as follows JSON：
{"name": "ayuliao", "age": 30}This paragraph JSON During transmission ,name and age These two field names will also be transferred , But these two field names have no business meaning , It is mainly used to obtain data , Suppose this data uses Protobuf transmission ,Protobuf Will not name and age Code into data stream , Only will ayuliao And 30 Code in , After receiving the data , I have a copy of the same as the sender proto file , Then press proto Decode the format in the file .
Think about it. , Without a field name , Reasonable decoding Protobuf data , It must be required that the transmission data is organized according to certain rules , such as ayuliao Must be in 30 front , So I will decode it first ayuliao, Decode again 30, thus , We introduce proto A syntax requirement for a file ,message The assignment ID in the structure cannot be repeated .
See the above proto file , There are two messge structure , External message Structure is Person,Person Among them are name And info Two attributes , among name The assigned identification number of is 1,info The assigned identification number of is 2, In the same message in , The assignment ID cannot be repeated , otherwise protobuf Not working properly .
Besides , From above proto You can also find ,message Different attributes in can have different qualifiers , Yes 3 Kind of ：
required： The data sent by the sender must contain the value of this field , The data received by the receiver must also be able to identify this field , vernacular , add required Modifier , Both sides of this field must use , Otherwise, the report will be wrong .
optional： Optional fields , The sender can optionally send this field , If the receiver can recognize this field, it will decode it accordingly , If you can't identify , Then ignore it directly .
repeated： Repeatable fields , The sender can contain multiple values each time , Similar to passing an array .
In daily use protobuf when , There are two common tips:
1. The allocation of identification numbers is generally divided by business , Fields between different businesses are not closely sorted by size , Such as ：
message data {
optional string name = 10001;
optional int32 age = 10002;
optional string job = 20001;
optional string hobby = 20002;
}Above proto, Basic information （name、age） With 1000 start , Other information （job、hobby） With 2000 start , So when you want to add it later , More clarity , For example, the basic information of gender should be added ：optional string sex = 10003;.
2. Many uses protobuf The communication system will set the qualifier of the field to optional, In this way, the system is upgraded , Old programs can communicate with new programs without upgrading , It's just that you can't recognize the new field , In this way, smooth upgrade can be achieved .
Here is an excerpt from the Internet proto The data types that files can use and when generated in different programming languages , Types mapped in the programming language , No need to remember , When needed , Just turn it over here .
The present , Microservice architecture is quite popular ,Python The commonly used communication technology in this area is Protobuf+gRPC, This article first introduces briefly Protobuf, Later, based on this article , Write an essay Protobuf+gRPC Use case of .
First , We need to download protoc compiler , adopt protoc, We can define proto The file is converted into the corresponding programming language .
Download address ：https://github.com/protocolbuffers/protobuf/releases, If you're like me , The use of windows 64 A system of , Then download win64 Version of protoc Can be .
And then , We define a simple proto file , be known as demo.proto, The contents are as follows ：
syntax = "proto3";
message Person {
string name = 1;
int32 id = 2;
string email = 3;
string phone = 4;
}adopt protoc take demo.proto Translate it into Python file , The order is as follows （protoc Replace the path with your own path ）：
& "C:\Users\admin\Downloads\protoc-21.1-win64\bin\protoc.exe" --python_out=. demo.proto--python_out Used to specify the build Python The path where the file should be stored , Followed by demo.proto File path （ Note that there are spaces for spacing ）.
After running the command , Will be generated with the name demo_pd2.py The file of .
And then we import demo_pd2 file , Use one of the Person Class can implement Protobuf Coding and decoding of .
Through a simple code demonstration ：
from asyncore import read
import demo_pb2
R = 'read'
W = 'write'
def fwrb(filepath, opt, data=None):
if opt == R:
with open(filepath, 'rb') as f:
return f.read()
elif opt == W:
with open(filepath, 'wb') as f:
f.write(data)
filepath = './person_protobuf_data'
person = demo_pb2.Person()
person.name = "ayuliao"
person.id = 6
person.email = "[email protected]"
person.phone = "13229483229"
person_protobuf_data = person.SerializeToString()
print(f'person_protobuf_data: {person_protobuf_data}')
fwrb(filepath, W, person_protobuf_data)
data_from_file = fwrb(filepath, R)
parse_person = demo_pb2.Person()
# No data
print(f'parse_person1 : {parse_person}')
# ParseFromString Function returns the length of the parsed data
parse_data_len = parse_person.ParseFromString(data_from_file)
# After the parsing ,parse_person Objects are filled with data
print(f'parse_person2 : {parse_person}')
print(f'parse_data_len: {parse_data_len}')
print(f'data_from_file lenght: {len(data_from_file)}')In the above code , It should be noted that ,ParseFromString The function does not directly return the parsed result , Instead, the result is filled directly into the parse_person In the object .
This article discusses protobuf Basic knowledge of ,protobuf It is common in applications requiring high communication quality , But my personal view is still ： Do not add entities if not necessary , If the application does not arrive, you need to use protobuf The point of , It's still a priority HTTP Well .
Last , about Python introduction 、Python Students interested in Automation , You can start with 《Python Office automation 》 This book , at present 5 On sale ~
