Early Proto Architecture & C Server Code Generation

#

This is the second devlog in the development story of WindRPC, a lightweight Remote Procedure Call (RPC) framework designed for communication between microcontrollers and host applications.

https://github.com/micro-artwork/windrpc

Introduction

#

In the previous post, I introduced the YAML specification descriptor (user_spec.yml) to provide an overview of the framework. However, during the earliest prototyping stage, development actually began with defining Protobuf .proto message structures directly.

The first technical problem to solve was: in a microcontroller (MCU) environment where gRPC cannot be used, how can a single binary stream differentiate and route requests across multiple services and procedure calls?

Early Proto Message Design

#

The initial concept aimed to encapsulate all services and procedures within top-level container messages.

Considering the distinct transmission roles of client and server nodes, I divided top-level messages into a matching pair: ClientMessage and ServerMessage.

The message direction was determined at the top level, while nested oneof fields inside were used to hierarchically group services and procedures:

• Transmitting Node Separation: Top-level ClientMessage (for client requests) and ServerMessage (for server responses and notifications).
• Hierarchical Dispatch: Nested oneof blocks mapping identical service and command names to procedure parameters (inputs) and return values (outputs).

In architectural diagrams, this design appeared logical and clean. It seemed like an elegant way to organize all services and procedure calls under a unified Protobuf schema.

Because gRPC keywords could not be used, the focus was on human readability and creating a schema that could directly map to generated executable code.

[ Early Nested oneof Message Hierarchy ]

1. Client Request Packet (ClientMessage)
ClientMessage (Top-level)
  └── [oneof payload] ──► Request
                             ├── bytes request_id
                             └── [oneof service]
                                    ├── common.Request
                                    ├── led.Request ──► [oneof command] ──► display_pixels (PixelData)
                                    └── power.Request

2. Server Response & Notification Packet (ServerMessage)
ServerMessage (Top-level)
  └── [oneof payload]
        ├── Response (Response to request)
        │     ├── bytes request_id
        │     └── [oneof service] ──► common.Response / power.Response
        └── Notification (Asynchronous event push)
              └── [oneof service] ──► power.Notification

// Early core proto definition (windrpc.proto)
syntax = "proto3";

package hlt.windrpc.core;

import "hlt/windrpc/service/common.proto";
import "hlt/windrpc/service/led.proto";
import "hlt/windrpc/service/power.proto";

message ClientMessage {
  oneof payload { Request request = 1; }
}

message Request {
  bytes request_id = 1;
  oneof service {
    hlt.windrpc.service.common.Request common = 6;
    hlt.windrpc.service.led.Request led = 7;
    hlt.windrpc.service.power.Request power = 8;
  }
}

message ServerMessage {
  oneof payload { 
    Response response = 1; 
    Notification notification = 2; 
  }
}

message Response {
  bytes request_id = 1;
  oneof service {
    hlt.windrpc.service.common.Response common = 6;
    hlt.windrpc.service.power.Response power = 8;
  }
}

// Individual service definition example (led.proto)
package hlt.windrpc.service.led;

message PixelData {
  repeated fixed32 colors = 1;
}

message Request {
  oneof command {
    PixelData display_pixels = 1;
  }
}

NanoPB Complexity & The Callback Trap

#

Although the design was conceptually clean, compiling these nested .proto schemas into C structures via NanoPB revealed severe friction in embedded C development.

Nested Struct Path Verbosity

#

Combining Protobuf package names, service names, and procedure names resulted in extremely long C structure type names and oneof tag identifiers generated by NanoPB.

Accessing a specific procedure payload required traversing lengthy C struct paths:

// Example of nested struct tag assignment in C
hlt_windrpc_core_ClientMessage msg = hlt_windrpc_core_ClientMessage_init_zero;
msg.which_payload = hlt_windrpc_core_ClientMessage_request_tag;

hlt_windrpc_core_Request *req = &msg.payload.request;
req->which_service = hlt_windrpc_core_Request_led_tag;
req->service.led.which_command = hlt_windrpc_service_led_Request_display_pixels_tag;

Simply setting a message type resulted in bloated C code with poor legibility.

NanoPB Decoding Callback Friction

#

A deeper issue occurred when writing server-side C decoding routines.

NanoPB uses pb_callback_t handlers for variable-length strings and repeated fields. Decoding nested oneof structures required implementing manual callback functions at every level of the message tree:

• Implementing field-level decoding callback functions (decode_string_callback, decode_request_id, etc.)
• Manually handling oneof tag branches and reading byte streams
• Managing buffers and contexts to prevent stack/buffer overflows

Writing all of this manually in server C code led to an explosive increase in code volume. Hand-coded C decoding routines were error-prone, frequently causing build errors and runtime deserialization faults due to tag mismatch typos or structural manipulation errors.

Relying entirely on nested .proto message types to enforce RPC dispatching made the server parsing pipeline excessively fragile.

Transition to C Server Code Automation

#

Initially, the goal was simply testing whether .proto files could be generated from YAML and exchanged cleanly between C and C# clients.

However, after suffering through the pain of writing server-side packet parsing and dispatch logic by hand in C, I concluded that requiring developers to write decoding routines manually was completely unsustainable.

This prompted a pivotal decision: extending the generator script (windrpc_gen.py) so that it outputs not only .proto schemas, but also the entire C server engine code and callback stubs automatically.

The target design allowed developers to remain unaware of NanoPB’s internal callbacks or tag branching mechanisms, simply filling in application logic (e.g., LED control, sensor reads) inside auto-generated C function handlers.

Conclusion

#

Attempting to hierarchically organize RPC calls using nested Protobuf oneof structures resulted in significant friction with complex C structs and decoding callbacks in NanoPB environments.

This realization led directly to automating C server engine generation, which I will cover in detail in the next post.