just speaking from experience here, i have done a lot of custom codecs and custom RPC APIs and i don't do them anymore... i used my experience to improve the way that fiatjaf's golang library for nostr is written and architected, but i would not write my own codecs now so readily or at least use tools that make it a lower debug time and easier maintenance thing, and actually, if i could avoid it, i'd not rewrite the RPC API framework and codec, and just use grpc/protobuf
i have a very nice guide on how to do grpc/protobuf apps in golang https://github.com/quanterall/kitchensink
the only exception to my advice about custom codecs is when you are writing a low level, high performance network transport
i was working on such a thing last year and it is the single case where extreme performance and optimisation is warranted
if the data volume is less than 1Mbit typical average throughput it's not worth optimising it
i mean client traffic not a server