[RFC] Where should we store our Kafka/Redpanda topic schemas?

[aldy505]

For consistency sake (and the aim for creating backward compatible stuff), we should document our Kafka/Redpanda topic schema somewhere.

A while ago (6 April 2023), I asked these questions on the Teknologi Umum group:

1. Do you have a repository or wiki that contains Kafka message schema contracts?
2. What protocol/format do you use for the schema contracts? Is it JSON, proto, or other formats?
3. What kind of things that you documents? Is it only schema & topic name without any notes, or as detailed as partition configuration, consumer group names, etc?
4. Is there any codegen that renders the schema to some programming language that is used throughout the code base?

I got some answers.

From @rizaramadan:

1. yes
2. protocol buffers
3. nothing much, some explain with comments, some don't
4. yes, to golang and typescript

From @lynxluna:

1. yes, shared repository
2. protocol buffers
3. no documentation
4. yes, it's important so no one actually touch the proto format. only senior engineers can generate the stub.

And so, how would we want to approach this? We already have a shared proto contracts, but that's for transport purposes (communication between services through gRPC).

• Should we create a new repository that explicitly (and exclusively) handles Kafka/Redpanda topic/message schema?
• Or.. should we reuse the proto repository, and create a distinction (or separate some of the directory) in which we're able to know whether it's for Kafka/Redpanda or gRPC?

[albilaga]

I do prefer in same proto repository. Why? To make it less scattered and easier to find. So when we want to find our all shared, we just need to find in 1 repository

[aldy505]

How would you suggest on separating the proto file from presentation (gRPC) purposes and Kafka purposes?

[rizaramadan]

how about:

• authentication
  • authentication.proto
  • events
    • user_registered.proto
• user
  • user.proto
  • events
    • user_deleted.proto

[aldy505]

I agree with @rizaramadan's approach. Yet, we have to ask other members about it first.

Ping @TokoBapak/backend

[lynxluna]

I believe that proto isa contract, hence the importance to make it shared and not scattered.

I propose not to use protobuf directly but create a pipeline to generate SDK and then auto-commit to a git repository or artifact repository. (e.g. jfrog if you're using uhuk Java) so that new people can just include the SDKs to their project.