Idempotent consumers: a library that dedupes on topic:partition:offset

[Violetta Pidvolotska (violetta98)]

Hello everyone!

I got tired of writing the same dedup code in every consumer and getting it slightly wrong, so I packaged it.

from idemkit import QueueConfig, RedisBackend
from idemkit.contrib.kafka import kafka_consumer

consumer = kafka_consumer(
    backend=RedisBackend.from_url("redis://localhost:6379"),
    group_id="payments",
    config=QueueConfig(max_attempts=5),
)

@consumer.handle
def process(msg) -> None:
    charge_customer(msg.value())

In your poll loop, consumer.dispatch_sync(msg) tells you whether to commit or leave the message. It doesn't import the Kafka SDK, you create the client yourself.

Why a "processed offsets" set isn't enough: it lets two consumers check at the same time, both see nothing, and both run. Here only one can win. It also handles a consumer dying mid-message without leaving the offset stuck.

https://github.com/idemkit/idemkit, Apache-2.0

If it's useful, I'd appreciate a star. It's new, so visibility helps a lot right now.