Reading tracker active
0 / 7 read
0%
← Home
Architecture Notes

Technical Writing

Notes on distributed systems, AI infrastructure, databases, and observability. Written for engineers who want depth, not surface-level takes.

SYSTEM IMPROVEMENT STUDY · KAFKA CONSUMER RE-ARCHITECTURELOAD GENERATORk6 · JMeterKAFKA PARTITIONP0KAFKA PARTITIONP1KAFKA PARTITIONP2CONSUMERC0CONSUMERC1CONSUMERC2RESULT1 hour → 36 seconds · 100× throughputaryan.devdistributed-systems · observability
distributed-systemsobservability

Kafka, Parallel Consumers, and the 6-Hour Testing Bottleneck

How I redesigned a Kafka consumer architecture to achieve 100x throughput improvement — eliminating offset safety bugs, rebalance storms, and crash recovery failures in a production API security testing pipeline.

May 25, 2026·6 min read
KAFKA INTERNALS · COMMIT LOG ARCHITECTUREPARTITION 0 · APPEND-ONLY COMMIT LOGSEGMENTS0off 0+SEGMENTS1off 100+SEGMENTS2off 200+SEGMENTS3off 300+SEGMENTS4off 400+→PRODUCERwriteCONSUMERreadIN-SYNC REPLICAS (ISR)LEADERFOLLOWER AFOLLOWER Baryan.devdistributed-systems · databases
distributed-systemsdatabases

Kafka Internals: What Every Engineer Should Actually Know

A ground-up explanation of how Kafka actually works — log segments, consumer groups, ISR, offset management, exactly-once semantics, and the tradeoffs hiding beneath every configuration knob.

May 22, 2026·6 min read