Kafka, Parallel Consumers, and the 6-Hour Testing Bottleneck

At one point, our API security testing pipeline had become the single biggest bottleneck in our entire platform.

We were executing nearly 70,000 deep security tests in a single run, and it was taking almost 6 hours. To make matters worse, this was a shared execution queue across all customers. One massive test suite from a single enterprise client would back up the line, skyrocketing wait times for everyone else.

The textbook solution? Horizontal scaling. Spin up isolated worker pools per customer, migrate to a completely distributed job-based architecture, and auto-scale aggressively.

But that came with a massive catch: infrastructure costs would explode.

Instead of throwing money and machinery at the problem, I was challenged to scale smarter. That constraint led us down a deep engineering rabbit hole involving Kafka consumer internals, thread orchestration, offset safety, and stateful crash recovery.

By rethinking our execution model from scratch, we improved throughput by nearly 100x. Here is how we built it.

The Core Subsystem Challenge

Our testing engine executes real API security attacks. Each individual test follows a strict lifecycle:

[Pick Endpoint] → [Apply Attack Payload] → [Wait for Response] → [Analyze Behavior] → [Store Findings]

This looks straightforward, but the operational reality is highly unpredictable. Some endpoints respond instantly. Some take 30 seconds. Others simply hang forever.

Sequential execution was fundamentally broken at our scale. To survive, our engine required:

• Massive concurrency — thousands of tests running simultaneously
• Strict timeout controls — one hung endpoint cannot block everything else
• Bulletproof crash safety — a pod death cannot lose results or duplicate them
• Zero duplicate findings — users lose trust instantly if the same vulnerability appears multiple times

Kafka felt like the natural backbone for this architecture. It gave us a durable queue, replayability, and partition-based consumption. But once we moved past basic tutorial-level use cases, we hit a wall.

The challenge wasn't consuming messages faster. The challenge was: how do you safely execute thousands of long-running, unpredictable tasks in parallel without breaking Kafka's offset guarantees?

Problem 1: Concurrency vs. Offset Safety

The naive way to parallelize a Kafka consumer looks like this: consume a batch of records, hand them off to a local thread pool, and commit the highest offset.

This works beautifully — until a worker pod crashes mid-batch.

Imagine 100 tests running concurrently. 70 finish in seconds, but 30 are still grinding away. Suddenly, the pod suffers an OOM kill or a node eviction.

• If you commit offsets early: The 30 incomplete tests are lost forever.
• If you commit offsets late (after the whole batch finishes): The 70 completed tests will re-execute on restart. In a security platform, duplicate executions mean duplicate findings, inaccurate vulnerability counts, and corrupted reports.

Instead of writing complex custom offset-tracking logic across worker threads, we brought in Confluent's Parallel Consumer library.

ParallelConsumerOptions<String, String> options =
    ParallelConsumerOptions.<String, String>builder()
        .consumer(consumer)
        .ordering(UNORDERED)
        .maxConcurrency(100)
        .commitMode(PERIODIC_CONSUMER_SYNC)
        .build();

The magic of the Parallel Consumer is that it internally tracks records on a granular, per-message level. It handles out-of-order completions seamlessly. If task #50 finishes before task #40, the library tracks that gap and safely handles the underlying Kafka commits.

By delegating execution tracking to the library, an entire category of distributed state bugs disappeared overnight.

Problem 2: Hanging APIs Destroying Throughput

In production, our biggest enemy wasn't heavy processing — it was unresponsiveness. Some customer gateways were broken; some test endpoints simply never responded.

Without strict timeout enforcement, a worker thread would block indefinitely. If you have a concurrency limit of 100, it only takes 100 hung endpoints to fully occupy your thread pool. Throughput collapses to zero, and the entire pipeline stalls.

Kafka cannot fix this. The application orchestration layer must handle it.

We wrapped every test execution inside a timed Future:

Future<?> future = executor.submit(() -> runTest(message));

try {
    future.get(4, TimeUnit.MINUTES);
} catch (TimeoutException e) {
    future.cancel(true); // Interrupt the worker thread
    createTimedOutResult(message);
}

If an API crossed the 4-minute threshold, the task was forcefully cancelled, the thread released back to the pool, and an explicit TEST_TIMED_OUT result persisted.

This visibility was crucial. We didn't want silent failures. Users needed to see exactly which tests timed out so they could diagnose their own network or gateway issues. Most importantly, one slow endpoint could no longer poison the well for the rest of the execution engine.

Problem 3: The Dreaded Kafka Rebalance Storms

This is one of the most common pitfalls when dealing with long-running tasks in Kafka.

A standard Kafka consumer expects your application to continuously invoke .poll() within a window defined by max.poll.interval.ms. If your processing logic takes longer than this interval, Kafka assumes the consumer node has died. It triggers a rebalance, revokes your partitions, and hands them to another node.

Because our security tests could run for several minutes, our worker threads were deeply occupied. To Kafka, our consumer looked dead.

The result was operational chaos: endless rebalances, partition thrashing, duplicate processing, and total throughput collapse.

The Parallel Consumer library architecture elegantly solves this by decoupling message heartbeating from message execution. A dedicated lightweight background thread continuously communicates with the Kafka broker to keep the session alive, while our worker threads execute long-running tests independently.

Even during 4-minute attack simulations, Kafka remained perfectly stable. No unnecessary rebalances, no partition movement.

Problem 4: Graceful Drain vs. Hard Interrupts

A testing run can conclude in three ways: it completes naturally, it hits a global execution timeout, or a user explicitly clicks "Cancel."

Treating these scenarios identically was an early operational mistake. If we always force-killed threads, valid in-flight results were lost. If we always waited gracefully, a user-initiated cancellation would hang for minutes.

We split the termination lifecycle into two explicit paths:

if (isCancelled || maxTimeExceeded) {
    // Hard Interrupt: Users expect immediate responsiveness
    executor.shutdownNow();
} else {
    // Graceful Drain: Ensure all active work finishes cleanly
    executor.shutdown();
    executor.awaitTermination(5, TimeUnit.MINUTES);
}

Natural completions preserve data integrity. Manual cancellations feel instantaneous to the end-user.

Problem 5: The Sneaky Startup Race Condition

This was the most subtle bug in the system. At startup, before Kafka has successfully established a connection and delivered the first batch of records, the consumer evaluates its state.

A naive implementation looks at the queue, sees that parallelConsumer.workRemaining() == 0, and interprets it as: "The queue is empty, all work is done." The application shuts down before processing a single message.

A tiny race condition leading to catastrophic production failures where worker pods would instantly crash-loop on startup.

The fix was elegantly simple: an AtomicBoolean flag.

AtomicBoolean firstRecordRead = new AtomicBoolean(false);

We updated the shutdown condition to only activate after at least one record had been successfully read from the broker. One line of code eliminated an entire class of initialization failures.

The Real Engineering Challenge: Stateful Crash Recovery

Optimizing concurrency within a healthy JVM is one thing. Ensuring that concurrency survives a sudden container crash is where real engineering happens.

Our pipeline uses a two-phase architecture executing inside the same container:

[Producer Phase: Generates Test Jobs] → [Consumer Phase: Executes via Kafka]

If a pod was evicted or restarted during a massive run, we faced a critical question: on restart, how do we know where we left off?

Without state recovery, the producer phase would execute again, pushing duplicate jobs into Kafka, causing corrupted summaries and double-writes to MongoDB. We engineered a two-layer recovery model.

Layer 1: Singleton Configuration State (In-Memory)

To prevent 100 concurrent worker threads from hammering MongoDB to independently fetch attack configurations, we implemented an in-memory thread-safe singleton (TestingConfigurations.getInstance()). Configuration loads exactly once on startup, workers read from shared memory, and database pressure drops to near-zero.

Layer 2: File-Based Progress Checkpoints (On-Disk)

To coordinate phases across pod restarts, we used a localized persistent file-based state machine on the container's persistent volume:

{
  "summaryId": "run_cfg_9921a",
  "CONSUMER_RUNNING": true
}

This file acted as our source of truth during an unexpected boot sequence:

            [Service Starts]
                    │
         ┌──────────┴──────────┐
         ▼                     ▼
[Recovery File Exists?]   [No File Found]
         │                     │
         ▼                     ▼
┌───────────────────┐   ┌───────────────────┐
│ SKIP Producer     │   │ Run Producer Phase│
│ Resume via Kafka  │   │ Write State File  │
└───────────────────┘   └─────────┬─────────┘
                                  ▼
                        ┌───────────────────┐
                        │ Start Consumer    │
                        └───────────────────┘

• Kafka Offsets handled message-level recovery
• The State File handled workflow-phase recovery

The Payoff

Final Thoughts

The most rewarding part of this project wasn't working with Kafka itself. It was managing everything around it.

Distributed systems tutorials make streaming look simple: you write a producer, you write a consumer, and messages flow. But production environments don't live in tutorials. They throw network partitions, hanging APIs, unexpected restarts, and race conditions at your code.

Infrastructure gives you the primitives. Reliability is still something you have to engineer.