test_detective

Testing Kafka Streams Topologies with TopologyTestDriver in Java

Kafka Streams applications can look deceptively simple. A few topics enter a topology, records are filtered or transformed, and a result topic receives the output. In production, though, behaviour depends on keys, timestamps, state stores, window boundaries, serializers, repartitioning, and the exact order in which events arrive.

TopologyTestDriver gives Java teams a fast way to exercise that behaviour without starting a Kafka broker. It runs a compiled Kafka Streams topology in memory, allowing tests to write records to input topics and read results from output topics. For backend teams in Sydney, Melbourne, or Brisbane, this makes feedback much quicker than waiting for a shared development cluster.

Testing approach Best use Main limitation
TopologyTestDriver Deterministic unit and component tests for stream logic Does not verify a real broker or cluster
Embedded Kafka Producer, consumer, and serialisation integration tests Slower and more operationally involved
Shared Kafka environment End-to-end validation across services Can be flaky, costly, and difficult to isolate
Production-like performance test Throughput, lag, partitioning, and resource behaviour Requires realistic infrastructure and data volume

Why the Topology Driver Matters

A Kafka Streams topology is a graph of processors rather than a single method call. A record may be deserialised, filtered, repartitioned, joined with another stream, aggregated into a state store, and serialised again before it reaches an output topic. Testing only individual helper methods misses the interactions between those stages.

The driver lets a test execute the graph as a small, controlled streaming application. You can provide an input record with a chosen key, value, and timestamp, then inspect the exact output produced by the topology. This is especially useful for rules such as “emit an alert after three failed payments in five minutes” or “ignore events older than the current session window”.

It also keeps most tests independent of infrastructure. A developer can run a suite from a laptop during an arvo coding session, without reserving a Kafka namespace or coordinating with a platform team. That speed encourages more examples around edge cases instead of relying on a few broad end-to-end scenarios.

What TopologyTestDriver Simulates

TopologyTestDriver simulates the Kafka Streams runtime closely enough for functional testing. It creates test input topics and output topics, applies configured serializers and deserializers, advances stream time, and maintains local state stores. It can therefore validate filtering, mapping, grouping, aggregation, joins, windows, and punctuations.

It does not behave like a complete Kafka cluster. There are no real brokers, network connections, consumer groups, replication events, broker rebalances, or production partition failures. A passing driver test cannot prove that a deployment will handle a dead broker or a badly configured security protocol.

That boundary is useful rather than problematic. Use the driver to test topology semantics and deterministic processing; use integration and system tests to cover Kafka clients, authentication, topic configuration, deployment manifests, and cross-service contracts. Keeping those responsibilities separate makes failures easier to diagnose.

Build a Testable Java Topology

A practical design starts with a topology factory that accepts configuration instead of reading everything from global application state. The production application can pass its normal StreamsBuilder, topic names, and SerDes, while a test can create the same graph with test-specific properties.

Topology topology = new StreamsBuilder()
    .stream("payments", Consumed.with(Serdes.String(), paymentSerde))
    .filter((id, payment) -> payment.amount().signum() > 0)
    .selectKey((id, payment) -> payment.customerId())
    .groupByKey(Grouped.with(Serdes.String(), paymentSerde))
    .aggregate(BigDecimal.ZERO,
        (customer, payment, total) -> total.add(payment.amount()),
        Materialized.with(Serdes.String(), Serdes.BigDecimal()))
    .toStream()
    .to("customer-totals", Produced.with(Serdes.String(), Serdes.BigDecimal()));

In a real test, the topology should normally be created by a named method rather than assembled inside the test itself. This prevents the test from accidentally describing a different graph. Keep topic names, application IDs, default SerDes, and state-store names explicit, because hidden defaults often create confusing failures.

Record names also matter. If a topology uses JSON, Avro, or custom domain objects, configure the same serializer and deserializer that production uses where practical. Using plain strings in the test while production uses a schema-aware format can conceal compatibility problems at the boundary.

Feed Records and Check Results

A basic driver test creates properties, builds the topology, and opens the driver in a try-with-resources block. Input and output topics are then represented by TestInputTopic and TestOutputTopic.

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payments-test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);

try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) {
    TestInputTopic<String, Payment> input =
        driver.createInputTopic("payments", stringSerializer, paymentSerializer);

    TestOutputTopic<String, BigDecimal> output =
        driver.createOutputTopic("customer-totals", stringDeserializer,
            bigDecimalDeserializer);

    input.pipeInput("event-1", payment("c-42", "12.50"));
    assertThat(output.readKeyValue()).isEqualTo(
        new KeyValue<>("c-42", new BigDecimal("12.50")));
}

Assertions should cover keys as carefully as values. A topology that calculates the right amount under the wrong key may send data to the wrong partition or customer. For multiple results, readKeyValuesToList() is convenient, while isEmpty() and getQueueSize() help assert that filtering rules have suppressed unwanted records.

Use descriptive fixtures rather than anonymous maps. A payment(...) factory or test data builder makes it clear which fields influence the rule under test. For an event pipeline handling promotions, withdrawals, or account funding, domain scenarios should include valid, rejected, duplicated, and incomplete events. A discussion of safe casino deposit bonus rules, for example, can provide ideas for testing eligibility, limits, and promotional-state transitions without coupling those examples to production data.

Control Time and Event Order

Time is one of the most common sources of weak Kafka Streams tests. Windowed aggregations and scheduled punctuation depend on stream time, which advances when records arrive with timestamps. Wall-clock time is generally irrelevant to a deterministic driver test unless the topology explicitly uses a wall-clock punctuator.

Create the input topic with an initial timestamp and an advance interval when you need predictable event times. Then send records with explicit timestamps:

TestInputTopic<String, Event> input = driver.createInputTopic(
    "events",
    stringSerializer,
    eventSerializer,
    Instant.parse("2025-01-01T00:00:00Z"),
    Duration.ofSeconds(1));

input.pipeInput("a", event("customer-1"), Instant.parse("2025-01-01T00:00:05Z"));
input.pipeInput("b", event("customer-1"), Instant.parse("2025-01-01T00:00:20Z"));

Test the boundaries deliberately: an event exactly at the window limit, an event just outside it, a late event, and two records with identical timestamps. Australian systems often process users across several time zones, from Perth to Sydney, so prefer UTC-based timestamps in fixtures. This avoids accidentally testing daylight-saving assumptions that differ between states such as Queensland and New South Wales.

If the topology uses a grace period, advance the stream far enough for the window to close before reading the expected result. A test that checks output too early may appear to prove that no result exists when the processor is simply waiting for more event time.

Inspect State Stores and Failure Paths

State stores are central to many stream applications. Aggregations, deduplication, joins, and session windows all depend on state that is not visible in an output topic immediately. When a store is materialised with a name, the driver can expose it for direct assertions.

KeyValueStore<String, BigDecimal> store =
    driver.getKeyValueStore("customer-totals-store");

assertThat(store.get("c-42")).isEqualByComparingTo("12.50");

Store assertions are valuable for checking intermediate state, but they should not replace output assertions. A topology may contain the correct aggregate while publishing the wrong record, topic, or key. Verify the externally observable result first, then inspect the store when it explains an internal invariant.

Failure tests deserve equal attention. Send malformed values, null keys, duplicate event IDs, and records that violate business constraints. Configure an appropriate deserialisation or production error handler and assert the intended behaviour. In a financial or payments context, silently dropping a bad event may be unacceptable; a dead-letter topic or explicit rejection event could be the required contract.

When testing joins, remember that both sides need compatible keys and timestamps. Feed each input topic independently and verify what happens when a matching record arrives before the other side, after the join window, or with a null key. These cases are often missed because the happy path is easy to express.

Fit Driver Tests Into Delivery

A useful test suite separates fast topology checks from infrastructure checks. Run driver tests on every Java build, including pull requests and local Gradle or Maven commands. They should be deterministic, isolated, and quick enough that developers do not avoid them.

Then add a smaller integration layer for real serializers, topic creation, security settings, and consumer-producer behaviour. In Australia, teams may deploy across AWS regions such as Sydney while supporting customers nationwide; test configuration should reflect the actual naming, retention, partition, and access-control conventions used by that environment. A driver cannot tell you whether an IAM policy, ACL, or Schema Registry permission is wrong.

Property-based tests can extend coverage for transformations and aggregations. Generate amounts, timestamps, customer identifiers, and event sequences, then assert invariants such as totals never becoming negative or duplicate IDs never increasing a balance. Keep generated cases reproducible by recording the random seed when a failure occurs.

Performance testing remains separate. The driver is excellent for thousands of carefully chosen behavioural examples, but it is not a replacement for measuring throughput, memory use, consumer lag, or repartition traffic on a real Kafka cluster. Those checks belong in a performance environment with realistic partitions and payload sizes.

A Practical Testing Routine

Start with one topology factory and one focused test for its simplest valid record. Add assertions for the output key, output value, and absence of unexpected records. Then expand the suite around the rules most likely to cause production incidents: timestamps, duplicate events, null data, late arrivals, state restoration assumptions, and serialisation failures.

Use stable topic names and close every driver with try-with-resources. Keep test fixtures small enough to understand at a glance, and give each test one behavioural purpose. When a test fails, the input sequence should make the reason apparent without requiring a debugger or a running Kafka cluster.

The concrete next step is to create a TopologyTestDriver test for one existing Java topology, feed it three timestamped records covering the normal path and one boundary case, and assert the complete output key-value sequence.