Testing Java Collections and Streams with JMH
Java collection and stream performance can look deceptively simple. A loop over an ArrayList, a lookup in a HashMap, or a filtered stream may appear easy to compare in a small program. In practice, the HotSpot JVM changes code while it runs, garbage collection interrupts execution, and the amount and shape of data can alter the result significantly.
Java Microbenchmark Harness, usually called JMH, provides the controls needed for reliable microbenchmarks. It handles warm-up iterations, forks, timing modes, compiler barriers, and statistical reporting, allowing developers to measure operations without relying on a casual System.nanoTime() experiment.
This matters when backend services process large JSON payloads, build response objects, or transform records before sending them through a REST API. A choice between a List, Set, Map, sequential stream, or parallel stream can affect throughput, allocation rate, and tail latency in production.
For Australian teams, these details can surface in systems serving customers across Sydney, Melbourne, Brisbane, and Perth, where workload patterns and infrastructure vary. A benchmark run on a developer laptop in Melbourne is useful, but it should be treated as evidence about that machine rather than a universal performance guarantee.
Why JMH Produces More Trustworthy Results
A naïve benchmark often measures the wrong thing. The first invocation may include class loading and JIT compilation, while later invocations benefit from optimised machine code. If the computed value is never used, the JIT compiler may remove the operation entirely. A loop can therefore report an impressive time for work that never really happened.
JMH addresses these problems with warm-up iterations and measurement iterations. Warm-up gives the JVM time to compile hot methods; measurement begins after the code has reached a more stable state. Forks start separate JVM processes, reducing contamination from earlier tests and making compiler behaviour more representative.
A basic benchmark can look like this:
@Benchmark
public int containsValue() {
return values.contains(target) ? 1 : 0;
}
The method should be placed in a JMH benchmark class and executed through a generated runner or the JMH Maven plugin. When a result must be consumed, use Blackhole or return the value from the benchmark method. Returning a value is often clearer for small operations because JMH consumes the result safely.
Benchmark configuration is part of the experiment. Mode.AverageTime reports time per operation, while Mode.Throughput reports operations per unit of time. SampleTime can reveal distribution and outliers. For server applications, average time alone may hide pauses caused by allocation or garbage collection, so throughput and percentile-style measurements are often more informative.
Designing Collection Benchmarks That Reflect Real Work
A fair collection comparison needs an explicit question. Comparing ArrayList, LinkedList, and HashSet without defining the operation produces little useful information. Are you measuring appending, indexed access, membership checks, iteration, removal, sorting, or memory consumption? Each data structure is optimised for a different access pattern.
Use JMH parameters to vary collection size and input position. A membership benchmark should test hits and misses, because a HashSet may behave differently from a list when the target is near the beginning or end. A lookup benchmark can include small, medium, and large datasets rather than a single convenient size.
@Param({"100", "10000", "1000000"})
int size;
@Param({"arrayList", "hashSet"})
String collectionType;
The fixture should be created outside the measured method when the benchmark is intended to measure lookup or iteration. @Setup(Level.Trial) is appropriate for data shared across all invocations, while @Setup(Level.Invocation) measures a different and much more expensive scenario because it rebuilds state for every operation.
Collection contents must also resemble production data. Sequential integers are easy for a CPU to process and may produce cache-friendly behaviour. Real domain objects with strings, nullable fields, and uneven key distributions can create different allocation and branch patterns. For a service handling customer records during an end-of-financial-year workload, a million simple integers is unlikely to represent the actual pressure on the system.
Do not confuse execution time with total cost. ArrayList may iterate quickly because its elements are stored contiguously, while LinkedList can incur pointer-chasing overhead. HashMap often provides fast average lookup, but resizing, poor hash distribution, and object allocation still matter. If the test concerns memory, add an allocation profiler or inspect heap usage separately rather than inferring it from one timing score.
Measuring Stream Pipelines Without Distorting Them
Streams introduce several variables: source collection, intermediate operations, terminal operation, boxing, ordering, and parallelism. A pipeline such as items.stream().filter(...).map(...).count() should be compared with an equivalent loop that performs the same logical work. The loop must produce an observable result, and both versions must process identical input.
Stream setup belongs outside the timed section when the benchmark is measuring traversal. Creating a new source list, generating random values, or rebuilding predicates inside the benchmark changes the question. If allocation is part of the production operation, measure it deliberately and label the benchmark accordingly.
Sequential streams can be competitive for readable transformations over sufficiently large datasets, but tiny collections may favour a plain loop because stream machinery has fixed overhead. Boxing can become significant when a pipeline converts primitive values to Integer, Long, or other objects. Prefer IntStream, LongStream, or DoubleStream when the computation naturally uses primitives, then verify the result with JMH rather than assuming it is faster.
Parallel streams require particular care. They use the common ForkJoinPool by default, so unrelated application work can compete for the same workers. The overhead of splitting, scheduling, and joining tasks may exceed the useful work for modest collections. On a laptop, a parallel stream may appear attractive; in a shared service hosted in an Australian cloud region, it may increase contention and worsen latency for concurrent requests.
When a pipeline has side effects or depends on ordering, the comparison becomes less reliable and less maintainable. Prefer pure transformations and a clear terminal operation. For APIs that return different error responses, functional behaviour should be tested separately; guidance on REST error handling is more appropriate than hiding exception behaviour inside a performance benchmark.
Interpreting Scores, Allocation, And Variance
A JMH result is a measurement with uncertainty, not a permanent property of a Java class. Report the Java version, JVM implementation, operating system, processor, collection size, benchmark mode, forks, warm-up count, and measurement count. A result from Temurin on an Apple Silicon machine should not be presented as equivalent to one from an x86 production host.
Run enough forks and iterations to expose variation, but avoid treating a longer run as automatically correct. Background processes, thermal throttling, CPU frequency changes, container limits, and garbage collection can affect results. Close unnecessary applications and avoid benchmarking while a laptop is performing a large download or build. A developer working from Sydney may see different numbers on battery power than when the machine is connected to mains power.
Use JMH profilers when the score raises a question. -prof gc can show allocation rate and collection counts. Hardware counters or Linux perf can help investigate cache misses and branch behaviour, although access may be restricted in containers or hosted CI agents. A benchmark that is faster but allocates substantially more memory may create worse pauses under sustained load.
Statistical significance should guide decisions. If two implementations differ by less than the normal error margin, prefer the clearer one unless a larger workload test shows a meaningful production effect. For an HTTP service, pair microbenchmark results with endpoint-level load tests. Network, serialisation, database calls, thread pools, and Australian internet variability can dominate a collection operation that takes only a few nanoseconds.
Turning Benchmarks Into Engineering Evidence
Keep benchmark source code under version control and run it with the same dependency versions as the application where possible. Name methods according to the operation and input, such as arrayListIteration or parallelStreamFilterLargeInput. Avoid clever benchmark abstractions that make it difficult to see what is actually measured.
A useful workflow starts with a hypothesis, isolates one operation, verifies equal output, runs JMH locally, and then confirms the result with a realistic load test. The benchmark should help explain a production observation, such as rising CPU usage after replacing a loop with a stream, rather than becoming a contest for the smallest synthetic number.
| Question | Useful JMH setup | Common interpretation |
|---|---|---|
| Which collection iterates fastest? | Same data, same traversal, AverageTime or Throughput |
Measures traversal cost and memory locality |
| How does lookup scale? | Several @Param sizes, hits and misses |
Shows size and access-pattern effects |
| Does a stream beat a loop? | Equivalent logic, consumed result, sequential mode first | Exposes pipeline overhead and boxing |
| Is parallel processing worthwhile? | Multiple sizes, sequential and parallel variants | Reveals scheduling overhead and contention |
| Which version allocates less? | -prof gc, fixed input and forks |
Connects timing with garbage-collection pressure |
Benchmarks should also be repeatable in continuous integration, although CI numbers are rarely suitable for strict performance gates without stable hardware. Store historical results and investigate substantial regressions. A small performance change may be acceptable if it improves readability, while a larger regression in a hot request path deserves a focused review.
Be cautious with external “benchmark” material that lacks source code, JVM settings, or workload details. A humorous example from a time-wasting collection of developer distractions may be entertaining, but performance decisions need reproducible experiments and production context. Treat published charts as leads for investigation, not as final evidence.
The central practice is simple: make the workload explicit, let JMH control execution, consume every result, and measure the costs that matter to the service. Java collections and streams do not have universally fastest implementations; their behaviour depends on data shape, operation, size, allocation, and concurrency. A well-designed JMH benchmark turns those variables into evidence that can support a sound engineering decision.