Benchmarking Java strings and regex with JMH

String handling often looks trivial until a service starts parsing large payloads, rewriting identifiers or validating thousands of records. A regex may save code yet introduce compilation cost, backtracking and unexpected allocation. Measuring those effects with a stopwatch is too noisy for reliable engineering decisions.

JMH microbenchmarks provide controlled measurements with warm-up iterations, forked JVMs and statistical reporting. They can reveal whether Java string manipulation is fast in isolation, why a regular expression creates garbage and how performance changes as input grows.

The results still need context. A microbenchmark measures a narrow operation rather than an entire application, so its findings should guide profiling and optimisation rather than replace load testing. This article shows how to build a credible benchmark harness, compare realistic alternatives and interpret the outcome in an Australian engineering environment.

Why string microbenchmarks need care

Business services frequently process customer names, log messages, JSON payloads and reference identifiers. These workloads make string operations appear inexpensive because each individual call seems quick. Under sustained traffic, however, repeated character copying, temporary collections and regex compilation can increase latency and garbage collection pressure.

The hidden cost depends on the Java version and the exact method used. substring copies characters in modern Java, split may compile a regular expression, and replaceAll treats both its pattern and replacement as regex syntax. Unicode handling and the chosen string representation add further variables that a simple loop over elapsed time will not explain.

A useful benchmark therefore compares complete, equivalent operations. It also distinguishes one-off setup from repeated work. Otherwise, a benchmark can accidentally suggest that precompiling a Pattern has no value simply because the pattern was already compiled before the timed region began.

Building a trustworthy JMH harness

JMH belongs in a dedicated test or benchmark module with the annotation processor enabled. A typical Maven dependency is:

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.37</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.37</version>
</dependency>

Configure the run at the class or method level. For routine comparisons, five one-second warm-up iterations, ten one-second measurement iterations and three JVM forks offer a sensible starting point. The warm-up lets the JIT compiler optimise the hot path, while separate forks prevent one run’s heap state and compilation artefacts from contaminating the next.

Keep benchmark inputs in JMH state objects and exclude their construction from the timed section unless setup cost is the subject of the measurement. A Blackhole prevents the JIT compiler from eliminating an unused result. Test data should be deterministic and shared by the methods being compared; random generation inside each invocation would measure the random generator as much as the string operation.

Comparing common string operations

Literal replacement and regular-expression replacement answer different questions. String.replace searches for literal characters, whereas replaceAll interprets a pattern. Their results can differ when the input contains regex metacharacters, so comparing them requires either equivalent expected output or a clearly stated reason for the difference.

@Benchmark
public void literalReplace(String input, Blackhole blackhole) {
    blackhole.consume(input.replace(" ", "-"));
}

@Benchmark
public void regexReplace(String input, Blackhole blackhole) {
    blackhole.consume(input.replaceAll("\\s+", "-"));
}

The same care applies to trim and strip, concatenation and StringBuilder, or substring and copying to a new array. Each has distinct semantics and implementation behaviour. The benchmark should measure the construction teams would actually use, including any formatting, normalisation or validation surrounding the core call.

Modern Java releases also change the performance landscape. Compact strings reduce the memory used for many Latin-character data, while StringConcatFactory can improve concatenation. Record the JDK vendor, version, enabled flags and garbage collector in the results, because a comparison from Java 11 should not be presented as equivalent to one from Java 21.

Separating regex compilation from matching

String.matches compiles a pattern on every invocation. By contrast, a reused Pattern performs compilation once and then creates matchers for subsequent calls. Benchmarking these separately exposes a recurring source of overhead in code that validates short values inside a loop.

Pattern is immutable and safe to share between threads; Matcher is stateful and is not. If a benchmark or production method reuses a matcher, keep it in thread-scoped state and avoid concurrent access. Also compare the intended matching semantics: matches requires the entire input to match, while find searches for the first matching region.

Regex performance can deteriorate sharply with backtracking. Nested quantifiers and ambiguous alternatives may create superlinear work on specially crafted input. JMH should use representative, bounded data rather than an unbounded hostile string that can stall the benchmark runner. Anchors, possessive quantifiers or a different parsing strategy may improve reliability, but each alternative must be tested for identical results.

Choosing representative test data

Short strings do not expose the same allocation and scanning behaviour as long records. Use @Param fields to vary input length and character composition, then report each combination separately. This makes it possible to distinguish constant overhead from growth that becomes significant in production-sized documents.

Australian data provides useful variety. Synthetic records can include spellings such as “optimisation”, “organisation” and “favour”, alongside names containing accented characters and scripts used across Australian communities. Including ASCII-only data alone can conceal Unicode processing costs and create false confidence for teams operating with multicultural customer bases.

Keep correctness checks outside the timed benchmark. A separate test can assert that every implementation produces the required output for the same fixtures. Production records should not be copied directly into a public benchmark repository; generated data preserves realism without exposing personal information under the Australian Privacy Act or internal security policies.

Running benchmarks across Australian teams

Sydney and Melbourne are major centres for Australian Java development, and cloud infrastructure in both locations can provide consistent runners. Timing labels still need care: Sydney and Melbourne observe daylight saving, Brisbane remains on AEST and Perth uses AWST. Report scheduled-run timestamps in UTC while retaining the local zone separately, especially when correlating results with deployment logs.

Hardware consistency matters more than geographic proximity. A benchmark run on a MacBook in Brisbane is not directly comparable with one on an AMD-based cloud instance in Melbourne. Pin the JDK, operating-system image, virtual machine size and JVM flags, and use dedicated runners for performance comparisons. Local laptops are useful for quick feedback but unstable for release gates.

Many Australian employers in banking, government, utilities, logistics and mining operate Java backbones with strict processing windows. In those environments, a small allocation improvement may matter during a nightly batch, while extra CPU spent on unnecessary regex compilation can affect shared infrastructure. When an Australian teammate says they will “have a proper look”, the phrase usually means a genuine investigation rather than a quick scan, which fits the need for reproducible evidence before changing a service.

Nightly continuous-delivery jobs are usually more suitable than per-pull-request benchmarks. Correctness tests can block every merge, while a scheduled JMH task stores JSON results and compares them with an accepted baseline. Statistical thresholds and historical charts are more effective than a single pass or fail rule, which can create noisy failures when shared runners briefly become busy.

Turning measurements into a focused change

JMH reports average time, sample-time percentiles or throughput depending on the selected mode. For average time, a lower nanoseconds-per-operation result is better; for throughput, a higher operations-per-second result is better. Examine confidence intervals before declaring a winner, and investigate samples whose distributions overlap or contain obvious outliers.

Allocation is often as important as latency. Running JMH with the garbage-collection profiler can show bytes allocated per operation, helping explain why a seemingly fast regex creates unnecessary pressure on the heap. Pair the timing result with correctness evidence and a representative workload before replacing production code.

The principal scenarios compare as follows:

Scenario	JMH comparison	Useful metric	Main question
Literal versus regex replacement	Equivalent output with replace and replaceAll	Average time and allocation rate	Which method fits the required semantics?
Compiled versus repeated patterns	Reused Pattern and String.matches	Average time and allocation rate	How much does compilation cost?
Whole-input versus partial matching	matches and find with the same pattern	Sample time and percentiles	Which matching contract is required?
Input-size sensitivity	Short, typical and long values	Results per parameter set	Does work grow proportionally?
Alternative implementations	Two implementations with identical output	Average time, GC metrics and variance	Is the improvement stable and useful?

The next concrete step is to add one JMH benchmark module to the affected service, pin it to the production JDK and run it on the same isolated CI runner used for baselines.