Testing Java Annotation Processors with Compile Testing

Annotation processors are compiler plugins that inspect Java source while javac builds it. A processor can validate a custom annotation, report a diagnostic, or generate another source file, so ordinary unit tests may pass even when compilation fails. The risky behaviour sits in the compiler lifecycle, where elements, annotations and processing rounds interact.

The Compile Testing library makes that lifecycle testable by compiling small fixtures and exposing diagnostics and generated files. Combined with JUnit 5 and Truth assertions, it provides fast feedback without starting a complete build. This is valuable to Australian teams in Sydney, Melbourne and Brisbane, where JVM libraries support products in fintech, public-sector services and established enterprise systems.

Why annotation processors need focused tests

A focused processor test should reproduce a javac invocation with just enough source to trigger one rule. Small fixtures make failure output readable and show whether the defect lies in element lookup, diagnostic formatting, source generation or interaction between processing rounds.

Common defects include reading the wrong annotation mirror, losing generic type information, emitting a file with unresolved imports, and producing output that compiles only on one JDK. These failures can remain invisible when helper methods are tested directly because the processor never receives compiler symbols or a proper processing environment.

A useful test pyramid puts most cases in fast processor fixtures, then adds a small integration suite for service discovery and build-tool behaviour. That split keeps feedback quick while still covering the parts a compiler-only fixture cannot model. A processor that generates code is therefore tested at two levels: the contract of each rule, and the contract with the real build.

Preparing a JUnit 5 test harness

Add com.google.testing.compile:compile-testing and JUnit Jupiter to the test dependencies, and pin versions through Gradle or Maven dependency management. Compile Testing uses Truth-style subjects, so include Truth explicitly if the project references its assertions directly. Use a JDK matching the project’s supported range; compiling only on a newer JDK can hide dependencies on APIs unavailable on an older release.

Create a helper that accepts source text and a processor instance. JavaFileObjects.forSourceLines turns package declarations and source lines into an in-memory Java file, while Compiler.javac() controls the compilation. The following pattern keeps the intent visible:

var compilation = Compiler.javac()
    .withProcessors(new JsonBindingProcessor())
    .compile(JavaFileObjects.forSourceLines(
        "example.Order",
        "package example;",
        "import example.bind.JsonBinding;",
        "@JsonBinding",
        "public class Order {}"));

assertThat(compilation).succeededWithoutWarnings();

The test should name the behaviour, such as generatesBindingForAnnotatedType, rather than a generic method such as testProcessor. Give each processor its own fixture builder and keep unrelated annotations out of the sample. Australian development teams often use a mix of Windows laptops and Linux or macOS CI workers, so avoid hard-coded paths and make file comparisons insensitive to platform line endings.

Checking diagnostics and generated code

Start with the successful path. Assert that compilation succeeds, then decide whether warnings are acceptable. A processor that is expected to generate code should normally pass with no warnings, but a separate warning fixture may deliberately cover an experimental option. succeededWithoutWarnings() is stricter than succeeded() and prevents accidental diagnostic output from accumulating unnoticed.

For failure paths, assert both status and content. Use failed() with hadErrorContaining() for a stable phrase, and hadErrorCount() when extra errors would reveal a cascading problem. Where supported, also inspect the diagnostic code, source file and line. Exact full-message assertions are brittle when javac changes wording, while checking only that compilation failed can accept the wrong reason.

Also test supported processor options. withOptions("-Akey=value") can reproduce command-line switches, while an omitted or malformed option should produce the intended warning or error. Restrict getSupportedAnnotationTypes() to the annotations owned by the processor where practical; claiming every annotation can cause the processor to run on unrelated compilation units.

Generated sources need their own assertions. Retrieve a file through compilation.generatedSourceFile() and verify its qualified name, then inspect the text for the expected class, field or method. A stronger check compiles the generated source again in a separate invocation without the generating processor; that exposes missing imports, invalid syntax and inaccessible types immediately. A text snapshot is useful for stable formatting, but focused assertions are better when unrelated whitespace changes often.

Normalise output before comparing snapshots, especially line endings and generated timestamps. Keep generated code deterministic by avoiding dates, hash-code iteration order and absolute paths. Australian English can matter when APIs expose words such as serialise, and fixtures should also include Unicode names such as José or Müller to protect source encoding. Those checks are inexpensive and prevent failures that otherwise appear only on particular workstations.

Choosing the right verification level

Compiler fixtures, direct unit tests and full project builds verify different contracts. Choose the lightest layer that exercises the behaviour at risk, then add slower checks only where the build introduces extra behaviour.

Verification approach Main contract covered Strongest use Main limitation
Compile Testing with JUnit Diagnostics, processing rounds and generated files Fast rules and edge cases Does not reproduce every build plugin or classloader rule
Direct calls to processor helpers Internal validation and formatting logic Isolated utility methods Misses javac symbols and lifecycle behaviour
Gradle TestKit or a Maven integration build Service discovery, compiler flags, incremental metadata and packaging Release-critical wiring Slower and requires a realistic sample project
Golden-file repository compilation Complete generated output across supported JDKs Compatibility and broad regression coverage Noisy diffs and expensive execution

Use the first approach for the routine suite and retain at least one real-build check for shared processor artefacts. This matters in Australian organisations that distribute common libraries across multiple teams: a processor may pass in isolation yet disappear from a shaded JAR, lose its service descriptor or behave differently under an incremental build.

Golden-file snapshots work best for libraries with intentionally stable generated APIs. For frequently changing implementation details, assert public signatures or behaviour instead; otherwise reviewers learn to ignore large snapshot diffs and genuine regressions become harder to spot.

Covering discovery and incremental builds

withProcessors() supplies the processor explicitly and bypasses Java’s ServiceLoader lookup. It is the right choice for focused fixtures, but it does not prove that META-INF/services/javax.annotation.processing.Processor names the class correctly. Add a test that reads the service resource, and verify the packaged JAR contains it as well as the processor class and any required dependencies.

Gradle projects can also declare incremental processor metadata under META-INF/gradle/incremental.annotation.processors, using the appropriate isolating or aggregating classification. Compile Testing performs a fresh compilation, so it cannot establish that a recompile touches only the expected source files. A Gradle TestKit fixture should run a clean build, then change one input and inspect up-to-date and generated outputs. A Maven Invoker test can cover equivalent service discovery and compiler-plugin configuration.

Atlassian’s Sydney presence, Melbourne’s large consulting and fintech sector, and Brisbane-based software teams illustrate the range of Australian employers that may maintain shared build tooling. Such organisations often publish common components through internal Maven or Gradle repositories, and government suppliers face similar packaging and release-evidence requirements. These paths add practical failure modes: omitted resources, dependency convergence, proxy behaviour and release-plugin differences. Reproducing one representative build locally, while the normal suite remains fast, gives stronger release evidence without making every commit wait on a full integration project.

Recommendations for dependable processor tests

A balanced suite moves from precise compiler behaviour to the real consuming project. The fast tests should explain failures clearly and leave one slower test for wiring that the library cannot emulate.

Run the processor suite on every supported JDK, even if local development uses only one version. Compiler APIs evolve, and release flags can reveal differences in diagnostics or generated source handling. A scheduled build can broaden the matrix, but the pull-request suite should still cover the version used by the product team.

The durable rule is to test the compiler contract, not merely the processor’s helper methods: javac must accept the inputs, the diagnostics must be useful, the generated code must compile, and the consuming build must discover and package the processor correctly.