test_detective

Functional tests for Java build tool plugins

Build plugins sit at the boundary between configuration and execution. A Maven mojo or Gradle task can pass unit tests yet fail when a real project resolves dependencies, enters a lifecycle phase, writes files, or reports an error. Build-level coverage creates a small fixture, applies the plugin, runs the relevant command, and inspects the observable result: exit status, generated artefacts, reports, and failure messages. For Java and Groovy tooling, this contract is more valuable than checking private helpers alone.

A reliable functional test starts with a minimal project resembling ordinary use. Maven can execute a goal against a generated POM and reactor; Gradle can launch a runner with a temporary project and injected plugin classpath. Keep inputs, Java version, locale, timezone, and repository settings deterministic, then cover a successful path and rejected input. Fast unit tests still handle edge-case logic, while functional coverage catches packaging, classpath, lifecycle, and configuration regressions before release.

Start with observable build behaviour

Treat the plugin as a contract with a build, not as a collection of classes. Each fixture should identify the available extension, goal, or task, required parameters, and expected output. Maven assertions might inspect generated source or an updated descriptor; Gradle assertions might inspect a task output, report, or project state. Assert the developer-visible result, not an incidental log line.

Create fixtures in temporary directories from small text resources. Give each run a unique project name, controlled permissions, and explicit encoding. Do not inherit a developer's settings.xml, shell aliases, global Gradle properties, or credentials. When a network service is unavoidable, use a local stub or fake repository and assert the request and error. Reproducibility across laptops, CI agents, and Maven or Gradle build services is the point.

Keep the build-level suite separate from fast unit tests, but connect them through the same domain model. A unit test can verify a filename rule; the functional fixture proves that Maven or Gradle supplies the project path in the expected form. Name fixtures after behaviour, such as generated sources or a failed configuration, so a CI report identifies the contract without dumping every internal method.

Build a Maven fixture around a real lifecycle

Maven offers two useful levels for this work. The Maven Plugin Testing Harness and MojoRule exercise mojo parameters, injected services, and project objects with little ceremony. They do not replace lifecycle checks: a mojo can behave differently after Maven discovers the plugin, resolves dependencies, and invokes it through a phase. For that, configure maven-invoker-plugin to run a copied fixture in an isolated directory, then use maven-verifier or a Groovy post-build script for assertions.

Make the fixture resemble a user's POM. A single module can prove basic goal binding; a small reactor reveals dependency order, inherited properties, and phase execution. Use a local repository under the fixture, then assert generated files and the absence of unrelated changes. Keep Groovy or Spock checks focused on stable files and values rather than Maven's complete diagnostic output. The Veloquips plugin examples provide a useful reference for that balance.

Version the execution environment as deliberately as the fixture. Run the Maven Wrapper against supported Maven and JDK combinations, and give the invoker a clean local repository so cached artefacts cannot conceal a missing dependency. A Groovy assertion can inspect a small manifest or generated file, while the Maven process remains responsible for the actual lifecycle. This separation keeps diagnosis fast when a failure comes from either plugin code or build-tool compatibility.

Exercise Gradle tasks with TestKit

Gradle's functional counterpart is Gradle TestKit. GradleRunner launches Gradle against a temporary project, while withPluginClasspath() supplies the plugin classes. A small settings.gradle or settings.gradle.kts and build script can apply an extension, register a task, and expose realistic inputs. build() and buildAndFail() then verify the command's outcome without reaching into task internals.

Unlike ProjectBuilder, TestKit exercises task registration, dependency resolution, source sets, configuration-cache behaviour, and the link between build script and task graph. Create a functionalTest source set under the java-gradle-plugin conventions, add the TestKit dependency, and select the Gradle version deliberately. Forward output for diagnosis, but assert files, values, exit codes, and stable failure text. Run a task twice to expose stale outputs and incorrect up-to-date checks.

TestKit also gives a natural place for asynchronous and external-process checks. Register a task that starts a worker, expose a bounded timeout, and assert completion or cancellation instead of sleeping for an arbitrary duration. Set GRADLE_USER_HOME and project caches to disposable locations when reproducing a clean build; otherwise a local result may depend on artefacts left by a developer's previous run.

Test inputs, failures and side effects

Most defects appear at the edges: a missing option, empty path, unsupported Java or Gradle version, unavailable repository, or unwritable file. Create a fixture for each contract branch and assert the non-zero status plus a stable diagnostic naming the input or dependency. Also prove that failure did not leave a half-written descriptor, corrupt a cache, or report success after a background process failed.

Treat external work as controlled infrastructure. Use a local HTTP stub for metadata, return deterministic responses, and await spawned processes so a test cannot pass while work continues. Keep projects and caches temporary, use a fixed clock or explicit timestamps, and bound retries. Australian teams in Sydney and Melbourne commonly use Linux, Windows, and macOS build agents; shell syntax, line endings, and file locks should not decide compatibility. For Melbourne-based builds, explicitly set the timezone during date-sensitive tests because daylight-saving transitions can change an offset or formatting unexpectedly.

Property precedence deserves its own fixture. Test defaults, project properties, user-provided options, and environment or system properties separately, then verify that secrets are not printed or copied into generated files. A plugin can behave correctly for one build tool and incorrectly for another when interpolation rules differ; paired scenarios expose that mismatch early.

Use a compact functional test matrix

A compact matrix prevents a large plugin from becoming unrelated smoke tests. Select fixtures for distinct build behaviours, reuse assertions where Maven and Gradle expose equivalent semantics, and let shared helpers create projects, isolate storage, and clean up. Each item in the lists below can become a fixture with a fixed command and visible result.

Balance successful builds with rejected inputs. Include a dependency or version failure, since compatibility bugs often survive a happy path. Run the smallest fixture during pull requests and broader reactor or lifecycle scenarios in the normal verification job.

Maven cases

Gradle cases

Give every fixture an expected exit code and a list of files it owns. Assert that unrelated files retain their content, catching accidental writes as well as missing outputs. Stable assertions keep CI failures short without hiding useful build output.

Compare harnesses and run them in CI

Choosing between harnesses follows the execution model. Maven invoker fixtures suit goals and reactor phases; Gradle TestKit suits tasks, plugins, and DSL configuration. Both can run in JUnit, but each needs its own wrapper, temporary workspace, and version matrix. Keep assertions semantic and treat supported JDKs as part of the fixture. The broader software testing guidance connects these checks with API, test-data, and performance suites in the delivery pipeline.

Use the project's Maven or Gradle wrapper locally, not a globally installed tool. In CI, provide a clean checkout, point maven.repo.local and GRADLE_USER_HOME at disposable caches, and set explicit proxy, locale, and timezone values. Australian organisations in banking, logistics, and government may connect Java builds to AWS Sydney or Azure Australia East; tests should not depend on a regional endpoint unless that is the contract. Isolation also helps teams distributed across Brisbane, Perth, Adelaide, and the eastern capitals. The practical choices align as follows:

Concern Maven fixture Gradle fixture Assertion
Execution Real mvn process in a copied project GradleRunner with plugin classpath Status and output
Configuration Mojo parameters and lifecycle phase Extension, task inputs, and DSL Parsed values and generated files
Isolation Temporary POM and local repository Temporary project and .gradle storage No unrelated changes
Failures Verifier or post-build checks buildAndFail() and forwarded output Stable diagnostic
Versions Maven Wrapper and JDK matrix Gradle version and JDK matrix Supported combinations

Keep each run small, named after its contract, and easy to diagnose. The concrete next step is one minimal Maven fixture paired with a matching Gradle TestKit fixture in the existing build.