Using testcontainers to perform Flyway migration and generate jOOQ code with Gradle.

Intro

Long time ago, when switching to another project, joining new team, or just working with different branches developers needed to perform tedious local setup. Multiple versions of Java, different database versions with local database installation, sometimes we even had to use different operation systems to run some of the tooling. But now, when humanity invented such tools as jEnv, Docker, and many others, our life improved significantly.

With Docker, we are now able to start different databases, message brokers (i.e. RabbitMQ), and many more with a simple docker run. Those are frequently used during compilation or integration/component tests.

In my opinion, testcontainers are taking it to a next level: you don’t need to start containers manually. Testcontainers would spin container up for you, configure, provide connection configuration to you, and then gracefully destroy it. Of course, it’s oversimplification of the testcontainers capabilities and usage, but let’s leave it as is for now.

What was bothering me, though, is that I still needed to run my PostgreSQL database container manually for Flyway migration and jOOQ code generation in Gradle.

Problem

When I was starting a pet project I had the following requirements:

1. PostgreSQL database.
2. DB is accessed from the JVM application, using jOOQ.
3. Migrations performed using the Flyway.
4. The project is built using Gradle.
5. The build shall work in the “Fire and forget” mode.

Thus, simplified, build should:

1. Use java source code and .sql migrations as an input.
2. Spin up the database within testcontainers.
3. Perform the Flyway migration inside the testcontainers.
4. Generate database access code using jOOQ, using the migrated database from the previous step.
5. Use generated jOOQ code + existing Java code to complete the build.
6. (Optional) Run the integration tests using the same testcontainers.

I was sure that it shall be fairly simple to run Flyway migration in testcontainers and use the result for jOOQ code generation in Gradle.

Spoiler alert: It is easy. But not if you didn’t touch Gradle for 2 years.

jOOQ and Flyway, as expected, had Gradle plugins. There were almost no doubt: now I would immediately find a guide how to glue it all together with testcontainers… Nope. I mean. Actually, I did find two examples.
Unfortunately, Guide 1 uses Java code to perform the Flyway migration and use testcontainers, which I wanted to avoid. In my opinion it’s best to keep the build logic as close as possible to the build system.
There was another example I found, which I liked more, but it uses reflection, which… you guessed it… I wanted to avoid as well. Maven users are just happy people, having a solution directly from jOOQ since 2021, and a working example from testcontainers

The build

If you want to skip the description, here’s the sample repo, which tries to reproduce mentioned testcontainers setup for Maven with Gradle.

Dependencies

This dependency description is based on an older repository revision with an old-style dependency definition, as it better suits the blog post. For the implementation using a version catalog, please refer to the newest revision of the main branch.

During the build we would use PostgreSQL testcontainers and Flyway, so first we need to add them to the buildscript dependencies.

buildscript {
// ...
    dependencies {
//      ...
//      Required dependencies to start testcontainers and do the migration during the build.
        classpath 'org.testcontainers:postgresql:1.19.3'
        classpath 'org.flywaydb:flyway-database-postgresql:10.4.0'
    }
}

Second, we want to apply jOOQ and Flyway plugins:

plugins {
// ...
//  Gradle plugins for jOOQ and Flyway.
    id 'org.flywaydb.flyway' version '10.4.0'
    id 'org.jooq.jooq-codegen-gradle' version '3.19.1'
}

Next, we would need to add dependencies required for the application to be built and running:

dependencies {
//  ...
//  Dependencies required for the application to use jooq and flyway in runtime.
    implementation "org.flywaydb:flyway-database-postgresql:$flyway_version"
    implementation "org.jooq:jooq:$jooq_version"

//  Your database driver
    runtimeOnly "org.postgresql:postgresql:$postgres_version"
//  Database driver for the build time.
    jooqCodegen "org.postgresql:postgresql:$postgres_version"

//  Testcontainers dependencies for the test tasks.
    testImplementation "org.testcontainers:junit-jupiter:$testcontainers_version"
    testImplementation "org.testcontainers:postgresql:$testcontainers_version"
}

That ends the trivial part.

Build service configuration

Next, we need to run testcontainers with the database, which shall be up during at least:

1. Flyway migration task.

2. jOOQ codegen task.

   And optionally, we might want this container to run during the testphase, so we can reuse it and save some container startup time:

3. (Optional) Test task.

This would require us to (a) start testcontainers before the (1) Flyway task and (b) stop the testcontainers no later than the build ends. This can not be done properly using normal Gradle tasks approach. Fortunately for us, there are Gradle Shared Build Services.

Gradle takes care of the build service lifecycle, so we can be sure that testcontainers would be correctly stopped as defined, and, at the same time, there’s a guarantee that it would be available during all the tasks we need it to be available.

So, first, let’s define our build service:

abstract class PostgresService implements BuildService<BuildServiceParameters.None>, AutoCloseable {
    private final PostgreSQLContainer container;

    PostgresService() {
//      Services are initialized lazily, on first request to them, so we start container immediately.
        container = new PostgreSQLContainer("postgres:16-alpine")
        container.start()
    }

    @Override
    void close() {
//      Ensure to stop container in the end
        container.stop()
    }

    PostgreSQLContainer getContainer() {
        return container
    }
}

And then, create a provider, which would allow tasks to use it:

// Here we register service for providing our database during the build.
Provider<PostgresService> dbContainerProvider = project.getGradle()
        .getSharedServices()
        .registerIfAbsent("postgres", PostgresService.class, {})

Flyway configuration

Unfortunately, neither the Flyway nor jOOQ plugins allows to delay the configuration initialization untill they are actually required.

Thus, almost everything else, unfortunately, is more of a hack than a good solution. At least, it is not canonical.

We need to configure database in Flyway taking the DB configuration from the service we’ve implemented earlier. Actually, the first time we ask for the container instance, service would create the container. Please notice that doFirst is not canonical here, but as Flyway does not allow us doing this natively, there’s not much we can do about it.

flywayMigrate {
//  We need access to our database during the build.
    usesService dbContainerProvider
//  Define location of the migrations.
    locations = ["filesystem:src/main/resources/db/migration"]
//  Flyway plugin won't define input files for us, so to follow Gradle convention define them.
    inputs.files(fileTree("src/main/resources/db/migration"))

    doFirst {
        def dbContainer = dbContainerProvider.get().container
//      Set up the flyway config.
        url = dbContainer.jdbcUrl
        user = dbContainer.username
        password = dbContainer.password
    }
}

jOOQ configuration

Good news are, we still can configure most of the jOOQ things during the Gradle configuration phase:

jooq {
//  jOOQ generation config
    configuration {
        logging = org.jooq.meta.jaxb.Logging.WARN
        generator {
            name = "org.jooq.codegen.DefaultGenerator"
            database {
                name = "org.jooq.meta.postgres.PostgresDatabase"
                includes = ".*"
                excludes = 'flyway_schema_history'
                inputSchema = 'public'
            }
            target {
                packageName = 'com.testcontainers.demo.jooq'
                directory = 'target/generated-sources/jooq'
            }
        }
    }
}

As testcontainers would start clean each time, to run jooqCodegen task, we must ensure that flywayMigrate task was executed. For this we simply need to depend on flywayMigrate task. Unfortunately, jOOQ plugin configures some things in afterEvaluate (see here). This makes it a big ugly on our side:

afterEvaluate {
//  For jOOQ to run we always need for flyway to be completed before.
    jooqCodegen.dependsOn flywayMigrate
}

Last but not least, we need to configure jOOQ DB connection params. Similarly to Flyway, jOOQ plugin does not allow delayed initialization, so we would still use doFirst, and, as in the previous example, it still runs in the afterEvaluate block, so we can update the previous example to:

afterEvaluate {
//  For jOOQ to run we always need for flyway to be completed before.
    jooqCodegen.dependsOn flywayMigrate
    jooqCodegen {
        doFirst {
            def dbContainer = dbContainerProvider.get().container
            jooq {
                configuration {
                    jdbc {
                        driver = "org.postgresql.Driver"
                        url = dbContainer.jdbcUrl
                        user = dbContainer.username
                        password = dbContainer.password
                    }
                }
            }
        }
    }
}

Enforce execution

UPDATE: Since jOOQ 3.19.2 the following step is required to ensure that code generation is triggered automatically. Thanks Martin Váňa for the update.

To ensure that migration and code generation are executed automatically before the build make JavaCompile task dependent on jooqCodegen:

tasks.withType(JavaCompile).configureEach {
    ...
    dependsOn jooqCodegen
}

Optional configuration for tests

If you want tests to run within same container as the one you use for the build, adding following to the buildscript shall do, assuming you’re using spring:

test {
//  ...
//  This is only necessary if you want to reuse container for the tests.
    usesService dbContainerProvider
    doFirst {
        def dbContainer = dbContainerProvider.get().container
        systemProperty('spring.datasource.url', dbContainer.jdbcUrl)
        systemProperty('spring.datasource.username', dbContainer.username)
        systemProperty('spring.datasource.password', dbContainer.password)
    }
}

Instead of a conclusion

As a result, with some hacks, we achieved the goal. Flyway and jOOQ are automatically executed during the build within PostgreSQL testcontainers.

Unfortunately, this solution has its downsides, i.e. neither the flywayMigrate nor jooqCodegen tasks can be built incrementally, as either inputs or outputs are undefined. Fortunately for us, though, Gradle is implemented in such a way that if the result of jooqCodegen task did not change, all the dependent build steps would be able to reuse the caching, so, in general, incremental build works with a minor exception.

Liked my post? Have some private feedback? Want to ask some questions, or maybe discuss a job opportunity? Feel free to reach out to me directly using any of the linked social network!