Skip to content

Manual migration script generation fails for 26.8.0 org custom changes when upgrading from before 26.0.0 #55275

Open

Description

@shawkins

Before reporting an issue

  • I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

storage

Bug Description

Two custom changes in jpa-changelog-26.8.0.xml run queries against the generation database while a manual migration script is generated, and they read columns and tables that an earlier changeset has not created yet when the starting version is older than 26.0.0.

26.8.0-org-idp-and-domain-routing-migration (JpaUpdate26_8_0_DomainIdpRoutingMigration) selects IDENTITY_PROVIDER.ORGANIZATION_ID and joins ORG and ORG_DOMAIN. It has no precondition.

26.8.0-idp-mapper-organization-id (JpaUpdate26_8_0_IdpMapperOrganizationId) selects IDENTITY_PROVIDER.ORGANIZATION_ID. Its onSqlOutput="IGNORE" precondition is not evaluated in script mode, so the Java always runs.

ORG and ORG_DOMAIN are created in 25.0.0 and IDENTITY_PROVIDER.ORGANIZATION_ID is added in 26.0.0.
For a database older than that, the generation database does not have them.
CustomKeycloakTask.isApplicable() only checks that REALM exists and has rows, so it does not skip the task, the query fails, and the script generation fails with a CustomChangeException.

Automatic migration is not affected because each earlier changeset is applied before the next one is evaluated.

This was found by reading the code against the guidance in docs/updating-database-schema.md and has not been reproduced yet.

The comments on 26.8.0-idp-mapper-organization-id, 26.8.0-drop-idp-organization-id and 26.8.0-backfill-group-org-id also say the columns "existed before 26.8" or are "always present on the generation database".
That only holds for starting versions after 26.0.0 (ORGANIZATION_ID) or 26.6.0 (KEYCLOAK_GROUP.ORG_ID).
The IGNORE choice on the last two is correct because they only emit SQL, so only the wording is wrong.

Version

26.8.0

Regression

  • The issue is a regression

Expected behavior

Manual migration from any supported earlier version, including one that predates 26.0.0 and has realms, generates a script.
A task that reads a table or column that is absent from the generation database emits nothing, because a database without that column has no data to migrate.

Actual behavior

Script generation is expected to fail when the generation database has realms but lacks IDENTITY_PROVIDER.ORGANIZATION_ID, ORG or ORG_DOMAIN.

How to Reproduce?

  1. Create a database with Keycloak 24.0.4 or earlier, such as 19.0.3, and create a realm with an identity provider.
  2. Start the 26.8.0 server with --migration-strategy=manual against that database.
  3. Check whether script generation fails in JpaUpdate26_8_0_DomainIdpRoutingMigration or JpaUpdate26_8_0_IdpMapperOrganizationId.

The existing tests do not cover this.
CI only runs MigrationTest with -Dmigration.mode=auto from 24.0.4, and ManualMigrationEmptyDatabaseDistTest starts from an empty database with no REALM table, so the tasks are skipped.

Anything else?

Suggested fix:

  • Make the two tasks skip when the generation database does not have the table or column they read. Do not change the changesets themselves, because their checksums cover the changes and modifySql, not comments or preconditions.
  • Correct the misleading comments in jpa-changelog-26.8.0.xml. Comments and preconditions are not part of the Liquibase checksum, which I confirmed with Liquibase 4.33.0.
  • Add a test that generates a manual script from an older version with existing data. MigrationTest with -Dmigration.mode=manual is the closest existing one.

The other released changelogs from 25.0.0 to 26.7.0 use TEST with indexExists and columnExists, and were not reviewed individually.
The open 27.0.0 changelog has no problems against the new guidance: it has no preconditions and only reads tables that exist at every starting version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storageStorage layer — JPA entities, database schemas, Liquibase migrations, cachingkind/bugCategorizes a PR related to a bugstatus/triageteam/core-shared

    Type

    Fields

    No fields configured for bug.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions