Skip to content

Ignore a left coordinator in the JDBC_PING2 health check until it writes again - #55248

Merged
ahus1 merged 2 commits into
keycloak:mainfrom
ahus1:is-55236-jdbc-ping-remove-left-coordinator
Nov 19, 2026
Merged

ahus1 merged 2 commits into
keycloak:mainfrom
ahus1:is-55236-jdbc-ping-remove-left-coordinator

Conversation

@ahus1

@ahus1 ahus1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #55236

When the coordinator of a small cluster dies, its row stays in the table with coord = true until it is stale, and with the lower address it wins the split-brain decision against the survivor. Ignoring that row lets the survivor stay ready.

Decisions and assumptions

  • Each node remembers when a member left its view. The health check ignores the coord = true row of such a member until the row has been updated after it left. Only a member itself writes its own coordinator row, so an updated row shows that it is alive again, and the usual tie-break applies (the older node with the lower address wins).
  • The row is ignored instead of deleted. Deleting it from the other side races with a live member rewriting its own row, which can leave a live coordinator without a row for up to one info-writer interval and invert the tie-break.
  • The check compares the row's last_update with the local clock, as the staleness check already does. It relies on synchronized clocks, and the resolution is one second: a live member that rewrites its row within the same second as it left is counted again with its next write.
  • Remembered entries older than the staleness timeout are dropped whenever members leave, as the rows would be ignored as stale anyway.
  • A real partition has a short window where the other side's row is ignored and both sides may look healthy, until that side writes again. Split-brain detection is polling-based anyway, and a crash is more common than a partition.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deleting a live former coordinator’s row can temporarily select the smaller partition as the winner, requiring prompt republication and a live-partition regression test rather than reliance on periodic refresh.

1 open issue
What changed in this PR

This PR addresses #55236 by removing a departed coordinator’s JDBC_PING2 row, allowing surviving nodes to recover readiness without waiting for that row to become stale.

Changes:

  • Deletes the previous coordinator’s row when the local node is coordinator and the previous coordinator is absent from the new view.
  • Adds a regression test verifying removal while preserving an unrelated coordinator’s row.
  • Adds a separate design draft for consolidating database-backed session handling.
File Description
model/​infinispan/​src/​test/​java/​org/​keycloak/​jgroups/​protocol/​JdbcPing2Test.java Tests departed-coordinator cleanup and preservation of unrelated rows.
model/​infinispan/​src/​main/​java/​org/​keycloak/​jgroups/​protocol/​KEYCLOAK_JDBC_PING2.java Removes the departed coordinator’s database row during view changes.
model/​infinispan/​session-storage-refactoring.md Documents a separate session-storage proposal and its transaction, isolation, and validation requirements.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The proposal relies on unavailable reference documents and incorrectly treats embedded-session binary-storage isolation as an existing guarantee.

2 open issues
1 resolved since last review

Review effort: Balanced

Closes keycloak#55236

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-55236-jdbc-ping-remove-left-coordinator branch from 5b3b57a to f6e31fa Compare November 16, 2026 20:13
@ahus1
ahus1 requested a balanced review from Copilot November 16, 2026 20:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the departed-coordinator database-row cleanup change.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

keycloak-github-bot[bot]

This comment was marked as outdated.

…again

Instead of deleting the row of the previous coordinator, ignore its row in the split-brain decision until the row is updated after it left the view.

Closes keycloak#55236

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1 ahus1 changed the title Remove the previous coordinator from JDBC_PING2 on view change Ignore a left coordinator in the JDBC_PING2 health check until it writes again Nov 17, 2026
@keycloak keycloak deleted a comment from keycloak-github-bot Bot Nov 17, 2026
@ahus1 ahus1 removed the flaky-test label Nov 17, 2026
@ahus1
ahus1 requested a balanced review from Copilot November 17, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change excluding departed coordinators’ unchanged rows from health checks.

No open issues 🎉

Review effort: Balanced

@ahus1
ahus1 requested a review from ruchikajha95 November 17, 2026 10:57

@ruchikajha95 ruchikajha95 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM . Thanks @ahus1 for this PR.

@ahus1
ahus1 marked this pull request as ready for review November 19, 2026 09:45
@ahus1
ahus1 requested review from a team as code owners November 19, 2026 09:45
@ahus1
ahus1 enabled auto-merge (squash) November 19, 2026 09:45
@ahus1
ahus1 merged commit 7148a5e into keycloak:main Nov 19, 2026
97 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two-node cluster: killing the JDBC_PING2 coordinator makes the survivor report split-brain and stay unready for ~40–70 s

4 participants