Skip to content

fix(dev_env): restart xmc-bridge and ingestion-gateway in gke-upgrade restart step - #988

Merged
VLukiantsev merged 4 commits into
mainfrom
fix/dev-env-restart-bridge
Sep 28, 2026
Merged

VLukiantsev merged 4 commits into
mainfrom
fix/dev-env-restart-bridge

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

gke-upgrade --only restart now also restarts xmc-bridge and ingestion-gateway, with xmc-bridge going first. Before this, images for those services that were pushed with --only images (:latest tag) never rolled out.

Motivation & Context

step_restart used a hard-coded list of services that left out xmc-bridge and ingestion-gateway. During live testing of the ingestion-gateway 1P component, a rebuilt xmc_bridge image (with updated embedded component charts) was pushed, but the running pod kept the old image digest. The bridge restarts first because it may own component releases, such as the ingestion-gateway workload.

Key Changes

  • tools/dev_env/upgrade_dev_env.py: add xmc-bridge (first) and ingestion-gateway to the restart list. resolve_deployment_name already skips deployments that aren't deployed, e.g. when components or metrics are disabled.
  • tools/dev_env/upgrade_dev_env_test.py: updated counts, plus tests for bridge-before-gateway ordering and for skipping when absent. The tests assert on the full kubectl commands.

Assumptions

  • A redundant restart of an optional deployment is harmless, so no feature-flag checks were added. If it is not deployed, resolve_deployment_name returns None.

Verification & Testing

  • Commands Executed: bazel test //tools/dev_env/... (14/14 pass), ./.githooks/pre-commit (pass).
  • Live: on GKE dev cluster ..., gke-upgrade --only restart ran kubectl rollout restart on both xmc-bridge-deployment and ingestion-gateway-deployment.

Related Issues / Bugs

Found while working on bug 548435582.

TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds xmc-bridge and ingestion-gateway to the list of services restarted during the development environment upgrade, updating the corresponding unit tests to verify the new service count and ordering. Feedback on these changes highlights that because kubectl rollout restart is asynchronous, simply ordering the services in a loop will not guarantee that xmc-bridge finishes updating before ingestion-gateway starts; the script should instead wait for the rollout to complete. Additionally, the test helper _restart_cmd should dynamically reference self.config.namespace rather than hardcoding the namespace string.

Comment thread tools/dev_env/upgrade_dev_env.py Resolved
Comment thread tools/dev_env/upgrade_dev_env_test.py Resolved
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds 'xmc-bridge' and 'ingestion-gateway' to the list of services restarted during a rolling restart in the development environment upgrade script, ensuring 'xmc-bridge' is triggered first. It also updates the unit tests to reflect the new service count and adds test cases to verify the restart order and the conditional skipping of these optional deployments. There are no review comments, so no further feedback is provided.

… restart step

step_restart used a hard-coded list that omitted xmc-bridge and
ingestion-gateway, so images pushed with --only images (e.g. a new
xmc_bridge embedding updated component charts) were never rolled out.
Deployments that are not present are already skipped.

BUG=b/548435582
TAG=agy
Address review: restart the bridge first so it runs the new image before
component releases roll, and assert on full kubectl commands in tests.

BUG=b/548435582
TAG=agy
…pace in test

kubectl rollout restart is asynchronous, so document that xmc-bridge is only
triggered first rather than fully rolled first. The restart-order test now
derives the namespace from the test config.

BUG=b/548435582
TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7
@VLukiantsev
VLukiantsev force-pushed the fix/dev-env-restart-bridge branch from 468e551 to 391f40b Compare September 25, 2026 13:19
@VLukiantsev
VLukiantsev marked this pull request as ready for review September 25, 2026 14:37
andrzejfudro
andrzejfudro previously approved these changes Sep 28, 2026
Comment thread tools/dev_env/upgrade_dev_env_test.py Resolved
…ntics

rollout restart is asynchronous and step_restart needs no rollout ordering,
so assert only that both restarts are triggered.

Comment link: #988 (comment)

BUG=b/548435582
TAG=agy
CONV=baca6192-74d3-4579-9f73-0a3263d9a2a7
@VLukiantsev
VLukiantsev added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 3af2007 Sep 28, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants