Skip to content

feat(health): implement pause scheduling domain logic and repository wiring - #894

Merged
andrzejfudro merged 3 commits into
mainfrom
feat/health-service-pause-scheduling
Sep 17, 2026
Merged

andrzejfudro merged 3 commits into
mainfrom
feat/health-service-pause-scheduling

Conversation

@andrzejfudro

@andrzejfudro andrzejfudro commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Author note:

Decoupled pause state from cache as the main purpose of it was caching http requests to other services (and checks that are triggered then in cascade) rather than reading database state (that gets easily invalidated). Added TODO comments that are also tracked as a separate tickets. In the follow up PR I will add publishing redis messages for pause and resumption.

Summary

Extracts pause scheduling domain logic from server.go into a dedicated PauseScheduler component backed by pausestate.Repository, wires it into Health Service via dependency injection, and dynamically reflects pause/resume states in GetHealthStatus.

Key Changes

  • cmd/health_service/pause_scheduling.go: Implements PauseScheduler managing the PauseSchedulingState persistence lifecycle:
    • Pause: updates state to SchedulingPaused: true and clears resumption metadata.
    • Resume: generates a new UUID v4 ResumptionRunID, sets SchedulingPaused: false, and clears ResumptionWorkerID.
    • IsPaused: checks whether scheduling is currently paused.
  • cmd/health_service/server.go: Injects PauseScheduler via newServer. Retains PauseScheduling and ResumeScheduling gRPC methods delegating core domain logic to PauseScheduler, sanitizing internal errors returned to the client. In GetHealthStatus, dynamically queries IsPaused(ctx) live on every request, completely decoupling pause state from cached component health checks. Adds a TODO tracking error propagation on fetch failure.
  • cmd/health_service/main.go: Instantiates PauseScheduler via initPauseScheduler and passes it to newServer. Includes a TODO tracking replacement of the in-memory fake with the real Spanner-backed repository.
  • cmd/health_service/server_test.go & pause_scheduling_test.go: Comprehensive unit tests covering pause/resume lifecycle, UUID generation, error message sanitization, and verifying live pause state evaluation on cache hits.

Assumptions

  • Treated PauseScheduler as a required struct invariant consistent with checker and guard, avoiding redundant defensive nil checks.
  • Redis stream publishing is strictly out of scope for this task and deferred to Task 8.
  • Extending the proto to report partial fetch errors is deferred to the Spanner wiring task, tracked via a TODO in server.go.

Verification & Testing

  • bazel test --nocache_test_results //cmd/health_service/... (Passed)
  • bazel build //cmd/health_service/... (Passed)
  • ./.githooks/pre-commit (Passed)
  • ./tools/analyze_diff_size.py --base-ref origin/main... (141 prod / 443 test lines)

Fixes bug 559548901

TAG=agy
CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4

…wiring

Extract pause scheduling domain logic into PauseScheduler backed by
pausestate.Repository. Wire PauseScheduler into Health Service via
dependency injection in newServer and main.go.

Implement state persistence lifecycle in PauseScheduler:
- Pause: sets SchedulingPaused=true and clears resumption metadata.
- Resume: generates UUID v4 ResumptionRunID, sets SchedulingPaused=false,
  and clears ResumptionWorkerID.
- IsPaused: reads current paused state from repository.

Keep gRPC service methods PauseScheduling and ResumeScheduling on
server.go, delegating core domain logic to PauseScheduler. Update
calculateHealthStatus to populate SystemInfo.is_scheduling_paused
directly from PauseScheduler. Invalidate health status cache upon
pause/resume state transitions.

Fixes bug 559548901

TAG=agy
CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces pause scheduling functionality to the health service, adding a PauseScheduler interface, its implementation, and corresponding tests. The health service server is updated to track and dynamically expose the scheduling pause state, invalidating its health status cache during state transitions. The code review feedback highlights several critical areas for improvement: replacing the in-memory fake repository used in production with a persistent implementation, resolving a potential race condition where slow concurrent health checks could overwrite the invalidated cache with stale data, and avoiding information disclosure by returning generic error messages instead of raw internal errors to gRPC clients.

Comment thread cmd/health_service/main.go Resolved
Comment thread cmd/health_service/server.go Resolved
Comment thread cmd/health_service/server.go Resolved
Comment thread cmd/health_service/server.go Resolved
- Dynamically populate pause scheduling state on every GetHealthStatus call,
  decoupling it from component health cache to eliminate cache races.
- Remove invalidateHealthStatusCache.
- Sanitize internal error messages in PauseScheduling and ResumeScheduling.
- Add TODO comments in main.go for Spanner repository and server.go for
  error propagation when pause scheduling check fails.

TAG=agy
CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4
@andrzejfudro
andrzejfudro marked this pull request as ready for review September 16, 2026 13:44
@github-actions
github-actions Bot requested a review from pachopepe September 16, 2026 13:45
pachopepe
pachopepe previously approved these changes Sep 16, 2026
Comment thread cmd/health_service/pause_scheduling.go Resolved
Comment thread cmd/health_service/pause_scheduling.go Resolved
Add an idempotency check in Resume() to return the current ResumptionRunID
when scheduling is already active, preserving active worker leases.

TAG=agy
CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4
@andrzejfudro
andrzejfudro added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit 0d3fa18 Sep 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants