Repository navigation
feat(health): implement pause scheduling domain logic and repository wiring - #894
Conversation
…wiring Extract pause scheduling domain logic into PauseScheduler backed by pausestate.Repository. Wire PauseScheduler into Health Service via dependency injection in newServer and main.go. Implement state persistence lifecycle in PauseScheduler: - Pause: sets SchedulingPaused=true and clears resumption metadata. - Resume: generates UUID v4 ResumptionRunID, sets SchedulingPaused=false, and clears ResumptionWorkerID. - IsPaused: reads current paused state from repository. Keep gRPC service methods PauseScheduling and ResumeScheduling on server.go, delegating core domain logic to PauseScheduler. Update calculateHealthStatus to populate SystemInfo.is_scheduling_paused directly from PauseScheduler. Invalidate health status cache upon pause/resume state transitions. Fixes bug 559548901 TAG=agy CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4
There was a problem hiding this comment.
Code Review
This pull request introduces pause scheduling functionality to the health service, adding a PauseScheduler interface, its implementation, and corresponding tests. The health service server is updated to track and dynamically expose the scheduling pause state, invalidating its health status cache during state transitions. The code review feedback highlights several critical areas for improvement: replacing the in-memory fake repository used in production with a persistent implementation, resolving a potential race condition where slow concurrent health checks could overwrite the invalidated cache with stale data, and avoiding information disclosure by returning generic error messages instead of raw internal errors to gRPC clients.
- Dynamically populate pause scheduling state on every GetHealthStatus call, decoupling it from component health cache to eliminate cache races. - Remove invalidateHealthStatusCache. - Sanitize internal error messages in PauseScheduling and ResumeScheduling. - Add TODO comments in main.go for Spanner repository and server.go for error propagation when pause scheduling check fails. TAG=agy CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4
Add an idempotency check in Resume() to return the current ResumptionRunID when scheduling is already active, preserving active worker leases. TAG=agy CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4
Author note:
Decoupled pause state from cache as the main purpose of it was caching http requests to other services (and checks that are triggered then in cascade) rather than reading database state (that gets easily invalidated). Added TODO comments that are also tracked as a separate tickets. In the follow up PR I will add publishing redis messages for pause and resumption.
Summary
Extracts pause scheduling domain logic from
server.gointo a dedicatedPauseSchedulercomponent backed bypausestate.Repository, wires it into Health Service via dependency injection, and dynamically reflects pause/resume states inGetHealthStatus.Key Changes
cmd/health_service/pause_scheduling.go: ImplementsPauseSchedulermanaging thePauseSchedulingStatepersistence lifecycle:Pause: updates state toSchedulingPaused: trueand clears resumption metadata.Resume: generates a new UUID v4ResumptionRunID, setsSchedulingPaused: false, and clearsResumptionWorkerID.IsPaused: checks whether scheduling is currently paused.cmd/health_service/server.go: InjectsPauseSchedulervianewServer. RetainsPauseSchedulingandResumeSchedulinggRPC methods delegating core domain logic toPauseScheduler, sanitizing internal errors returned to the client. InGetHealthStatus, dynamically queriesIsPaused(ctx)live on every request, completely decoupling pause state from cached component health checks. Adds a TODO tracking error propagation on fetch failure.cmd/health_service/main.go: InstantiatesPauseSchedulerviainitPauseSchedulerand passes it tonewServer. Includes a TODO tracking replacement of the in-memory fake with the real Spanner-backed repository.cmd/health_service/server_test.go&pause_scheduling_test.go: Comprehensive unit tests covering pause/resume lifecycle, UUID generation, error message sanitization, and verifying live pause state evaluation on cache hits.Assumptions
PauseScheduleras a required struct invariant consistent withcheckerandguard, avoiding redundant defensive nil checks.server.go.Verification & Testing
bazel test --nocache_test_results //cmd/health_service/...(Passed)bazel build //cmd/health_service/...(Passed)./.githooks/pre-commit(Passed)./tools/analyze_diff_size.py --base-ref origin/main...(141 prod / 443 test lines)Fixes bug 559548901
TAG=agy
CONV=68e2d5f5-411d-477e-955c-76fbbdf8ece4