🌐 US-Proxy
class="logged-out env-production page-responsive" style="word-wrap: break-word;" >
Skip to content

fix: wait for HTTP server drain during graceful shutdown - #1745

Merged
AmanGIT07 merged 4 commits into
mainfrom
fix/connect-shutdown-drain
Jul 14, 2026
Merged

fix: wait for HTTP server drain during graceful shutdown#1745
AmanGIT07 merged 4 commits into
mainfrom
fix/connect-shutdown-drain

Conversation

@AmanGIT07

Copy link
Copy Markdown
Contributor

Summary

Graceful shutdown now finishes draining in-flight requests before the
process tears down its dependencies.

Changes

  • ServeConnect tracks its shutdown goroutines with a sync.WaitGroup
    and waits for them after ListenAndServe returns.
  • A failed server.Shutdown returns early instead of also logging the
    "shutdown complete" message.
  • The metrics server shutdown gets the same grace period as the connect
    server, and its error is logged instead of dropped.

Technical Details

http.Server.Shutdown closes the listener first and drains active
requests afterwards, so ListenAndServe returns before the drain
finishes. ServeConnect now waits for the shutdown goroutines, so it
only returns once requests are done.

Test Plan

  • Added TestServeConnectReturnsAfterShutdownOnContextCancel, runs
    the real server with and without the metrics listener
  • go test -race ./pkg/server/... passes
  • golangci-lint run pkg/server/ reports 0 issues

🤖 Generated with Claude Code

ServeConnect returned as soon as the listener closed, before in-flight
requests finished draining, so callers tore down the database while
requests were still running. Track the shutdown goroutines with a
WaitGroup and wait for them before returning. Bound the metrics server
shutdown with the same grace period and log its error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
frontier Ready Ready Preview, Comment Jul 14, 2026 6:56am

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Improved graceful shutdown so both the main service and the optional metrics endpoint shut down in a coordinated, timeout-based way.
    • Ensures in-flight requests are allowed to complete and shutdown errors/success are logged consistently.
    • Adjusted behavior so the app returns promptly on non-standard server startup failures, rather than waiting unnecessarily.
  • Tests
    • Added coverage for graceful shutdown draining in-flight requests.
    • Added coverage that the service exits cleanly after context cancellation for both core-only and metrics-enabled modes.

Walkthrough

ServeConnect now coordinates graceful shutdown for the connect and optional metrics HTTP servers with a shared WaitGroup, timed shutdown contexts, and shutdown outcome logging. Tests cover in-flight request draining and context-cancelled shutdown for both server configurations.

Changes

ServeConnect Shutdown Coordination

Layer / File(s) Summary
WaitGroup-based shutdown implementation
pkg/server/server.go
Configures h2c shutdown handling, tracks connect and metrics shutdown goroutines, applies timed shutdown contexts, logs outcomes, and waits for completion after normal server closure.
Shutdown behavior tests
pkg/server/server_test.go
Adds in-flight request, dynamic-port, and readiness coverage, and verifies shutdown for connect-only and connect-plus-metrics scenarios.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pkg/server/server.go (1)

268-276: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Early return on ListenAndServe failure skips shutdownWG.Wait.

If ListenAndServe returns a non-ErrServerClosed error (e.g., port bind failure), ServeConnect returns at line 269 without waiting for the shutdown goroutine, which is still blocked on <-ctx.Done(). The goroutine will eventually unblock when the caller cancels ctx, but it outlives the function call. Consider returning after shutdownWG.Wait() on this path as well, or documenting that the error path intentionally skips the wait.

🔧 Proposed fix to wait on all return paths
 	// Start server
 	if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+		// Ensure shutdown goroutines complete even on the error path.
+		shutdownWG.Wait()
 		return fmt.Errorf("connect server failed: %w", err)
 	}

Note: the caller must cancel ctx for the shutdown goroutines to unblock; if they don't, Wait will block indefinitely. Alternatively, keep the early return but document the intentional skip.

🧹 Nitpick comments (1)
pkg/server/server_test.go (1)

43-87: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider testing actual in-flight request drain behavior.

The test verifies ServeConnect returns after context cancellation, but doesn't verify the core PR claim: that in-flight requests finish draining before ServeConnect returns. A long-running handler that blocks until a signal would prove the drain actually waits. Without this, a regression that removes shutdownWG.Wait() would still pass this test (since Shutdown closes the listener quickly).

💡 Suggested drain-verification test case
func TestServeConnectDrainsInflightRequests(t *testing.T) {
	logger := slog.New(slog.NewTextHandler(io.Discard, nil))

	var cfg Config
	cfg.Connect.Port = freePort(t)

	// Register a slow handler via a custom mux that ServeConnect can use.
	// Since ServeConnect builds its own mux, you'd need to either:
	// 1. Add a test-only hook to inject handlers, or
	// 2. Use the /ping endpoint with a deliberate delay via a wrapper.
	//
	// Minimal approach: verify that after cancel, an in-flight /ping
	// request still completes successfully.

	// This would require either a test hook in ServeConnect or
	// a separate test that directly exercises server.Shutdown + WaitGroup.
}

This may require a small test hook in ServeConnect to inject a slow handler; if that's too invasive, consider a unit test for the shutdown coordination logic in isolation.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f1a4f9c2-0e14-4c56-b9e7-c5b10d0ebd5e

📥 Commits

Reviewing files that changed from the base of the PR and between 8ad9342 and 05a9de9.

📒 Files selected for processing (2)
  • pkg/server/server.go
  • pkg/server/server_test.go

Comment thread pkg/server/server.go Outdated
@coveralls

coveralls commented Jul 9, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 29312885340

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.3%) to 45.212%

Details

  • Coverage increased (+0.3%) from the base build.
  • Patch coverage: 12 uncovered changes across 1 file (26 of 38 lines covered, 68.42%).
  • 116 coverage regressions across 4 files.

Uncovered Changes

File Changed Covered %
pkg/server/server.go 38 26 68.42%

Coverage Regressions

116 previously-covered lines in 4 files lost coverage.

File Lines Losing Coverage Coverage
core/invitation/service.go 66 50.54%
cmd/serve.go 42 0.0%
internal/store/postgres/invitation_repository.go 7 79.51%
pkg/server/server.go 1 64.29%

Coverage Stats

Coverage Status
Relevant Lines: 37669
Covered Lines: 17031
Line Coverage: 45.21%
Coverage Strength: 12.7 hits per line

💛 - Coveralls

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@rohilsurana rohilsurana left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes a real problem. http.Server.Shutdown closes the listener first and finishes requests after. So ListenAndServe returns while requests are still running. StartServer in cmd/serve.go then runs its deferred Close() calls right away (DB client, session service, billing services). Before this change, those could close while requests were still being served.

The WaitGroup approach is correct. The wait cannot hang, because server.Shutdown returns within the 10s grace period. If the server fails to start, the function returns before the Wait(), so a bad port cannot deadlock. The metrics server shutdown also improves: it now has a timeout, logs errors, and gets waited on.

Two things worth fixing before merge (details inline):

  1. The wait does not cover h2c connections. That is most gRPC traffic on this server.
  2. The new test passes with or without the fix, so it does not prove the new behavior.

Not part of this PR, but related: ServeUI has no shutdown handling at all. Its http.ListenAndServe ignores the context.

Comment thread pkg/server/server.go
Comment thread pkg/server/server.go Outdated
Comment thread pkg/server/server_test.go
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Share the http2.Server between the h2c handler and http2.ConfigureServer
so Shutdown sends GOAWAY to hijacked h2c connections. Extract the
duplicated shutdown goroutine into a gracefulShutdown helper that skips
its completion log when the server never started, and add a slow-handler
test that fails without the drain wait.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/server/server.go (1)

207-210: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Set ReadHeaderTimeout to mitigate Slowloris attacks.

Both HTTP servers are instantiated without a read timeout. Without a timeout, a slow or malicious client can hold connections open indefinitely, potentially exhausting server resources. Setting ReadHeaderTimeout is a standard best practice to bound how long the server waits while reading request headers.

  • pkg/server/server.go#L207-L210: add ReadHeaderTimeout: 5 * time.Second to the connect server configuration.
  • pkg/server/server.go#L232-L235: add ReadHeaderTimeout: 5 * time.Second to the metrics server configuration.

Source: Linters/SAST tools


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 216c8697-294b-43c1-a019-c2a7b6243242

📥 Commits

Reviewing files that changed from the base of the PR and between 86a2168 and a068555.

📒 Files selected for processing (2)
  • pkg/server/server.go
  • pkg/server/server_test.go

@AmanGIT07
AmanGIT07 merged commit ba080b5 into main Jul 14, 2026
8 checks passed
@AmanGIT07
AmanGIT07 deleted the fix/connect-shutdown-drain branch July 14, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants