Skip to content

feat(server): graceful shutdown, atomic server state, and health endpoint Improvements #3243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 16, 2025

Conversation

sachaarbonel
Copy link
Contributor

This PR introduces robust server lifecycle management, atomic server state tracking, and improved health endpoint reporting to the whisper.cpp HTTP server.

Key Changes

  • Graceful Shutdown:
    • Implements cross-platform signal handling (SIGINT/SIGTERM, Ctrl+C) for clean server shutdown.
    • Double Ctrl+C forces immediate termination for developer convenience.
  • Atomic Server State:
    • Adds an atomic server_state variable to track model loading and readiness.
    • Ensures thread-safe state transitions during model (re)loading.
  • Health Endpoint Enhancement:
    • /health endpoint now returns 503 and a loading message if the model is not ready.
    • Returns 200 and {"status":"ok"} when the server is ready.
  • Threaded Server Execution:
    • Runs the server in a dedicated thread to allow responsive signal handling and proper resource cleanup.
  • Smart Pointer Usage:
    • Refactors server object management to use std::unique_ptr for safer resource handling.
  • General Refactoring:
    • Updates all server method calls for smart pointer usage.
    • Adds a cleanup routine to free resources on shutdown.

Motivation

  • Improve production robustness and observability.
  • Enable safe and predictable shutdowns for deployment and development.
  • Provide accurate health status for orchestration and monitoring systems.

Testing

  • Manual testing on macOS and Linux for signal handling and health endpoint responses.
  • Verified model loading, reloading, and shutdown behavior.
  • Confirmed /health endpoint returns correct status during all server states.

Copy link
Member

@danbev danbev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manually tested on Linux, macos, and windows 👍

@sachaarbonel
Copy link
Contributor Author

hey @danbev I adressed your review. Can I get a review again please?

@danbev danbev merged commit 107c303 into ggml-org:master Jun 16, 2025
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants