Graceful Shutdown¶
By default, GizmoSQL stops immediately when it receives SIGINT (Ctrl-C) or
SIGTERM (the signal Kubernetes sends to terminate a pod): any in-flight queries
are interrupted and their clients get an error.
Graceful shutdown changes that. When enabled, the first SIGINT/SIGTERM
puts the server into a draining state instead of stopping right away:
- Already-running queries and their in-progress result fetches are allowed to finish (or hit their per-query timeout).
- New sessions and new statements are rejected with a retriable
UNAVAILABLEerror —"GizmoSQL instance is shutting down; not accepting new statements. Retry against another instance." - Once all in-flight work has drained, the server shuts down on its own.
This is the behavior you want for rolling deployments on Kubernetes (or any container orchestrator): when a pod is being replaced, its in-flight analytical queries complete cleanly instead of failing, while the load balancer routes new work to other replicas.
Enabling it¶
Graceful shutdown is opt-in. Enable it with the CLI flag or the environment variable:
| Setting | CLI flag | Environment variable | Default |
|---|---|---|---|
| Enable graceful shutdown | --graceful-shutdown |
GIZMOSQL_GRACEFUL_SHUTDOWN |
false (immediate stop) |
| Maximum drain time | --shutdown-grace-period-seconds |
GIZMOSQL_SHUTDOWN_GRACE_PERIOD_SECONDS |
300 (0 = wait indefinitely) |
These set the boot-time values. Both are also adjustable on a live server without a restart — see Adjusting it on a running server below.
gizmosql_server \
--database-filename=./data.db \
--graceful-shutdown=true \
--shutdown-grace-period-seconds=300
Or via environment variables (e.g. in a container):
How it works¶
- First signal → drain. On the first
SIGINT/SIGTERM, the server marks itself as draining. NewExecute, prepared-statement creation/execution, and update calls returnUNAVAILABLE; new client sessions are refused. Statements and result fetches that were already running continue. - Wait for in-flight work. The server waits until every in-flight query execution and result-stream fetch has completed.
- Grace-period cap. If draining takes longer than
--shutdown-grace-period-seconds, any remaining queries are interrupted and the server stops anyway. This guarantees the process always terminates — for example when a query has no timeout (query_timeout = 0) and runs forever. Set the cap to0to wait indefinitely and rely solely on per-query timeouts. - Second signal → force. Sending a second
SIGINT/SIGTERMwhile draining aborts the drain and stops the server immediately (the "press Ctrl-C again to force quit" pattern), interrupting any still-running queries.
Forcing interrupts queries, but can't preempt an uninterruptible call
A forced stop (grace-period elapsed or second signal) calls DuckDB's query
interrupt on every in-flight statement, so normal queries abort within a
fraction of a second and the server stops promptly. A handful of operations
don't check the interrupt flag — notably sleep_ms() — and will run to
completion regardless; only SIGKILL (or process exit) stops those. This is a
DuckDB limitation, not a GizmoSQL one, and doesn't affect real analytical
workloads.
Adjusting it on a running server¶
Both knobs are live-adjustable, just like gizmosql.query_timeout and the
statement-queue settings. They appear in the gizmosql_settings() view and can be
changed with SET GLOBAL — no restart required:
| Setting | SET GLOBAL name |
Type |
|---|---|---|
| Enable graceful shutdown | gizmosql.graceful_shutdown |
BOOLEAN |
| Maximum drain time | gizmosql.shutdown_grace_period_seconds |
INTEGER (seconds, 0 = unlimited) |
Both are server-wide (GLOBAL scope) and may only be changed by an admin
user:
-- Inspect the current values (effective + global + default + env var).
SELECT name, value, global_value, default_value, env_var
FROM gizmosql_settings()
WHERE name LIKE 'gizmosql.%shutdown%' OR name = 'gizmosql.graceful_shutdown';
-- Turn graceful shutdown on for a running instance.
SET GLOBAL gizmosql.graceful_shutdown = true;
-- Give long-running queries more (or less) time to drain.
SET GLOBAL gizmosql.shutdown_grace_period_seconds = 600;
When to use this:
- Enable on demand before a maintenance window — flip
gizmosql.graceful_shutdown = trueright before you roll a pod, even if the server booted with it off. - Extend the cap mid-drain —
gizmosql.shutdown_grace_period_secondsis re-read on every drain-loop iteration, so raising it while a drain is already in progress gives the remaining queries more time. Lowering it forces a sooner cutoff.
When the enable flag is latched
gizmosql.graceful_shutdown is consulted at the moment the first
SIGINT/SIGTERM arrives. Toggling it after a drain has already begun has
no effect on that drain — but the grace-period cap can still be changed
mid-drain.
Kubernetes configuration¶
The kubelet sends SIGTERM, waits up to the pod's
terminationGracePeriodSeconds, and then sends SIGKILL. For the drain to
complete, the pod's grace period must be at least as long as GizmoSQL's
shutdown grace period (plus a little slack):
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
# Give the kubelet enough time for GizmoSQL to drain before SIGKILL.
terminationGracePeriodSeconds: 330 # >= shutdown-grace-period-seconds
containers:
- name: gizmosql
image: gizmodata/gizmosql:latest
env:
- name: GIZMOSQL_GRACEFUL_SHUTDOWN
value: "true"
- name: GIZMOSQL_SHUTDOWN_GRACE_PERIOD_SECONDS
value: "300"
Match the two grace periods
If terminationGracePeriodSeconds is shorter than
GIZMOSQL_SHUTDOWN_GRACE_PERIOD_SECONDS, the kubelet's SIGKILL will
preempt the drain and in-flight queries will still be killed. Always set the
pod grace period to the GizmoSQL grace period plus a buffer.
During the drain the server's gRPC health check stops reporting healthy when the
process finally exits; combined with the UNAVAILABLE errors on new statements,
this lets a Kubernetes Service / load balancer stop routing new connections to
the terminating pod while existing queries finish.
Library (C API)¶
Embedders running a server via the GizmoSQL library can trigger the same drain
programmatically — the in-process equivalent of delivering one SIGINT/SIGTERM:
#include "gizmosql_library.h"
// ... RunFlightSQLServer(..., /*graceful_shutdown=*/true, /*grace=*/300) on a thread ...
RequestGracefulShutdown(); // begin draining; call again to force immediate stop
When graceful shutdown is not enabled, RequestGracefulShutdown() falls back
to an immediate stop (equivalent to ShutdownFlightServer()).