On this page0
Unfamiliar word? See Glossary: words used in these docs. Looking for the tasks in order? Begin at Start Here.
Reference
Failure Modes
What a person, a script and an administrator see when one part of an instance stops working: the database, the rate-limit store, a single service, the AI model runtime, the tunnel that connects the server to the internet, the disk, your backup destination, your identity provider, your mail server, or a request that takes too long.
What You Would Notice
Your instance is a set of separate services on one server, each doing one job: sign-in, people and roles, devices, compliance, recruitment and so on. They sit behind a single front door called the gateway. When one service stops, the gateway keeps serving the others, so a fault in one area does not take the whole product down. Each service is started with a restart policy, so one that stops is started again without anyone stepping in.
There are two places to look. The sign-in page shows one status line: All Systems Operational, Partial Service Disruption or System Maintenance in Progress. After signing in, System → System Health names the service that is unhealthy (see Check system health). For what to do about a message on screen, start with When something goes wrong.
No Uptime Commitment Is Published
Zonitix publishes no uptime SLA and no recovery-time commitment. That covers both a recovery-time objective and a recovery-point objective: neither is published. A standard instance is one server, with one PostgreSQL database on it that every service connects to. This page describes how that one server behaves; it makes no claim about failover to a second server. We host and monitor your server. Zonitix releases are installed automatically in a weekly maintenance window when that is switched on and email is set up, which is the choice pre-selected in the first-run setup wizard; an administrator can apply them by hand instead. Operating-system security patches follow the same window or are applied by hand. Backups and restores are your administrator’s to set up and run.
What the product does instead is fixed in its code: a stopped container is started again by its restart policy, and the database is backed up on a schedule. The schedule, how far back a restore can take you and who can run one are under Recovery Points: How Far Back, How Recent. Each scheduled restore drill records how long that restore took, and the Off-box backup destination dialog shows the figure in seconds beside each recent run; see Restore-Drill Schedule. That figure is a record of one past run, not a commitment.
When A Backup Later Fails
Setting backups up once does not mean they keep working: a credential expires, a bucket is deleted, the nightly backup stops. Once a first backup has been verified, the instance checks every hour that backups are still reaching your storage. It reports a failure in three cases: the backup destination has been switched off or removed; the storage has not answered three hourly checks in a row; or no backup has reached your storage for more than 3 days.
A failing backup never locks the product. Everyone keeps working. What changes is what the people who can fix it are shown:
- A warning banner across the top of every screen, for people who can open the System screens. It starts Backups are failing., gives the cause, and says that Zonitix keeps working but a lost server would mean lost data. It cannot be dismissed and stays until backups arrive again.
- A dialog, for the administrators who can change the backup destination or run a backup. Its title is Your backups are not reaching your storage. It has no close button and cannot be clicked away; it goes when you press I understand or Open Backups & Recovery. It shows the cause, the date of the last successful backup and the date the failure began, and appears once per browser session for each failure.
- An email and a notification in the bell, to the same administrators, with the subject Zonitix backups are not reaching your storage. Both are repeated every 3 days while the failure lasts, and sent again if the cause changes. The bell notification is written first, so it arrives when no mail server is set up.
To clear it, correct the destination or its credentials under System → Backups & Recovery and let a backup reach the bucket; the steps are in Set up backups. Until then the backups written each night stay on the server, which is the one place a lost server takes with it.
If something here is not working as you expect, start with When something goes wrong.
The plain explanation is above. This part is for IT and developers: the exact settings, permission names, API addresses and data fields behind it. Closed: select the heading above to open it. While it is closed, your browser’s find-on-page (Ctrl+F or Cmd+F) does not search this part.
How A Failure Reaches A Caller
Every request passes through the gateway. Most are proxied to a service, and the gateway applies the same four rules to each service behind it. The fifth rule covers the routes the gateway serves itself.
- Waiting. A proxied request waits up to 300 seconds for the service by default. Session validation waits 5 seconds. Calls the gateway makes itself wait 15 seconds, or 90 seconds for the few marked long-running.
- No answer. If the service cannot be reached, resets the connection or runs out of time, the reply is
504 { "error": "Gateway Timeout" }. The same body is used for every one of those causes. - Circuit breaker. Five consecutive failures against one service (no answer, or any 5xx reply) open a breaker for that service. While it is open the gateway answers
503 { "error": "Service temporarily unavailable" }at once, without calling the service. After 15 seconds requests are forwarded again: the first success restores normal service, and a single further failure opens the breaker for another 15 seconds. The breaker is per service, so the other services are untouched. - The service’s own fault. A 5xx raised inside a service is forwarded as it is: the shared envelope with
Internal Server Errorand acorrelationId. - Routes the gateway serves itself. Some routes are answered by the gateway calling a service on the caller’s behalf, for example the Security Lab routes under
/api/v1/auth/security-lab. They share the same breaker but answer through the shared envelope, not the two proxy replies above:500withInternal Server Errorand acorrelationIdwhen the service cannot be reached or does not answer within 15 seconds, and503with the same generic message while the breaker is open. In production the message is the same for every 5xx from these routes; the cause is in the instance log under thecorrelationId.
The envelope, the status table and which requests are safe to send again are on Error Handling.
What Happens When A Part Is Down
| What is down | What a caller gets | What an administrator sees | What to do |
|---|---|---|---|
| The database (PostgreSQL) | A signed-in or token request is refused at the gateway’s session check, which reads the database through the sign-in service: 500 with the error "Auth service error", and after five in a row 503 "Auth service unavailable". A route that needs no session fails inside its service: 500 "Internal Server Error" with a correlationId, then, after five such failures in a row from one service, 503 "Service temporarily unavailable" for that service for 15 seconds at a time. | Each service’s readiness check answers 503 with db: "down", so its container is marked unhealthy. The status line on the sign-in page stops reading "All Systems Operational". | Retry once it is back; for a write, read the record first to see whether it was saved. The database container is set to restart on its own. If it does not come back, contact Zonitix support: we host the server. |
| Redis (the store behind rate limits) | Routes protected by a strict limit refuse rather than run unprotected: 503 with Retry-After: 5 and the error "Rate limiting is temporarily unavailable, so this request was refused. Please retry shortly." This covers sign-in, sign-in links, the second factor, step-up, first-run setup, device agent check-ins, AI requests, document signing, Security Lab routes and public form submissions. Limits that only shape traffic let requests through instead: public read pages, training content, SCIM and the per-token limit on personal API tokens. | Sign-in is refused until Redis is back. | Retry after the interval. No data is at risk: the refusal happens before the request reaches a service. |
| One service behind the gateway | Its routes answer 504 with the error "Gateway Timeout" (sent for any failure to reach the service, not only a timeout). After five failures in a row they answer 503 "Service temporarily unavailable" without calling the service; after 15 seconds requests are forwarded again: the first success restores normal service, and a single further failure brings the 503 back for another 15 seconds. Routes served by other services are not affected. | System Health: the Containers card reads "Degraded" and names the service in red, and the header shows a count of unhealthy containers. The sign-in page reads "Partial Service Disruption". | A stopped service is started again by its restart policy. Reads are safe to retry. For a write that returned 504, read the record back before sending it again (see Retries below the status table on Error Handling). |
| The sign-in (identity) service | Every route that needs a session or a token answers 503 with the error "Auth service unavailable". The gateway waits up to 5 seconds for session validation before giving that answer. | The app cannot load any signed-in screen. The sign-in page status line reports the disruption. | Retry once the service is back. |
| The tunnel, or the network edge in front of the server | No request from outside reaches the gateway, so none of the replies on this page is sent: whatever a browser or a script receives comes from the network edge, not from your instance, and carries no correlationId. Every address of the instance is affected alike: the app pages, the API, SCIM and device agent check-ins. | The sign-in page does not load, so neither its status line nor System Health can be read. On the server the services, the database and the scheduled jobs keep running. Connections the server opens itself, such as the license check-in and backup uploads, do not travel through the tunnel. | The tunnel client is one container and is set to start again whenever it stops. If the instance stays unreachable, contact Zonitix support: we host the server. For a write that was in flight when the connection dropped, read the record back before sending it again. |
| The backup destination (your bucket cannot be reached) | Nothing changes for callers. Scheduled database backups are still written on the server. | Each upload that fails is written to the audit log with the error text: DB_BACKUP_UPLOAD_FAILED for a database dump, a base backup or a file archive, AUDIT_ARCHIVE_UPLOAD_FAILED for an archived audit log. Nothing is dropped: the upload check runs every 10 minutes (every 30 minutes for audit-log archives) and tries again every file that has no confirmed copy. After three hourly checks in a row without an answer from the storage, or more than 3 days without a backup arriving, the backup health check raises the “Backups are failing” banner, a dialog, a bell notification and an email (BACKUP_HEALTH_FAILING). The product is not locked. | Correct the destination or its credentials under System → Backups & Recovery; the next check sends what is waiting. Once any dump has reached the bucket, a dump on the server is removed on age (14 days by default) only after its own copy is confirmed there, so backups that cannot be uploaded stay on the server’s disk until the destination is back. |
| The server’s disk filling up | The gateway has no separate reply for a full disk. A request fails only if the service handling it cannot write, and is then answered like any other fault inside that service: 500 "Internal Server Error" with a correlationId. | System Health shows a DISK gauge with used and total space. From 90% used, a red notice on that page includes "Disk is almost full." The nightly file archive refuses to run when free space is less than the archive’s size plus 10% and 256 MB, and a database dump that comes out smaller than 1 KB is treated as failed and not kept. | Contact Zonitix support: the disk is part of the server we host. If backups are accumulating because the backup destination cannot be reached, correct that first (the row above). |
| Your identity provider (single sign-on) | What the person sees depends on whether the instance can build the provider’s sign-in address without calling the provider. With SAML it always can, and with OpenID Connect it can for one hour after it last read the provider’s discovery document: the browser is sent to the provider and shows the browser’s own connection error, not a Zonitix message. When the instance cannot build the address (OpenID Connect with no discovery document held and the provider unreachable, or a wrong setting), the browser returns to the sign-in page with "Single sign-on is not set up correctly on this instance. Ask your administrator to check the SSO settings." (the address ends ?sso_error=config). A failure after the person has signed in at the provider returns "Single sign-on did not complete. Try again; if it keeps happening, contact your administrator." (?sso_error=failed). | Sessions that are already open stay open: a session is checked against the instance’s own database, not against the provider. Sign-in with a password, a passkey or an emailed link does not call the provider, for accounts that have one of those. | Wait for the provider to come back, or check the single sign-on settings if the provider itself is up. |
| Your mail server (set up, but refusing or unreachable) | A first sign-in invite answers 424 with "The invite email to … could not be sent", followed by the mail server’s error, and the account keeps the status it had. The test send in the mail settings answers 424 with the mail server’s error text. A request for an emailed sign-in link gets the same reply whether or not the email left, so the person sees no error and no email arrives. | A failed first sign-in invite is written to the audit log as USER_INVITE_EMAIL_FAILED, and a failed test send as SMTP_TEST_FAILED. Any other message the mail server refuses is written to the server log with the recipient and the error, and is not sent again. | Correct the mail settings, run the test send, then repeat the action that needed the email. Each kind of email is covered under When A Send Fails on Outgoing Email (SMTP). |
| The AI model runtime | AI requests answer 500 with "AI processing failed" (or "AI engine unreachable" for chat) after one automatic retry. The AI report review answers 503 with { "offline": true, "advisory": true, "error": "AI service unavailable" } and no analysis. Five failures in a row make the gateway answer 503 for AI routes for 15 seconds at a time. | The AI service stays up and reports itself degraded, with the reason (runtime unreachable, or the selected model not installed yet), so the sign-in page reads "Partial Service Disruption". Features that do not use AI keep working. | Retry later. If the selected model is missing, the AI service tries to download it and then retries the request itself. |
| The health reading itself | GET /api/v1/compliance/reports/system-health fails like any route of the compliance service (504, then 503). | System Health shows "Could not reach the health service." and the indicator beside the refresh button changes from LIVE to STALE; the figures on screen are the previous reading. If only the container list cannot be read, a warning says the Docker socket is not reachable and the four gauges stay live. | Wait for the next 30-second refresh or press Refresh now. |
| The host updater agent | An update, backup or restore request that cannot be handed to the agent answers 503. In production the body is the generic envelope { "error": "Internal Server Error", "correlationId": "…" }; the reason (control channel not writable) is in the instance log under that correlationId. A request that was queued but is not picked up stays queued, and any further request answers 409 while it does. | The update status reads "Host updater agent not installed." and the action buttons are disabled. If the status cannot be read at all, the Updates and Backups screens say so and retry every few seconds. | The agent is a component on the server itself, outside the app. Contact Zonitix support. |
| The Zonitix license server (not reachable from your instance) | Nothing changes for callers. The instance keeps the plan it last received. | The failed check-in is written to the audit log as LICENSE_SYNC_FAILED. An unreachable license server never pauses a module: a pause needs an answer from the server after the paid-through date. | Nothing. The next check-in that succeeds clears it. |
| A request that takes longer than the gateway waits | By default the gateway waits up to 300 seconds for a proxied request and then answers 504 "Gateway Timeout". A few long operations (Security Lab scans and summaries) have a 90-second wait and answer 503 with code STILL_RUNNING and Retry-After: 60 while the work continues on the server. | The screen shows the message from the reply. For STILL_RUNNING it says the work is still running and to reload in a minute. | Do not send a STILL_RUNNING request again; read the result after the interval. After a 504 on a write, check whether the change was made before repeating it. |
What each kind of email does when the mail server refuses it is under When A Send Fails, and which path a request takes from the network edge to the gateway is under Request Path And TLS.
What a lapsed subscription does is a separate case, described on Your Customer Dashboard (Instance Status). Recovering from the loss of the server itself is on Backup, Restore & Updates.
The Backup Health Check
The check runs in the compliance service 90 seconds after it starts and then every hour, and does nothing until backup setup holds a verified first backup. Each run tests the connection to the destination and reads the newest dump recorded as uploaded to the destination configured now. The three failing reasons are destination_disabled, destination_unreachable (three consecutive hourly connection failures) and no_recent_backup (more than 3 days since the newest upload; when the current destination holds none yet, since the first verified backup or the last change of destination, whichever is later). The verdict is stored and returned as health by GET /api/v1/compliance/reports/backup-setup to callers who hold view_system: { state: "ok" | "failing" | "unknown", reason, detail, since, alertedAt, lastCheckAt, lastGoodAt, destFailCount, failDays }. Before the first verdict is stored it is { state: "unknown" } alone.
A new failure, a change of reason, or 3 days since the last alert writes BACKUP_HEALTH_FAILING to the audit log and notifies the holders of approve_system_updates in the bell and by email. While the verdict is failing it is re-evaluated every 5 minutes, and straight after the destination is saved, so the banner clears soon after a fix; those in-between checks do not count towards the three-in-a-row rule. The first healthy verdict after a failure is audited as BACKUP_HEALTH_RECOVERED. The dialog is shown by the dashboard to holders of approve_system_updates and is not shown while the instance is locked for backup setup; the acknowledgement is kept in the browser session, keyed on the reason and the time the failure began.
The Two Health Endpoints
Both are answered by the gateway and need no session, so an outside monitor can poll them.
GET /api/v1/healthanswers200 { "status": "UP", "service": "api-gateway" }. It says only that the gateway process is answering.GET /api/v1/health/clusterprobes each service the instance is expected to run, with a 2-second limit per probe, and answers{ "status": "…", "timestamp": "…" }. The HTTP status is200only when the status isoperational, and503otherwise. It never lists the services.
| status | Meaning | Sign-in page |
|---|---|---|
| operational | Every expected service passed its readiness check and none reported itself degraded. | All Systems Operational |
| degraded | At least one service is answering, and at least one is down or reports itself degraded. | Partial Service Disruption |
| maintenance | No expected service is answering. The sign-in page also shows this when it cannot reach the gateway at all. | System Maintenance in Progress |
A service’s readiness check connects to the database and answers 503 when it cannot; the same check is the container health check of most services, which is what turns a service unhealthy on System Health. Services for a module that is not active on your instance are skipped rather than counted as down. The sign-in page re-reads the status every 30 seconds.