Backup, Restore & Updates

How backups are taken on your instance, why the two operations that can lose data — a database restore and a backup deletion — need two different administrators, why backup setup is mandatory, and how updates are installed: by themselves in a weekly maintenance window, or by hand.

What It Does

The operations that can lose data — overwriting the live database with an old backup, or deleting a backup outright — are not one-click for anyone. A restore or a deletion is requested by one administrator and must be approved by a different administrator before anything runs. Even then, the server takes a fresh safety backup of the current state before it restores, so a mistaken restore is itself recoverable. A system update applied by hand is different: an administrator’s own request runs straight away, and a request from anyone else waits for an administrator to approve it. An update installed by the automatic schedule needs no request.

What Is Backed Up, And When

The schedule is fixed on the server; you do not set it in the app. Everything below stays on the server only until you configure an off-box destination, after which each item is encrypted on the server and copied to your bucket. The times below are when each timer is due; the short random delay on each start, and how long each copy is kept, are listed once in The schedule and retention defaults in code.

This retention is why a row can show no local under STORED: the copy on the server was pruned and the bucket copy is the one left. A database dump in that state still has a Restore button while the off-box destination is enabled and the object is still in the bucket; the server fetches it first (the reads behind this are under Technical detail below). The file-store archive is not in the database restore-point list. It has its own File backups panel on the same screen, with its own Restore button and the same two-administrator approval (see Restoring The File Store). Zonitix does not perform restores on your behalf.

Mandatory Backup Setup

Every instance, on every plan, has to send its backups to storage its owner controls before the rest of the dashboard opens. Nothing here is optional and no plan switches it off. It is done once, on System → Backups & Recovery, in two stages; the steps are in Set up backups.

1. Storage and a first verified backup. Connect a bucket you own and run a backup. The instance uploads it, then reads it back from your storage and decrypts it. Only a backup that survives that check counts. A new instance is locked from the first sign-in (the backup setup lock) until this stage is done, and it unlocks as soon as the backup is verified.
2. Both recovery keys in escrow. The backup key and the application data key each have to be split between trustees (see Key Escrow), which takes two administrators and three trustees. You have 14 days from the first verified backup. The product stays open during those days; if the deadline passes it locks again until both keys are escrowed.

Once both stages are done, setup is complete for good. Nothing that happens later locks the product again, including a backup that starts failing: that raises a warning instead, described on Failure Modes.

What Locked Means

The lock applies to the dashboard’s screens, for everyone who signs in and not only for administrators. These screens stay open, because they are what an administrator needs to finish: System → Backups & Recovery; Administration → Users, Groups and Roles; Security Settings; System → System Health; System → Updates; System → License & Entitlements; My Profile. Opening any other screen does one of two things:

• An administrator who can complete the waiting stage is taken to Backups & Recovery, where the checklist shows what is left.
• Everyone else sees Zonitix is not ready to use yet, with one of two lines: “Your administrator needs to finish setting up backups for your organisation.” or “Your administrator needs to finish securing the recovery keys for your organisation’s backups.” It adds that nothing is wrong with their account and that the page will open as soon as that is done.

The lock only ever follows a clear answer that setup is not done: the dashboard never locks on an answer it could not read.

An Instance That Was Already Live

An instance whose first-run setup was already finished when this requirement reached it is not locked on the spot. It gets 30 days, counted from the update that brought the requirement, with a banner on every screen for the people who look after the system. If storage is still not connected and verified when the 30 days end, it locks like a new instance. Its key-escrow deadline is the later of two dates: 14 days after the first verified backup, or the end of the 30 days.

The people who can connect storage and run a backup are also reminded by email and in the notification bell: when the 30 days start, with 7 days left, the day before, and when the lock begins. The key-escrow deadline has the same pattern on every instance: when the first backup is verified, with 3 days left, the day before, and when the lock begins. Each reminder is sent once.

The Banner

People who can open the System screens see a banner across the top of the app until setup is complete. It cannot be dismissed. In its last 24 hours a deadline is shown with the time of day and the hours left. Someone who can see the banner but cannot complete the stage is told to ask an administrator.

Recovery Points: How Far Back, How Recent

In plain words: a restore puts the database back to the moment a backup was taken, so anything entered after that backup is lost. With the built-in schedule the newest scheduled backup is at most about a day old. Your administrators run restores themselves, and it always takes two of them. Your instance also keeps the finer-grained record that a recovery to an exact moment would need, but nobody can use it through the product: there is no button and no API call for it, and the only thing that reads it is an automatic weekly test.

The standard deployment starts one PostgreSQL database on your server, and every service connects to that one database. The recovery points on this page are therefore backups of that database, taken on the schedule in What Is Backed Up, And When, above.

Which restore returns you to which moment, who can run it, and the schedule each backup follows are in the technical part below.

How Updates Are Installed

We host and monitor your server. Zonitix releases are installed automatically in a weekly maintenance window when that is switched on and email is set up, which is the choice pre-selected in the first-run setup wizard; an administrator can apply them by hand instead. Operating-system security patches follow the same window or are applied by hand. Backups and restores are your administrator’s to set up and run.

Both ways are on System → Updates. By hand, an administrator presses the button for a release or for server patches when it suits them; the steps are in Apply updates. Automatically, the instance does the same work by itself once a week, at a day and time you choose, called the maintenance window.

A new instance starts with automatic updates on: the first-run setup wizard asks, with the automatic choice already selected and a window of Saturday at 02:00. An instance that was already live before automatic release installs existed is left as it was: releases are applied by hand until an administrator switches the schedule on. In both cases nothing is installed by itself until outgoing email is set up, because you could not be warned first.

What Happens In A Maintenance Window

1. The Zonitix release goes first. The instance checks that it is healthy, backs up the database, installs the release, and then checks itself from the outside the way a browser would. If that check fails it goes back to the version it was running. The database is not rolled back.
2. Then the server’s own packages. The database is backed up before these too, and nothing is installed if that backup fails. Server packages cannot be un-installed automatically, so if the instance is not healthy afterwards the run is reported as failed.
3. If the release did not go in cleanly, the server packages are skipped for that window and run in the next one.

Expect a few minutes of interruption while services restart. The people who look after updates are emailed the day before, an hour before, when the window starts and when it ends; the last email names the release and says whether it was applied, rolled back or left waiting. Everyone signed in sees a notice across the app during the hour before a window and while it runs. A window that could not start on time, because the server was off or another update or backup was still running, is tried again each minute for 45 minutes; if it still cannot start it is reported as missed and is not run at a later hour.

What You Choose

The window itself is a day of the week, a start time, a timezone and a maximum duration from 30 to 240 minutes. Turn off automatic updates on the same screen stops all of it; releases and patches are then applied by hand.

Releases That Are Never Installed By Themselves

• A release that changes the database in a way an automatic rollback could not undo, or that carries no clearance for automatic install. When an update would bring in several releases at once, every one of them has to be cleared.
• With Security releases only: a security release that belongs to a newer release than the one your instance runs, because installing it would also install that release’s features. A security release of the release you run is installed.
• A release that an earlier window had to roll back. It is not tried again by itself; the next newer release is.
• Anything on the pre-release channel that is not a published release.

Such a release is not lost. The email at the end of the window says it is waiting, and an administrator applies it from System → Updates when ready. A release is also left alone when the instance is not healthy before the window’s update begins: nothing is changed, and the email says so.

If something here is not working as you expect, start with When something goes wrong.

The Backup Setup Gate

The decision is one pure function over two stored rows and the current time. The origin row (backup_setup_origin) is written once, by the database migration that runs on every deploy: { mode: "fresh", seededAt } when first-run setup had not been completed at that moment, otherwise { mode: "existing", seededAt, graceEndsAt } with graceEndsAt 30 days later. It is never rewritten, so a new instance that is redeployed before finishing setup is not handed a grace period. The state row (backup_setup_state) is owned by the compliance service and holds provenAt, provenBy, provenFile, lastVerify, escrowDeadlineAt, completedAt, completedBy and the ledger of reminders already sent.

The phase is complete when completedAt is set, escrow when only provenAt is set, and storage otherwise. completedAt is a latch: no later event produces locked: true.

The escrow deadline is computed once, at the moment the first backup is verified, and stored: the later of that moment plus 14 days and, on an existing instance, graceEndsAt. So an instance that was already live never has a key-escrow deadline that falls before its 30 days end.

What Counts As A Verified Backup

An upload is not proof. The service takes the newest database dump the backup ledger records as uploaded to the destination configured now (a dump sent to a previous destination does not count), streams it back from your storage, decrypts it with the backup key this instance holds, and decompresses it. It passes only when the object was encrypted, the decryption authenticates, at least 1,024 bytes come out, and the content starts with the PostgreSQL dump header and ends with its end-of-dump marker. A failure is stored in lastVerify with the reason, and the background check does not retry the same object for 15 minutes (it records the reason verify_failed_recently for that run); Check again always retries.

Setup completes when both backup_encryption_key and encryption_key have escrow shares issued for the key in use (escrow status active or degraded; a key rotated since its shares were issued does not count). The state is advanced by three things: a watchdog that runs 90 seconds after the service starts and then every hour, the Check again call below, and every key-escrow issuance that succeeds, so finishing the ceremony after the deadline unlocks the instance without waiting for the hour.

The Two Calls

gate is { phase, mode, completed, proven, graceEndsAt, escrowDeadlineAt, locked, inGrace, reason }, with the two dates as ISO strings or null. steps is the checklist: destination (done, bucket, provider), encryption, firstBackup (done, provenAt, file, uploaded, lastVerify), keys (done and one item per key) and agentInstalled. policy is { graceDays: 30, escrowDays: 14 }. The checklist names your bucket, which is why a caller without the System screens gets the gate alone.

After a verify call, result.reason says why nothing advanced: no_destination, no_backup_key, no_uploaded_backup, verify_failed (with the reason in error), keys_not_escrowed or complete. All of them answer 200.

Where the lock is applied. The server computes the decision and the dashboard applies it to its screens. The dashboard reads the gate at sign-in, again when second-factor verification completes, then once a minute, and every 15 seconds while locked. It redirects or shows the waiting screen only on locked: true, and never on an answer it could not read. After a failed read it also drops the backup health verdict, so no warning is raised from an answer it could not refresh.

What Is Audited

• BACKUP_SETUP_VERIFY_FAILED: a read-back failed, with the file name and the reason.
• BACKUP_SETUP_PROVEN: the first backup was verified, with the file name, its size after decryption and the escrow deadline.
• BACKUP_SETUP_COMPLETED: a verified backup exists and both keys are escrowed.
• BACKUP_SETUP_REMINDER: one entry per reminder sent. Each reminder also goes to the notification bell and by email to the holders of approve_system_updates; the bell is written first, so it arrives when no mail server is configured.

What You Can Return To, And Who Can Do It

The detail behind Recovery Points: How Far Back, How Recent, in the plain part above.

Zonitix does not perform restores on your behalf, and a restore never runs on one person’s say-so: the approval is refused with 403 when the approver is the requester. Before a database restore the host agent takes a safety dump of the current state and stops if that dump fails, so the state you restored over is itself a restore point. The steps are in Restore from a backup.

The Schedule And Retention Defaults In Code

“Server time” is the clock of the server host: these timers are written without a time zone, so they follow whatever zone the host is set to. See Clocks and scheduled jobs.

Two consequences follow from these defaults. Until an off-box destination is set, every copy above is on the same server as the database, so losing the server loses the backups with it. And because the product restores from dumps, the change log shortens the gap only for a recovery that replays it, which the product does not offer as an action today. On the Backups & Recovery screen a base backup carries the label PITR only and has no Restore button. Zonitix keeps what a point-in-time recovery would be built from: a weekly base backup and the record of changes written after it. But no screen and no API call restores your live database to a chosen moment, so a row marked PITR only is not something you can restore from the app. No commitment that Zonitix will carry one out for you is published, so plan your recovery around the dumps: what you can restore is a nightly backup whose Restore button works, and anything entered after it must be entered again. Whether your backups are being produced, copied and test-restored is shown on System → Backups & Recovery; the restore drill history is the evidence that a copy in your bucket can actually be restored.

Restore & Delete: The Two-Person Loop

Restore and backup-deletion each split into a request and an approve call. Both calls require approve_system_updates — so both actors are administrators — and the compliance service enforces requester ≠ approver at approval time. There is a single shared “pending approval” slot, so a request parks until it is approved, rejected, or cancelled; a second request while one is pending returns 409.

Requesting A Restore

The only field is target, and it must be a logical dump filename matching zonitix_*.sql.gz with no path component — validated here and re-validated in the host agent. Physical base tarballs (*.tar.gz) are refused here: no screen and no API call restores your live database to a chosen moment. The request is also refused with a 400 if the dump’s PostgreSQL major version differs from the live server’s — a cross-major restore is a supervised migration, not a click (the guard fails open only if either version is unknown).

The status codes to expect: 202 on a parked request, 403 when approver equals requester (restore or delete), 404 approving when nothing is pending, and 409 when a request is already pending or an operation is already in flight. The approve call carries no body — it acts on the single pending slot. Neither call needs a step-up, so a personal API token whose owner holds the permission can make them; from a browser session cookie instead, the write must also carry an Origin (or Referer) header for your instance’s own address. Every request/approve/reject/cancel raises an audit event (DB_RESTORE_REQUESTED, DB_RESTORE_APPROVED, DB_BACKUP_DELETE_REQUESTED, DB_BACKUP_DELETE_APPROVED, …).

Restoring The File Store

A database backup does not contain files. Signed documents and the instance signing key, uploaded compliance evidence and candidate résumés are kept in the nightly file-store archive instead. System → Backups & Recovery lists those archives in the File backups panel (columns WHEN, ARCHIVE, SIZE, STORED), each with a Restore button. An administrator uses it when files were lost or damaged, or straight after a database restore, picking the archive closest to, and not older than, the database backup that was restored.

It is the same two-person loop as a database restore, on the same single pending slot with the same 30-minute expiry. The request body is { "target": "<archive filename>" }; the name must match zonitix_files_*.tar.gz with no path component, otherwise the call returns 400. The approve call has no body. It returns 404 when nothing is pending, 409 when the pending request is not a file restore, and 403 when the approver is the requester. Reject and cancel use update-reject and update-cancel, as for every other request.

An archive is restorable when a copy is on the server, or when it is recorded as uploaded and an off-box destination is enabled; in the second case fetch is true and the host agent downloads it from your storage first. foundInStorage marks an archive that is in your bucket but in neither the server’s backup directory nor its backup records (for example after the server was rolled back to an earlier state); it is still restorable. The remote listing is cached for five minutes.

What The Host Agent Does On Approval

1. Re-validates the target name, and fetches the archive from your storage if it is not on the server (phase fetching). A failed fetch ends the run with nothing changed.
2. Checks the archive before anything is touched (phase verifying). An archive that fails the check is refused.
3. Stops the services that use the file stores. Signing, recruiting and compliance pause for about a minute.
4. Takes a safety copy of the current files (phase backing-up). If that copy cannot be written, the restore is aborted and nothing is changed. The two newest safety copies are kept on the server.
5. Extracts the archive over the file stores (phase restoring). Files added since the archive was taken are kept; a file that exists in both places is replaced by the archived copy.
6. Starts the services again and checks that the instance is healthy. If the extraction itself fails midway, the safety copy is put back.

The agent’s own steps land in the audit trail next to the request and approval: FILES_ARCHIVE_FETCHED, FILES_SAFETY_ARCHIVED, FILES_RESTORE_COMPLETED, and on a failure FILES_RESTORE_REFUSED, FILES_RESTORE_ABORTED, FILES_RESTORE_FAILED or FILES_RESTORE_UNHEALTHY.

Off-Box Destination & Connection Test

The destination is configured in the product, not on the host. It supports S3 / S3-compatible (AWS, R2, B2, Wasabi, GCS, MinIO) and Azure Blob Storage. Reading the config is view_system and comes back masked — you get hasSecret: true|false, never the secret itself. Saving or testing needs approve_system_updates.

The test is deliberately non-destructive to your saved config: it builds a transient client from the values you POST and runs the exact operations the backup pipeline relies on — reach the bucket, authenticate, write a probe object, read it back, delete it — returning each of the five checks separately so a failure names the missing permission. Nothing is written to the database.

Notes: leaving secretAccessKey blank on an edit keeps the stored one (it is not wiped), but switching provider discards it — an S3 secret is not an Azure account key. Azure has no reusable-provider hook, so credentialSource there is forced to manual. When enabled: true, a missing bucket or missing manual credentials is a 400.

Restore-Drill Schedule (INF-02 Evidence)

An off-box copy that has never been restored from is an assumption, not a backup. The scheduled restore drill pulls the newest dump back down and restores it into a throwaway database, and its pass/fail history is the evidence behind the INF-02 control. You set the cadence in the product; the write needs approve_system_updates, and the current schedule + run history read back through GET /reports/backup-destination under restoreDrill.

What A Drill Measures, Stores And Shows

Every drill that runs writes one row, and the row holds the measurement. The host script times the whole drill from start to finish (fetching the newest dump from your destination, restoring it into the throwaway database and checking the newest file archive) in whole seconds, and sends it as durationMs with the name and size of the dump it pulled and the number of tables it restored.

On screen, the Off-box backup destination dialog lists the five newest runs under Automatically test that these backups restore: the result, the time, the duration in seconds and the number of tables. The stored row also keeps the start and finish times and the script’s exit code, in the restore_drill_runs table. The duration is the drill’s, not a live restore’s: a live restore first takes a safety dump of the current database, and the drill does not.

Read-Only: Restore Points, Governance Trail, WAL/PITR

Three read endpoints feed the restore-point list and the audit panel on System → Backups & Recovery. They are read-only: the change-log listing shows what is being kept and does not offer a restore to a chosen moment. The one write here re-ships a local backup whose bucket copy has gone missing. The three reads require view_system — the same permission that shows System Health, Updates and Backups & Recovery in the sidebar; the re-upload requires approve_system_updates. Each is gated on its permission alone, never on a license flag, so a plan change cannot take down the ops surface. A caller holding only view_reporting gets a 403.

Each restore point carries restorable (true for a logical dump that has a local file, or one that can be fetched), fetch (true when there is no local file and the host agent will download the dump from your storage before restoring it), versionCompatible (PostgreSQL major matches the live server), offboxMissing (the run claims an off-box copy but the object is gone from the bucket), previousDestination (it is missing only because the destination was changed after it was shipped) and foundInStorage (a dump that is in your bucket but in neither the server’s backup directory nor its backup records, for example after the server was rolled back to an earlier state; shown with a found in storage chip and still restorable). A dump with no local file is fetchable only while an off-box destination is enabled and the object is still in the bucket. On the screen, a row that cannot be restored reads off-box only (no destination is enabled to fetch it from), not available (the object is gone) or PITR only (a base backup). WAL segments are kept out of the restore-point list on purpose — there are thousands of them, because a segment is closed at least every 300 seconds while the database is changing; they are the change log between base backups, not things you restore to, so they get their own summary and drill-down.

The Patch / Update Control Loop

An update applied by hand shares the same request/approve machinery. The reads describe what is deployed and how far behind it is; the writes queue and approve a routine update through the same single pending slot the restore/delete flow uses. The automatic schedule hands its steps to the host agent through that same slot, already approved; its settings, its two calls and what a window does are under The Automatic Update Schedule below.

Reads

patch-status prefers the host agent’s locally git-computed figure (no GitHub rate limit); it falls back to the GitHub compare API and will report an error string if that is rate-limited or the deployed commit is unknown. host-patches returns available: false when the agent is not installed. system-health degrades to host-only metrics when the Docker socket is not mounted — it never issues a Docker control call.

host-security is what the Host & SSH security card on Threat Defense shows: whether SSH password authentication is disabled, the root-login setting, whether fail2ban is active for SSH and how many addresses it has banned, failed SSH logins in the last 24 hours, the maximum authentication tries, and when the host last published these figures. The card’s chip reads Live, Stale, Awaiting host or Unavailable; a value the host has not measured is shown as Unknown rather than guessed. It is read-only.

The release channel is set on the Release channel card at the bottom of System → Updates: Stable (recommended) installs only published Zonitix releases, including security releases, while Main (pre-release) makes the instance apply the newest unreleased code and is for test environments only. Anyone holding view_system sees the card; changing it needs approve_system_updates and a fresh re-authentication, choosing Main asks for a confirmation (Switch to Main), and the updater picks the change up at its next check. Stay on Stable.

The PUT body is { "channel": "stable" | "main" }. The reply repeats the channel state with changed and previous. A real change is audited as UPDATE_CHANNEL_CHANGED (at warning level when the new channel is main); a refused one as UPDATE_CHANNEL_CHANGE_REFUSED.

maintenance-status feeds the notice every signed-in user sees across the top of the app. It reports running while the host agent is applying an update or a restore (a plain backup does not count) and for the whole of an open scheduled window, including the pause between its steps, and scheduled during the 60 minutes before a saved automatic window on an instance with email configured. It deliberately carries no release tag, host detail or requester name. The app polls it once a minute.

The Request → Approve Loop

This is exactly where the permission split shows itself. A holder of request_system_updates who is not an administrator files a request that is parked (status: pending_approval) and does not reach the agent. A user on the administrator role — which holds every permission, approve_system_updates included — is the approver, so their own request auto-approves and dispatches (status: dispatched). This applies to a signed-in session only. A personal API token never carries a role, so a request made with an administrator’s token is parked like any other and needs an update-approve call; a token that carries approve_system_updates may make it, the same token included, because updates are not two-person. A parked request expires after 30 minutes. Cancel is scoped to the original requester: a different admin must Reject instead, and the service returns 403 if the caller is not the requester.

Status codes to assert: 202 on a queued request, 400 for a mode outside the five valid values, 409 when an update is already queued or awaiting approval, 404 approving/cancelling with nothing pending, and 503 when the control channel is not writable (the host updater agent is not installed); in production that reply carries the generic message Internal Server Error and a correlationId, not the reason. Every transition lands in the audit trail: UPDATE_REQUESTED, UPDATE_DISPATCHED, UPDATE_APPROVED, UPDATE_REJECTED, UPDATE_CANCELLED.

What The Host Agent Does With A Request

Every application or OS update, requested by hand or started by the automatic schedule, starts with a database backup; if that backup fails, the run stops with nothing changed. An application update is then fetched, built and restarted, and the instance is checked for health. If the check fails, the agent rolls the code back to the version that was running; the database is not rolled back. OS packages have no rollback: an unhealthy instance after an OS update ends as failed. A mode: os-full request, made from the app or by the schedule, permits a restart: when an installed package requires one, the server restarts and the agent writes the final verdict after it comes back. An os-security request never restarts the server; a restart that is still required is reported in the status message.

update-status exposes the agent’s status.phase. Working phases: backing-up, pulling, building, verifying, rolling-back, rebooting, restoring, fetching. Terminal phases: done, rolled-back, failed. The reply also carries pendingTtlMs, the lifetime of a parked request (30 minutes).

The Automatic Update Schedule

day is one of sun to sat, time a 24-hour HH:MM, and timezone an IANA zone name the server itself recognises. All three are validated before anything is written, so a refused save stores nothing. maxMinutes is held between 30 and 240 (default 60). appFull means every release and implies appSecurity; with both off, releases are applied by hand. Only switching enabled from off to on is refused without email: an instance that is already on may still move its window, and the scheduler starts nothing until email works. With no timezone saved, the window follows the instance’s regional timezone and timezoneSaved is false. Each save is audited as AUTO_UPDATE_CONFIGURED with the resulting schedule.

Defaults. A new instance is seeded with enabled, appSecurity, appFull and osSecurity on, osFull off, and Saturday 02:00. The choice made on the wizard’s Updates step is applied over that the first time the schedule is read, audited as AUTO_UPDATE_CONFIGURED by setup-wizard: the automatic choice saves the chosen day, time and timezone, and the manual choice sets enabled, appSecurity and appFull to off. A setting that was never written reads as off for enabled and the two release switches, which is why an instance that was already live keeps applying releases by hand after it is updated.

What The Scheduler Does

A scheduler in the compliance service wakes once a minute and keeps the current window’s progress in the database, so a restart in the middle of a release (which restarts that service too) neither loses nor repeats it. It does nothing while enabled is off or smtpConfigured is false. Otherwise, for each weekly window:

• Reminders. One email in the 24 hours before the window and one in the last hour, to the holders of request_system_updates. A window set for less than an hour ahead gets only the second.
• Start. It hands the first step to the host agent as an already-approved request by auto-scheduler and emails that maintenance has started. The steps, in order: the release (full or security), then the server packages (os-full or os-security). Only os-full carries permission to restart the server. With no update type selected, nothing is sent.
• Busy channel. If another update, backup or restore holds the request slot, it retries each minute. A window still not started 45 minutes after it was due is reported as missed and dropped.
• Between steps. When the release step ends as rolled back, failed, aborted, expired, timed out or stopped by an unhealthy instance, the server-package step is recorded as skipped. If automatic updates are switched off during a window, the step in progress is seen through to its report and no further step starts.
• Finish. One email with a line per step, and one audit entry, AUTO_UPDATE_WINDOW_FINISHED, carrying the window, the release and previous release, the release outcome and every step’s result. A window that ended in a failure, a timeout or a rollback, or that could not check for releases, is logged at error level.
• Backstop. A run with no result after the maximum duration plus 30 minutes (never more than three hours) is closed as timed-out. If the host agent never picked the request up, the request is withdrawn so it cannot run late, audited as UPDATE_REQUEST_WITHDRAWN.

On a server where an operations alert address was set at provisioning, a window that fails, times out, rolls back or cannot be checked also sends that address an email naming the instance, the window and each step’s outcome. The Email reminders card on the schedule screen says the same: failures also alert Zonitix operations.

What The Host Agent Checks Before An Unattended Release

The scheduler only asks. The host agent decides, on the server, before the backup and before anything changes. In order: with security releases only, a candidate that is not a security release is dropped, and one from a newer release line is replaced by the newest security release of the deployed line or left waiting; the target must be a published release; it must not be the release an earlier unattended run rolled back; every release between the deployed version and the target must be marked as cleared for automatic install in its release tag (a single one marked otherwise, or not marked, holds the whole update); an end-to-end check address must be known; and that check must pass now. The check fetches the instance’s own web address and the gateway’s readiness aggregate, /health/cluster, through the public route, so containers that merely report healthy are not enough. Each refusal is audited by the agent as APP_PATCH_AUTO_SKIPPED with the reason, and an already-current instance as APP_PATCH_UP_TO_DATE.

After the gates the agent runs the same sequence as a release applied by hand: database backup (a failed backup ends the step as aborted, audited APP_PATCH_ABORTED), install, end-to-end check, and on failure a redeploy of the version that was deployed before, followed by the same check. The rollback point is the version last deployed successfully, not whatever is checked out on disk.

These are the values of releaseOutcome in the AUTO_UPDATE_WINDOW_FINISHED entry. The finished email’s subject follows them: completed; completed with a release waiting for you; could not check for new releases; rolled back; status unknown; failed.

How A Release Is Checked Before It Is Built

This is what the host agent and the deploy script check between fetching a release and starting it, on the Stable channel. Each check that fails stops the run before anything is built or restarted.

Fetching. The host agent fetches the list of release tags from the Zonitix release repository with git fetch --tags. Where the server reaches the repository with its own SSH key, that key is created on the server, and the connection accepts only the repository host’s host key that the server has on file. If the fetch fails and no newer release is already on the server, the agent records that it could not check for releases instead of reporting the server as up to date.

Building. By default the server builds the release’s container images itself, from the source at the tag. A server set to pull pre-built images instead checks every image before it uses any of them: the image is resolved to its digest, its recorded source revision must equal the commit being deployed, and its cosign signature must verify against the release repository’s image-build workflow for that same commit. If one image fails, none is used, the deploy is refused and the server does not fall back to building.

What happens after the build (the health check, and the return to the previous version when it fails) is under What The Host Agent Does With A Request, above.

Where The Backup Key Lives

Key Escrow Endpoints

The Off-Box Envelope

An encrypted off-box object is AES-256-GCM in a small envelope: the 4-byte marker ZBK2, a 4-byte key id, a 12-byte IV, the ciphertext, then the 16-byte authentication tag (36 bytes of overhead). The AES key is the SHA-256 of the backup encryption key, and the key id is the first 4 bytes of the SHA-256 of that AES key, so a reader knows which key sealed an object before trying to open it. The backup code can hold a ring of the current key plus earlier ones, so an object written under an earlier key is read when that key is supplied, and the offline decryptor accepts earlier keys; which keys a running instance is given today is under Keys And Rotation on Data Isolation. Objects in the older ZBK1 envelope, which has no key id, are still read too.