Message Ops

Use public HTTPS for browser access, private API ingress and no Worker ingress. Keep database certificate verification enabled and use separate migration/runtime identities. Production API/Worker containers run non-root without a shell; collect diagnostics through platform logs, health and telemetry rather than installing tools in them.

Trust and data handling

PostgreSQL trust uses DATABASE_SSL_CA_BASE64 where a private CA is required. Supply it consistently to API, Worker, migration and bootstrap clients. Node outbound HTTPS can extend trust with a mounted PEM and NODE_EXTRA_CA_CERTS on API/Worker. Messaging trust is configured per environment with Verify broker TLS certificate and its broker CA field. Do not disable verification as a connectivity workaround.

HTTPS SEMP is recommended. Explicit HTTP SEMP is supported with an application warning and exposes its traffic/credentials in transit; use it only under your approved network policy. It does not weaken other TLS controls. Verify network routes from the actual server execution path, not only your browser.

Messages remain on the broker. PostgreSQL contains configuration, audit, jobs, credentials and temporary references. Reference previews are pattern-masked before storage; headers/properties/metadata may be stored raw for message replay and identity matching, then masked on read for callers without content permission. Protect the database and backups accordingly. Masking is not a complete data-loss prevention control.

Generate and store independent keys

The required keys and consumer mappings are in Configuration Reference. On a trusted operator host, set VAULT and SECRET_NAME for one intended secret:

umask 077
openssl rand -base64 32 > key.txt
az keyvault secret set --vault-name "$VAULT" --name "$SECRET_NAME" --file key.txt --output none
rm -f key.txt

Use an approved protected temporary workspace; deletion is not guaranteed secure erasure. Generate independent keys and retain them in the approved vault/backup system. Losing encryption keys can make stored credentials unrecoverable.

Rotate encryption keys

SSO, MFA and broker encryption support ordered key lists: the first key encrypts and listed keys can decrypt. API and Worker must share the same SSO/broker lists.

1. Back up required keys and identify stored values/backups that need them.
2. Distribute OLD,NEW to every relevant reader and restart processes so they can read either key before any writer changes.
3. Promote to NEW,OLD on all consumers and restart. New writes use the new primary.
4. Rewrite affected values through their supported application workflows. Reading alone does not guarantee re-encryption; there is no universal background re-encryption job.
5. Retire old keys only after every affected value and required backup has a tested recovery path.

Compliance signing is different: only the first key signs and verifies, with no old-key fallback. Retain old signing keys and evidence provenance securely if old evidence must remain verifiable. Plan this separately from encryption rotation.

Rotate database credentials

The privileged migration credential and restricted runtime credential have separate procedures.

For a Terraform-managed migration password, increment postgres_administrator_password_version by one and review/apply the plan. It must update in place, not replace the database/server. If the server update and vault write diverge, increment again and reapply; reapplying the same version does not resend an ephemeral password. Do not change this credential out of band, because write-only password drift is not detected by Terraform. Start and verify a new named migration execution afterward.

For a manually provisioned migration password, coordinate the database-owner password and its vault secret using your approved private administration path, then verify a new migration execution.

For the runtime message_ops_app password:

1. Enter a maintenance window, stop new work and allow running jobs to settle.
2. Use the privileged PostgreSQL path and interactive \password message_ops_app, with history suppressed.
3. Update the runtime vault secret from a protected, percent-encoded URL file.
4. Restart API and Worker active revisions so they read the new value, then verify readiness and a real queued test operation.
5. Restore traffic only after database and secret values are confirmed to agree.

Existing connections may survive a password change; reconnects fail until cutover. This in-place procedure does not promise zero downtime. Do not improvise a sibling role: future grants and audit-purge access target message_ops_app. Restoring an old image does not restore a previous password.

Keep all secret references versionless. Application/job processes read environment secrets at startup, not continuously. For a controlled rotation, explicitly restart/redeploy and verify instead of assuming secret refresh has completed. Running job executions keep their original environment; start a new execution after a job-secret rotation. Preserve previous values until cutover is verified where overlap is supported.

Broker-account rotation is an application operation: update credentials under System → Environments, test connectivity and a permitted test job, then retire the old broker credential. Environment password edits do not require restarting API/Worker. Provider and telemetry credentials similarly have their own application workflows.

Maintenance containment

Use the shared delete/move controls on both API and Worker, or the environment read-only control, to contain specific mutations. They do not undo in-flight requests or stop every type of work. For a database restore, block new requests and stop Worker processing through your infrastructure maintenance procedure. Never use Terraform’s deployment-existence flags as a pause switch.