API and Worker write structured JSON logs to standard output and can export traces, metrics and logs over **OTLP/HTTP**. Browser tracing is optional and passes through the API so collector credentials are not exposed to the browser. There is no local Prometheus `/metrics` endpoint; use your collector to integrate a Prometheus-compatible backend.

## Configure destinations

Open **Admin → Observability** with the appropriate observability permission or `admin:manage`. Changes and connection tests require recent MFA.

Use **Add Endpoint** for up to ten named destinations. Enabled destinations receive copies simultaneously; their order does not define priority or failover. The master telemetry switch stops all exports without discarding endpoint settings. At least one configuration row remains, but all can be disabled.

<table><caption class="sr-only">Configure destinations</caption><thead><tr><th scope="col">Field</th><th scope="col">Purpose</th></tr></thead><tbody><tr><td>Telemetry export</td><td>Master enable switch.</td></tr><tr><td>Endpoint name/enabled</td><td>Identify and activate an independent destination.</td></tr><tr><td>OTLP base endpoint</td><td>Collector base URL; signal paths default to <code>/v1/traces</code>, <code>/v1/metrics</code>, <code>/v1/logs</code>.</td></tr><tr><td>Signal overrides</td><td>Optional complete URL for each signal.</td></tr><tr><td>Secret header name/value</td><td>Optional collector authentication; the value is encrypted and write-only.</td></tr><tr><td>Deployment environment</td><td>Resource/span label for this installation.</td></tr><tr><td>Sampling ratio</td><td>Trace sampling fraction from 0 to 1.</td></tr><tr><td>Metric export interval</td><td>Milliseconds, minimum 1,000.</td></tr><tr><td>Browser tracing</td><td>Enables browser traces on the next page load; no browser metrics/log export.</td></tr></tbody></table>

Endpoint cards and advanced fields can be collapsed without losing edits. **Save Observability** saves the whole collection atomically. Correct identified errors and save again; a rejected endpoint leaves the prior collection intact. Removing an endpoint takes effect on save.

After saving, use that endpoint’s **Test Connection**. It tests the saved traces destination and credential with an empty OTLP HTTP POST, including for a disabled endpoint. It does not prove metrics/log ingestion or downstream storage.

A stored authentication secret is bound to the header name and recipient origins (scheme/host/port). Changing them requires a replacement secret in the same save, even for disabled endpoints. Path-only changes on the same origin can retain the secret. Clear the header name to remove authentication explicitly.

## Apply and verify

The handling API replica applies saved settings immediately; other API/Worker replicas normally reload within 30 seconds. `OBSERVABILITY_RELOAD_INTERVAL_MS=0` disables periodic reload. Reload the browser for browser tracing changes. Invalid/unsafe destinations leave the last working exporter in place.

Deployment export policy remains authoritative. Customer-site installations default to `customer_collector`, disabled export and no external-export approval. Administrators can enable an approved customer collector. `custom_otlp_endpoint` requires approved external export; `disabled` policy prevents export regardless of endpoint switches. The administration API does not override this governance policy.

Production connection-test/browser-relay endpoints require safe HTTPS destinations and reject restricted address ranges/redirects. Verify the intended collector is supported by these controls before rollout. A private CA extends certificate trust but does not remove address restrictions.

Look for these service names in your collector:

-   `message-operations-api`
-   `message-operations-worker`
-   `message-operations-web` (traces only)

Perform an API action and an approved test job; confirm traces, fresh metrics and sanitized logs at the intended destination. A successful connection test alone is insufficient. When upgrading an older installation, update filters that still select `solace-operations-*`; historical telemetry retains its old identity.

## Health and daily checks

API `/health/live` returns 200 while the process responds. `/health/ready` returns 200 only when its read-only database/ledger/queue checks pass, otherwise 503. Worker uses the same paths on its internal health port (default 9090) and additionally requires the consumer to be registered. A later database outage changes readiness without requiring restart. Liveness remains independent of database availability.

Probe Web on `/` at port 8080. Public Web `/health/ready` proxies API readiness; it does not expose Worker readiness. Check exact new revisions rather than a healthy older revision.

Review request/error rate and latency, authentication denials, job queue wait/failures, broker failures, database storage, audit-write errors and telemetry freshness. Health does not verify broker authorization or completed business processing.

## Useful signals

<table><caption class="sr-only">Useful signals</caption><thead><tr><th scope="col">Signal</th><th scope="col">What to investigate</th></tr></thead><tbody><tr><td><code>messageops.privileged_broker_actions</code></td><td>Operation/outcome trends by environment.</td></tr><tr><td><code>messageops.operation_messages</code></td><td>Message-operation volume.</td></tr><tr><td><code>messageops.idempotency_replays</code>, <code>messageops.idempotency_conflicts</code></td><td>Client retries and conflicting request reuse.</td></tr><tr><td><code>messageops.worker.jobs.started</code>, <code>messageops.worker.jobs.processed</code>, <code>messageops.worker.jobs.failed</code>, <code>messageops.worker.job.duration</code>, <code>messageops.worker.job.retries</code></td><td>Worker execution health.</td></tr><tr><td><code>messageops.worker.queue.wait_time</code></td><td>Backlog/consumer delays.</td></tr><tr><td><code>messageops.worker.job.items.processed</code>, <code>messageops.worker.messages.processed</code>, <code>messageops.worker.messages.failed</code>, <code>messageops.worker.message.processing.duration</code></td><td>Per-item throughput and failures.</td></tr><tr><td><code>messageops.worker.search.messages.scanned</code>, <code>messageops.worker.search.messages.matched</code></td><td>Bounded search workload and yield.</td></tr><tr><td><code>messageops.semp.request.count</code></td><td>Physical SEMP requests, including failures/retries.</td></tr><tr><td><code>messageops.semp.cache.hit.count</code>, <code>messageops.semp.cache.miss.count</code>, <code>messageops.semp.cache.refresh.count</code></td><td>Metadata cache effectiveness.</td></tr></tbody></table>

Use low-cardinality environment labels; keep message/user/correlation IDs in logs or traces rather than metric labels.

Trace context links API requests, `job.enqueue` and Worker `job.process`; broker management uses `semp.request`. Correlate with job IDs and audit records. Dedicated spans for every admin or synchronous domain action are not provided in this release.

## Diagnose telemetry and SEMP failures

Logs redact credentials, tokens, cookies and sensitive fields. Do not add raw request/message payloads to operational log collection. Additional ingress/access logs must omit sensitive query strings and referrers; application redaction cannot recall copies in upstream systems.

For missing metrics, confirm the saved metrics override (or base `/v1/metrics`), header, export policy and receiver pipeline. Generate activity and wait for the configured interval. API/Worker output distinguishes `exporter configured`, `exporter disabled`, `metrics export: success`, `no_measurements`, `export_failed`, `timeout` and transport/collection failures. Repeated identical outcomes are suppressed. Collector outages retry at the normal cadence and do not terminate the application; a successful export confirms transport acceptance, not dashboard retention.

SEMP failure logs include environment, API kind, sanitized path, attempt, error code and upstream status where available. Trace/span IDs connect them to tracing. Upstream diagnostic descriptions are bounded and sanitized; treat them as untrusted text. Safe GET retries apply to server/network/timeout failures, not 4xx errors. Check credentials and broker permissions separately from networking. [SEMP Traffic](https://portal.cjcit.io/knowledge-base/message_ops/1.7/administration/semp-traffic/) explains cached/stale metadata and usage reporting.