API and Worker write structured JSON logs to standard output and can export traces, metrics and logs over OTLP/HTTP. Browser tracing is optional and passes through the API so collector credentials are not exposed to the browser. There is no local Prometheus /metrics endpoint; use your collector to integrate a Prometheus-compatible backend.
Configure destinations
Open Admin → Observability with the appropriate observability permission or admin:manage. Changes and connection tests require recent MFA.
Use Add Endpoint for up to ten named destinations. Enabled destinations receive copies simultaneously; their order does not define priority or failover. The master telemetry switch stops all exports without discarding endpoint settings. At least one configuration row remains, but all can be disabled.
| Field | Purpose |
|---|---|
| Telemetry export | Master enable switch. |
| Endpoint name/enabled | Identify and activate an independent destination. |
| OTLP base endpoint | Collector base URL; signal paths default to /v1/traces, /v1/metrics, /v1/logs. |
| Signal overrides | Optional complete URL for each signal. |
| Secret header name/value | Optional collector authentication; the value is encrypted and write-only. |
| Deployment environment | Resource/span label for this installation. |
| Sampling ratio | Trace sampling fraction from 0 to 1. |
| Metric export interval | Milliseconds, minimum 1,000. |
| Browser tracing | Enables browser traces on the next page load; no browser metrics/log export. |
Endpoint cards and advanced fields can be collapsed without losing edits. Save Observability saves the whole collection atomically. Correct identified errors and save again; a rejected endpoint leaves the prior collection intact. Removing an endpoint takes effect on save.
After saving, use that endpoint’s Test Connection. It tests the saved traces destination and credential with an empty OTLP HTTP POST, including for a disabled endpoint. It does not prove metrics/log ingestion or downstream storage.
A stored authentication secret is bound to the header name and recipient origins (scheme/host/port). Changing them requires a replacement secret in the same save, even for disabled endpoints. Path-only changes on the same origin can retain the secret. Clear the header name to remove authentication explicitly.
Apply and verify
The handling API replica applies saved settings immediately; other API/Worker replicas normally reload within 30 seconds. OBSERVABILITY_RELOAD_INTERVAL_MS=0 disables periodic reload. Reload the browser for browser tracing changes. Invalid/unsafe destinations leave the last working exporter in place.
Deployment export policy remains authoritative. Customer-site installations default to customer_collector, disabled export and no external-export approval. Administrators can enable an approved customer collector. custom_otlp_endpoint requires approved external export; disabled policy prevents export regardless of endpoint switches. The administration API does not override this governance policy.
Production connection-test/browser-relay endpoints require safe HTTPS destinations and reject restricted address ranges/redirects. Verify the intended collector is supported by these controls before rollout. A private CA extends certificate trust but does not remove address restrictions.
Look for these service names in your collector:
message-operations-apimessage-operations-workermessage-operations-web(traces only)
Perform an API action and an approved test job; confirm traces, fresh metrics and sanitized logs at the intended destination. A successful connection test alone is insufficient. When upgrading an older installation, update filters that still select solace-operations-*; historical telemetry retains its old identity.
Health and daily checks
API /health/live returns 200 while the process responds. /health/ready returns 200 only when its read-only database/ledger/queue checks pass, otherwise 503. Worker uses the same paths on its internal health port (default 9090) and additionally requires the consumer to be registered. A later database outage changes readiness without requiring restart. Liveness remains independent of database availability.
Probe Web on / at port 8080. Public Web /health/ready proxies API readiness; it does not expose Worker readiness. Check exact new revisions rather than a healthy older revision.
Review request/error rate and latency, authentication denials, job queue wait/failures, broker failures, database storage, audit-write errors and telemetry freshness. Health does not verify broker authorization or completed business processing.
Useful signals
| Signal | What to investigate |
|---|---|
messageops.privileged_broker_actions | Operation/outcome trends by environment. |
messageops.operation_messages | Message-operation volume. |
messageops.idempotency_replays, messageops.idempotency_conflicts | Client retries and conflicting request reuse. |
messageops.worker.jobs.started, messageops.worker.jobs.processed, messageops.worker.jobs.failed, messageops.worker.job.duration, messageops.worker.job.retries | Worker execution health. |
messageops.worker.queue.wait_time | Backlog/consumer delays. |
messageops.worker.job.items.processed, messageops.worker.messages.processed, messageops.worker.messages.failed, messageops.worker.message.processing.duration | Per-item throughput and failures. |
messageops.worker.search.messages.scanned, messageops.worker.search.messages.matched | Bounded search workload and yield. |
messageops.semp.request.count | Physical SEMP requests, including failures/retries. |
messageops.semp.cache.hit.count, messageops.semp.cache.miss.count, messageops.semp.cache.refresh.count | Metadata cache effectiveness. |
Use low-cardinality environment labels; keep message/user/correlation IDs in logs or traces rather than metric labels.
Trace context links API requests, job.enqueue and Worker job.process; broker management uses semp.request. Correlate with job IDs and audit records. Dedicated spans for every admin or synchronous domain action are not provided in this release.
Diagnose telemetry and SEMP failures
Logs redact credentials, tokens, cookies and sensitive fields. Do not add raw request/message payloads to operational log collection. Additional ingress/access logs must omit sensitive query strings and referrers; application redaction cannot recall copies in upstream systems.
For missing metrics, confirm the saved metrics override (or base /v1/metrics), header, export policy and receiver pipeline. Generate activity and wait for the configured interval. API/Worker output distinguishes exporter configured, exporter disabled, metrics export: success, no_measurements, export_failed, timeout and transport/collection failures. Repeated identical outcomes are suppressed. Collector outages retry at the normal cadence and do not terminate the application; a successful export confirms transport acceptance, not dashboard retention.
SEMP failure logs include environment, API kind, sanitized path, attempt, error code and upstream status where available. Trace/span IDs connect them to tracing. Upstream diagnostic descriptions are bounded and sanitized; treat them as untrusted text. Safe GET retries apply to server/network/timeout failures, not 4xx errors. Check credentials and broker permissions separately from networking. SEMP Traffic explains cached/stale metadata and usage reporting.