# Operating LMC Exporter ## Overview `LMC Exporter` exposes LSEG/LMC shared-memory managed object data as Prometheus-compatible metrics. It can emit periodic stdout snapshots or serve metrics through an HTTP scrape endpoint, which defaults to `/metrics` when HTTP mode is enabled. This article explains how to operate the exporter after it is built and configured. It focuses on: - monitoring exporter and target health - configuring HTTP scraping - using scoped metric sets for operational views - tuning render and worker behavior - scaling from one target to multiple targets - troubleshooting scrape readiness and routing errors NoteScope This article covers operational use of `lmc_exporter`. It does not document custom metric mapping development, generator workflows, or contributor tasks. ## Prerequisites Before operating `lmc_exporter`, ensure that you have: - A Linux environment supported by the repository build targets: `OL7_64_GCC482`, `RHEL8_64_GCC831`, or `RHEL9_64_GCC1141`. - A built `src/lmc_exporter` executable. - An LMC install tree with the required headers and libraries. - A valid local license file, or a configured Gateway that can grant the complete exporter workload. Licensing completes before the exporter creates the LMC shared-memory pool. - One or more shared-memory `server_key` values for the LMC applications you want to scrape. - A Prometheus-compatible scraper, or direct access to the exporter HTTP endpoint for manual checks with `curl`. NoteGateway platform support Gateway licensing is qualified on Enterprise Linux 8 and 9 x86\_64. Oracle Linux 7 support applies to local-file licensing only. To confirm the binary version: ``` ./src/lmc_exporter --version ``` ## How It Works At runtime, `lmc_exporter`: 1. Loads built-in defaults, an optional TOML config file, and then any CLI overrides. 2. Builds one or more scrape target definitions. 3. Validates the local license, or acquires the full configured workload from Gateway. 4. Creates the LMC shared-memory server pool. 5. Adds configured server keys to the pool. 6. Creates a target registry. 7. Creates a `MetricBuilder` and `LMCExporterProbe` for each target. 8. Starts either: - the HTTP scrape server, or - the periodic stdout metrics printer. 9. Discovers visible roots for each configured `server_key`. 10. Selects the best matching root, optionally using an `instance` hint. 11. Recursively binds child objects and variables under the selected root. 12. Renders Prometheus text once the tracked subtree is ready. Before the first full sync completes, HTTP scrapes return `503 Service Unavailable` with a readiness message. After initial sync, newly added child objects or variables do not take the target back to startup readiness. The exporter continues serving the last rendered snapshot and adds the new metrics after those paths sync. ## Configuration ### Core operational settings The following TOML keys are directly relevant to operations.

Core operational settings
Key	Purpose	Operational guidance
`metrics_http_enabled`	Enables the HTTP metrics listener.	Set to `true` for Prometheus scraping.
`metrics_bind_address`	IPv4 address for the listener.	Use `127.0.0.1` for local-only access or `0.0.0.0` for network scrapes.
`metrics_port`	HTTP listener port.	Default runtime port is `9464`.
`metrics_path`	Base scrape path.	Usually `/metrics`.
`metrics_interval_seconds`	Stdout snapshot interval.	Applies when HTTP mode is not used.
`render_min_interval_ms`	Minimum spacing between demand renders for one target.	Default is `250`; increase to reduce render CPU during update bursts, or set `0` for render-per-update behavior.
`target_workers`	Number of logical in-process worker partitions.	`0` enables automatic sizing from target count, with a minimum of `1` and a cap of `64`.
`worker_execution_mode`	Worker model selector.	Use `logical` unless you are explicitly validating `threaded`.
`worker_assignment_mode`	Target assignment strategy.	Use `hash` for stable distribution or `least_loaded` to assign new targets by estimated load.
`worker_rebalance_enabled`	Enables low-frequency worker target rebalancing.	Disabled by default. Enable only when you need runtime balancing across workers.
`debug`	Enables verbose sync, update, and lookup diagnostics.	Use temporarily for troubleshooting.
`license_mode`	Selects `local_file` or `gateway` licensing.	`local_file` is the default. See Gateway Licensing before enabling Gateway mode.
`license_file`	Customer license JSON file.	Required only in `local_file` mode.

### Example operational TOML ``` context_id = "shmApp" pool_name = "pool" metrics_http_enabled = true metrics_bind_address = "0.0.0.0" metrics_port = 9464 metrics_path = "/metrics" metrics_interval_seconds = 10 render_min_interval_ms = 250 target_workers = 0 worker_execution_mode = "logical" worker_assignment_mode = "hash" worker_rebalance_enabled = false debug = false license_file = "./scratch/example.lic" [[metric_scopes]] name = "process_health" metrics = [ "ads_managed_process_sink_dist_state", "ads_managed_process_sink_dist_process_id", "ads_shmem_mo_server_stats_num_clients" ] [[targets]] app = "ads" server_key = "82" [[targets]] app = "ads" server_key = "456" [[targets]] app = "adh" server_key = "999" ``` NoteLocal and Gateway enforcement In `local_file` mode, `lmc_exporter` validates the configured license before LMC discovery starts. If the license is expired but still within the 36-hour grace period, the process can start with warning logs. After the grace period is exhausted, startup is blocked or a running process exits on the next enforcement check. In `gateway` mode, temporary renewal failures retain operation only until the current lease expires. Lease expiry or established entitlement loss closes collection and output, then terminates the exporter with status `1`. Shutdown does not release an expired or lost lease; a valid release is limited by the remaining lease lifetime and `gateway_release_timeout_ms`. ## Usage ### Start HTTP scrape mode ``` ./src/lmc_exporter \ --server-key 82 \ --app ads \ --enable-metrics-http \ --metrics-port 9464 ``` ``` ./src/lmc_exporter --config ./lmcExporterConfig.toml ``` ### Scrape a single target If only one target is configured, the base metrics path can resolve that target automatically: ``` curl "http://127.0.0.1:9464/metrics" ``` When multiple targets are configured, include routing parameters. ``` curl "http://127.0.0.1:9464/metrics?server_key=82" ``` You can also route by application and resolved instance: ``` curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads" ``` If more than one configured target shares the same `app` and `instance`, include `server_key` as well: ``` curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82" ``` ### Scrape a scoped metric set Metric scopes expose named subsets of the rendered Prometheus output. This is useful for health checks, dashboards, or reducing scrape payload size for specific views. ``` [[metric_scopes]] name = "process_health" metrics = [ "ads_managed_process_sink_dist_state", "ads_managed_process_sink_dist_process_id", "ads_shmem_mo_server_stats_num_clients" ] ``` Scrape the scope as a path segment: ``` curl "http://127.0.0.1:9464/metrics/process_health?server_key=82" ``` Or use the `scope` query parameter: ``` curl "http://127.0.0.1:9464/metrics?scope=process_health&server_key=82" ``` NoteScoped scrapes still need target routing When multiple targets are configured, scoped scrapes still require target-routing parameters such as `server_key`, or `app` plus `instance`. ## Monitoring LMC Exporter Health ### Check scrape readiness Use the base or scoped metrics endpoint and inspect the HTTP status code. ``` curl -i "http://127.0.0.1:9464/metrics?server_key=82" ``` Expected operational states:

Check scrape readiness
Status	Meaning	Action
`200 OK`	Target is ready and the exporter returned Prometheus text.	Continue normal monitoring.
`400 Bad Request`	Required routing parameters are missing or ambiguous.	Add `server_key`, or provide both `app` and `instance`.
`404 Not Found`	The requested target or metric scope was not found.	Verify `server_key`, `app`, `instance`, and scope name.
`503 Service Unavailable`	The target is not ready for scrape.	Wait for initial sync or investigate the readiness message.

A `503` response includes a message similar to: ``` scrape not ready: objects 10/12 synced, variables 120/140 synced, waiting on variable sync ``` ### Monitor internal exporter metrics Each rendered target includes internal exporter metrics alongside the application metrics. Use these metrics to monitor readiness, sync state, queue pressure, worker utilization, refresh behavior, and cache effectiveness.

Monitor internal exporter metrics
Metric	Type	What it indicates
`lmc_exporter_internal_target_ready`	Gauge	`1` when the target is serving scrapes; `0` when it returns `503`.
`lmc_exporter_internal_target_initial_sync_complete`	Gauge	`1` after the tracked subtree completes its initial full sync.
`lmc_exporter_internal_target_pending_sync_items`	Gauge	Objects and variables that have not synced for the first time.
`lmc_exporter_internal_scrape_render_cache_hits_total`	Counter	Scrapes served from the rendered-output cache.
`lmc_exporter_internal_scrape_render_cache_misses_total`	Counter	Scrapes that required refreshed rendered output.
`lmc_exporter_internal_scrape_scope_cache_hits_total`	Counter	Scoped scrapes served from the scope cache.
`lmc_exporter_internal_scrape_scope_cache_misses_total`	Counter	Scoped scrapes that required rebuilding scoped output.
`lmc_exporter_worker_busy_idle_ratio`	Gauge	Ratio of worker busy refresh time to idle queue-wait time.
`lmc_exporter_worker_dequeue_latency_avg_ms`	Gauge	Average worker queue wait latency.
`lmc_exporter_worker_dequeue_latency_p50_ms`	Gauge	Median sampled worker queue wait latency.
`lmc_exporter_worker_dequeue_latency_p95_ms`	Gauge	95th percentile sampled worker queue wait latency.
`lmc_exporter_worker_dequeue_latency_p99_ms`	Gauge	99th percentile sampled worker queue wait latency.
`lmc_exporter_worker_rebalance_total`	Counter	Number of target moves performed by runtime worker rebalancing.
`lmc_exporter_worker_rebalance_last_duration_ms`	Gauge	Duration of the most recent rebalance pass.
`lmc_exporter_target_refresh_total`	Counter	Number of refresh cycles executed for the target.
`lmc_exporter_target_refresh_duration_p50_ms`	Gauge	Median sampled target refresh duration.
`lmc_exporter_target_refresh_duration_p95_ms`	Gauge	95th percentile sampled target refresh duration.
`lmc_exporter_target_refresh_duration_p99_ms`	Gauge	99th percentile sampled target refresh duration.

### Recommended alert signals Use the internal metrics to build operational alerts around target availability and exporter pressure. ``` lmc_exporter_internal_target_ready == 0 ``` ``` lmc_exporter_internal_target_initial_sync_complete == 0 ``` ``` lmc_exporter_internal_target_pending_sync_items > 0 ``` ``` rate(lmc_exporter_worker_rebalance_total[5m]) > 0 ``` ``` lmc_exporter_worker_dequeue_latency_p95_ms > 1000 ``` Tune thresholds for your environment. ## Operational Best Practices ### Prefer TOML for production configuration Use `--config` with a TOML file for production runs. TOML supports runtime defaults, metric scopes, and preloaded target tables in one file. ``` ./src/lmc_exporter --config ./lmcExporterConfig.toml ``` CLI flags override TOML values. Use CLI overrides for temporary changes, not for long-lived production configuration. ### Use `server_key` routing for multi-target deployments In multi-target mode, `server_key` is the simplest and least ambiguous routing parameter. ``` curl "http://127.0.0.1:9464/metrics?server_key=82" ``` Use `app` and `instance` routing only when the resolved instance is known and unique. ### Keep health scopes small For operational dashboards and alerting, define a small `process_health` scope instead of scraping and evaluating the full application payload every time. ``` [[metric_scopes]] name = "process_health" metrics = [ "ads_managed_process_sink_dist_state", "ads_managed_process_sink_dist_process_id", "ads_shmem_mo_server_stats_num_clients" ] ``` ### Use filters to limit subscription scope For target definitions, use `classes`, `seed_path_groups`, or `seed_paths` to limit the tracked subtree. These filters limit both the subscribed subtree and the metrics exposed for that target. ``` [[targets]] app = "ads" server_key = "82" classes = ["ManagedProcess.SinkDist", "ConsumerDataStream"] seed_path_groups = ["core", "users"] ``` ### Use `--debug` only during investigation Debug mode emits verbose sync, update, and lookup diagnostics. Enable it while investigating readiness, target selection, or variable update behavior, then disable it to reduce log volume. ``` ./src/lmc_exporter --config ./lmcExporterConfig.toml --debug ``` ## Performance Tuning ### Tune render coalescing `render_min_interval_ms` controls the minimum spacing between demand renders for a single target. Default behavior: ``` render_min_interval_ms = 250 ``` Operational effects: - Higher values reduce render CPU during bursts because multiple updates are coalesced into one deferred render. - Lower values reduce the maximum delay between LMC updates and rendered scrape output. - `0` disables spacing and restores render-per-update behavior. Use a higher value when: - update churn is high - worker dequeue latency is increasing - scrape output does not need sub-second freshness Use a lower value when: - freshness is more important than render CPU - the target count is small - internal latency metrics show the exporter has enough headroom ### Use scoped metric sets to reduce scrape processing Scoped metric sets are filtered from rendered Prometheus output and cached by scope. Repeated scoped scrapes can be served from the scope cache when the underlying snapshot and scope configuration have not changed. Use scoped metrics for: - health checks - focused dashboards - high-frequency operational scrapes ### Watch cache effectiveness Use these metrics to understand whether scrapes are using cached output or causing extra work: ``` rate(lmc_exporter_internal_scrape_render_cache_hits_total[5m]) ``` ``` rate(lmc_exporter_internal_scrape_render_cache_misses_total[5m]) ``` ``` rate(lmc_exporter_internal_scrape_scope_cache_hits_total[5m]) ``` ``` rate(lmc_exporter_internal_scrape_scope_cache_misses_total[5m]) ``` Frequent cache misses during stable periods may indicate that target data is changing often, scopes are changing, or scrapes are reaching targets immediately after render updates. ### Watch worker pressure Monitor worker metrics when increasing target count or widening metric scope. ``` lmc_exporter_worker_busy_idle_ratio ``` ``` lmc_exporter_worker_dequeue_latency_p95_ms ``` ``` lmc_exporter_target_refresh_duration_p95_ms ``` Sustained increases in queue latency or refresh duration usually indicate that you should reduce target scope, increase worker capacity, or review scrape frequency. ## Scaling Considerations ### Scale target definitions deliberately A single TOML file can preload multiple targets: ``` metrics_http_enabled = true metrics_port = 9464 [[targets]] app = "ads" server_key = "82" seed_path_groups = ["core", "admin", "sink_ripc", "users"] [[targets]] app = "ads" server_key = "456" seed_path_groups = ["core", "source_ssl", "service_generator"] [[targets]] app = "adh" server_key = "999" seed_path_groups = [ "consumer_mqos_service_service", "mqos_service_attributes", "mqos_service_datastreams" ] ``` Each `[[targets]]` entry must define `app` and `server_key`. `instance` is optional when `server_key` is provided and can be resolved from the selected root at runtime. ### Configure worker count Use automatic worker sizing first: ``` target_workers = 0 ``` Automatic sizing uses the configured target count, with a minimum of `1` and a cap of `64`. For explicit sizing: ``` target_workers = 4 ``` Non-zero worker values must be between `1` and `64`. ### Choose an assignment mode Use `hash` for stable target placement: ``` worker_assignment_mode = "hash" ``` Use `least_loaded` when adding targets and you want assignment based on the smallest current estimated load: ``` worker_assignment_mode = "least_loaded" ``` ### Enable rebalancing only when needed Runtime worker rebalancing is disabled by default. ``` worker_rebalance_enabled = false ``` Enable it when worker metrics show persistent imbalance across workers: ``` worker_rebalance_enabled = true ``` When enabled, the exporter can move targets between workers and reports rebalance activity through `lmc_exporter_worker_rebalance_total` and `lmc_exporter_worker_rebalance_last_duration_ms`. NoteScaling recommendation Start with `target_workers = 0`, `worker_assignment_mode = "hash"`, and filtered target scopes. Add workers or enable rebalancing only after internal worker metrics show sustained queue pressure or uneven load. ## Examples ### Minimal single-target HTTP scrape ``` app = "ads" server_keys = ["82"] metrics_http_enabled = true metrics_bind_address = "0.0.0.0" metrics_port = 9464 metrics_path = "/metrics" license_file = "./scratch/example.lic" ``` Run: ``` ./src/lmc_exporter --config ./lmcExporterConfig.toml ``` Scrape: ``` curl "http://127.0.0.1:9464/metrics" ``` ### Multi-target health scope ``` metrics_http_enabled = true metrics_port = 9464 render_min_interval_ms = 250 target_workers = 0 [[metric_scopes]] name = "process_health" metrics = [ "ads_managed_process_sink_dist_state", "ads_managed_process_sink_dist_process_id", "ads_shmem_mo_server_stats_num_clients" ] [[targets]] app = "ads" server_key = "82" seed_path_groups = ["core"] [[targets]] app = "ads" server_key = "456" seed_path_groups = ["core"] ``` Scrape each target’s health subset: ``` curl "http://127.0.0.1:9464/metrics/process_health?server_key=82" curl "http://127.0.0.1:9464/metrics/process_health?server_key=456" ``` ### Legacy target file Use the legacy format only when you need a minimal target-only file. ``` target app=ads server_key=82 target app=ads server_key=456 ``` Run with: ``` ./src/lmc_exporter --targets-config ./targets.txt --enable-metrics-http --metrics-port 9464 ``` ## Troubleshooting ### HTTP endpoint returns `503` Cause: the target is not ready for scrape. During startup, all tracked objects and variables must sync before the first scrape can succeed. Resolution: 1. Inspect the response body for the readiness message. 2. Confirm the configured `server_key` is correct. 3. Confirm the selected root exists and matches the optional `instance` hint. 4. Run with `--debug` to capture sync, lookup, and update diagnostics. 5. Monitor `lmc_exporter_internal_target_initial_sync_complete` and `lmc_exporter_internal_target_pending_sync_items`. ### HTTP endpoint returns `400` Cause: target routing is missing or ambiguous. Resolution: - If multiple targets are configured, include `server_key`. - If omitting `server_key`, provide both `app` and `instance`. - If multiple targets share the same `app` and `instance`, include `server_key`. Example: ``` curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82" ``` ### HTTP endpoint returns `404` Cause: the requested target or scope does not exist, or the selected scope does not match any rendered metrics. Resolution: 1. Verify the `server_key`, `app`, and `instance` values. 2. Verify the `[[metric_scopes]]` `name`. 3. Confirm that each metric name in the scope matches the rendered Prometheus metric names. 4. For wildcard scopes, verify that the wildcard pattern matches the rendered metric names. ### No metrics are emitted Cause: the app mapping or filters do not select any metrics. Resolution: 1. Confirm the target `app` is correct. 2. Confirm metric mappings exist for that application. 3. Review configured `classes`, `seed_path_groups`, and `seed_paths`. 4. Temporarily reduce filtering to confirm whether the target emits metrics without the restrictive scope. ### Multi-target scrape cannot disambiguate the target Cause: more than one target matches the supplied route. Resolution: Use `server_key` in the scrape query. ``` curl "http://127.0.0.1:9464/metrics?server_key=82" ``` ### Startup fails before LMC discovery Cause: license validation fails. Resolution: 1. Confirm `license_file` points to the customer license JSON file. 2. Confirm the license has not expired beyond the 36-hour grace period. 3. Check startup stderr logs for license validation messages. 4. Re-run after updating the configured license file. ### Worker latency or refresh duration is high Cause: target refresh work is exceeding worker capacity or render output is too expensive. Resolution: 1. Check `lmc_exporter_worker_dequeue_latency_p95_ms`. 2. Check `lmc_exporter_target_refresh_duration_p95_ms`. 3. Increase `render_min_interval_ms` to coalesce more updates. 4. Narrow `classes`, `seed_path_groups`, or `seed_paths`. 5. Use scoped metric sets for high-frequency operational scrapes. 6. Increase `target_workers` or enable `worker_rebalance_enabled` after confirming sustained pressure.