Operating LMC Exporter

Overview

LMC Exporter exposes LSEG/LMC shared-memory managed object data as Prometheus-compatible metrics. It can emit periodic stdout snapshots or serve metrics through an HTTP scrape endpoint, which defaults to /metrics when HTTP mode is enabled.

This article explains how to operate the exporter after it is installed and configured. It focuses on:

• monitoring exporter and target health
• configuring HTTP scraping
• using scoped metric sets for operational views
• tuning render and worker behavior
• scaling from one target to multiple targets
• troubleshooting scrape readiness and routing errors

Prerequisites

Before operating lmc_exporter, ensure that you have:

• A matching EL8 or EL9 x86_64 tar archive installed using Installation, including runtime libraries and the active-install link.
• An account with permission to access the target LMC shared memory. Commands below use the EL8 binary; use lmc_exporter-el9 on EL9.
• A valid local license file, or a configured Gateway that can grant the complete exporter workload. Licensing completes before the exporter creates the LMC shared-memory pool.
• One or more shared-memory server_key values for the LMC applications you want to scrape.
• A Prometheus-compatible scraper, or direct access to the exporter HTTP endpoint for manual checks with curl.

To confirm the binary version:

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --version

How It Works

At runtime, lmc_exporter:

1. Loads built-in defaults, an optional TOML config file, and then any CLI overrides.
2. Builds one or more scrape target definitions.
3. Validates the local license, or acquires the full configured workload from Gateway.
4. Creates the LMC shared-memory server pool.
5. Adds configured server keys to the pool.
6. Creates a target registry.
7. Sets up metric collection for each target.
8. Starts either:
   • the HTTP scrape server, or
   • the periodic stdout metrics printer.
9. Discovers visible roots for each configured server_key.
10. Selects the best matching root, optionally using an instance hint.
11. Recursively binds child objects and variables under the selected root.
12. Renders Prometheus text once the tracked subtree is ready.

Before the first full sync completes, HTTP scrapes return 503 Service Unavailable with a readiness message. After initial sync, newly added child objects or variables do not take the target back to startup readiness. The exporter continues serving the last rendered snapshot and adds the new metrics after those paths sync.

Partial TOML examples below illustrate settings to add to your existing target and licensing configuration.

Configuration

Core operational settings

The following TOML keys are directly relevant to operations.

Example operational TOML

context_id = "shmApp"
pool_name = "pool"

metrics_http_enabled = true
metrics_bind_address = "0.0.0.0"
metrics_port = 9464
metrics_path = "/metrics"

metrics_interval_seconds = 10
render_min_interval_ms = 250
target_workers = 0
worker_execution_mode = "logical"
worker_assignment_mode = "hash"
worker_rebalance_enabled = false
debug = false

license_file = "/opt/particle_horizon/local_data/lmc_exporter/lmc_exporter.lic"

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

[[targets]]
app = "ads"
server_key = "82"

[[targets]]
app = "ads"
server_key = "456"

[[targets]]
app = "adh"
server_key = "999"

Usage

Start HTTP scrape mode

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 \
  --server-key 82 \
  --app ads \
  --enable-metrics-http \
  --metrics-port 9464 \
  --license-file /opt/particle_horizon/local_data/lmc_exporter/lmc_exporter.lic

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --config ./lmcExporterConfig.toml

Scrape a single target

If only one target is configured, the base metrics path can resolve that target automatically:

curl "http://127.0.0.1:9464/metrics"

When multiple targets are configured, include routing parameters.

curl "http://127.0.0.1:9464/metrics?server_key=82"

You can also route by application and resolved instance:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads"

If more than one configured target shares the same app and instance, include server_key as well:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82"

Scrape a scoped metric set

Metric scopes expose named subsets of the rendered Prometheus output. This is useful for health checks, dashboards, or reducing scrape payload size for specific views.

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

Scrape the scope as a path segment:

curl "http://127.0.0.1:9464/metrics/process_health?server_key=82"

Or use the scope query parameter:

curl "http://127.0.0.1:9464/metrics?scope=process_health&server_key=82"

Upgrade to 0.19.0

Review Compatibility

Existing configuration keys, CLI flags, and public endpoints are retained. Existing v0.18.0 configurations remain valid, but ADH adds labels to existing series and adds new metrics. Review ADH metric and label changes, dashboards, recording rules, joins, and storage/cardinality impact before rollout.

Review restrictive ADH filters to include the new objects when required. Verify timestamp values with the exporter timezone set to match the source. To add RRCP Trans, validate the live role layout and readiness/recovery behavior; Gateway licensing requires Keystone Gateway v0.2.1, the rrcp_trans entitlement, and stable component IDs before restarting.

Stage and Back Up

The commands below use the EL8 installation layout from Installation. Use el9 for EL9. If your current deployment uses another layout, record its binary path, service configuration, and startup method before changing it.

Download the supplied archive and checksum file into the current directory. Verify and extract to a separate staging directory:

sha256sum -c lmc_exporter-v0.19.0-el8-x86_64.tar.gz.sha256sum
upgrade_stage=$(mktemp -d)
tar -xzf lmc_exporter-v0.19.0-el8-x86_64.tar.gz -C "$upgrade_stage"

Record the current version and active-install destination. Stop the exporter cleanly before backing up state or replacing the binary:

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --version
readlink -f /opt/particle_horizon/lmc_exporter
sudo systemctl stop lmc-exporter
backup_dir=$(sudo mktemp -d /opt/lmc-exporter-backup.XXXXXX)
sudo cp -a /opt/particle_horizon/local_config/lmc_exporter "$backup_dir/config"
sudo cp -a /opt/particle_horizon/local_data/lmc_exporter "$backup_dir/data"
sudo systemctl cat lmc-exporter | sudo tee "$backup_dir/service.txt" >/dev/null

Also back up any license, credential, service override, and Gateway state files stored outside these directories using your normal secure backup procedure. Preserve ownership and permissions. Keep the existing Gateway installation-id in place and reuse it on this host; do not create a new identity or copy it to another installation.

Activate the New Binary

Copy only the new versioned software directory. Do not overwrite the existing configuration, data, or service unit with archive defaults:

sudo cp -a "$upgrade_stage/particle_horizon/SOFTWARE/lmc_exporter-v0.19.0-el8-x86_64" /opt/particle_horizon/SOFTWARE/
sudo ln -sfnT SOFTWARE/lmc_exporter-v0.19.0-el8-x86_64 /opt/particle_horizon/lmc_exporter
/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --version

The active-install path must be a symlink, not a real directory. If it is a directory, preserve it and adapt your service’s ExecStart to the new versioned binary instead of replacing that directory. Ensure the new binary is readable and executable by the service account. Preserve your reviewed configuration and absolute license path.

Restart and validate:

sudo systemctl start lmc-exporter
sudo systemctl status lmc-exporter
sudo journalctl -u lmc-exporter -n 100 --no-pager
curl --fail --show-error "http://127.0.0.1:9464/metrics?server_key=82"

Run sudo systemctl daemon-reload before starting if you changed the service unit. Replace the example address, port, path, and key with your configured values. Repeat the scrape for every target after initial sync, check application metrics and timestamp values, and confirm Gateway renewal if enabled.

Roll Back if Validation Fails

Stop the service, restore the recorded previous active-install link or ExecStart path, and restore the previous configuration if you changed it. Remove newly added rrcp_trans targets when returning to v0.18.0, which does not support that mapping. Preserve the current Gateway installation identity. Reload systemd if the unit changed, then start the service and repeat the checks above. Keep the prior binary and protected backups until validation is complete.

Monitoring LMC Exporter Health

Check scrape readiness

Use the base or scoped metrics endpoint and inspect the HTTP status code.

curl -i "http://127.0.0.1:9464/metrics?server_key=82"

Expected operational states:

A 503 response includes a message similar to:

scrape not ready: objects 10/12 synced, variables 120/140 synced, waiting on variable sync

Monitor internal exporter metrics

Each rendered target includes internal exporter metrics alongside the application metrics. Use these metrics to monitor readiness, sync state, and cache effectiveness.

Recommended alert signals

Alert on Prometheus up == 0 for the configured exporter scrape job to detect failed scrapes, including 503 responses. A 503 response contains a readiness message, not a Prometheus metrics payload, so internal readiness metrics alone cannot detect an unavailable target.

Use the internal metrics to build operational alerts around target availability and exporter pressure.

lmc_exporter_internal_target_ready == 0

lmc_exporter_internal_target_initial_sync_complete == 0

lmc_exporter_internal_target_pending_sync_items > 0

Tune thresholds for your environment.

Operational Best Practices

Prefer TOML for production configuration

Use --config with a TOML file for production runs. TOML supports runtime defaults, metric scopes, and preloaded target tables in one file.

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --config ./lmcExporterConfig.toml

CLI flags override TOML values. Use CLI overrides for temporary changes, not for long-lived production configuration.

Use server_key routing for multi-target deployments

In multi-target mode, server_key is the simplest and least ambiguous routing parameter.

curl "http://127.0.0.1:9464/metrics?server_key=82"

Use app and instance routing only when the resolved instance is known and unique.

Keep health scopes small

For operational dashboards and alerting, define a small process_health scope instead of scraping and evaluating the full application payload every time.

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

Use filters to limit subscription scope

For target definitions, use classes, seed_path_groups, or seed_paths to limit the tracked subtree. These filters limit both the subscribed subtree and the metrics exposed for that target.

[[targets]]
app = "ads"
server_key = "82"
classes = ["ManagedProcess.SinkDist", "ConsumerDataStream"]
seed_path_groups = ["core", "users"]

Use --debug only during investigation

Debug mode emits verbose sync, update, and lookup diagnostics. Enable it while investigating readiness, target selection, or variable update behavior, then disable it to reduce log volume.

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --config ./lmcExporterConfig.toml --debug

Performance Tuning

Tune render coalescing

render_min_interval_ms controls the minimum spacing between demand renders for a single target.

Default behavior:

render_min_interval_ms = 250

Operational effects:

• Higher values reduce render CPU during bursts because multiple updates are coalesced into one deferred render.
• Lower values reduce the maximum delay between LMC updates and rendered scrape output.
• 0 disables spacing and restores render-per-update behavior.

Use a higher value when:

• update churn is high
• scrape latency or exporter CPU usage is increasing
• scrape output does not need sub-second freshness

Use a lower value when:

• freshness is more important than render CPU
• the target count is small
• host CPU and scrape latency show the exporter has enough headroom

Use scoped metric sets to reduce scrape processing

Scoped metric sets are filtered from rendered Prometheus output and cached by scope. Repeated scoped scrapes can be served from the scope cache when the underlying snapshot and scope configuration have not changed.

Use scoped metrics for:

• health checks
• focused dashboards
• high-frequency operational scrapes

Watch cache effectiveness

Use these metrics to understand whether scrapes are using cached output or causing extra work:

rate(lmc_exporter_internal_scrape_render_cache_hits_total[5m])

rate(lmc_exporter_internal_scrape_render_cache_misses_total[5m])

rate(lmc_exporter_internal_scrape_scope_cache_hits_total[5m])

rate(lmc_exporter_internal_scrape_scope_cache_misses_total[5m])

Frequent cache misses during stable periods may indicate that target data is changing often, scopes are changing, or scrapes are reaching targets immediately after render updates.

Watch collection pressure

Monitor exporter CPU and memory with your host monitoring tools, and use Prometheus scrape duration and failures to identify collection pressure. Worker dequeue percentiles, busy/idle ratios, rebalance counters, and per-target refresh percentiles are not exposed as metrics in this version.

Check the service log for warning stale scrape snapshot, warning target update lag, and warning snapshot mutex contention. If CPU or scrape latency rises as you add targets, narrow the subscribed paths and review scrape frequency. Change worker settings only after measuring the effect on your deployment.

Scaling Considerations

Scale target definitions deliberately

A single TOML file can preload multiple targets:

metrics_http_enabled = true
metrics_port = 9464

[[targets]]
app = "ads"
server_key = "82"
seed_path_groups = ["core", "admin", "sink_ripc", "users"]

[[targets]]
app = "ads"
server_key = "456"
seed_path_groups = ["core", "source_ssl", "service_generator"]

[[targets]]
app = "adh"
server_key = "999"
seed_path_groups = [
  "consumer_mqos_service_service",
  "mqos_service_attributes",
  "mqos_service_datastreams"
]

Each [[targets]] entry must define app and server_key. instance is optional when server_key is provided and can be resolved from the selected root at runtime.

Configure worker count

Use automatic worker sizing first:

target_workers = 0

Automatic sizing uses the configured target count, with a minimum of 1 and a cap of 64.

For explicit sizing:

target_workers = 4

Non-zero worker values must be between 1 and 64.

Choose an assignment mode

Use hash for stable target placement:

worker_assignment_mode = "hash"

Use least_loaded when adding targets and you want assignment based on the smallest current estimated load:

worker_assignment_mode = "least_loaded"

Enable rebalancing only when needed

Runtime worker rebalancing is disabled by default.

worker_rebalance_enabled = false

Enable it only after validating a benefit for your workload:

worker_rebalance_enabled = true

When enabled, the exporter can move targets between workers. Inspect runtime diagnostics and compare host resource usage and scrape latency before and after the change.

Examples

Minimal single-target HTTP scrape

app = "ads"
server_keys = ["82"]

metrics_http_enabled = true
metrics_bind_address = "0.0.0.0"
metrics_port = 9464
metrics_path = "/metrics"

license_file = "/opt/particle_horizon/local_data/lmc_exporter/lmc_exporter.lic"

Run:

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --config ./lmcExporterConfig.toml

Scrape:

curl "http://127.0.0.1:9464/metrics"

Multi-target health scope

license_file = "/opt/particle_horizon/local_data/lmc_exporter/lmc_exporter.lic"

metrics_http_enabled = true
metrics_port = 9464
render_min_interval_ms = 250
target_workers = 0

[[metric_scopes]]
name = "process_health"
metrics = [
  "ads_managed_process_sink_dist_state",
  "ads_managed_process_sink_dist_process_id",
  "ads_shmem_mo_server_stats_num_clients"
]

[[targets]]
app = "ads"
server_key = "82"
seed_path_groups = ["core"]

[[targets]]
app = "ads"
server_key = "456"
seed_path_groups = ["core"]

Scrape each target’s health subset:

curl "http://127.0.0.1:9464/metrics/process_health?server_key=82"
curl "http://127.0.0.1:9464/metrics/process_health?server_key=456"

Legacy target file

Use the legacy format only when you need a minimal target-only file.

target app=ads server_key=82
target app=ads server_key=456

Run with:

/opt/particle_horizon/lmc_exporter/bin/lmc_exporter-el8 --targets-config ./targets.txt --enable-metrics-http --metrics-port 9464 --license-file /opt/particle_horizon/local_data/lmc_exporter/lmc_exporter.lic

Troubleshooting

HTTP endpoint returns 503

Cause: the target is not ready for scrape. During startup, all tracked objects and variables must sync before the first scrape can succeed.

Resolution:

1. Inspect the response body for the readiness message.
2. Confirm the configured server_key is correct.
3. Confirm the selected root exists and matches the optional instance hint.
4. Run with --debug to capture sync, lookup, and update diagnostics.
5. Monitor scrape success with Prometheus up and inspect the 503 response body. Internal metrics are available again after a successful scrape.

HTTP endpoint returns 400

Cause: target routing is missing or ambiguous.

Resolution:

• If multiple targets are configured, include server_key.
• If omitting server_key, provide both app and instance.
• If multiple targets share the same app and instance, include server_key.

Example:

curl "http://127.0.0.1:9464/metrics?app=ads&instance=phoads01.1.ads&server_key=82"

HTTP endpoint returns 404

Cause: the requested target or scope does not exist, or the selected scope does not match any rendered metrics.

Resolution:

1. Verify the server_key, app, and instance values.
2. Verify the [[metric_scopes]] name.
3. Confirm that each metric name in the scope matches the rendered Prometheus metric names.
4. For wildcard scopes, verify that the wildcard pattern matches the rendered metric names.

No metrics are emitted

Cause: the app mapping or filters do not select any metrics.

Resolution:

1. Confirm the target app is correct.
2. Confirm metric mappings exist for that application.
3. Review configured classes, seed_path_groups, and seed_paths.
4. Temporarily reduce filtering to confirm whether the target emits metrics without the restrictive scope.

Multi-target scrape cannot disambiguate the target

Cause: more than one target matches the supplied route.

Resolution:

Use server_key in the scrape query.

curl "http://127.0.0.1:9464/metrics?server_key=82"

Startup fails before LMC discovery

Cause: license validation fails.

Resolution:

1. Confirm license_file points to the customer license JSON file.
2. Confirm the license has not expired beyond the 36-hour grace period.
3. Check startup stderr logs for license validation messages.
4. Re-run after updating the configured license file.

Exporter CPU or scrape latency is high

Cause: the collection workload or render output may exceed the available capacity.

Resolution:

1. Check host CPU and memory usage, scrape duration, and scrape failures.
2. Increase render_min_interval_ms to coalesce more updates.
3. Narrow classes, seed_path_groups, or seed_paths.
4. Use scoped metric sets to reduce high-frequency scrape payloads; scopes do not reduce subscriptions.
5. Validate worker-count or rebalancing changes against the same workload before retaining them.