HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

telemetry Block

Placementtelemetry

The telemetry block configures Nomad's publication of metrics and telemetry to third-party systems.

telemetry {
  publish_allocation_metrics = true
  publish_node_metrics       = true
}

This section of the documentation only covers the configuration options for telemetry block. To understand the architecture and metrics themselves, please see the Telemetry guide.

telemetry Parameters

Due to the number of configurable parameters to the telemetry block, parameters on this page are grouped by the telemetry provider.

Common

The following options are available on all telemetry configurations.

  • in_memory_collection_interval (duration: 10s) Configures the in-memory sink collection interval. This sink is always configured and backs the JSON metrics API endpoint. This option is particularly useful for debugging or development purposes, where aggressive collection is required.

  • in_memory_retention_period (duration: 1m) Configures the in-memory sink retention period. This sink is always configured and backs the JSON metrics API endpoint. This option is particularly useful for debugging or development purposes.

  • disable_hostname (bool: false) - Specifies if gauge values should be prefixed with the local hostname.

  • collection_interval (duration: 1s) - Specifies the time interval at which the Nomad agent collects telemetry data. For metrics tools that scrape metrics (for example, Prometheus), you should ensure this value is less than or equal to the value of any scrape interval.

  • use_node_name (bool: false) - Specifies if gauge values should be prefixed with the name of the node, instead of the hostname. If set it will override disable_hostname value.

  • publish_allocation_metrics (bool: false) - Specifies if Nomad should publish runtime metrics of allocations.

  • include_alloc_metadata_in_metrics (bool: false) - This controls whether allocation metadata is included in metric labels. Enabling this option may result in high cardinality labels. You should also configure allowed_metadata_keys_in_metrics.

  • allowed_metadata_keys_in_metrics (list: []) - This filters the metadata keys to be included in the metric publishing. By default it does not filter out any keys and thus include all metadata.

  • publish_node_metrics (bool: false) - Specifies if Nomad should publish runtime metrics of nodes.

  • filter_default (bool: true) - This controls whether to allow metrics that have not been specified by the filter. Defaults to true, which will allow all metrics when no filters are provided. When set to false with no filters, no metrics will be sent.

  • prefix_filter (list: []) - This is a list of filter rules to apply for allowing/blocking metrics by prefix. A leading "+" will enable any metrics with the given prefix, and a leading "-" will block them. If there is overlap between two rules, the more specific rule will take precedence. Blocking will take priority if the same prefix is listed multiple times.

['-nomad.raft', '+nomad.raft.apply', '-nomad.memberlist']

  • disable_dispatched_job_summary_metrics (bool: false) - Specifies if Nomad should ignore jobs dispatched from a parameterized job when publishing job summary statistics. Since each job has a small memory overhead for tracking summary statistics, it is sometimes desired to trade these statistics for more memory when dispatching high volumes of jobs.

  • disable_quota_utilization_metrics (bool: false) - Specifies if Nomad should publish metrics about quota utilization (a Nomad Enterprise feature). Since each quota utilization check requires a relatively expensive check against Nomad's state store, users with many namespaces and many quotas may want to disable these metrics.

  • disable_allocation_hook_metrics (bool: false) - Specifies if the Nomad client should publish metrics related to allocation and task hooks.

statsite

These telemetry parameters apply to statsite.

  • statsite_address (string: "") - Specifies the address of a statsite server to forward metrics data to.
telemetry {
  statsite_address = "statsite.company.local:8125"
}

statsd

These telemetry parameters apply to statsd.

  • statsd_address (string: "") - Specifies the address of a statsd server to forward metrics to.
telemetry {
  statsd_address = "statsd.company.local:8125"
}

datadog

These telemetry parameters apply to DataDog statsd.

  • datadog_address (string: "") - Specifies the address of a DataDog statsd server to forward metrics to.

  • datadog_tags (list: []) - Specifies a list of global tags that will be added to all telemetry packets sent to DogStatsD. It is a list of strings, where each string looks like "my_tag_name:my_tag_value".

telemetry {
  datadog_address = "dogstatsd.company.local:8125"
  datadog_tags    = ["my_tag_name:my_tag_value"]
}

prometheus

These telemetry parameters apply to Prometheus.

  • prometheus_metrics (bool: false) - Specifies whether the agent should make Prometheus formatted metrics available at /v1/metrics?format=prometheus.

circonus

These telemetry parameters apply to Circonus.

  • circonus_api_token (string: "") - Specifies a valid Circonus API Token used to create/manage check. If provided, metric management is enabled.

  • circonus_api_app (string: "nomad") - Specifies a valid app name associated with the API token.

  • circonus_api_url (string: "https://api.circonus.com/v2") - Specifies the base URL to use for contacting the Circonus API.

  • circonus_submission_interval (string: "10s") - Specifies the interval at which metrics are submitted to Circonus.

  • circonus_submission_url (string: "") - Specifies the check.config.submission_url field, of a Check API object, from a previously created HTTPTRAP check.

  • circonus_check_id (string: "") - Specifies the Check ID (not check bundle) from a previously created HTTPTRAP check. The numeric portion of the check._cid field in the Check API object.

  • circonus_check_force_metric_activation (bool: false) - Specifies if force activation of metrics which already exist and are not currently active. If check management is enabled, the default behavior is to add new metrics as they are encountered. If the metric already exists in the check, it will not be activated. This setting overrides that behavior.

  • circonus_check_instance_id (string: "<hostname>:<application>") - Serves to uniquely identify the metrics coming from this instance. It can be used to maintain metric continuity with transient or ephemeral instances as they move around within an infrastructure. By default, this is set to hostname:application name (e.g. "host123:nomad").

  • circonus_check_search_tag (string: <service>:<application>) - Specifies a special tag which, when coupled with the instance id, helps to narrow down the search results when neither a Submission URL or Check ID is provided. By default, this is set to service:app (e.g. "service:nomad").

  • circonus_check_display_name (string: "") - Specifies a name to give a check when it is created. This name is displayed in the Circonus UI Checks list.

  • circonus_check_tags (string: "") - Comma separated list of additional tags to add to a check when it is created.

  • circonus_broker_id (string: "") - Specifies the ID of a specific Circonus Broker to use when creating a new check. The numeric portion of broker._cid field in a Broker API object. If metric management is enabled and neither a Submission URL nor Check ID is provided, an attempt will be made to search for an existing check using Instance ID and Search Tag. If one is not found, a new HTTPTRAP check will be created. By default, this is a random Enterprise Broker is selected, or, the default Circonus Public Broker.

  • circonus_broker_select_tag (string: "") - Specifies a special tag which will be used to select a Circonus Broker when a Broker ID is not provided. The best use of this is to as a hint for which broker should be used based on where this particular instance is running (e.g. a specific geographic location or datacenter, dc:sfo).

Edit this page on GitHub