LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
モニタリングツールであるPrometheusの始め方の紹介です。
作業は、Ubuntu 20.04 LTSで行っていますが、その他のディストリビューションでも基本同じかと思います。
作業の流れは、Prometheusの本家サイトにある通りです。
※今回紹介する方法以外にも、kubernetesにコンテナとしてデプロイする方法もあります。
まず、ここから、Linux用のprometheus-2.19.0.linux-amd64.tar.gzをダウンロードします。
※2020年6月11日時点
takahiro@ubuntu2004server:~$ cd /var/tmp/ takahiro@ubuntu2004server:/var/tmp$ wget https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz --2020-06-11 16:36:54-- https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz Resolving github.com (github.com)... 52.69.186.44 Connecting to github.com (github.com)|52.69.186.44|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/8c466b00-aa67-11ea-8348-73b449fef4eb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200611%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200611T023654Z&X-Amz-Expires=300&X-Amz-Signature=333c0e2d4f3a642ed4088e09449e5244bd64b4c4ecd41498c14456afc2722d68&X-Amz-SignedHeaders=host&actor_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.19.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2020-06-11 16:36:54-- https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/8c466b00-aa67-11ea-8348-73b449fef4eb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200611%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200611T023654Z&X-Amz-Expires=300&X-Amz-Signature=333c0e2d4f3a642ed4088e09449e5244bd64b4c4ecd41498c14456afc2722d68&X-Amz-SignedHeaders=host&actor_id=0&repo_id=6838921&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.19.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.169.219 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.169.219|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 64161216 (61M) [application/octet-stream] Saving to: ‘prometheus-2.19.0.linux-amd64.tar.gz’ prometheus-2.19.0.l 100%[===================>] 61.19M 2.86MB/s in 25s 2020-06-11 16:37:20 (2.42 MB/s) - ‘prometheus-2.19.0.linux-amd64.tar.gz’ saved [64161216/64161216] takahiro@ubuntu2004server:/var/tmp$ ll drwxrwxrwt 6 root root 4096 Jun 11 02:36 ./ drwxr-xr-x 13 root root 4096 Apr 23 07:35 ../ -rw-rw-r-- 1 takahiro takahiro 64161216 Jun 9 16:40 prometheus-2.19.0.linux-amd64.tar.gz
続いて、tarballの解凍。
takahiro@ubuntu2004server:/var/tmp$ tar zxvf prometheus-2.19.0.linux-amd64.tar.gz prometheus-2.19.0.linux-amd64/ prometheus-2.19.0.linux-amd64/consoles/ prometheus-2.19.0.linux-amd64/consoles/prometheus.html prometheus-2.19.0.linux-amd64/consoles/node-overview.html prometheus-2.19.0.linux-amd64/consoles/node-disk.html prometheus-2.19.0.linux-amd64/consoles/index.html.example prometheus-2.19.0.linux-amd64/consoles/prometheus-overview.html prometheus-2.19.0.linux-amd64/consoles/node.html prometheus-2.19.0.linux-amd64/consoles/node-cpu.html prometheus-2.19.0.linux-amd64/tsdb prometheus-2.19.0.linux-amd64/NOTICE prometheus-2.19.0.linux-amd64/prometheus prometheus-2.19.0.linux-amd64/LICENSE prometheus-2.19.0.linux-amd64/prometheus.yml prometheus-2.19.0.linux-amd64/console_libraries/ prometheus-2.19.0.linux-amd64/console_libraries/menu.lib prometheus-2.19.0.linux-amd64/console_libraries/prom.lib prometheus-2.19.0.linux-amd64/promtool takahiro@ubuntu2004server:/var/tmp$ ll total 62684 drwxrwxrwt 6 root root 4096 Jun 11 02:38 ./ drwxr-xr-x 13 root root 4096 Apr 23 07:35 ../ drwxr-xr-x 4 takahiro takahiro 4096 Jun 9 16:37 prometheus-2.19.0.linux-amd64/ -rw-rw-r-- 1 takahiro takahiro 64161216 Jun 9 16:40 prometheus-2.19.0.linux-amd64.tar.gz
そして、Prometheusの設定。
takahiro@ubuntu2004server:/var/tmp$ cd prometheus-2.19.0.linux-amd64/
takahiro@ubuntu2004server:/var/tmp/prometheus-2.19.0.linux-amd64$ ls
console_libraries LICENSE prometheus promtool
consoles NOTICE prometheus.yml tsdb
takahiro@ubuntu2004server:/var/tmp/prometheus-2.19.0.linux-amd64$ cat -n prometheus.yml
1 # my global config
2 global:
3 scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
4 evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
5 # scrape_timeout is set to the global default (10s).
6
7 # Alertmanager configuration
8 alerting:
9 alertmanagers:
10 - static_configs:
11 - targets:
12 # - alertmanager:9093
13
14 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
15 rule_files:
16 # - "first_rules.yml"
17 # - "second_rules.yml"
18
19 # A scrape configuration containing exactly one endpoint to scrape:
20 # Here it's Prometheus itself.
21 scrape_configs:
22 # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
23 - job_name: 'prometheus'
24
25 # metrics_path defaults to '/metrics'
26 # scheme defaults to 'http'.
27
28 static_configs:
29 - targets: ['192.168.1.254:9090']
29行目にあるtargetsに書かれたところが監視対象の指定となり、上記の場合 Prometheus自身が監視対象となっています。
Prometheusの起動。
コンフィグとしてprometheus.ymlを指定して、Prometheusを起動します。
takahiro@ubuntu2004server:/var/tmp/prometheus-2.19.0.linux-amd64$ sudo ./prometheus --config.file=prometheus2.yml [sudo] password for takahiro: level=info ts=2020-06-11T16:41:02.989Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d level=info ts=2020-06-11T16:41:02.989Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.0, branch=HEAD, revision=5d7e3e970602c755855340cb190a972cebdd2ebf)" level=info ts=2020-06-11T16:41:02.989Z caller=main.go:338 build_context="(go=go1.14.4, user=root@d4cf5c7e268d, date=20200609-10:29:59)" level=info ts=2020-06-11T16:41:02.989Z caller=main.go:339 host_details="(Linux 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 ubuntu2004server (none))" level=info ts=2020-06-11T16:41:02.989Z caller=main.go:340 fd_limits="(soft=1024, hard=1048576)" level=info ts=2020-06-11T16:41:02.989Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-06-11T16:41:02.991Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-06-11T16:41:02.991Z caller=main.go:678 msg="Starting TSDB ..." level=info ts=2020-06-11T16:41:02.995Z caller=head.go:645 component=tsdb msg="Replaying WAL and on-disk memory mappable chunks if any, this may take a while" level=info ts=2020-06-11T16:41:02.995Z caller=head.go:706 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-06-11T16:41:02.995Z caller=head.go:709 component=tsdb msg="WAL replay completed" duration=188.168μs level=info ts=2020-06-11T16:41:02.996Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-06-11T16:41:02.996Z caller=main.go:695 msg="TSDB started" level=info ts=2020-06-11T16:41:02.996Z caller=main.go:799 msg="Loading configuration file" filename=prometheus2.yml level=info ts=2020-06-11T16:41:02.997Z caller=main.go:827 msg="Completed loading of configuration file" filename=prometheus2.yml level=info ts=2020-06-11T16:41:02.997Z caller=main.go:646 msg="Server is ready to receive web requests." level=info ts=2020-06-11T16:41:29.645Z caller=compact.go:495 component=tsdb msg="write block" mint=1591843274307 maxt=1591848000000 ulid=01EAGYBH32WM7ZSSSJSCZKJSHF duration=331.694171ms level=info ts=2020-06-11T16:41:29.653Z caller=head.go:792 component=tsdb msg="Head GC completed" duration=2.134251ms level=info ts=2020-06-11T17:00:14.897Z caller=compact.go:495 component=tsdb msg="write block" mint=1591848000000 maxt=1591855200000 ulid=01EAH2VQBFXGCGNM3BQ5YPG319 duration=577.457902ms level=info ts=2020-06-11T17:00:14.925Z caller=head.go:792 component=tsdb msg="Head GC completed" duration=21.590017ms
Prometheusを起動したら、ブラウザからhttp://targetsに記載したIPアドレス:9090/へアクセスします。
Prometheusで取得している情報を確認するため、監視対象(Targets)の情報を表示させます。
EndpointにあるURLをクリックします。
少し長いですが全体は以下にようになり、それぞれPrometheusで取得できているメトリクスと呼ばれる項目です。
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.4475e-05
go_gc_duration_seconds{quantile="0.25"} 2.3584e-05
go_gc_duration_seconds{quantile="0.5"} 4.223e-05
go_gc_duration_seconds{quantile="0.75"} 5.9766e-05
go_gc_duration_seconds{quantile="1"} 0.000192084
go_gc_duration_seconds_sum 0.008880847
go_gc_duration_seconds_count 176
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 40
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.2692912e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 6.37409024e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.509852e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.591842e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 1.0159255019806047e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.594504e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.2692912e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.0738816e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.5550848e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 90290
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.6913152e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6289664e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5918631793169043e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.682132e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 6944
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 219504
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 245760
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.2946768e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.007132e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 819200
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 819200
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.3482496e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 10
# HELP net_conntrack_dialer_conn_attempted_total Total number of connections attempted by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_attempted_total counter
net_conntrack_dialer_conn_attempted_total{dialer_name="alertmanager"} 0
net_conntrack_dialer_conn_attempted_total{dialer_name="default"} 0
net_conntrack_dialer_conn_attempted_total{dialer_name="prometheus"} 1
# HELP net_conntrack_dialer_conn_closed_total Total number of connections closed which originated from the dialer of a given name.
# TYPE net_conntrack_dialer_conn_closed_total counter
net_conntrack_dialer_conn_closed_total{dialer_name="alertmanager"} 0
net_conntrack_dialer_conn_closed_total{dialer_name="default"} 0
net_conntrack_dialer_conn_closed_total{dialer_name="prometheus"} 0
# HELP net_conntrack_dialer_conn_established_total Total number of connections successfully established by the given dialer a given name.
# TYPE net_conntrack_dialer_conn_established_total counter
net_conntrack_dialer_conn_established_total{dialer_name="alertmanager"} 0
net_conntrack_dialer_conn_established_total{dialer_name="default"} 0
net_conntrack_dialer_conn_established_total{dialer_name="prometheus"} 1
# HELP net_conntrack_dialer_conn_failed_total Total number of connections failed to dial by the dialer a given name.
# TYPE net_conntrack_dialer_conn_failed_total counter
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="unknown"} 0
# HELP net_conntrack_listener_conn_accepted_total Total number of connections opened to the listener of a given name.
# TYPE net_conntrack_listener_conn_accepted_total counter
net_conntrack_listener_conn_accepted_total{listener_name="http"} 36
# HELP net_conntrack_listener_conn_closed_total Total number of connections closed that were made to the listener of a given name.
# TYPE net_conntrack_listener_conn_closed_total counter
net_conntrack_listener_conn_closed_total{listener_name="http"} 39
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 18.98
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 23
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.04607744e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.59184326264e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.055744e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP prometheus_api_remote_read_queries The current number of remote read queries being executed or waiting.
# TYPE prometheus_api_remote_read_queries gauge
prometheus_api_remote_read_queries 0
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="HEAD",goversion="go1.14.4",revision="5d7e3e970602c755855340cb190a972cebdd2ebf",version="2.19.0"} 1
# HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.5918432629974217e+09
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
# TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
# HELP prometheus_engine_query_duration_seconds Query timings
# TYPE prometheus_engine_query_duration_seconds summary
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} 8.011e-06
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} 8.72e-06
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} 8.72e-06
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 0.0028019949999999994
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 35
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} 1.6708e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} 3.3095e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} 3.3095e-05
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 0.0009892709999999997
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 35
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.5"} 1.7224e-05
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.9"} 4.8574e-05
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.99"} 4.8574e-05
prometheus_engine_query_duration_seconds_sum{slice="queue_time"} 0.00112571
prometheus_engine_query_duration_seconds_count{slice="queue_time"} 35
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="result_sort"} 6.657300000000001e-05
prometheus_engine_query_duration_seconds_count{slice="result_sort"} 18
# HELP prometheus_engine_query_log_enabled State of the query log.
# TYPE prometheus_engine_query_log_enabled gauge
prometheus_engine_query_log_enabled 0
# HELP prometheus_engine_query_log_failures_total The number of query log failures.
# TYPE prometheus_engine_query_log_failures_total counter
prometheus_engine_query_log_failures_total 0
# HELP prometheus_http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE prometheus_http_request_duration_seconds histogram
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.1"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.2"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.4"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="1"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="3"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="8"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="20"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="60"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="120"} 4
prometheus_http_request_duration_seconds_bucket{handler="/",le="+Inf"} 4
prometheus_http_request_duration_seconds_sum{handler="/"} 0.000155281
prometheus_http_request_duration_seconds_count{handler="/"} 4
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.1"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.2"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.4"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="1"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="3"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="8"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="20"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="60"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="120"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="+Inf"} 9
prometheus_http_request_duration_seconds_sum{handler="/api/v1/label/:name/values"} 0.009479952
prometheus_http_request_duration_seconds_count{handler="/api/v1/label/:name/values"} 9
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="0.1"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="0.2"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="0.4"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="1"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="3"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="8"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="20"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="60"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="120"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/metadata",le="+Inf"} 2
prometheus_http_request_duration_seconds_sum{handler="/api/v1/metadata"} 0.0040098370000000005
prometheus_http_request_duration_seconds_count{handler="/api/v1/metadata"} 2
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.1"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.2"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.4"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="1"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="3"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="8"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="20"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="60"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="120"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="+Inf"} 17
prometheus_http_request_duration_seconds_sum{handler="/api/v1/query"} 0.018056896
prometheus_http_request_duration_seconds_count{handler="/api/v1/query"} 17
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.1"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.2"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="0.4"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="1"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="3"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="8"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="20"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="60"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="120"} 18
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query_range",le="+Inf"} 18
prometheus_http_request_duration_seconds_sum{handler="/api/v1/query_range"} 0.024129813
prometheus_http_request_duration_seconds_count{handler="/api/v1/query_range"} 18
prometheus_http_request_duration_seconds_bucket{handler="/config",le="0.1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="0.2"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="0.4"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="3"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/config",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/config"} 0.001804822
prometheus_http_request_duration_seconds_count{handler="/config"} 1
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.1"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.2"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="0.4"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="1"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="3"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="8"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="20"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="60"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="120"} 6
prometheus_http_request_duration_seconds_bucket{handler="/graph",le="+Inf"} 6
prometheus_http_request_duration_seconds_sum{handler="/graph"} 0.012010925999999998
prometheus_http_request_duration_seconds_count{handler="/graph"} 6
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.1"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.2"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="0.4"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="1"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="3"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="8"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="20"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="60"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="120"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/metrics",le="+Inf"} 1334
prometheus_http_request_duration_seconds_sum{handler="/metrics"} 5.441442855999996
prometheus_http_request_duration_seconds_count{handler="/metrics"} 1334
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.1"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.2"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="0.4"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="1"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="3"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="8"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="20"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="60"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="120"} 46
prometheus_http_request_duration_seconds_bucket{handler="/static/*filepath",le="+Inf"} 46
prometheus_http_request_duration_seconds_sum{handler="/static/*filepath"} 0.07611005
prometheus_http_request_duration_seconds_count{handler="/static/*filepath"} 46
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="0.1"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="0.2"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="0.4"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="1"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="3"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="8"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="20"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="60"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="120"} 3
prometheus_http_request_duration_seconds_bucket{handler="/targets",le="+Inf"} 3
prometheus_http_request_duration_seconds_sum{handler="/targets"} 0.007361082
prometheus_http_request_duration_seconds_count{handler="/targets"} 3
# HELP prometheus_http_requests_total Counter of HTTP requests.
# TYPE prometheus_http_requests_total counter
prometheus_http_requests_total{code="200",handler="/api/v1/label/:name/values"} 9
prometheus_http_requests_total{code="200",handler="/api/v1/metadata"} 2
prometheus_http_requests_total{code="200",handler="/api/v1/query"} 17
prometheus_http_requests_total{code="200",handler="/api/v1/query_range"} 18
prometheus_http_requests_total{code="200",handler="/config"} 1
prometheus_http_requests_total{code="200",handler="/graph"} 6
prometheus_http_requests_total{code="200",handler="/metrics"} 1334
prometheus_http_requests_total{code="200",handler="/static/*filepath"} 46
prometheus_http_requests_total{code="200",handler="/targets"} 3
prometheus_http_requests_total{code="302",handler="/"} 4
# HELP prometheus_http_response_size_bytes Histogram of response size for HTTP requests.
# TYPE prometheus_http_response_size_bytes histogram
prometheus_http_response_size_bytes_bucket{handler="/",le="100"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="1000"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="10000"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="100000"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="1e+06"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="1e+07"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="1e+08"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="1e+09"} 4
prometheus_http_response_size_bytes_bucket{handler="/",le="+Inf"} 4
prometheus_http_response_size_bytes_sum{handler="/"} 116
prometheus_http_response_size_bytes_count{handler="/"} 4
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="100"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1000"} 1
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="10000"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="100000"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+06"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+07"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+08"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="1e+09"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/label/:name/values",le="+Inf"} 9
prometheus_http_response_size_bytes_sum{handler="/api/v1/label/:name/values"} 11515
prometheus_http_response_size_bytes_count{handler="/api/v1/label/:name/values"} 9
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="10000"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="100000"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="1e+06"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="1e+07"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="1e+08"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="1e+09"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/metadata",le="+Inf"} 2
prometheus_http_response_size_bytes_sum{handler="/api/v1/metadata"} 6944
prometheus_http_response_size_bytes_count{handler="/api/v1/metadata"} 2
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1000"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="10000"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="100000"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+06"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+07"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+08"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="1e+09"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query",le="+Inf"} 17
prometheus_http_response_size_bytes_sum{handler="/api/v1/query"} 3088
prometheus_http_response_size_bytes_count{handler="/api/v1/query"} 17
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1000"} 11
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="10000"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="100000"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+06"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+07"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+08"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="1e+09"} 18
prometheus_http_response_size_bytes_bucket{handler="/api/v1/query_range",le="+Inf"} 18
prometheus_http_response_size_bytes_sum{handler="/api/v1/query_range"} 15182
prometheus_http_response_size_bytes_count{handler="/api/v1/query_range"} 18
prometheus_http_response_size_bytes_bucket{handler="/config",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/config",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/config",le="10000"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="100000"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="1e+06"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="1e+07"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="1e+08"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="1e+09"} 1
prometheus_http_response_size_bytes_bucket{handler="/config",le="+Inf"} 1
prometheus_http_response_size_bytes_sum{handler="/config"} 4385
prometheus_http_response_size_bytes_count{handler="/config"} 1
prometheus_http_response_size_bytes_bucket{handler="/graph",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/graph",le="10000"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="100000"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+06"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+07"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+08"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="1e+09"} 6
prometheus_http_response_size_bytes_bucket{handler="/graph",le="+Inf"} 6
prometheus_http_response_size_bytes_sum{handler="/graph"} 35868
prometheus_http_response_size_bytes_count{handler="/graph"} 6
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="10000"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="100000"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+06"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+07"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+08"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="1e+09"} 1334
prometheus_http_response_size_bytes_bucket{handler="/metrics",le="+Inf"} 1334
prometheus_http_response_size_bytes_sum{handler="/metrics"} 1.0220106e+07
prometheus_http_response_size_bytes_count{handler="/metrics"} 1334
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1000"} 6
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="10000"} 21
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="100000"} 38
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+06"} 46
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+07"} 46
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+08"} 46
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="1e+09"} 46
prometheus_http_response_size_bytes_bucket{handler="/static/*filepath",le="+Inf"} 46
prometheus_http_response_size_bytes_sum{handler="/static/*filepath"} 1.987248e+06
prometheus_http_response_size_bytes_count{handler="/static/*filepath"} 46
prometheus_http_response_size_bytes_bucket{handler="/targets",le="100"} 0
prometheus_http_response_size_bytes_bucket{handler="/targets",le="1000"} 0
prometheus_http_response_size_bytes_bucket{handler="/targets",le="10000"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="100000"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="1e+06"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="1e+07"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="1e+08"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="1e+09"} 3
prometheus_http_response_size_bytes_bucket{handler="/targets",le="+Inf"} 3
prometheus_http_response_size_bytes_sum{handler="/targets"} 17590
prometheus_http_response_size_bytes_count{handler="/targets"} 3
# HELP prometheus_notifications_alertmanagers_discovered The number of alertmanagers discovered and active.
# TYPE prometheus_notifications_alertmanagers_discovered gauge
prometheus_notifications_alertmanagers_discovered 0
# HELP prometheus_notifications_dropped_total Total number of alerts dropped due to errors when sending to Alertmanager.
# TYPE prometheus_notifications_dropped_total counter
prometheus_notifications_dropped_total 0
# HELP prometheus_notifications_queue_capacity The capacity of the alert notifications queue.
# TYPE prometheus_notifications_queue_capacity gauge
prometheus_notifications_queue_capacity 10000
# HELP prometheus_notifications_queue_length The number of alert notifications in the queue.
# TYPE prometheus_notifications_queue_length gauge
prometheus_notifications_queue_length 0
# HELP prometheus_remote_storage_highest_timestamp_in_seconds Highest timestamp that has come into the remote storage via the Appender interface, in seconds since epoch.
# TYPE prometheus_remote_storage_highest_timestamp_in_seconds gauge
prometheus_remote_storage_highest_timestamp_in_seconds 1.591863224e+09
# HELP prometheus_remote_storage_samples_in_total Samples in to remote storage, compare to samples out for queue managers.
# TYPE prometheus_remote_storage_samples_in_total counter
prometheus_remote_storage_samples_in_total 743215
# HELP prometheus_remote_storage_string_interner_zero_reference_releases_total The number of times release has been called for strings that are not interned.
# TYPE prometheus_remote_storage_string_interner_zero_reference_releases_total counter
prometheus_remote_storage_string_interner_zero_reference_releases_total 0
# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{quantile="0.5"} NaN
prometheus_rule_evaluation_duration_seconds{quantile="0.9"} NaN
prometheus_rule_evaluation_duration_seconds{quantile="0.99"} NaN
prometheus_rule_evaluation_duration_seconds_sum 0
prometheus_rule_evaluation_duration_seconds_count 0
# HELP prometheus_rule_group_duration_seconds The duration of rule group evaluations.
# TYPE prometheus_rule_group_duration_seconds summary
prometheus_rule_group_duration_seconds{quantile="0.01"} NaN
prometheus_rule_group_duration_seconds{quantile="0.05"} NaN
prometheus_rule_group_duration_seconds{quantile="0.5"} NaN
prometheus_rule_group_duration_seconds{quantile="0.9"} NaN
prometheus_rule_group_duration_seconds{quantile="0.99"} NaN
prometheus_rule_group_duration_seconds_sum 0
prometheus_rule_group_duration_seconds_count 0
# HELP prometheus_rule_group_iterations_missed_total The total number of rule group evaluations missed due to slow rule group evaluation.
# TYPE prometheus_rule_group_iterations_missed_total counter
prometheus_rule_group_iterations_missed_total 1
# HELP prometheus_rule_group_iterations_total The total number of scheduled rule group evaluations, whether executed or missed.
# TYPE prometheus_rule_group_iterations_total counter
prometheus_rule_group_iterations_total 0
# HELP prometheus_sd_consul_rpc_duration_seconds The duration of a Consul RPC call in seconds.
# TYPE prometheus_sd_consul_rpc_duration_seconds summary
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="services",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="services",endpoint="catalog"} 0
# HELP prometheus_sd_consul_rpc_failures_total The number of Consul RPC call failures.
# TYPE prometheus_sd_consul_rpc_failures_total counter
prometheus_sd_consul_rpc_failures_total 0
# HELP prometheus_sd_discovered_targets Current number of discovered targets.
# TYPE prometheus_sd_discovered_targets gauge
prometheus_sd_discovered_targets{config="config-0",name="notify"} 0
prometheus_sd_discovered_targets{config="prometheus",name="scrape"} 1
# HELP prometheus_sd_dns_lookup_failures_total The number of DNS-SD lookup failures.
# TYPE prometheus_sd_dns_lookup_failures_total counter
prometheus_sd_dns_lookup_failures_total 0
# HELP prometheus_sd_dns_lookups_total The number of DNS-SD lookups.
# TYPE prometheus_sd_dns_lookups_total counter
prometheus_sd_dns_lookups_total 0
# HELP prometheus_sd_failed_configs Current number of service discovery configurations that failed to load.
# TYPE prometheus_sd_failed_configs gauge
prometheus_sd_failed_configs{name="notify"} 0
prometheus_sd_failed_configs{name="scrape"} 0
# HELP prometheus_sd_file_read_errors_total The number of File-SD read errors.
# TYPE prometheus_sd_file_read_errors_total counter
prometheus_sd_file_read_errors_total 0
# HELP prometheus_sd_file_scan_duration_seconds The duration of the File-SD scan in seconds.
# TYPE prometheus_sd_file_scan_duration_seconds summary
prometheus_sd_file_scan_duration_seconds{quantile="0.5"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.9"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.99"} NaN
prometheus_sd_file_scan_duration_seconds_sum 0
prometheus_sd_file_scan_duration_seconds_count 0
# HELP prometheus_sd_kubernetes_events_total The number of Kubernetes events handled.
# TYPE prometheus_sd_kubernetes_events_total counter
prometheus_sd_kubernetes_events_total{event="add",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="add",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="add",role="node"} 0
prometheus_sd_kubernetes_events_total{event="add",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="add",role="service"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="node"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="service"} 0
prometheus_sd_kubernetes_events_total{event="update",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="update",role="ingress"} 0
prometheus_sd_kubernetes_events_total{event="update",role="node"} 0
prometheus_sd_kubernetes_events_total{event="update",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="update",role="service"} 0
# HELP prometheus_sd_received_updates_total Total number of update events received from the SD providers.
# TYPE prometheus_sd_received_updates_total counter
prometheus_sd_received_updates_total{name="notify"} 2
prometheus_sd_received_updates_total{name="scrape"} 2
# HELP prometheus_sd_updates_total Total number of update events sent to the SD consumers.
# TYPE prometheus_sd_updates_total counter
prometheus_sd_updates_total{name="notify"} 1
prometheus_sd_updates_total{name="scrape"} 1
# HELP prometheus_target_interval_length_seconds Actual intervals between scrapes.
# TYPE prometheus_target_interval_length_seconds summary
prometheus_target_interval_length_seconds{interval="15s",quantile="0.01"} 14.999888781
prometheus_target_interval_length_seconds{interval="15s",quantile="0.05"} 14.9999813
prometheus_target_interval_length_seconds{interval="15s",quantile="0.5"} 15.000031674
prometheus_target_interval_length_seconds{interval="15s",quantile="0.9"} 15.000053574
prometheus_target_interval_length_seconds{interval="15s",quantile="0.99"} 15.000653256
prometheus_target_interval_length_seconds_sum{interval="15s"} 19950.049036158016
prometheus_target_interval_length_seconds_count{interval="15s"} 1330
# HELP prometheus_target_metadata_cache_bytes The number of bytes that are currently used for storing metric metadata in the cache
# TYPE prometheus_target_metadata_cache_bytes gauge
prometheus_target_metadata_cache_bytes{scrape_job="prometheus"} 8842
# HELP prometheus_target_metadata_cache_entries Total number of metric metadata entries in the cache
# TYPE prometheus_target_metadata_cache_entries gauge
prometheus_target_metadata_cache_entries{scrape_job="prometheus"} 152
# HELP prometheus_target_scrape_pool_reloads_failed_total Total number of failed scrape loop reloads.
# TYPE prometheus_target_scrape_pool_reloads_failed_total counter
prometheus_target_scrape_pool_reloads_failed_total 0
# HELP prometheus_target_scrape_pool_reloads_total Total number of scrape loop reloads.
# TYPE prometheus_target_scrape_pool_reloads_total counter
prometheus_target_scrape_pool_reloads_total 0
# HELP prometheus_target_scrape_pool_sync_total Total number of syncs that were executed on a scrape pool.
# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{scrape_job="prometheus"} 1
# HELP prometheus_target_scrape_pools_failed_total Total number of scrape pool creations that failed.
# TYPE prometheus_target_scrape_pools_failed_total counter
prometheus_target_scrape_pools_failed_total 0
# HELP prometheus_target_scrape_pools_total Total number of scrape pool creation attempts.
# TYPE prometheus_target_scrape_pools_total counter
prometheus_target_scrape_pools_total 1
# HELP prometheus_target_scrapes_cache_flush_forced_total How many times a scrape cache was flushed due to getting big while scrapes are failing.
# TYPE prometheus_target_scrapes_cache_flush_forced_total counter
prometheus_target_scrapes_cache_flush_forced_total 0
# HELP prometheus_target_scrapes_exceeded_sample_limit_total Total number of scrapes that hit the sample limit and were rejected.
# TYPE prometheus_target_scrapes_exceeded_sample_limit_total counter
prometheus_target_scrapes_exceeded_sample_limit_total 0
# HELP prometheus_target_scrapes_sample_duplicate_timestamp_total Total number of samples rejected due to duplicate timestamps but different values
# TYPE prometheus_target_scrapes_sample_duplicate_timestamp_total counter
prometheus_target_scrapes_sample_duplicate_timestamp_total 0
# HELP prometheus_target_scrapes_sample_out_of_bounds_total Total number of samples rejected due to timestamp falling outside of the time bounds
# TYPE prometheus_target_scrapes_sample_out_of_bounds_total counter
prometheus_target_scrapes_sample_out_of_bounds_total 0
# HELP prometheus_target_scrapes_sample_out_of_order_total Total number of samples rejected due to not being out of the expected order
# TYPE prometheus_target_scrapes_sample_out_of_order_total counter
prometheus_target_scrapes_sample_out_of_order_total 0
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="prometheus"} 0.000151235
prometheus_target_sync_length_seconds_count{scrape_job="prometheus"} 1
# HELP prometheus_template_text_expansion_failures_total The total number of template text expansion failures.
# TYPE prometheus_template_text_expansion_failures_total counter
prometheus_template_text_expansion_failures_total 0
# HELP prometheus_template_text_expansions_total The total number of template text expansions.
# TYPE prometheus_template_text_expansions_total counter
prometheus_template_text_expansions_total 0
# HELP prometheus_treecache_watcher_goroutines The current number of watcher goroutines.
# TYPE prometheus_treecache_watcher_goroutines gauge
prometheus_treecache_watcher_goroutines 0
# HELP prometheus_treecache_zookeeper_failures_total The total number of ZooKeeper failures.
# TYPE prometheus_treecache_zookeeper_failures_total counter
prometheus_treecache_zookeeper_failures_total 0
# HELP prometheus_tsdb_blocks_loaded Number of currently loaded data blocks
# TYPE prometheus_tsdb_blocks_loaded gauge
prometheus_tsdb_blocks_loaded 2
# HELP prometheus_tsdb_checkpoint_creations_failed_total Total number of checkpoint creations that failed.
# TYPE prometheus_tsdb_checkpoint_creations_failed_total counter
prometheus_tsdb_checkpoint_creations_failed_total 0
# HELP prometheus_tsdb_checkpoint_creations_total Total number of checkpoint creations attempted.
# TYPE prometheus_tsdb_checkpoint_creations_total counter
prometheus_tsdb_checkpoint_creations_total 0
# HELP prometheus_tsdb_checkpoint_deletions_failed_total Total number of checkpoint deletions that failed.
# TYPE prometheus_tsdb_checkpoint_deletions_failed_total counter
prometheus_tsdb_checkpoint_deletions_failed_total 0
# HELP prometheus_tsdb_checkpoint_deletions_total Total number of checkpoint deletions attempted.
# TYPE prometheus_tsdb_checkpoint_deletions_total counter
prometheus_tsdb_checkpoint_deletions_total 0
# HELP prometheus_tsdb_compaction_chunk_range_seconds Final time range of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_range_seconds histogram
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="100"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="25600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="102400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="409600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1.6384e+06"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6.5536e+06"} 3348
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="2.62144e+07"} 3348
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="+Inf"} 3348
prometheus_tsdb_compaction_chunk_range_seconds_sum 6.587955e+09
prometheus_tsdb_compaction_chunk_range_seconds_count 3348
# HELP prometheus_tsdb_compaction_chunk_samples Final number of samples on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_samples histogram
prometheus_tsdb_compaction_chunk_samples_bucket{le="4"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="6"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="9"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="13.5"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="20.25"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="30.375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="45.5625"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="68.34375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="102.515625"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="153.7734375"} 2344
prometheus_tsdb_compaction_chunk_samples_bucket{le="230.66015625"} 3348
prometheus_tsdb_compaction_chunk_samples_bucket{le="345.990234375"} 3348
prometheus_tsdb_compaction_chunk_samples_bucket{le="+Inf"} 3348
prometheus_tsdb_compaction_chunk_samples_sum 442545
prometheus_tsdb_compaction_chunk_samples_count 3348
# HELP prometheus_tsdb_compaction_chunk_size_bytes Final size of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_size_bytes histogram
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="32"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="48"} 1782
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="72"} 2932
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="108"} 3031
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="162"} 3076
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="243"} 3249
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="364.5"} 3287
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="546.75"} 3321
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="820.125"} 3330
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1230.1875"} 3347
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1845.28125"} 3348
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="2767.921875"} 3348
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="+Inf"} 3348
prometheus_tsdb_compaction_chunk_size_bytes_sum 250216
prometheus_tsdb_compaction_chunk_size_bytes_count 3348
# HELP prometheus_tsdb_compaction_duration_seconds Duration of compaction runs
# TYPE prometheus_tsdb_compaction_duration_seconds histogram
prometheus_tsdb_compaction_duration_seconds_bucket{le="1"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="2"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="4"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="8"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="16"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="32"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="64"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="128"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="256"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="512"} 2
prometheus_tsdb_compaction_duration_seconds_bucket{le="+Inf"} 2
prometheus_tsdb_compaction_duration_seconds_sum 0.909110087
prometheus_tsdb_compaction_duration_seconds_count 2
# HELP prometheus_tsdb_compaction_populating_block Set to 1 when a block is currently being written to the disk.
# TYPE prometheus_tsdb_compaction_populating_block gauge
prometheus_tsdb_compaction_populating_block 0
# HELP prometheus_tsdb_compactions_failed_total Total number of compactions that failed for the partition.
# TYPE prometheus_tsdb_compactions_failed_total counter
prometheus_tsdb_compactions_failed_total 0
# HELP prometheus_tsdb_compactions_skipped_total Total number of skipped compactions due to disabled auto compaction.
# TYPE prometheus_tsdb_compactions_skipped_total counter
prometheus_tsdb_compactions_skipped_total 0
# HELP prometheus_tsdb_compactions_total Total number of compactions that were executed for the partition.
# TYPE prometheus_tsdb_compactions_total counter
prometheus_tsdb_compactions_total 2
# HELP prometheus_tsdb_compactions_triggered_total Total number of triggered compactions for the partition.
# TYPE prometheus_tsdb_compactions_triggered_total counter
prometheus_tsdb_compactions_triggered_total 335
# HELP prometheus_tsdb_head_active_appenders Number of currently active appender transactions
# TYPE prometheus_tsdb_head_active_appenders gauge
prometheus_tsdb_head_active_appenders 0
# HELP prometheus_tsdb_head_chunks Total number of chunks in the head block.
# TYPE prometheus_tsdb_head_chunks gauge
prometheus_tsdb_head_chunks 2810
# HELP prometheus_tsdb_head_chunks_created_total Total number of chunks created in the head
# TYPE prometheus_tsdb_head_chunks_created_total counter
prometheus_tsdb_head_chunks_created_total 6158
# HELP prometheus_tsdb_head_chunks_removed_total Total number of chunks removed in the head
# TYPE prometheus_tsdb_head_chunks_removed_total counter
prometheus_tsdb_head_chunks_removed_total 3348
# HELP prometheus_tsdb_head_gc_duration_seconds Runtime of garbage collection in the head block.
# TYPE prometheus_tsdb_head_gc_duration_seconds summary
prometheus_tsdb_head_gc_duration_seconds_sum 0.023858978
prometheus_tsdb_head_gc_duration_seconds_count 2
# HELP prometheus_tsdb_head_max_time Maximum timestamp of the head block. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_head_max_time gauge
prometheus_tsdb_head_max_time 1.591863224307e+12
# HELP prometheus_tsdb_head_max_time_seconds Maximum timestamp of the head block.
# TYPE prometheus_tsdb_head_max_time_seconds gauge
prometheus_tsdb_head_max_time_seconds 1.591863224e+09
# HELP prometheus_tsdb_head_min_time Minimum time bound of the head block. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_head_min_time gauge
prometheus_tsdb_head_min_time 1.5918552e+12
# HELP prometheus_tsdb_head_min_time_seconds Minimum time bound of the head block.
# TYPE prometheus_tsdb_head_min_time_seconds gauge
prometheus_tsdb_head_min_time_seconds 1.5918552e+09
# HELP prometheus_tsdb_head_samples_appended_total Total number of appended samples.
# TYPE prometheus_tsdb_head_samples_appended_total counter
prometheus_tsdb_head_samples_appended_total 743215
# HELP prometheus_tsdb_head_series Total number of series in the head block.
# TYPE prometheus_tsdb_head_series gauge
prometheus_tsdb_head_series 562
# HELP prometheus_tsdb_head_series_created_total Total number of series created in the head
# TYPE prometheus_tsdb_head_series_created_total counter
prometheus_tsdb_head_series_created_total 562
# HELP prometheus_tsdb_head_series_not_found_total Total number of requests for series that were not found.
# TYPE prometheus_tsdb_head_series_not_found_total counter
prometheus_tsdb_head_series_not_found_total 0
# HELP prometheus_tsdb_head_series_removed_total Total number of series removed in the head
# TYPE prometheus_tsdb_head_series_removed_total counter
prometheus_tsdb_head_series_removed_total 0
# HELP prometheus_tsdb_head_truncations_failed_total Total number of head truncations that failed.
# TYPE prometheus_tsdb_head_truncations_failed_total counter
prometheus_tsdb_head_truncations_failed_total 0
# HELP prometheus_tsdb_head_truncations_total Total number of head truncations attempted.
# TYPE prometheus_tsdb_head_truncations_total counter
prometheus_tsdb_head_truncations_total 2
# HELP prometheus_tsdb_isolation_high_watermark The highest TSDB append ID that has been given out.
# TYPE prometheus_tsdb_isolation_high_watermark gauge
prometheus_tsdb_isolation_high_watermark 2662
# HELP prometheus_tsdb_isolation_low_watermark The lowest TSDB append ID that is still referenced.
# TYPE prometheus_tsdb_isolation_low_watermark gauge
prometheus_tsdb_isolation_low_watermark 2662
# HELP prometheus_tsdb_lowest_timestamp Lowest timestamp value stored in the database. The unit is decided by the library consumer.
# TYPE prometheus_tsdb_lowest_timestamp gauge
prometheus_tsdb_lowest_timestamp 1.591843274307e+12
# HELP prometheus_tsdb_lowest_timestamp_seconds Lowest timestamp value stored in the database.
# TYPE prometheus_tsdb_lowest_timestamp_seconds gauge
prometheus_tsdb_lowest_timestamp_seconds 1.591843274e+09
# HELP prometheus_tsdb_mmap_chunk_corruptions_total Total number of memory-mapped chunk corruptions.
# TYPE prometheus_tsdb_mmap_chunk_corruptions_total counter
prometheus_tsdb_mmap_chunk_corruptions_total 0
# HELP prometheus_tsdb_out_of_bound_samples_total Total number of out of bound samples ingestion failed attempts.
# TYPE prometheus_tsdb_out_of_bound_samples_total counter
prometheus_tsdb_out_of_bound_samples_total 0
# HELP prometheus_tsdb_out_of_order_samples_total Total number of out of order samples ingestion failed attempts.
# TYPE prometheus_tsdb_out_of_order_samples_total counter
prometheus_tsdb_out_of_order_samples_total 0
# HELP prometheus_tsdb_reloads_failures_total Number of times the database failed to reload block data from disk.
# TYPE prometheus_tsdb_reloads_failures_total counter
prometheus_tsdb_reloads_failures_total 0
# HELP prometheus_tsdb_reloads_total Number of times the database reloaded block data from disk.
# TYPE prometheus_tsdb_reloads_total counter
prometheus_tsdb_reloads_total 3
# HELP prometheus_tsdb_retention_limit_bytes Max number of bytes to be retained in the tsdb blocks, configured 0 means disabled
# TYPE prometheus_tsdb_retention_limit_bytes gauge
prometheus_tsdb_retention_limit_bytes 0
# HELP prometheus_tsdb_size_retentions_total The number of times that blocks were deleted because the maximum number of bytes was exceeded.
# TYPE prometheus_tsdb_size_retentions_total counter
prometheus_tsdb_size_retentions_total 0
# HELP prometheus_tsdb_storage_blocks_bytes The number of bytes that are currently used for local storage by all blocks.
# TYPE prometheus_tsdb_storage_blocks_bytes gauge
prometheus_tsdb_storage_blocks_bytes 407886
# HELP prometheus_tsdb_symbol_table_size_bytes Size of symbol table on disk (in bytes)
# TYPE prometheus_tsdb_symbol_table_size_bytes gauge
prometheus_tsdb_symbol_table_size_bytes 176
# HELP prometheus_tsdb_time_retentions_total The number of times that blocks were deleted because the maximum time limit was exceeded.
# TYPE prometheus_tsdb_time_retentions_total counter
prometheus_tsdb_time_retentions_total 0
# HELP prometheus_tsdb_tombstone_cleanup_seconds The time taken to recompact blocks to remove tombstones.
# TYPE prometheus_tsdb_tombstone_cleanup_seconds histogram
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.005"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.01"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.025"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.05"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.1"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.25"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="0.5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="1"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="2.5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="5"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="10"} 0
prometheus_tsdb_tombstone_cleanup_seconds_bucket{le="+Inf"} 0
prometheus_tsdb_tombstone_cleanup_seconds_sum 0
prometheus_tsdb_tombstone_cleanup_seconds_count 0
# HELP prometheus_tsdb_vertical_compactions_total Total number of compactions done on overlapping blocks.
# TYPE prometheus_tsdb_vertical_compactions_total counter
prometheus_tsdb_vertical_compactions_total 0
# HELP prometheus_tsdb_wal_completed_pages_total Total number of completed pages.
# TYPE prometheus_tsdb_wal_completed_pages_total counter
prometheus_tsdb_wal_completed_pages_total 252
# HELP prometheus_tsdb_wal_corruptions_total Total number of WAL corruptions.
# TYPE prometheus_tsdb_wal_corruptions_total counter
prometheus_tsdb_wal_corruptions_total 0
# HELP prometheus_tsdb_wal_fsync_duration_seconds Duration of WAL fsync.
# TYPE prometheus_tsdb_wal_fsync_duration_seconds summary
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.5"} NaN
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.9"} NaN
prometheus_tsdb_wal_fsync_duration_seconds{quantile="0.99"} NaN
prometheus_tsdb_wal_fsync_duration_seconds_sum 0.04735605
prometheus_tsdb_wal_fsync_duration_seconds_count 2
# HELP prometheus_tsdb_wal_page_flushes_total Total number of page flushes.
# TYPE prometheus_tsdb_wal_page_flushes_total counter
prometheus_tsdb_wal_page_flushes_total 2921
# HELP prometheus_tsdb_wal_segment_current WAL segment index that TSDB is currently writing to.
# TYPE prometheus_tsdb_wal_segment_current gauge
prometheus_tsdb_wal_segment_current 2
# HELP prometheus_tsdb_wal_truncate_duration_seconds Duration of WAL truncation.
# TYPE prometheus_tsdb_wal_truncate_duration_seconds summary
prometheus_tsdb_wal_truncate_duration_seconds_sum 0
prometheus_tsdb_wal_truncate_duration_seconds_count 0
# HELP prometheus_tsdb_wal_truncations_failed_total Total number of WAL truncations that failed.
# TYPE prometheus_tsdb_wal_truncations_failed_total counter
prometheus_tsdb_wal_truncations_failed_total 0
# HELP prometheus_tsdb_wal_truncations_total Total number of WAL truncations attempted.
# TYPE prometheus_tsdb_wal_truncations_total counter
prometheus_tsdb_wal_truncations_total 0
# HELP prometheus_tsdb_wal_writes_failed_total Total number of WAL writes that failed.
# TYPE prometheus_tsdb_wal_writes_failed_total counter
prometheus_tsdb_wal_writes_failed_total 0
# HELP prometheus_web_federation_errors_total Total number of errors that occurred while sending federation responses.
# TYPE prometheus_web_federation_errors_total counter
prometheus_web_federation_errors_total 0
# HELP prometheus_web_federation_warnings_total Total number of warnings that occurred while sending federation responses.
# TYPE prometheus_web_federation_warnings_total counter
prometheus_web_federation_warnings_total 0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1334
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
例えば「prometheus_remote_storage_samples_in_total」という項目の情報を知りたい場合、以下の「Expression」とある欄に項目名を書き、「Execute」をクリックします。
そして、「Graph」というタブを選ぶと、グラフで値の変化を見ることができます。
※表示する日時や時間、表示するグラフのデータ間隔などは変更可能。
なお、監視対象の追加を行う場合は、Prometheusの起動時に使用したymlファイルの29行目にTargetsに,(カンマ)で区切り対象を追加したり、
21行目から29行目のスクレイプ設定を追加してから、ctrl+cでprometheusを停止、prometheusの起動をすればOKです。
また、ユーザがログアウトした後もPrometheusを起動させておく場合、
rootユーザ権限でnohupをつけて実行しておけば大丈夫です。
root@ubuntu2004server: nohup ./prometheus --config.file=prometheus2.yml
