安装监控插件

 wget  tar -zxvf elasticsearch_exporter-1.0.4rc1.linux-amd64.tar.gz cd elasticsearch_exporter-1.0.4rc1.linux-amd64/ nohup ./elasticsearch_exporter --web.listen-address ":9109"  --es.uri  &  

启动成功后，可以访问 ，看抓取的信息

什么是Elasticsearch？

Elasticsearch是一个开源的分布式文档存储和搜索引擎，可以近乎实时地存储和检索数据结构。 由Shay Banon开发并于2010年发布，它在很大程度上依赖于Apache Lucene，这是一个用 java 编写的全文搜 索引 擎。 Elasticsearch以结构化JSON文档的形式表示数据，并通过RESTful API和Web客户端访问PHP，Python和 Ruby 等语言的全文搜索。 它也具有弹性，因为它易于水平扩展 – 只需添加更多节点即可分配负载。 如今，许多公司，包括维基百科，eBay，GitHub和Datadog，都使用它来存储，搜索和分析大量数据。

Elasticsearch的元素 在我们开始探索性能指标之前，让我们来看看Elasticsearch的工作原理。在Elasticsearch中，集群由一个或多个节点组成，如下所示：

每个节点都是Elasticsearch的单个运行实例，其elasticsearch.yml配置文件指定它所属的集群（cluster.name）以及它可以是什么类型的节点。也可以通过命令行参数指定配置文件中设置的任何属性（包括集群名称）。上图中的集群由一个专用主节点和五个数据节点组成。

监控图表

指标解析 ##搜索和索引性能

elasticsearch_indices_search_query_total查询总数 吞吐量elasticsearch_indices_search_query_time_seconds查询总时间 性能elasticsearch_indices_search_fetch_total提取总数elasticsearch_indices_search_fetch_time_seconds花费在提取上的总时间##索引请求

elasticsearch_indices_indexing_index_total索引的文件总数elasticsearch_indices_indexing_index_time_seconds_total索引文档总时间elasticsearch_indices_indexing_delete_total索引的文件删除总数elasticsearch_indices_indexing_delete_time_seconds_total索引的文件删除总时间elasticsearch_indices_refresh_total索引刷新总数elasticsearch_indices_refresh_time_seconds_total刷新指数的总时间elasticsearch_indices_flush_total索引刷新总数到磁盘elasticsearch_indices_flush_time_seconds将索引刷新到磁盘上的总时间 累计flush时间##JVM内存和垃圾回收

elasticsearch_jvm_gc_collection_seconds_sumGC run time in seconds垃圾回收时间elasticsearch_jvm_gc_collection_seconds_countCount of JVM GC runs垃圾搜集数elasticsearch_jvm_memory_committed_bytesJVM memory currently committed by area最大使用内存限制elasticsearch_jvm_memory_max_bytes配置的最大jvm值elasticsearch_jvm_memory_pool_max_bytesJVM内存最大池数elasticsearch_jvm_memory_pool_peak_max_bytes最大的JVM内存峰值elasticsearch_jvm_memory_pool_peak_used_bytes池使用的JVM内存峰值elasticsearch_jvm_memory_pool_used_bytes目前使用的JVM内存池elasticsearch_jvm_memory_used_bytesJVM memory currently used by area 内存使用量##集群健康和节点可用性

elasticsearch_cluster_health_status集群状态，green（ 所有的主分片和副本分片都正常运行）、yellow（所有的主分片都正常运行，但不是所有的副本分片都正常运行）red（有主分片没能正常运行）值为1的即为对应状态elasticsearch_cluster_health_number_of_data_nodesnode节点的数量elasticsearch_cluster_health_number_of_in_flight_fetch正在进行的碎片信息请求的数量elasticsearch_cluster_health_number_of_nodes集群内所有的节点elasticsearch_cluster_health_number_of_pending_tasks尚未执行的集群级别更改elasticsearch_cluster_health_initializing_shards正在初始化的分片数elasticsearch_cluster_health_unassigned_shards未分配分片数elasticsearch_cluster_health_active_primary_shards活跃的主分片总数elasticsearch_cluster_health_active_shards活跃的分片总数（包括复制分片）elasticsearch_cluster_health_relocating_shards当前节点正在迁移到其他节点的分片数量，通常为0，集群中有节点新加入或者退出时该值会增加##资源饱和度

elasticsearch_thread_pool_completed_count 线程池 操作完成（bulk、index、search、force_merge）elasticsearch_thread_pool_active_count线程池线程活动（bulk、index、search、force_merge）elasticsearch_thread_pool_largest_count线程池最大线程数（bulk、index、search、force_merge）elasticsearch_thread_pool_queue_count线程池中的排队线程数（bulk、index、search、force_merge）elasticsearch_thread_pool_rejected_count线程池的被拒绝线程数（bulk、index、search、force_merge）elasticsearch_indices_fielddata_memory_size_bytesfielddata缓存的大小（字节）elasticsearch_indices_fielddata_evictions来自fielddata缓存的驱逐次数elasticsearch_indices_filter_cache_evictions来自过滤器缓存的驱逐次数（仅版本2.x）elasticsearch_indices_filter_cache_memory_size_bytes过滤器高速缓存的大小（字节）（仅版本2.x）elasticsearch_cluster_health_number_of_pending_tasks待处理任务数elasticsearch_indices_get_time_seconds

elasticsearch_indices_get_missing_total丢失的文件的GET请求总数elasticsearch_indices_get_missing_time_seconds花费在文档丢失的GET请求上的总时间elasticsearch_indices_get_exists_time_seconds

elasticsearch_indices_get_exists_total

elasticsearch_indices_get_total

##主机级别的系统和网络指标

elasticsearch_process_cpu_percentPercent CPU used by process CPU使用率elasticsearch_filesystem_data_free_bytesFree space on block device in bytes 磁盘可用空间elasticsearch_process_open_files_countOpen file descriptors ES进程打开的文件描述符elasticsearch_transport_rx_packets_totalCount of packets receivedES节点之间网络入流量elasticsearch_transport_tx_packets_totalCount of packets sentES节点之间网络出流量

prometheus配置

 - job_name: 'elasticsearch'      scrape_interval: 60s      scrape_timeout:  30s      metrics_path: "/metrics"      static_configs:      - targets:        - '192.168.50.153:9109'        labels:          service: elasticsearch      relabel_configs:      - source_labels: [__address__]        regex: '(.*)\:9109'        target_label:  'instance'        replacement:   '$1'      - source_labels: [__address__]        regex:         '.*\.(.*)\.lan.*'        target_label:  'environment'        replacement:   '$1'  

之后运行重读prometheus配置命令

 ./reload-prometheus.sh  

grafana模板

报警配置

 groups:- name: elasticsearchStatsAlert  rules:  - alert: Elastic_Cluster_Health_RED    expr: elasticsearch_cluster_health_status{color="red"}==1    for: 1m    labels:      severity: critical    annotations:      summary: "Instance {{ $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster {{ $labels.cluster }}"      description: "Instance {{ $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster {{ $labels.cluster }}."  - alert: Elastic_Cluster_Health_Yellow    expr: elasticsearch_cluster_health_status{color="yellow"}==1    for: 1m    labels:      severity: critical    annotations:      summary: " Instance {{ $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster {{ $labels.cluster }}"      description: "Instance {{ $labels.instance }}: not all primary and replica shards are allocated in elasticsearch cluster {{ $labels.cluster }}."  - alert:  elastic search_JVM_Heap_Too_High    expr: elasticsearch_jvm_memory_used_bytes{area="heap"} / elasticsearch_jvm_memory_max_bytes{area="heap"} > 0.8    for: 1m    labels:      severity: critical    annotations:      summary: "ElasticSearch node {{ $labels.instance }} heap usage is high "      description: "The heap in {{ $labels.instance }} is over 80% for 15m."  - alert: Elasticsearch_health_up    expr: elasticsearch_cluster_health_up !=1    for: 1m    labels:      severity: critical    annotations:      summary: " ElasticSearch node: {{ $labels.instance }} last scrape of the ElasticSearch cluster health failed"      description: "ElasticSearch node: {{ $labels.instance }} last scrape of the ElasticSearch cluster health failed"  - alert: Elasticsearch_Too_Few_Nodes_Running    expr: elasticsearch_cluster_health_number_of_nodes < 12    for: 1m    labels:      severity: critical    annotations:      summary: "There are only {{$value}} < 12 ElasticSearch nodes running "      description: "lasticSearch running on less than 12 nodes(total 14)"  - alert: Elasticsearch_Count_of_JVM_GC_Runs    expr: rate(elasticsearch_jvm_gc_collection_seconds_count{}[5m])>5    for: 1m    labels:      severity: critical    annotations:      summary: "ElasticSearch node {{ $labels.instance }}: Count of JVM GC runs > 5 per sec and has a value of {{ $value }} "      description: "ElasticSearch node {{ $labels.instance }}: Count of JVM GC runs > 5 per sec and has a value of {{ $value }}"  - alert: Elasticsearch_GC_Run_Time    expr: rate(elasticsearch_jvm_gc_collection_seconds_sum[5m])>0.3    for: 1m    labels:      severity: critical    annotations:      summary: " ElasticSearch node {{ $labels.instance }}: GC run time in seconds > 0.3 sec and has a value of {{ $value }}"      description: "ElasticSearch node {{ $labels.instance }}: GC run time in seconds > 0.3 sec and has a value of {{ $value }}"  - alert: Elasticsearch_json_parse_failures    expr: elasticsearch_cluster_health_json_parse_failures>0    for: 1m    labels:      severity: critical    annotations:      summary: " ElasticSearch node {{ $labels.instance }}: json parse failures > 0 and has a value of {{ $value }}"      description: "ElasticSearch node {{ $labels.instance }}: json parse failures > 0 and has a value of {{ $value }}"  - alert: Elasticsearch_breakers_tripped    expr: rate(elasticsearch_breakers_tripped{}[5m])>0    for: 1m    labels:      severity: critical    annotations:      summary: " ElasticSearch node {{ $labels.instance }}: breakers tripped > 0 and has a value of {{ $value }}"      description: "ElasticSearch node {{ $labels.instance }}: breakers tripped > 0 and has a value of {{ $value }}"  - alert: Elasticsearch_health_timed_out    expr: elasticsearch_cluster_health_timed_out>0    for: 1m    labels:      severity: critical    annotations:      summary: " ElasticSearch node {{ $labels.instance }}: Number of cluster health checks timed out > 0 and has a value of {{ $value }}"      description: "ElasticSearch node {{ $labels.instance }}: Number of cluster health checks timed out > 0 and has a value of {{ $value }}"  

elasticsearch-7.x监控

 Since Elasticsearch 7.0.0 : ./bin/elasticsearch-plugin install -b Since Elasticsearch 6.0.0 : ./bin/elasticsearch-plugin install -b On Elasticsearch 5.x.x : ./bin/elasticsearch-plugin install -b On old 2.x.x versions : ./bin/plugin install Do not forget to restart the node after the installation!Note that the plugin needs the following special permissions:java. lang .RuntimePermission accessClassInPackage.sun.miscjava.lang.RuntimePermission accessDeclaredMembersjava.lang.reflect.ReflectPermission suppressAccessChecksIf you have a lot of indices and think this data is irrelevant, you can disable in the main configuration file:prometheus.indices: falseTo disable exporting cluster settings use:prometheus.cluster.settings: false  

 UninstallSince Elasticsearch 6.0.0: ./bin/elasticsearch-plugin remove prometheus-exporterOn Elasticsearch 5.x.x: ./bin/elasticsearch-plugin remove prometheus-exporterOn old 2.x.x versions: ./bin/plugin remove prometheus-exporter  

prometheus配置

 - job_name: elasticsearch  scrape_interval: 10s  metrics_path: "/_prometheus/metrics"  static_configs:  - targets:    - node1:9200    - node2:9200    - node3:9200  

带账号密码：

 - job_name: 'elastic-cluster'      scrape_interval: 10s      metrics_path: '/_prometheus/metrics'      static_configs:      - targets:        - 'node1:9200'        - 'node2:9200'        - 'node3:9200'      basic_auth:        username: 'elastic'        password: 'elastic'  

grafana