1、EsRejectedExecutionException異常示例

java.util.concurrent.ExecutionException: RemoteTransportException[[node-client10][10.93.21.21:9300][indices:data/write/update]]; nested: RemoteTransportException[[node-client
09][10.93.18.35:9300][indices:data/write/update[s]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.netty.MessageChannelHandler$Reque
[email protected] on EsThreadPoolExecutor[index, queue capacity 
 
= 200, [email protected]7f6431[Running, pool size = 12, active
threads = 12, queued tasks = 200, completed tasks = 15656095]]];

2、EsRejectedExecutionException異常解釋

EsRejectedExecutionException異常，從字面意思上看是ES拒絕執行請求。這個異常的觸發場景如下。

使用Elasticsearch的時候，在並發查詢量大的情況下，訪問流量超過了集群中單個Elasticsearch實例的處理能力，Elasticsearch服務端會觸發保護性的機制，拒絕執行新的訪問，並且拋出EsRejectedExecutionException異常。

這個保護機制與異常觸發是由Elasticsearch API實現中的thread pool與配套的queue決定的。

在示例中，Elasticsearch為index操作分配的線程池，pool size=12， queue capacity=200，當12個線程處理不過來，並且隊列中緩沖的tasks超過200個，那麽新的task就會被簡單的丟棄掉，並且拋出EsRejectedExecutionException異常。

官方的詳細解釋鏈接在這裏：

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html

這裏是幾個重要線程池的默認配置，其中很多配置都與processors（cpu core）有關。

1）processors

processors指的是cpu的core數，這個核數可以被自動的探知並且在線程池的配置中自動引入，processors不必在elasitcsearch.yml中配置，當然你可以重寫配置。

重寫方法:

 processor: 8

若在一臺機器上部署多個Elasticsearch node，可以將cpu cores平均分配給不同的node，例如2nodes部署在12個core的機器上，可以配置processors = 6。

2）線程池與隊列

一個Elasticsearch節點會有多個線程池，但重要的是下面四個：
索引（index）：主要是索引數據和刪除數據操作（默認是cached類型）
搜索（search）：主要是獲取，統計和搜索操作（默認是cached類型）
批量操作（bulk）：主要是對索引的批量操作（默認是cached類型）
更新（refresh）：主要是更新操作（默認是cached類型）

generic
For generic operations (e.g., background node discovery). Thread pool type is scaling.
index
For index/delete operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.
search
For count/search/suggest operations. Thread pool type is fixed with a size of int((# of available_processors * 3) / 2) + 1, queue_size of 1000.
get
For get operations. Thread pool type is fixed with a size of # of available processors, queue_size of 1000.
bulk
For bulk operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.
snapshot
For snapshot/restore operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(5, (# of available processors)/2).
warmer
For segment warm-up operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(5, (# of available processors)/2).
refresh
For refresh operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(10, (# of available processors)/2).
listener
Mainly for java client executing of action when listener threaded is set to true. Thread pool type is scaling with a default max of min(10, (# of available processors)/2).

3）線程池類型

fixed

The fixed thread pool holds a fixed size of threads to handle the requests with a queue (optionally bounded) for pending requests that have no threads to service them.

The size parameter controls the number of threads, and defaults to the number of cores times 5.

The queue_size allows to control the size of the queue of pending requests that have no threads to execute them. By default, it is set to -1 which means its unbounded. When a request comes in and the queue is full, it will abort the request.

fixed線程池保持固定個數的線程來處理請求隊列。

size參數設置線程的個數，默認設置是cpu核心數的5倍。

queue_size可以控制待處理請求隊列的大小。默認是設置為-1，意味著無限制。當一個請求到來但隊列滿了的時候，reject_policy參數可以控制它的行為。默認是abort，會使那個請求失敗。設置成caller會使該請求在io線程中執行。

threadpool:   
    index:   
        type: fixed   
        size: 30   
        queue: 1000   
        reject_policy: caller  

scaling

The scaling thread pool holds a dynamic number of threads. This number is proportional to the workload and varies between the value of the core and max parameters.

The keep_alive parameter determines how long a thread should be kept around in the thread pool without it doing any work.

scaling線程池保持著動態數量的線程。在core和max數量之間的動態變化，keep_alive配置的時間，維持了線程長時間未被調用。

thread_pool:
    warmer:
        core: 1
        max: 8
        keep_alive: 2m

blocking

在不同的版本有不同的類型。blocking類型的線程池特征是

blocking線程池允許設置一個最小值（min，默認為1）和線程池大小（size，默認為cpu核心數的5倍）。它也有一個等待隊列，隊列的大小（queue_size ）默認是1000，當這隊列滿了的時候。它會根據定好的等待時間（wait_time，默認是60秒）來調用io線程，如果沒有執行就會報錯。

threadpool:   
    index:   
        type: blocking   
        min: 1   
        size: 30   
        wait_time: 30s  

EsRejectedExecutionException排錯與線程池類型