python+elasticsearch集群

官方文檔：https://elasticsearch-py.readthedocs.io/en/master/
介紹
python提供了操作ElasticSearch 接口,因此要用python來操作ElasticSearch,首先要安裝python的ElasticSearch包,用命令 pip install elasticsearch安裝或下載安裝：https://pypi.python.org/pypi/elasticsearch/6.3.1
基本使用(本人是在django項目中使用)

1.配置setteing

ES_SETTINGS = {
"hosts": [{"host": '10.35.206.28', "port": 9200}, {"host": '10.35.206.27', "port": 9200},
{"host": '10.35.206.30', "port": 9200}],
"index": record',
"type": 'prometheus'
}

hosts：The configuration is mostly multiple implementation clusters

2.Elasticsearch客戶端創建

from django.conf import settings
from elasticsearch import Elasticsearch
es = Elasticsearch(hosts=settings.ES_SETTINGS['hosts'])

3.使用helpers插入es

from elasticsearch import helpers
data = {
"_index": "record",
"_type": "prometheus",
"_id": str(instance.id) + "prometheus",
"_score": "",
"_source": {
"recordId": instance.id,
"ipPort": ip + ":9100",
"proId": proId,
"proName": proName,
"msg": json.dumps(res.json()),
"recordTrans": instance.tran_info,
"time": time.time(),
}
}
a = helpers.bulk(es, actions=[data])

4.使用search查詢(You can only query at most once10000)

def prometheus_search_new(recordId, ipPort, proName):
query_body = {
"query": {
"bool": {
"must": [
{
"term": {
"recordId": recordId
}
},
{
"term": {
"ipPort.keyword": ipPort
}
},
{
"term": {
"proName.keyword": proName
}
},
]
},
},
"sort": {"time": {"order": "desc"}},
"from": 0,
"size": 10000
}
try:
result = es.search(index=settings.ES_PROMETHEUS_SETTINGS['index'],
doc_type=settings.ES_PROMETHEUS_SETTINGS['type'],
body=query_body)
hits = result['hits']['hits']
except Exception as e:
hits = []
return hits

5.使用helpers查詢（Paging next write can be achieved）

es_result = helpers.scan(
client=es,
query=query_body,
scroll='5m',
index=settings.ES_SETTINGS ['index'],
doc_type=settings.ES_SETTINGS ['type'],
timeout='1m',
preserve_order=True
)

6.返回結果

{
"took": 13,
"timed_out": false,
"_shards": {
"total": 65,
"successful": 65,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 152883,
"max_score": null,
"hits": [
{
"_index": "new_stat",
"_type": "stat",
"_id": "q1jWAG0BhYz49-yfOJms",
"_score": null,
"_source": {
"_index": "record",
"_type": "prometheus",
"_id": "7131prometheus",
"_version": 1,
"_score": 1,
"_source": {
"ipPort": "10.13.31.212:9100",
"recordTrans": "[{'enable': True, 'url': u'http://10.13.31.212:8008/v1/music/process', 'tran_name': u'\u53ee\u5f53\u5f71\u89c6', 'virtualNum': 100, 'method': u'POST', 'tran_id': 22355}]",
"proName": "cup_sys",
"recordId": 7131,
"time": 1551510988.541931,
"proId": 11,
"msg": "{"status": "success", "data": {"resultType": "matrix", "result": []}}"
}
}
]
}
}

另外

elasticsearch通過from~sizeWhen querying pagination,會受到max_result_window影響,It can be applied according to the actual situation,設置：

curl -XPUT "http://ip:port/index/_settings" -d '{"index": {"max_result_window": 1000000}}'