[Hindsight] how to output to elasticsearch?

Michael Trinkala mtrinkala at mozilla.com
Mon Apr 10 16:40:01 UTC 2017


The docs for the ES output plugin are here:
https://mozilla-services.github.io/lua_sandbox_extensions/elasticsearch/sandboxes/heka/output/elasticsearch_bulk_api.html

You will want to use the mozlog encoder (the docs appear to be a copy and
paste of the payload encoder, I will update them but it takes no
configuration beyond the common options)
https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/elasticsearch/io_modules/encoders/elasticsearch/mozlog.lua


On Mon, Apr 10, 2017 at 8:57 AM, 黄文彪 <huangwenbiao at 126.com> wrote:

> in heka will like below:
> ### Output ############################################################
> ########
> [ESJsonEncoder]
> type_name = "%{Type}"
> es_index_from_timestamp = true
> index = "%{Type}-%{%Y.%m.%d}"
>     [ESJsonEncoder.field_mappings]
>     Timestamp = "@timestamp"
>     Severity = "level"
>
> [elastic_output]
> type = "ElasticSearchOutput"
> server = "http://ip:po <http://ip:port>rt"
> message_matcher = "Type != 'heka.all-report' && Type != 'heka.memstat'"
> encoder = "ESJsonEncoder"
> username = "yyyy"
> password = "xxxx"
>    [elastic_output.buffering]
>     max_file_size = 16777216  # 16MB
>     max_buffer_size = 2147483648 <(214)%20748-3648> # changed 2048MB
>     full_action = "block" #shutdown,drop,block
>
> and in hindsight,i have output/input/0.log with these contents:
> lsb_heka_cat input/0.log
> :Uuid: e4ea9a56-7fe4-4611-a7b9-cc6d0e88c67f
> :Timestamp: 2017-04-10T14:47:48.000000000Z
> :Type: da
> :Logger: input.input_da
> :Severity: 7
> :Payload: <nil>
> :EnvVersion: <nil>
> :Pid: <nil>
> :Hostname: hzxs-montest-1
> :Fields:
>     | name: optime type: 3 representation: <nil> value: 3
>     | name: platform type: 0 representation: <nil> value: ICM
>     | name: to type: 0 representation: <nil> value: wendy.s.lo at qisda.com
>     | name: channel type: 0 representation: <nil> value: remote
>     | name: proxy type: 0 representation: <nil> value:
>     | name: state type: 0 representation: <nil> value: sent
>     | name: delay type: 3 representation: <nil> value: 3
>     | name: da_msg_id type: 0 representation: <nil> value:
> AQAAfwDX35elAuZY52l6Aw--.55707S2
>     | name: from type: 0 representation: <nil> value: eva.mao at quectel.com
>     | name: subject type: 0 representation: <nil> value: some subjects
>     | name: failed_msg type: 0 representation: <nil> value:
>     | name: tid type: 3 representation: <nil> value: 2179852032
> <(217)%20985-2032>
>     | name: size type: 3 representation: <nil> value: 305452
>
> i have found https://mozilla-services.github.io/lua_sandbox_extensions/
> elasticsearch/index.html,
> but there is no readme to show me how to configure it in hindsight.
> please help!
>
>
>
>
> _______________________________________________
> Hindsight mailing list
> Hindsight at mozilla.org
> https://mail.mozilla.org/listinfo/hindsight
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20170410/a84583ca/attachment.html>


More information about the Hindsight mailing list