[Hindsight] how to output to elasticsearch?

Michael Trinkala mtrinkala at mozilla.com
Tue Apr 11 02:26:53 UTC 2017


There is a pull request to add that functionality but it was never
completed.
https://github.com/mozilla-services/lua_sandbox_extensions/pull/51/files

On Mon, Apr 10, 2017 at 7:04 PM, 黄文彪 <huangwenbiao at 126.com> wrote:

> thx!
> does the elasticsearch_bulk_api.lua has user/password authentication like
> in heka?
>
>
>
>
>
> At 2017-04-11 00:40:01, "Michael Trinkala" <mtrinkala at mozilla.com> wrote:
>
> The docs for the ES output plugin are here:
> https://mozilla-services.github.io/lua_sandbox_extensions/elasticsearch/
> sandboxes/heka/output/elasticsearch_bulk_api.html
>
> You will want to use the mozlog encoder (the docs appear to be a copy and
> paste of the payload encoder, I will update them but it takes no
> configuration beyond the common options) https://github.com/mozilla-
> services/lua_sandbox_extensions/blob/master/elasticsearch/io_modules/
> encoders/elasticsearch/mozlog.lua
>
>
> On Mon, Apr 10, 2017 at 8:57 AM, 黄文彪 <huangwenbiao at 126.com> wrote:
>
>> in heka will like below:
>> ### Output ############################################################
>> ########
>> [ESJsonEncoder]
>> type_name = "%{Type}"
>> es_index_from_timestamp = true
>> index = "%{Type}-%{%Y.%m.%d}"
>>     [ESJsonEncoder.field_mappings]
>>     Timestamp = "@timestamp"
>>     Severity = "level"
>>
>> [elastic_output]
>> type = "ElasticSearchOutput"
>> server = "http://ip:po <http://ip:port>rt"
>> message_matcher = "Type != 'heka.all-report' && Type != 'heka.memstat'"
>> encoder = "ESJsonEncoder"
>> username = "yyyy"
>> password = "xxxx"
>>    [elastic_output.buffering]
>>     max_file_size = 16777216  # 16MB
>>     max_buffer_size = 2147483648 <(214)%20748-3648> # changed 2048MB
>>     full_action = "block" #shutdown,drop,block
>>
>> and in hindsight,i have output/input/0.log with these contents:
>> lsb_heka_cat input/0.log
>> :Uuid: e4ea9a56-7fe4-4611-a7b9-cc6d0e88c67f
>> :Timestamp: 2017-04-10T14:47:48.000000000Z
>> :Type: da
>> :Logger: input.input_da
>> :Severity: 7
>> :Payload: <nil>
>> :EnvVersion: <nil>
>> :Pid: <nil>
>> :Hostname: hzxs-montest-1
>> :Fields:
>>     | name: optime type: 3 representation: <nil> value: 3
>>     | name: platform type: 0 representation: <nil> value: ICM
>>     | name: to type: 0 representation: <nil> value: wendy.s.lo at qisda.com
>>     | name: channel type: 0 representation: <nil> value: remote
>>     | name: proxy type: 0 representation: <nil> value:
>>     | name: state type: 0 representation: <nil> value: sent
>>     | name: delay type: 3 representation: <nil> value: 3
>>     | name: da_msg_id type: 0 representation: <nil> value:
>> AQAAfwDX35elAuZY52l6Aw--.55707S2
>>     | name: from type: 0 representation: <nil> value: eva.mao at quectel.com
>>     | name: subject type: 0 representation: <nil> value: some subjects
>>     | name: failed_msg type: 0 representation: <nil> value:
>>     | name: tid type: 3 representation: <nil> value: 2179852032
>> <(217)%20985-2032>
>>     | name: size type: 3 representation: <nil> value: 305452
>>
>> i have found https://mozilla-services.github.io/lua_sandbox_
>> extensions/elasticsearch/index.html,
>> but there is no readme to show me how to configure it in hindsight.
>> please help!
>>
>>
>>
>>
>> _______________________________________________
>> Hindsight mailing list
>> Hindsight at mozilla.org
>> https://mail.mozilla.org/listinfo/hindsight
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20170410/7a6d6aa2/attachment-0001.html>


More information about the Hindsight mailing list