[Hindsight] Multiple Syslog Templates

Michael Trinkala mtrinkala at mozilla.com
Thu Nov 24 00:36:09 UTC 2016


The 'if' is broken, it is using the 5424 template instead of the grammar
(so it is doing a string match with the data as the pattern) You can
eliminate it entirely.
local grammar = rfc5424_grammar + rfc3164_grammar
local fields = grammar:match(data)

Trink

On Wed, Nov 23, 2016 at 12:38 PM, Thota, Madhukar <
madhukar.thota at libertymutual.com> wrote:

> Hi Trink
>
>
>
> Thanks for the input. I created combined grammar as a decoder as you
> suggested. Here is my first version of decoder:
>
>
>
> local syslog = require "lpeg.syslog"
>
>
>
> local rfc3164_template  = "<%PRI%>%TIMESTAMP% %HOSTNAME%
> %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
>
> local rfc5424_template  = "<%PRI%>%PROTOCOL-VERSION%
> %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID%
> %STRUCTURED-DATA% %msg%\n"
>
> local rfc3164_grammar   = syslog.build_rsyslog_grammar(rfc3164_template)
>
> local rfc5424_grammar   = syslog.build_rsyslog_grammar(rfc5424_template)
>
>
>
>
>
> local pairs = pairs
>
> local type  = type
>
>
>
> local inject_message = inject_message
>
>
>
> local M = {}
>
> setfenv(1, M) -- Remove external access to contain everything in the module
>
>
>
> local msg = {}
>
>
>
> function decode(data, dh)
>
>     local fields = {}
>
>     if rfc5424_template:match(data) then
>
>          fields = rfc5424_template:match(data)
>
>     else
>
>          fields = rfc3164_grammar:match(data)
>
>     end
>
>     if not fields then return "parse failed" end
>
>     if fields.pri then
>
>         msg.Severity = fields.pri.severity
>
>         fields.syslogfacility = fields.pri.facility
>
>         fields.pri = nil
>
>     else
>
>         msg.Severity = fields.syslogseverity or
> fields["syslogseverity-text"]
>
>         or fields.syslogpriority or fields["syslogpriority-text"]
>
>
>
>         fields.syslogseverity = nil
>
>         fields["syslogseverity-text"] = nil
>
>         fields.syslogpriority = nil
>
>         fields["syslogpriority-text"] = nil
>
>     end
>
>
>
>     if fields.syslogtag then
>
>         fields.programname = fields.syslogtag.programname
>
>         msg.Pid = fields.syslogtag.pid
>
>         fields.syslogtag = nil
>
>     end
>
>
>
>     msg.Hostname = fields.hostname or fields.source
>
>     fields.hostname = nil
>
>     fields.source = nil
>
>
>
>     msg.Payload = fields.msg
>
>     fields.msg = nil
>
>
>
>     msg.Fields = fields
>
>
>
>     if dh then
>
>         if not msg.Uuid then msg.Uuid = dh.Uuid end
>
>         if not msg.Logger then msg.Logger = dh.Logger end
>
>         if not msg.Hostname then msg.Hostname = dh.Hostname end
>
>         if not msg.Timestamp then msg.Timestamp = dh.Timestamp end
>
>         if not msg.Type then msg.Type = dh.Type end
>
>         if not msg.Payload then msg.Payload = dh.Payload end
>
>         if not msg.EnvVersion then msg.EnvVersion = dh.EnvVersion end
>
>         if not msg.Pid then msg.Pid = dh.Pid end
>
>         if not msg.Severity then msg.Severity = dh.Severity end
>
>
>
>         if type(dh.Fields) == "table" then
>
>             for k,v in pairs(dh.Fields) do
>
>                 if msg.Fields[k] == nil then
>
>                     msg.Fields[k] = v
>
>                 end
>
>             end
>
>         end
>
>     end
>
>
>
>     inject_message(msg)
>
> end
>
>
>
> return M
>
>
>
> I am testing the RFC5424 template with existing syslog decoder and with my
> new decoder and I see the both decoders are not parsing RFC5424 formar but
> when I tested using lpeg.trink.com, I was able to parse with the template
> I am using. Not sure why syslog decoder not parsing RFC5242 messages.
>
>
>
> Template: “<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n”
>
>
>
> Examples I used:
>
>
>
> <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su
> root' failed for lonvick on /dev/pts/8
>
>
>
> <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47
> [exampleSDID at 32473 iut="3" eventSource="Application" eventID="1011"]
> BOMAn application event log entry...
>
>
>
>
>
> Thanks for the help
>
>
>
> -M
>
>
>
>
>
> *From: *Michael Trinkala <mtrinkala at mozilla.com>
> *Date: *Wednesday, November 23, 2016 at 10:58 AM
> *To: *Madhukar Thota <madhukar.thota at LibertyMutual.com>
> *Cc: *"hindsight at mozilla.org" <hindsight at mozilla.org>
> *Subject: *Re: [Hindsight] Multiple Syslog Templates
>
>
>
> Decoder modules are designed to be single instance. See
> https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-
> convention
> <https://protect2.fireeye.com/url?k=695595e9-d528-4c35-a3d8-7358cc207902&u=https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-convention>.
> However, you have a few options:
> 1) run multiple inputs by running the old/new syslog on different
> interfaces or ports (highly recommended)
>
> 2) Use the existing building blocks to: (more dev work and less efficient
> since you have to run two grammars against one of the inputs)
>
>     a) create your own decoder that produces the combined grammar you are
> looking for
>     b) roll everything you need into a single input plugin
>
> Trink
>
>
>
> On Wed, Nov 23, 2016 at 6:30 AM, Thota, Madhukar <
> madhukar.thota at libertymutual.com> wrote:
>
> Friends,
>
>
>
> I am building a syslog listener setup with hindsight which needs to
> process both RFC5424 and RFC3164 formatted messages. Is it possible to
> define multiple syslog templates with syslog decoder?
>
>
>
> Thanks
>
> M
>
>
>
>
>
>
> _______________________________________________
> Hindsight mailing list
> Hindsight at mozilla.org
> https://mail.mozilla.org/listinfo/hindsight
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20161123/571e8512/attachment-0001.html>


More information about the Hindsight mailing list