[Hindsight] Multiple Syslog Templates

Thota, Madhukar madhukar.thota at LibertyMutual.com
Wed Nov 23 20:38:40 UTC 2016


Hi Trink

Thanks for the input. I created combined grammar as a decoder as you suggested. Here is my first version of decoder:

local syslog = require "lpeg.syslog"

local rfc3164_template  = "<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
local rfc5424_template  = "<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
local rfc3164_grammar   = syslog.build_rsyslog_grammar(rfc3164_template)
local rfc5424_grammar   = syslog.build_rsyslog_grammar(rfc5424_template)


local pairs = pairs
local type  = type

local inject_message = inject_message

local M = {}
setfenv(1, M) -- Remove external access to contain everything in the module

local msg = {}

function decode(data, dh)
    local fields = {}
    if rfc5424_template:match(data) then
         fields = rfc5424_template:match(data)
    else
         fields = rfc3164_grammar:match(data)
    end
    if not fields then return "parse failed" end
    if fields.pri then
        msg.Severity = fields.pri.severity
        fields.syslogfacility = fields.pri.facility
        fields.pri = nil
    else
        msg.Severity = fields.syslogseverity or fields["syslogseverity-text"]
        or fields.syslogpriority or fields["syslogpriority-text"]

        fields.syslogseverity = nil
        fields["syslogseverity-text"] = nil
        fields.syslogpriority = nil
        fields["syslogpriority-text"] = nil
    end

    if fields.syslogtag then
        fields.programname = fields.syslogtag.programname
        msg.Pid = fields.syslogtag.pid
        fields.syslogtag = nil
    end

    msg.Hostname = fields.hostname or fields.source
    fields.hostname = nil
    fields.source = nil

    msg.Payload = fields.msg
    fields.msg = nil

    msg.Fields = fields

    if dh then
        if not msg.Uuid then msg.Uuid = dh.Uuid end
        if not msg.Logger then msg.Logger = dh.Logger end
        if not msg.Hostname then msg.Hostname = dh.Hostname end
        if not msg.Timestamp then msg.Timestamp = dh.Timestamp end
        if not msg.Type then msg.Type = dh.Type end
        if not msg.Payload then msg.Payload = dh.Payload end
        if not msg.EnvVersion then msg.EnvVersion = dh.EnvVersion end
        if not msg.Pid then msg.Pid = dh.Pid end
        if not msg.Severity then msg.Severity = dh.Severity end

        if type(dh.Fields) == "table" then
            for k,v in pairs(dh.Fields) do
                if msg.Fields[k] == nil then
                    msg.Fields[k] = v
                end
            end
        end
    end

    inject_message(msg)
end

return M

I am testing the RFC5424 template with existing syslog decoder and with my new decoder and I see the both decoders are not parsing RFC5424 formar but when I tested using lpeg.trink.com, I was able to parse with the template I am using. Not sure why syslog decoder not parsing RFC5242 messages.

Template: “<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n”

Examples I used:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID at 32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...


Thanks for the help

-M


From: Michael Trinkala <mtrinkala at mozilla.com>
Date: Wednesday, November 23, 2016 at 10:58 AM
To: Madhukar Thota <madhukar.thota at LibertyMutual.com>
Cc: "hindsight at mozilla.org" <hindsight at mozilla.org>
Subject: Re: [Hindsight] Multiple Syslog Templates

Decoder modules are designed to be single instance. See https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-convention<https://protect2.fireeye.com/url?k=695595e9-d528-4c35-a3d8-7358cc207902&u=https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-convention>.  However, you have a few options:
1) run multiple inputs by running the old/new syslog on different interfaces or ports (highly recommended)
2) Use the existing building blocks to: (more dev work and less efficient since you have to run two grammars against one of the inputs)
    a) create your own decoder that produces the combined grammar you are looking for
    b) roll everything you need into a single input plugin
Trink

On Wed, Nov 23, 2016 at 6:30 AM, Thota, Madhukar <madhukar.thota at libertymutual.com<mailto:madhukar.thota at libertymutual.com>> wrote:
Friends,

I am building a syslog listener setup with hindsight which needs to process both RFC5424 and RFC3164 formatted messages. Is it possible to define multiple syslog templates with syslog decoder?

Thanks
M



_______________________________________________
Hindsight mailing list
Hindsight at mozilla.org<mailto:Hindsight at mozilla.org>
https://mail.mozilla.org/listinfo/hindsight

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20161123/a2c19257/attachment-0001.html>


More information about the Hindsight mailing list