[Hindsight] Multiple Syslog Templates

Thota, Madhukar madhukar.thota at LibertyMutual.com
Wed Nov 23 20:38:40 UTC 2016

Hi Trink

Thanks for the input. I created combined grammar as a decoder as you suggested. Here is my first version of decoder:

local syslog = require "lpeg.syslog"

local rfc3164_template  = "<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
local rfc5424_template  = "<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
local rfc3164_grammar   = syslog.build_rsyslog_grammar(rfc3164_template)
local rfc5424_grammar   = syslog.build_rsyslog_grammar(rfc5424_template)

local pairs = pairs
local type  = type

local inject_message = inject_message

local M = {}
setfenv(1, M) -- Remove external access to contain everything in the module

local msg = {}

function decode(data, dh)
    local fields = {}
    if rfc5424_template:match(data) then
         fields = rfc5424_template:match(data)
         fields = rfc3164_grammar:match(data)
    if not fields then return "parse failed" end
    if fields.pri then
        msg.Severity = fields.pri.severity
        fields.syslogfacility = fields.pri.facility
        fields.pri = nil
        msg.Severity = fields.syslogseverity or fields["syslogseverity-text"]
        or fields.syslogpriority or fields["syslogpriority-text"]

        fields.syslogseverity = nil
        fields["syslogseverity-text"] = nil
        fields.syslogpriority = nil
        fields["syslogpriority-text"] = nil

    if fields.syslogtag then
        fields.programname = fields.syslogtag.programname
        msg.Pid = fields.syslogtag.pid
        fields.syslogtag = nil

    msg.Hostname = fields.hostname or fields.source
    fields.hostname = nil
    fields.source = nil

    msg.Payload = fields.msg
    fields.msg = nil

    msg.Fields = fields

    if dh then
        if not msg.Uuid then msg.Uuid = dh.Uuid end
        if not msg.Logger then msg.Logger = dh.Logger end
        if not msg.Hostname then msg.Hostname = dh.Hostname end
        if not msg.Timestamp then msg.Timestamp = dh.Timestamp end
        if not msg.Type then msg.Type = dh.Type end
        if not msg.Payload then msg.Payload = dh.Payload end
        if not msg.EnvVersion then msg.EnvVersion = dh.EnvVersion end
        if not msg.Pid then msg.Pid = dh.Pid end
        if not msg.Severity then msg.Severity = dh.Severity end

        if type(dh.Fields) == "table" then
            for k,v in pairs(dh.Fields) do
                if msg.Fields[k] == nil then
                    msg.Fields[k] = v


return M

I am testing the RFC5424 template with existing syslog decoder and with my new decoder and I see the both decoders are not parsing RFC5424 formar but when I tested using lpeg.trink.com, I was able to parse with the template I am using. Not sure why syslog decoder not parsing RFC5242 messages.


Examples I used:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID at 32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

Thanks for the help


From: Michael Trinkala <mtrinkala at mozilla.com>
Date: Wednesday, November 23, 2016 at 10:58 AM
To: Madhukar Thota <madhukar.thota at LibertyMutual.com>
Cc: "hindsight at mozilla.org" <hindsight at mozilla.org>
Subject: Re: [Hindsight] Multiple Syslog Templates

Decoder modules are designed to be single instance. See https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-convention<https://protect2.fireeye.com/url?k=695595e9-d528-4c35-a3d8-7358cc207902&u=https://mozilla-services.github.io/lua_sandbox_extensions/#decoder-api-convention>.  However, you have a few options:
1) run multiple inputs by running the old/new syslog on different interfaces or ports (highly recommended)
2) Use the existing building blocks to: (more dev work and less efficient since you have to run two grammars against one of the inputs)
    a) create your own decoder that produces the combined grammar you are looking for
    b) roll everything you need into a single input plugin

On Wed, Nov 23, 2016 at 6:30 AM, Thota, Madhukar <madhukar.thota at libertymutual.com<mailto:madhukar.thota at libertymutual.com>> wrote:

I am building a syslog listener setup with hindsight which needs to process both RFC5424 and RFC3164 formatted messages. Is it possible to define multiple syslog templates with syslog decoder?


Hindsight mailing list
Hindsight at mozilla.org<mailto:Hindsight at mozilla.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20161123/a2c19257/attachment-0001.html>

More information about the Hindsight mailing list