<div dir="ltr"><div>It seems like the output is not being flushed to disk. When it is running for the first time take a look at output/input/0.log. I suspect the file size will show 0 until HS is shutdown and the file is closed (or you send more data through). If you see the file update and HS is still not picking it up that will also be informative. I am not seeing the problem on any of my Linux boxes so I suspect it is Docker related.<br><br></div>Trink<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 7, 2016 at 4:10 PM, Thota, Madhukar <span dir="ltr"><<a href="mailto:madhukar.thota@libertymutual.com" target="_blank">madhukar.thota@libertymutual.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" link="blue" vlink="purple" lang="EN-US">
<div class="m_4749411645183650262WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Hi trink,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I am running hindsight as Docker image. When I ran the container first time, I don’t see any output, but if I stop and start the container, I see all messages from previous run.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><b>hindsight.cp</b> file:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">_G['analysis->output.heka_<wbr>debug'] = '0:0'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">_G['analysis'] = '0:0'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">_G['input->analysis0'] = '0:0'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">_G['input->output.heka_debug'] = '0:0'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">_G['input'] = '0:12600'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:Calibri">hindsight.tsv<u></u><u></u></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Plugin Inject Message Count Inject Message Bytes Process Message Count<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">input.tcp 200 11600 0 0 161647 224615 0 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">output.heka_debug 0 0 0 0 42876 49050 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:Calibri">hindsight stdout:<u></u><u></u></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1478563528815493029 [info] <wbr>hindsight starting<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1478563528825646642 [info] <wbr>input.tcp starting<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1478563528828448499 [info] <wbr>analysis_plugins starting <wbr>thread: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1478563528837132261 [info] <wbr>output.heka_debug starting<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Michael Trinkala <<a href="mailto:mtrinkala@mozilla.com" target="_blank">mtrinkala@mozilla.com</a>><br>
<b>Date: </b>Monday, November 7, 2016 at 9:20 AM<br>
<b>To: </b>Madhukar Thota <madhukar.thota@LibertyMutual.<wbr>com><br>
<b>Cc: </b>"<a href="mailto:hindsight@mozilla.org" target="_blank">hindsight@mozilla.org</a>" <<a href="mailto:hindsight@mozilla.org" target="_blank">hindsight@mozilla.org</a>><br>
<b>Subject: </b>Re: [Hindsight] No output<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I would need to see the Hindsight log output and the hindsight.cp file. Best guess: If you have been testing and manually deleting stuff out of the output directory make sure you also delete the hindsight.cp
file.<u></u><u></u></p>
</div>
<p class="MsoNormal">Trink<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Sun, Nov 6, 2016 at 8:30 PM, Thota, Madhukar <<a href="mailto:madhukar.thota@libertymutual.com" target="_blank">madhukar.thota@libertymutual.<wbr>com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am playing with hindsight to test simple flow using tcp input and heka_debug.lua in output. I can see my messages are coming to tcp input and but
it’s not going to output.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am using Master branch for hindsight, lua_sandbox and extensions.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Here is my flow:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">hindsight.cfg:</span></b><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">output_path = "output"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">output_size = 1024 * 1024 * 1024</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sandbox_load_path = nil -- disable dynamic loading</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sandbox_run_path = "/etc/hindsight/run"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sandbox_install_path = "/usr/share/luasandbox/<wbr>sandboxes/heka"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">analysis_lua_path = "/usr/lib/luasandbox/modules/?<wbr>.lua"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">analysis_lua_cpath = "/usr/lib/luasandbox/modules/?<wbr>.so"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">io_lua_path = analysis_lua_path .. ";/usr/lib/luasandbox/io_<wbr>modules/?.lua"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">io_lua_cpath = analysis_lua_cpath .. ";/usr/lib/luasandbox/io_<wbr>modules/?.so"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">input_defaults = {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">}</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">analysis_defaults = {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">}</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">output_defaults = {</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">}</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">Input: tcp.cfg</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">filename = "tcp.lua"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">instruction_limit = 0</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">address = "0.0.0.0"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">port = 5555</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">default_headers = nil</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">decoder_module = "decoders.payload"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">Ouptut: heka_debug.cfg</span></b><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">filename = "heka_debug.lua"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">message_matcher = "TRUE"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In hindsight.tsv file I can four message but zero output.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Plugin Inject Message Count Inject Message Bytes Process Message Count Process Message Failures Current Memory
Max Memory Max Output Max Instructions Message Matcher Avg (ns) Message Matcher SD (ns) Process Message Avg (ns) Process Message SD (ns) Timer Event Avg (ns) Timer Event SD (ns)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">input.tcp
<b>4</b> 228 0 0 116095 162396 0 0 0 0 0 0 0 0</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">output.heka_debug 0 0 0 0 42836 49050 0 72 0
0 0 0 8566 1028</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any help is appreciated.</span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> </span></b><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
Hindsight mailing list<br>
<a href="mailto:Hindsight@mozilla.org" target="_blank">Hindsight@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/hindsight" target="_blank">https://mail.mozilla.org/<wbr>listinfo/hindsight</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Hindsight mailing list<br>
<a href="mailto:Hindsight@mozilla.org">Hindsight@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/hindsight" rel="noreferrer" target="_blank">https://mail.mozilla.org/<wbr>listinfo/hindsight</a><br>
<br></blockquote></div><br></div>