[Hindsight] Starting with hindsight- confused about whats happening

Michael Trinkala mtrinkala at mozilla.com
Tue Nov 15 15:59:56 UTC 2016


The benchmark reads from a Nginx log file with the following log_format
`$remote_addr $msec "$request" $status $body_bytes_sent "$http_referer"
"$http_user_agent" "$http_x_forwarded_for"` so you would actually need to
configure input_test.cfg to control the filename (not single.cfg) but since
you didn't get a file open error I am assuming that is what you meant.  If
the file you provided is not in that format the parser will not find any
records and you won't get any data.  Try this in your input file:

62.195.113.219 1391794828.893 "POST /v1/recovery_email/verify_code
HTTP/1.1" 200 22 "
https://accounts.dev.lcip.org/v1/verify_email?uid=9b6e44b8e67a4647bfe13595c06d2f9b&code=b4bf632f"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101
Firefox/29.0" "-"
62.195.113.219 1391794828.893 "GET /v1/recovery_email/status HTTP/1.1" 200
82 "-" "Mozilla/5.0 (Mobile; rv:29.0) Gecko/29.0 Firefox/29.0" "-"
62.195.113.219 1391794828.893 "POST /v1/session/destroy HTTP/1.1" 200 37
"-" "Mozilla/5.0 (Mobile; rv:29.0) Gecko/29.0 Firefox/29.0" "-"
62.195.113.219 1391794828.893 "POST /v1/auth/start HTTP/1.1" 200 588 "-"
"Mozilla/5.0 (Mobile; rv:29.0) Gecko/29.0 Firefox/29.0" "-"
54.224.179.251 1391794828.893 "POST /v1/auth/start HTTP/1.1" 200 932 "-"
"-" "-"
54.224.179.251 1391794828.893 "POST /v1/auth/finish HTTP/1.1" 200 141 "-"
"-" "-"
54.224.179.251 1391794828.893 "POST /v1/session/create HTTP/1.1" 200 274
"-" "-" "-"

1) stop hindsight (HS) if it is running
2) wipe the internal queue `rm -rf output`
3) restart HS with debug logging 'hindsight single.cfg 7'
4) you should see  '### [info] input.input_test detaching received: 0 msg:
7' in the HS log
--- in a separate terminal window
5) take a look at the internal queues
    - output/input/0.log  # you can `lsb_heka_cat` that to see the contents
(7 messages above)
    - output/analysis/0.log # you can `lsb_heka_cat -f` that and should see
a new analysis counter message every 5 seconds; the payload in this case
will max at 7
6) output/hindsight.tsv contains the statistics for the running plugins
(updated every 6 seconds, the input will have already exited)
7) output/hindsight.cp contains the checkpoints for the input/output
plugins and the analysis thread
-- end of second terminal window
8) since you are running with debug output you will see the print message
from the output plugin (the count) every 60 seconds in the HS log
9) ctrl-c to shutdown HS

The toml file is there to run the equivalent Heka test; it is not needed by
HS.

Trink


On Tue, Nov 15, 2016 at 12:31 AM, VINEET BERLIA <shanky221341 at gmail.com>
wrote:

> Hi,
>
> I was trying to get around using hindsight replacing heka, for one of our
> clients. I have done the setup successfully.
>
> I tried the examples in benchmark folder. But can't understand whats
> actually happening?:- for eg:-
>
> With following setup in
> .
> ├── run
> │   ├── analysis
> │   │   ├── counter.cfg
> │   │   ├── counter.fcfg
> │   │   └── counter.lua
> │   ├── input
> │   │   ├── input_test.cfg
> │   │   └── input_test.lua
> │   └── output
> │       ├── counter.cfg
> │       └── counter.lua
> ├── single.cfg
> └── single.toml
>
> I run:-  hindsight single.cfg
>
> and get the following output:-
>
> vineet at Vineet-ThinkPad-T420:~/Desktop/heka_test$ hindsight single.cfg
> 1479198461994525451 [info] hindsight starting
> 1479198462004416356 [info] input.input_test starting
> 1479198462004946356 [info] input.input_test detaching received: 0 msg: 0
> 1479198462005641516 [info] analysis.counter adding to thread: 0
> 1479198462005738750 [info] analysis_plugins starting thread: 0
> 1479198462006463212 [info] output.counter starting
>
>
> and then I have to exit hindsight.
>
> Whats actually happening here? i have configured in single.cfg to read
> from a log file. but whats is the output I should get and where?
>
> Can you pls explain the flow of execution from input to output?
>
> For eg from nginnx log files to just console output?
>
> And why do we need single.toml file here? that was in heka right?
>
> Thanks
> KR
> Vineet
>
> _______________________________________________
> Hindsight mailing list
> Hindsight at mozilla.org
> https://mail.mozilla.org/listinfo/hindsight
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20161115/c0dfc90d/attachment.html>


More information about the Hindsight mailing list