[Hindsight] number of log files in output/input dir

Michael Trinkala mtrinkala at mozilla.com
Tue Dec 27 17:40:42 UTC 2016


There is a pruning plugin that is installed with Hindsight, see
https://github.com/mozilla-services/hindsight/blob/master/sandboxes/input/prune_input.lua

Trink
<https://github.com/mozilla-services/hindsight/blob/master/sandboxes/input/prune_input.lua>

On Tue, Dec 27, 2016 at 7:50 AM, Thomas.Bostelmann <
Thomas.Bostelmann at target.com> wrote:

> All, is it possible to configure the number and size of log files in the
> output/input directory?
>
> I’m finding my running instance of hindsight filling up the ouput/input
> directory will a large number of files (GBs in size) which is causing the
> VMs I’m using to run out of disk space.
>
> I’ve pulled down the latest hindsight code and have it deployed with the
> following configuration settings (thanks in advance for any help):
>
> output_path             = "output"
> -- output_size             = 1024 * 1024 * 1024
> output_size             = 64 * 1024 * 1024
> sandbox_load_path       = "load" -- enable dynamic loading
> sandbox_run_path        = "run"
> -- sandbox_install_path    = "/usr/share/luasandbox/sandboxes/heka" -- default
>
> --[[ redhat --]]
> analysis_threads        = 1
> analysis_lua_path        = "/lua_sandbox_extensions/release/install/lib/luasandbox/modules/?.lua"
> analysis_lua_cpath       = "/lua_sandbox_extensions/release/install/lib/luasandbox/modules/?.so"
> io_lua_path             = analysis_lua_path .. ";/lua_sandbox_extensions/heka/io_modules/?.lua;/thota/run/modules/?.lua;/lua_sandbox_extensions/syslog/io_modules/?.lua;/lua_sandbox_extensions/syslog/modules/?.lua;/lua_sandbox_extensions/heka/modules/?.lua;/lua_sandbox_extensions/syslog/io_modules/?.lua;/lua_sandbox_extensions/syslog/modules/?.lua;/lua_sandbox_extensions/release/install/lib/luasandbox/io_modules/?.lua"
> io_lua_cpath            = analysis_lua_cpath .. ";/lua_sandbox_extensions/release/kafka/?.so;/lua_sandbox_extensions/release/install/lib/luasandbox/io_modules/?.so"
>
> --[[ debian
> analysis_threads        = 1
> analysis_lua_path       = "/usr/lib/luasandbox/modules/?.lua"
> analysis_lua_cpath      = "/usr/lib/luasandbox/modules/?.so"
> io_lua_path             = analysis_lua_path ..  ";/usr/lib/luasandbox/io_modules/?.lua"
> io_lua_cpath            = analysis_lua_cpath .. ";/usr/lib/luasandbox/io_modules/?.so"
>  --]]
>
> max_message_size        = 8e6 -- default
> backpressure            = 0 -- default
> backpressure_disk_free  = 4 -- default
> -- hostname                = -- default gethostname()
>
> analysis_defaults = {
> --    output_limit          = 1024 * 64,        -- default
> --    memory_limit          = 1024 * 1024 * 8,  -- default
> --    instruction_limit     = 1000000,          -- default
> --    preserve_data         = false,            -- default
> --    restricted_headers    = false,            -- default
>     ticker_interval       = 60,
> --    shutdown_on_terminate = false,            -- default
>     }
>
> input_defaults = {
> --    output_limit          = 1024 * 64,        -- default
> --    memory_limit          = 1024 * 1024 * 8,  -- default
>     instruction_limit     = 0,
> --    preserve_data         = false,            -- default
> --    restricted_headers    = true,             -- default
> --    ticker_interval       = 0,                -- default
> --    shutdown_on_terminate = false,            -- default
>     }
>
> output_defaults = {
> --    output_limit          = 1024 * 64,        -- default
> --    memory_limit          = 1024 * 1024 * 8,  -- default
> --    instruction_limit     = 1000000,          -- default
> --    preserve_data         = false,            -- default
> --    restricted_headers    = false,            -- default
>     ticker_interval       = 60,
> --    shutdown_on_terminate = false,            -- default
> --    remove_checkpoints_on_terminate = false,  -- default
>     }
>
>
> _______________________________________________
> Hindsight mailing list
> Hindsight at mozilla.org
> https://mail.mozilla.org/listinfo/hindsight
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/hindsight/attachments/20161227/13321e53/attachment-0001.html>


More information about the Hindsight mailing list