new de-orphaned fhr data available

[Brendan Colloran]

Anurag, let's talk about this stuff on Friday and come up with a plan? Put something on my calendar at your convenience.

-bc

----- Original Message -----
> From: "John Jensen" <jjensen at mozilla.com>
> To: "Brendan Colloran" <bcolloran at mozilla.com>
> Cc: "Saptarshi Guha" <joy at mozilla.com>, "Analytics team" <quants at mozilla.com>, "Anurag Phadke" <aphadke at mozilla.com>,
> "Thirunavukarasu Meyarivan" <tmary at mozilla.com>, "FHR-dev" <fhr-dev at mozilla.org>
> Sent: Tuesday, March 4, 2014 6:22:36 PM
> Subject: Re: new de-orphaned fhr data available
> 
> Hi all,
> 
> AFAICT this is going to be our main source-of-truth FHR dataset for some
> time, so we should get this process routinized as much as is practicable.
> Anurag, how problematic would this be?
> 
> John
> 
> ----- Original Message -----
> > From: "Brendan Colloran" <bcolloran at mozilla.com>
> > To: "Saptarshi Guha" <joy at mozilla.com>
> > Cc: "Analytics team" <quants at mozilla.com>, "Anurag Phadke"
> > <aphadke at mozilla.com>, "Thirunavukarasu Meyarivan"
> > <tmary at mozilla.com>, "FHR-dev" <fhr-dev at mozilla.org>
> > Sent: Tuesday, March 4, 2014 1:51:06 PM
> > Subject: Re: new de-orphaned fhr data available
> > 
> > Mmm, well we haven't a Formal Discussion with IT or anything, and we
> > haven't
> > followed CAB procedures, so I guess everything is subject to change without
> > warning until we decide to stop doing things the cowboy way. But the job
> > runs once a week (on Thursday), and for the time being I will consistently
> > be putting the final cleaned data on hdfs at:
> > 
> > /user/bcolloran/fhrDeorphaned_YYYY-MM-DD
> > 
> > The pipeline takes as long as it takes, which seems to be about 5ish days.
> > But since it's also variable based on cluster load, I'll continue sending
> > email alerts out until we formalize things a bit. But if you wish you can
> > look for new data at the above path.
> > 
> > Because the data is big and the space is limited, I'm also only retaining
> > the
> > two most recent de-orphaned data sets. Older data will be deleted when the
> > new data is done being processed.
> > 
> > -Brendan
> > 
> > ----- Original Message -----
> > > From: "Saptarshi Guha" <sguha at mozilla.com>
> > > To: "Brendan Colloran" <bcolloran at mozilla.com>
> > > Cc: "Analytics team" <quants at mozilla.com>, "Anurag Phadke"
> > > <aphadke at mozilla.com>, "Thirunavukarasu Meyarivan"
> > > <tmary at mozilla.com>, "FHR-dev" <fhr-dev at mozilla.org>
> > > Sent: Tuesday, March 4, 2014 1:29:26 PM
> > > Subject: Re: new de-orphaned fhr data available
> > > 
> > > Will this be automated? cron job? So that the sampler can inspect for the
> > > presence of a file
> > > and then do it's thing.
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Brendan Colloran" <bcolloran at mozilla.com>
> > > > To: "Analytics team" <quants at mozilla.com>, "Anurag Phadke"
> > > > <aphadke at mozilla.com>, "Thirunavukarasu Meyarivan"
> > > > <tmary at mozilla.com>
> > > > Cc: "FHR-dev" <fhr-dev at mozilla.org>
> > > > Sent: Tuesday, March 4, 2014 11:17:30 AM
> > > > Subject: new de-orphaned fhr data available
> > > > 
> > > > on hdfs at:
> > > > /user/bcolloran/fhrDeorphaned_2014-02-27
> > > > 
> > > > saptarshi, please reply-all when you have samples ready. thank!
> > > > 
> > > > -brendan
> > > > 
> > > 
> > 
> 
> --
> John Jensen | Director, Metrics | Mozilla Corporation
>