new de-orphaned fhr data available

[Brendan Colloran]

Mmm, well we haven't a Formal Discussion with IT or anything, and we haven't followed CAB procedures, so I guess everything is subject to change without warning until we decide to stop doing things the cowboy way. But the job runs once a week (on Thursday), and for the time being I will consistently be putting the final cleaned data on hdfs at:

/user/bcolloran/fhrDeorphaned_YYYY-MM-DD

The pipeline takes as long as it takes, which seems to be about 5ish days. But since it's also variable based on cluster load, I'll continue sending email alerts out until we formalize things a bit. But if you wish you can look for new data at the above path.

Because the data is big and the space is limited, I'm also only retaining the two most recent de-orphaned data sets. Older data will be deleted when the new data is done being processed.

-Brendan

----- Original Message -----
> From: "Saptarshi Guha" <sguha at mozilla.com>
> To: "Brendan Colloran" <bcolloran at mozilla.com>
> Cc: "Analytics team" <quants at mozilla.com>, "Anurag Phadke" <aphadke at mozilla.com>, "Thirunavukarasu Meyarivan"
> <tmary at mozilla.com>, "FHR-dev" <fhr-dev at mozilla.org>
> Sent: Tuesday, March 4, 2014 1:29:26 PM
> Subject: Re: new de-orphaned fhr data available
> 
> Will this be automated? cron job? So that the sampler can inspect for the
> presence of a file
> and then do it's thing.
> 
> 
> ----- Original Message -----
> > From: "Brendan Colloran" <bcolloran at mozilla.com>
> > To: "Analytics team" <quants at mozilla.com>, "Anurag Phadke"
> > <aphadke at mozilla.com>, "Thirunavukarasu Meyarivan"
> > <tmary at mozilla.com>
> > Cc: "FHR-dev" <fhr-dev at mozilla.org>
> > Sent: Tuesday, March 4, 2014 11:17:30 AM
> > Subject: new de-orphaned fhr data available
> > 
> > on hdfs at:
> > /user/bcolloran/fhrDeorphaned_2014-02-27
> > 
> > saptarshi, please reply-all when you have samples ready. thank!
> > 
> > -brendan
> > 
>