Usability improvements for Firefox automation initiative - Status update #7

[Armen Zambrano G.]

On this update we will look at the progress made in the last two weeks.

A reminder that this quarter’s main focus is on:
* Debugging tests on interactive workers (only Linux on TaskCluster)
* Improve end to end times on Try (Thunder Try project)

For all bugs and priorities you can check out the project management 
page for it:
https://wiki.mozilla.org/EngineeringProductivity/Projects/Debugging_UX_improvements

Status update:
Debugging tests on interactive workers
---------------------------------------------------
Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1262260

Upcoming:
* Android xpcshell
* Blog/newsgroup post


Thunder Try - Improve end to end times on try
---------------------------------------------

Project #1 - Artifact builds on automation
##########################################
Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1284882

Accomplished recently:
* The following platforms are now supported: linux, linux64, macosx64, 
win32, win64
* An option was added to download symbols for our compiled artifacts 
during the artifact build

Upcoming:
* Debug artifact builds on try. (Right now --artifact always results in 
an opt artifact build.)
* Android artifact builds on try, thanks to nalexander.

Project #2 - S3 Cloud Compiler Cache
####################################
Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1280641

Some of the issues found last quarter for this project was around NSS 
which also was in need of replacing. This project was put on hold until 
the NSS work was completed. We’re going to resume this for Q4.

Project #3 - Metrics
####################
Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1286856

Accomplished recently:
* Brittle running example here: 
http://people.mozilla.org/~klahnakoski/temp/End-to-End.html
* Problem with low populations; 90th percentile is effectively 
everything, and a couple of outliers impacts the End-to-End time shown.

Upcoming:
* Figure out what to do with these small populations:
* Ignore them - too small to be statistically significant
* Aggregate them - All the rarely run suites can be pushed into a 
“Other” category
* Show some other statistic:  Maybe median is better?
* Show median of past day, and 90% for the week:  That can show the 
longer trend, and short term situation, for better overall feel.

Project #4 - Build automation improvements
##########################################
Accomplished recently:
* Bug 1306167 - Updated build machines to use SSD. Linux PGO builds now 
take half the time

https://bugzilla.mozilla.org/show_bug.cgi?id=1306167#c11

Project #5 - Run Web platform tests from the source checkout
############################################################
Nothing to add on this edition.

-- 
Zambrano Gasparnian, Armen
Engineering productivity
http://armenzg.blogspot.ca