Autophone Update 2014-02-14

Bob Clary bob at bclary.com
Fri Feb 14 12:21:18 PST 2014


Autophone Update 2014-02-14

Bug 852264 <https://bugzilla.mozilla.org/show_bug.cgi?id=852264> appears 
to have significantly helped reduce noise in the results since 
2014-02-05 though not to the level desired.

Outage on 2014-02-12 due to Bug 972346 
<https://bugzilla.mozilla.org/show_bug.cgi?id=972346>. Fixed on 2014-02-13.

Began live testing on 2014-02-13 of Bug 968905 
<https://bugzilla.mozilla.org/show_bug.cgi?id=968905>. This allows the 
devices to be partitioned across tests/repositories. Restarted with 
final version of patch on 2014-02-14 11AM PT. The partitioning appears 
to be able to keep up with the load.

Mochitest-Canvas testing re-enabled on 2014-02-12. Nexus 5 is reporting 
results as well though not visible until autolog is reconfigured. See 
Bug 972369 <https://bugzilla.mozilla.org/show_bug.cgi?id=972369>. Having 
issues with Nexus 5 / Android 4.4 which prevents tests from running 
correctly.

We have been running without ipfw+dummynet traffic shaping for a week. 
The results have not shown any improvement in noise when compared to the 
previous week after Bug 852264 landed.

There is an unexplained performance improvement after the 2014-02-05 
landing of Bug 852264 and an unexplained lack of change after the change 
to not use ifpw+dummynet on 2014-02-08.

I have noticed that the results obtained from my development environment 
shows less noise than the production system. To investigate:

* As part of today's restart, I have directed all output from the local 
phonedash web server to /dev/null and changed the logging level to 
WARNING which will reduce extraneous output on the host server. I am 
interested in if this has any affect on the stability of the results.

* Beginning with builds on 2014-02-15, I will use the phonedash server 
on my workstation instead of the one on the autophone host.

Bugs on deck:

* Bug 968920 - <https://bugzilla.mozilla.org/show_bug.cgi?id=968920>
                Autophone - use standard error percentage
                as a discriminator
* Bug 969518 - <https://bugzilla.mozilla.org/show_bug.cgi?id=969518>
                Autophone - attempt to use reverse tethering
                so we can use ethernet over usb instead of wifi

The current device/test/repository partition is:

s1s2tests
=========
c8_aa_21_ac_0c_b5_droid-pro = mozilla-central fx-team
98_4b_4a_13_01_84_droid-pro = mozilla-inbound

00_23_76_b1_92_39_nexus-one = mozilla-central fx-team
00_23_76_96_cc_6f_nexus-one = mozilla-inbound

78_d6_f0_cf_d2_17_nexus-s = mozilla-central fx-team
78_d6_f0_cf_8d_11_nexus-s = mozilla-inbound

98_0c_82_33_ec_8d_samsung-gs2 = mozilla-central fx-team
20_64_32_4f_e9_4f_samsung-gs2 = mozilla-inbound

5c_0a_5b_b4_31_32_samsung-gs3 = mozilla-central fx-team
38_aa_3c_1e_a8_ee_samsung-gs3 = mozilla-inbound

98_d6_f7_69_37_05_nexus-4-jdq39 = mozilla-central fx-team
98_d6_f7_69_34_aa_nexus-4-jdq39 = mozilla-inbound

cc_fa_00_a8_82_61_nexus-5-kot49h = mozilla-central fx-team
cc_fa_00_a8_82_d3_nexus-5-kot49h = mozilla-inbound

d8_50_e6_7a_86_a2_nexus-7-jss15q = mozilla-central fx-team
d8_50_e6_7a_7b_de_nexus-7-jss15q = mozilla-inbound

Mochitest-Canvas Skia GL
========================
80_96_b1_ee_55_e0_atrix = mozilla-central
98_d6_f7_69_37_05_nexus-4-jdq39 = mozilla-central
10_bf_48_cd_c0_0b_nexus-7-jdq39 = mozilla-central
cc_fa_00_a8_82_61_nexus-5-kot49h = mozilla-central
d8_50_e6_7a_86_a2_nexus-7-jss15q = mozilla-central

/bc


More information about the mobile-firefox-dev mailing list