[Go Faster] Planning for deployments involving system add-ons or updates to system add-ons

Andrei Vaida andrei.vaida at softvisioninc.eu
Thu Mar 30 14:32:42 UTC 2017


Hi everyone,

@Cory a ballpark schedule at least for larger or more complex (as in 
requiring a high amount of manual testing) system add-ons would be 
great. I think having two rounds of testing (once for the * .xpi file 
and once for the release-sysaddon update channel) is a good thing, two 
QA filters is better than one and we all want to ship any system add-on 
in its best shape possible -- this is something that I think we should 
keep.

I'm aware that the whole point of Go Faster is to ship 
features/fixes/updates as fast as possible, that's why I was rather 
thinking of a raw and always-changing calendar that my team could check, 
even on a daily-basis. Using the Go Faster Public Calendar 
<https://calendar.google.com/calendar/embed?src=mozilla.com_fd39k26ps9k9t7b0ej5s1fmt7c@group.calendar.google.com&ctz=America/Los_Angeles> 
sounds good to me.

@Ben QE is usually notified when the system add-on is ready to be 
tested. Automating parts of this process would definitely help.

Overall, this is what we're covering for each system add-on through 
manual tests:
that the update.xml file associated to the builds we're covering are 
correctthat the system add-on or system add-on update is successfully 
installed on the update channelthat the features associated to that 
system add-on work as intendedbased on the complexity of the system 
add-on in question, this can get quite laboriousthat the other system 
add-ons (usually just Pocket) are not affected in an unintended way by 
the system add-onthis usually consists of a spot-check on Pocket core 
functionality only, but if we need to pay attention to other system 
add-ons or updates that are coincidentally deployed around the same 
time, then this too can get laboriousWe're doing this on 4 platforms (2 
x Windows, 1 x Mac, 1 x Linux), 2 Firefox build architectures (32-bit, 
64-bit), 3 locales (en-US, de, ru) and we're checking both the Firefox 
build versions that should get the system add-on and a few build 
versions that should not get it. Here's an example: 1325872 
<https://public.etherpad-mozilla.org/p/test_plan_for_bug_1325872>.

Thank you both for following up on this so quickly!

Best,
Andrei

------ Original Message ------
From: "Ben Hearsum" <bhearsum at mozilla.com>
To: "Cory Price" <cprice at mozilla.com>
Cc: "Andrei Vaida" <andrei.vaida at softvisioninc.eu>; "Ritu Kothari" 
<rkothari at mozilla.com>; "Jason Thomas" <jthomas at mozilla.com>; "Florin 
Mezei" <florin.mezei at softvisioninc.eu>; "cornel.ionce at softvisioninc.eu" 
<cornel.ionce at softvisioninc.eu>; "Robert Helmer" <rhelmer at mozilla.com>; 
"Felipe Gomes" <fgomes at mozilla.com>; gofaster at mozilla.org
Sent: 2017-03-30 4:11:42 PM
Subject: Re: [Go Faster] Planning for deployments involving system 
add-ons or updates to system add-ons

>On Wed, Mar 29, 2017 at 03:50:45PM -0700, Cory Price wrote:
>>  + some other folks because this is a great topic, so thanks for 
>>bringing it
>>  up.
>>
>>  On Wed, Mar 29, 2017 at 2:14 AM, Andrei Vaida 
>><andrei.vaida at softvisioninc.eu
>>  > wrote:
>>
>>  > Hi Cory,
>>  >
>>  > I was wondering if the Go Faster team has some sort of schedule 
>>available
>>  > for the system add-ons or system add-on updates they're planning to 
>>ship.
>>  >
>>  > The process we set up last year for test channel verification
>>  > 
>><https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process#Verification_of_Test_Channel> 
>>has
>>  > been working really well so far, but I think my team could be more
>>  > efficient if there were a release schedule we could access, to 
>>prepare for
>>  > upcoming deployments in advance. It wouldn't have to be a very 
>>precise
>>  > schedule, just enough so we know that one or more system add-ons 
>>are
>>  > planned to be deployed "sometime in the week of...".
>>  >
>>  > Such a schedule would help us a lot, especially since there were 
>>often
>>  > cases when my team had to fit in this kind of testing in 
>>particularly busy
>>  > days, without knowing and planning for them before hand.
>>  >
>>  > What do you think? Does Go Faster already have such a schedule? And 
>>if
>>  > not, would it be possible to build one and share it with our team?
>>  >
>>
>>  There currently is not such a schedule. The premise of the Go Faster
>>  workflow is to turn things around quickly, with the ability to land
>>  priority fixes and feature updates in short order.
>
>I'm coming at this without deep knowledge of this part of the process, 
>but I'm curious when QE currently gets looped in? Are they notified as 
>soon as we know there's a new System Addon coming, or only when it's 
>ready to be tested?
>
>>
>>  For some larger full feature system add-ons such as bug 1351424, or 
>>bug
>>  1351454 building a schedule is reasonable, but the majority of system
>>  add-ons we release (at least recently) have been things we have to 
>>fix
>>  right away.
>>
>>  I don't know what the best way to handle this is. One thing I've been
>>  thinking of was revisiting the requirement of two QA verification 
>>steps in
>>  the process[0], maybe asking for just one round of QA verification?
>
>Depending on how deep the "verification of test channel" is, this might 
>be something that can be automated. We have a couple of existing 
>frameworks that do automated app-update verification. Specifically, if 
>these are just "ensure system addon gets installed" (explicitly no 
>functional verification of the addon), it might be possible to adapt 
>them or do something similar for System Addons. This might be worth 
>bringing up with Benson's team, since they're already working on 
>automated pipeline for System Addon updates.
>
>- Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/gofaster/attachments/20170330/d7f5e790/attachment.html>


More information about the Gofaster mailing list