06/10/2015 Flame KK v18D-1 319 MB Gaia-ui Automation mozilla-central Mozilla RIL Full Flash Build 37/42 smoketests, 95/129 non-smoketests

Oliver Nelson onelson at qanalydocs.com
Wed Jun 10 19:28:52 UTC 2015


*Large problems performing reporting due to bugs [1] 1172343 and [2]
1173416*
[1] https://bugzil.la/1172343 -- Failures in various tests in
self.marionette.start_session() : IOError: Connection to Marionette server
is lost
** forces many tests to be re run in order to determine validity of
failures (ensure no device errors are missed)
[2] https://bugzil.la/1173416 -- Failing to flash devices when building
adhoc.bitbar runs: "FATAL: Device session is null"
** when attempting to run multiple adhocs (as necessary by the previous
bug) many runs will fail to find a device and fail, bottlenecking the
ability to confirm errors/ For this reason, many tests that errored under
the assumped of bug 1172343 were not checked today.

*Smoketests:*

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.smoke.bitbar/220/

*Test failures - 0 replicable manually:*

   - [Not reproduced manually] test_sms_with_attachments - Failed Jenkins
   adhoc; 0/5 passing -- Continuing assessment of issue

*bug 1172343 IO error -- Performed adhoc with no repeat to run through
tests*
** Unable to perform tests timely due to https://bugzil.la/1173416 --
devices consistently failing to flash, failing build[Pass]
test_settings_cell_data
**
http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc.bitbar/90/console
* test_settings_usb_storage
* test_power_button_long_press
* test_play_3gp_video

*Expected failures / disabled tests:*

   - [Expected fail] test_ftu_skip_tour.TestFtu - https://bugzil.la/1134257
   Default value for 'About Firefox OS, Yes! Send Data.' causing fail in
   ftu_skip_tour
   - [Disabled] test_rocketbar_add_collection_save_bookmark -
   https://bugzil.la/1162112 - TimeoutException: TimeoutException: Timed
   out after 30.1 seconds
   - [Disabled] test_call_log_all_calls.py - https://bugzil.la/1096385 -
   Investigate failure in test_call_log_all_calls.py, call log is displayed
   erroneously
   - [Disabled] test_dialer_set_up_conference_call.py -
   https://bugzil.la/1117813 Make PlivoUtil use a valid 'answer_url' with a
   valid Plivo XML
   - [Disabled] test_setup_and_send_active_sync_email.py -
   https://bugzil.la/1028192 - Problems with ActiveSync accounts
   - [Disabled] test_receive_active_sync_email.py -
   https://bugzil.la/1028192 - Problems with ActiveSync accountsc
   - [Disabled] test_fmradio_find_stations.py - https://bugzil.la/1000863
   Investigate test_fmradio_find_stations.py failure
   - [Disabled] test_ftu_skip_tour_desktopb2g.py - Desktop Only
   - [Disabled] test_settings_bluetooth.py - Bug 1134244 -
   test_settings_bluetooth.py:test_toggle_bluetooth_settings is failing on
   bluetooth_settings.enable_visible_to_all()
   - [Disabled] test_settings_airplane_mode.py - https://bugzil.la/1131070


*Non-smoketests:*

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1.bitbar/94/
http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2.bitbar/94/

*Test failures - 0 replicable manually:*

*non-smoke.1:*

   - [Not reproduced manually]
   test_dialer_miss_call_from_known_contact_notification -- Reproduced via
   Jenkins adhoc; 0/5 passing -- Continuing assessment of issue
   - [Not reproduced manually] test_dialer_receive_call_with_contact_photo
   -- Reproduced via Jenkins adhoc; 0/5 passing -- Continuing assessment of
   issue
   - [Not reproduced manually]
   test_add_to_an_existing_contact_from_messages -- Reproduced via Jenkins
   adhoc; 0/5 passing -- Continuing assessment of issue
   - [Not reproduced manually] test_sms_notification -
   https://bugzil.la/1172618 - TimeoutException: Timed out after 300.1
   seconds with message: Notification did not appear
   - [Not reproduced manually]
   test_sms_notification_removed_when_sms_deleted -
   https://bugzil.la/1172460 - TimeoutException: Timed out after 300.1
   seconds with message: Notification did not appear
   - [Not reproduced manually] test_gallery_empty -- Not reproduced via
   Jenkins adhoc; 5/5 passing


*bug 1172343 IO error -- Performed adhoc with no repeat to run through
tests*
** Unable to perform tests timely due to https://bugzil.la/1173416 --
devices consistently failing to flash, failing build
**
http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc.bitbar/92/console/
* test_settings_passcode
* test_settings_device_info
* test_settings_battery_power_save_mode
* test_settings_change_ring_tone
* test_geolocation_prompt
* test_system_message
* test_inter_app_comm
* test_privileged_app_video_capture_prompt
* test_privileged_app_device_music_prompt
* test_privileged_app_device_sdcard_prompt
* test_privileged_app_geolocation_prompt
* test_play_ogv_video
* test_play_mp4_video

*non-smoke.2:*

   - [Not reproduced manually] test_homescreen_column_layout -- Not
   reproduced via Jenkins adhoc; 5/5 passing


*bug 1172343 IO error -- Performed adhoc with no repeat to run through
tests*
** Unable to perform tests timely due to https://bugzil.la/1173416 --
devices consistently failing to flash, failing build
**
http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc.bitbar/91/console
[Pass] test_settings_change_keyboard_language
[Pass] test_settings_power_save_mode
[Pass] test_settings_root
[Pass] test_system_notification_bar
[Pass] test_quick_settings
[Pass] test_system_message_pending
[Pass] test_privileged_app_audio_capture_prompt
[Pass] test_privileged_app_contacts_prompt
[Pass] test_privileged_app_device_picture_prompt
[Pass] test_privileged_app_device_video_prompt
[Pass] test_video_empty
[Pass] test_play_webm_video
[Pass] test_play_ogg_video

*Expected failures / disabled tests:*

   - [Expected fail] test_lockscreen_time_check - https://bugzil.la/1157316
   - JavascriptException: TypeError: document.getElementById(...) is null
   - [Expected fail] test_import_gmail_no_network.py -
   https://bugzil.la/1155763 -xfail
   - [Expected fail] test_settings_change_time_format -
   https://bugzil.la/1141654 test_settings_change_time_format consistently
   failing, seems to hang and fail on the homescreen
   - [Expected fail] test_call_log_groups.TestCallLogGroups -
   https://bugzil.la/1117804 - [v2.2] Investigate test_call_log_groups
   failure
   - [Expected fail] test_change_language_settings_rtl -
   https://bugzil.la/1137766 - Error in both test_settings_change_language
   tests: exception "Element '%s' could not be foundin select wrapper"
   - [Expected fail] test_change_language_settings -
   https://bugzil.la/1137766 - Error in both test_settings_change_language
   tests: exception "Element '%s' could not be foundin select wrapper"
   - [Disabled] test_call_log_all_calls.py - https://bugzil.la/1096385 -
   Investigate failure in test_call_log_all_calls.py, call log is displayed
   erroneously
   - [Disabled] test_dialer_set_up_conference_call.py -
   https://bugzil.la/1117813 - Make PlivoUtil use a valid 'answer_url' with
   a valid Plivo XML
   - [Disabled] test_setup_and_send_active_sync_email.py - Bug 1028192 -
   Problems with ActiveSync accounts
   - [Disabled] test_receive_active_sync_email.py -
   https://bugzil.la/1028192 - Problems with ActiveSync accounts
   - [Disabled] test_fmradio_find_stations.py - https://bugzil.la/1000863 -
   Investigate test_fmradio_find_stations.py failure
   - [Disabled] test_ftu_skip_tour_desktopb2g.py - Test runs only on desktop
   - [Disabled] test_settings_bluetooth.py - https://bugzil.la/1134244 -
   test_settings_bluetooth.py:test_toggle_bluetooth_settings is failing on
   bluetooth_settings.enable_visible_to_all()
   - [Disabled] test_settings_airplane_mode.py - https://bugzil.la/1131070


*Build under test:*

Device firmware (base)     L1TC100118D0
Device firmware (date)     10 Jun 2015 08:33:42
Device firmware (incremental)     eng.cltbld.20150610.043331
Device firmware (release)     4.4.2
Device identifier     flame
Device memory     219772 kB
Device serial     1b41c231
Device uptime     0 days 0 hours 2 minutes 9 seconds
Gaia date     09 Jun 2015 20:49:53
Gaia revision     9844b7c9d442
Gecko build     20150610010201
Gecko revision     5787d784485e
Gecko version     41.0a1

*Summary:*

*Smoketests:*

179 tests ran in 4991.2 seconds.
37 passed, 136 skipped, 0 failed, 5 errors.
1 expected failures, 0 unexpected passes.

*Non-smoketest:*

124 tests ran in 7948.4 seconds.
44 passed, 59 skipped, 0 failed, 19 errors.
2 expected failures, 0 unexpected passes.

126 tests ran in 7141.1 seconds.
51 passed, 59 skipped, 0 failed, 14 errors.
2 expected failures, 0 unexpected passes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/fxosqa-report-watchlist/attachments/20150610/c01c72a6/attachment-0001.html>


More information about the Fxosqa-report-watchlist mailing list