Viewing resource usage of builds

Gregory Szorc gps at mozilla.com
Tue Aug 20 19:36:37 UTC 2013


Now in inbound, |mach build| will record *system* resource usage during 
builds. You can later view the resource usage by running |mach 
resource-usage| (mach will tell you at the end of the build if you can 
run this).

The goal of this feature is to help people diagnose why builds are slow 
and how builds are changing over time. You should be able to easily see 
where CPU isn't at 100%, when I/O is slowing you down, when you are 
swapping, etc. We should eventually be able to use this information to 
tell you things like "your build could be faster if you had an SSD, more 
memory, or more CPU cores." (We previously rolled out similar resource 
monitoring to mozharness and hope to one day have builds and automation 
producing the same machine readable data format so we can reuse the HTML 
viewer to easily view resource usage of many different kinds of jobs. 
The goal here is to identify poor hardware resource efficiency so we can 
improve it.)

We only achieved the initial milestone by landing this feature. We still 
have a lot of work to do. Bug 901601 tracks enabling this on non-mach 
builds, which will start recording resource usage on automation 
infrastructure. Bug 907297 tracks making the HTML display better. Bug 
892342 tracks making the resource monitor more accurate. These will all 
hopefully land in due time.

If you notice anything weird, please raise it in bug 883209 if it is 
critical or file a follow-up in Core :: Build Config.

The HTML display is currently pretty poor. I'm not a good web developer 
:) The HTML display is not part of the build, so I will rubber stamp 
pretty much any patch that improves it. Please scratch your own itches! 
Bug 907297 contains a list of suggested improvements.

For comparison purposes, [1] contains the output from a clobber build 
with empty ccache on my MBP. As you can see, there are large chunks of 
the build where we aren't CPU bound. Many of these are during recursive 
make traversal, which is one reason we're focusing on moving away from 
recursive make.

Finally, this is just a friendly reminder that if you build 
mozilla-central and you don't have a machine with at least 4 *physical* 
cores, 16 GB RAM, and an SSD, you should upgrade your hardware.

[1] https://people.mozilla.com/~gszorc/resources-references.png

Gregory



More information about the firefox-dev mailing list