[rust-dev] perf metrics and ratchets

Graydon Hoare graydon at mozilla.com
Tue Jul 16 17:33:45 PDT 2013


Hi,

I've added some new machinery to the testing systems in rust and wanted
to draw a little attention to them. At least this will all be true
shortly when https://github.com/mozilla/rust/pull/7829 lands.

They're (lightly) documented at the bottom of:

  https://github.com/mozilla/rust/wiki/Doc-unit-testing

and

  https://github.com/mozilla/rust/wiki/Note-testsuite

The point-form version is:

  - #[bench] tests can now save their results as json.
    run the testrunner with --save-metrics=foo.json

  - If you want to avoid regressing on something, use
    --ratchet-metrics=foo.json instead; it will save and
    reload with each run, and consider any regression beyond
    a given noise threshold to be a test-fail.

    (Removing, renaming, omitting or adding metrics does not cause
     failure. it emits a warning and assumes you meant to do it.)

  - If you configure with --ratchet-bench then all the crate #[bench]
    benchmarks will be ratcheted in your workspace. We will turn this
    on on the buildbot soon-ish. At that point, adding a #[bench]
    function will make it impervious to perf regressions, so add them
    regularly but also with some care and thought.

  - If you're on a particularly noisy machine, you can override the
    inferred noise thresholds with --ratchet-noise-percent=N

  - There are new compiletests in src/test/codegen. These consist
    of pairs of files, foo.rs and foo.cc, each containing a non-mangled
    extern C function 'test'. The runner will compile the rust one with
    rustc and the C++ one with clang, extract their LLVM bitcode,
    disassemble it and compare the sizes. The result is recorded in
    a metrics file and _ratcheted by default_.

  - The codegen tests also leave behind foo-extract.ll and
    foo-clang-extract.ll files, which are helpful for eyeballing
    problems in our codegen. Please populate the codegen directory
    liberally. All you need to do is write a function in rust and
    a function that "does the same thing" in C++.

If you have any questions, or find none of it works for you and want
help making it work, please let me know. I'll hopefully turn on
--ratchet-bench and export the metrics files from the buildbot soon.

-Graydon


More information about the Rust-dev mailing list