[rust-dev] perf metrics and ratchets
graydon at mozilla.com
Tue Jul 16 17:33:45 PDT 2013
I've added some new machinery to the testing systems in rust and wanted
to draw a little attention to them. At least this will all be true
shortly when https://github.com/mozilla/rust/pull/7829 lands.
They're (lightly) documented at the bottom of:
The point-form version is:
- #[bench] tests can now save their results as json.
run the testrunner with --save-metrics=foo.json
- If you want to avoid regressing on something, use
--ratchet-metrics=foo.json instead; it will save and
reload with each run, and consider any regression beyond
a given noise threshold to be a test-fail.
(Removing, renaming, omitting or adding metrics does not cause
failure. it emits a warning and assumes you meant to do it.)
- If you configure with --ratchet-bench then all the crate #[bench]
benchmarks will be ratcheted in your workspace. We will turn this
on on the buildbot soon-ish. At that point, adding a #[bench]
function will make it impervious to perf regressions, so add them
regularly but also with some care and thought.
- If you're on a particularly noisy machine, you can override the
inferred noise thresholds with --ratchet-noise-percent=N
- There are new compiletests in src/test/codegen. These consist
of pairs of files, foo.rs and foo.cc, each containing a non-mangled
extern C function 'test'. The runner will compile the rust one with
rustc and the C++ one with clang, extract their LLVM bitcode,
disassemble it and compare the sizes. The result is recorded in
a metrics file and _ratcheted by default_.
- The codegen tests also leave behind foo-extract.ll and
foo-clang-extract.ll files, which are helpful for eyeballing
problems in our codegen. Please populate the codegen directory
liberally. All you need to do is write a function in rust and
a function that "does the same thing" in C++.
If you have any questions, or find none of it works for you and want
help making it work, please let me know. I'll hopefully turn on
--ratchet-bench and export the metrics files from the buildbot soon.
More information about the Rust-dev