[rust-dev] Test suite uses too many open fds
banderson at mozilla.com
Wed Jul 17 10:37:48 PDT 2013
On 07/17/2013 10:35 AM, Brian Anderson wrote:
> On 07/13/2013 01:19 PM, Kevin Ballard wrote:
>> Last night I tracked down why `make check-stage2-std` no longer works
>> on my computer. This is documented in issue
>> https://github.com/mozilla/rust/issues/7772. In summary, the test
>> runner now uses num_cpus*2 threads instead of 4 threads, which ends up
>> being 16 on my machine, and this causes the process to run out of fds.
>> Why is the test suite using so many fds? lsof says most of them are
>> PIPEs. What are those being used for?
>> Rust-dev mailing list
>> Rust-dev at mozilla.org
> I replied in the issue tracker, but the new scheduler is based on
> libuv, which works by waiting on fds, and the test suite does
> aggressive testing of the scheduler, which eats up a lot of fds. I had
> to raise the ulimit on the mac bots to land it.
> There are two overcommits involved in the test suite now that could be
> contributing to this problem: first, as you pointed out, the
> multithreaded scheduler tests create num_cpus * 2 scheduler threads
> each; second, the normal test runner creates num_cpus * 4 test tasks.
> Each scheduler I believe needs a minimum of 2 fds, one for the kqueue
> (I think), and a second for an async handle. So at any time while
> running stdtest you may need at least num_cpus * 2 * 4 * 2 = 64 fds,
> and undoubtedly there are even more that I'm not aware of.
> There may also be some inefficiencies here too, e.g. we create too
> many idle handles (though I don't think those require an fd).
> Rust-dev mailing list
> Rust-dev at mozilla.org
Uh, that math was wrong. num_cpus * 2 * num_cpus * 4 * 2 = 256 :)
More information about the Rust-dev