[rust-dev] Test suite uses too many open fds

Brian Anderson banderson at mozilla.com
Wed Jul 17 10:37:48 PDT 2013


On 07/17/2013 10:35 AM, Brian Anderson wrote:
> On 07/13/2013 01:19 PM, Kevin Ballard wrote:
>> Last night I tracked down why `make check-stage2-std` no longer works
>> on my computer. This is documented in issue
>> https://github.com/mozilla/rust/issues/7772. In summary, the test
>> runner now uses num_cpus*2 threads instead of 4 threads, which ends up
>> being 16 on my machine, and this causes the process to run out of fds.
>>
>> Why is the test suite using so many fds? lsof says most of them are
>> PIPEs. What are those being used for?
>>
>> -Kevin
>> _______________________________________________
>> Rust-dev mailing list
>> Rust-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/rust-dev
>
> I replied in the issue tracker, but the new scheduler is based on 
> libuv, which works by waiting on fds, and the test suite does 
> aggressive testing of the scheduler, which eats up a lot of fds. I had 
> to raise the ulimit on the mac bots to land it.
>
> There are two overcommits involved in the test suite now that could be 
> contributing to this problem: first, as you pointed out, the 
> multithreaded scheduler tests create num_cpus * 2 scheduler threads 
> each; second, the normal test runner creates num_cpus * 4 test tasks.
>
> Each scheduler I believe needs a minimum of 2 fds, one for the kqueue 
> (I think), and a second for an async handle. So at any time while 
> running stdtest you may need at least num_cpus * 2 * 4 * 2 = 64 fds, 
> and undoubtedly there are even more that I'm not aware of.
>
> There may also be some inefficiencies here too, e.g. we create too 
> many idle handles (though I don't think those require an fd).
> _______________________________________________
> Rust-dev mailing list
> Rust-dev at mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev

Uh, that math was wrong. num_cpus * 2 * num_cpus * 4 * 2 = 256 :)


More information about the Rust-dev mailing list