[rust-dev] RFC: Future of the Build System

Erick Tryzelaar erick.tryzelaar at gmail.com
Fri Jan 10 10:38:38 PST 2014


I'm personally in favor of us making our own build system, but I am
completely biased as I've already written one multistage cross-compiling
build system in Python:

https://github.com/felix-lang/fbuild

The reason why went crazy and built one was for a couple reasons. Before
Rust I worked on the Felix language, which had some complex needs for the
build system. First off, we wanted to support using MSVC without Cygwin.
Second, Building Felix used three languages as part of it's build. The
compiler was written in O'Caml, the runtime in C++, and the standard
library was in Felix. We also used a couple parser generators. Third, Felix
was a cross compiler, so we would configure and build 3 compilers. The
first generated a compiler for architecture A that could make a compiler
for architecture B who's output targeted architecture C.

As you can see, it was a complex problem. Before I started working on
FBuild, I evaluated a ton of other build systems, but they all had a bunch
of issues trying to support something like this. The biggest one was due to
the very nature of a declarative build system. There really are two phases
to these systems. First is a phase to gather up all the dependencies and
build a tree of work to do. Second is the evaluation of that tree. Things
get rather messy when you need to run a command and use it's output to
declare more dependencies. A simple example of this is using `gcc -M` to
get all the C file dependencies. This is typically supported by all build
systems. A more complex one is something like `doxygen`, which requires
manual inspection of the file system to determine dependencies.

When you combine the two phases with multiple stages it gets really
complicated. Code generators like Autoconf and CMake use macro
substitutions which can be very difficult to decipher. Systems like SCons
can be a little better because you can run arbitrary code at a given step
in the dependency tree, but there's some impedance mismatches with how you
define the build and run code. You can't do something like this:

print 'before'

Program('foo.c')

print 'after

And have the prints bracket the building of 'foo.c'. Futhermore, you need
to work with a shared environment to declare command options, and I found
that to be difficult to work with when doing a multistage build. Finally,
it still seems to be suffering from some poor performance issues:

http://blog.melski.net/2013/12/11/update-scons-is-still-really-slow/

Waf and Tup were still pretty new at the time I started FBuild, and if I
recall correctly, I think they still had trouble doing a simple multistage
build.

What I ended up doing with FBuild was to make a build system that was
centered around the idea of caching function calls. I made it in Python,
and it had a similar style to SCons, where you would write:

def build(ctx):
    c = fbuild.builders.c.guess_static(ctx)

    print 'before'

    lib = c.build_lib('static', ['lib.c'])
    exe = c.build_exe('exe', ['exe.c'], libs=[lib])

    print 'after'

But each function call actually built the output (or loaded the results
from the cache), and the printouts actually happen when they look to happen.

Doing multiple stages in this way was really simple. You just use functions:

def my_c_compiler(ctx, c):
    exe = c.build_exec('my-gcc', ['my_gcc.c'])
    return fbuild.builders.c.gcc.static(ctx, exe)

def build(ctx):
    c = fbuild.builders.c.guess_static(ctx)
    new_c = my_c_compiler(ctx, c)
    new_new_c = my_c_compiler(ctx, new_c)

It worked well, and was able to run on all our platforms and support some
nifty auto configuration support too for many c libraries. It could also
build in parallel, but there were some limitations to it because Python did
not play well with exceptions, threads, and signals.

I've been planning on reimplementing FBuild in rust for quite some time,
but I haven't had the time yet to do it beyond a couple experiments. We
already have some pieces of this approach in libextra though. Graydon took
my caching-of-functions idea and turned it into `extra::workcache`. It
wouldn't be *that* hard to port my ideas over from python though. When I
get home tonight I'll try to upload my work in progress somewhere.




On Fri, Jan 10, 2014 at 7:53 AM, Gaetan <gaetan at xeberon.net> wrote:

> I also agree that the way waf works is quite weird to use, but this is
> actually how most of the installers works under windows or some on MacOS:
> the code of the installers is embedded into the distributed package. This
> violates the debian packaging rules, and I think this will be the same with
> rust.
>
> I however really like this approach. You don't have a huge among of
> dependencies to setup in order to build your package. This is not a problem
> on debian where apt does to marveillous job, but on Windows/Mac/generic
> linux/arm, .. this is a nightmare.
>
> One more point in favor of CMake to build rust: you can build
> android/arm/misc (thanks to Makefile) with the same CMakeList.txt files, or
> with external module inclusion. I also agree that its syntax used to be
> quite ugly (the famous if() / else() / endif()), but not so much in the
> lastest versions...
> But if you require python for building, scons is the perfect, natural
> candidate.
>
> -----
> Gaetan
>
>
>
> 2014/1/10 SiegeLord <slabode at aim.com>
>
>> On 01/10/2014 06:19 AM, Robert Knight wrote:
>>
>>> Hello,
>>>
>>> CMake does have a few things going for it:
>>>
>>
>> One more consideration is that LLVM can be built with CMake afaik, so if
>> we switch to CMake we may be able to drop the autotools dependency, which
>> is a more annoying dependency to fulfill (on Windows) than CMake (I don't
>> know if Rust has other components that require autotools though).
>>
>> Along the same lines, we also require Python for whatever reason, so
>> SCons would be a natural option too (it can't build LLVM though). I'd only
>> use SCons conditional on it accepting a Rust dependency scanner into its
>> source: using its current custom scanner infrastructure is not practical as
>> I found out.
>>
>> As for waf... they and Debian have been having a tiff (e.g. see
>> http://waf-devel.blogspot.com/2012/01/debian.html ,
>> https://lists.debian.org/debian-devel/2012/02/msg00207.html ). I would
>> not suggest it based on that.
>>
>> -SL
>>
>>
>> _______________________________________________
>> Rust-dev mailing list
>> Rust-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/rust-dev
>>
>
>
> _______________________________________________
> Rust-dev mailing list
> Rust-dev at mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140110/fbc441de/attachment-0001.html>


More information about the Rust-dev mailing list