[rust-dev] rustpkg, package identifiers, filename conventions

[Graydon Hoare]

Hi,

We've recently replaced the cargo package manager with a new (very
young) tool called rustpkg; in the process it's (temporarily) broken
most of the packages installable through cargo, but we'll be fixing this
up before the next release.

The main new feature of rustpkg is that it permits writing custom build
logic (and custom build tasks) in rust itself, as an escape hatch: if
there's a file called pkg.rs in a package repository it is compiled,
linked against rustpkg itself and run for various tasks.

That's a nice feature and all, but we don't want to force (or even
_suggest_) use of the escape hatch by default. We'd like rustpkg to do
most things by inferring-sensible-defaults when looking at a repository.
In particular, building all the most obvious / findable binary and
library crates contained therein, based on simple rules. Preferably
rules that don't require parsing all the files in the repo and sorting
them into a module-graph just to figure out the targets. There might be
a lot of files (cf. src/test/run-pass/*.rs)

The simplest rule I've come up with is based on a convention for chosen
filenames and their immediately-containing dir names. Specifically the
names "lib.rs" for libraries and "bin.rs" for binaries:

  foo/lib.rs ==> build libfoo-<hash>.so / .dll, a library
  foo/bin.rs ==> build foo / foo.exe, an executable

This is a convention that'll require a certain amount of reorganization
to existing code, but it permits multi-library / multi-binary packages
in a single source repo, which is an important use-case. Lots of time
people want >1 crate in the same git tree. Moreover it makes for a
pretty simple naming heuristic that can be derived directly from
URL-path-fragment package identifiers, such that:

   extern mod foo (id = "github.com/graydon/foo");

will pick up a single-library package, assuming there's a single file
lib.rs in that repo, and give it the name libfoo-<hash>.so, because
'foo' is the immediate parent-dir of the lib.rs file. Whereas:

   extern mod foo (id = "github.com/graydon/foo/thing");

will look for a thing/lib.rs dir-and-stem inside the
github.com/graydon/foo repo, and build-and-link _that_ into
libthing-<hash>.so.

This should also permit mapping such identifiers to filesystem paths in
GOPATH-like workspaces, permitting local workspaces overlaid on global
ones and branching (via your VCS -- it does versions!) inside local
workspaces and whatnot. In case it's not clear from the sketch here,
it's very similar to what Go does -- I think they did a very tidy job
and I'm interested in copying its good parts. It feels like we didn't
quite hit the mark with cargo and this is an important thing to get right.

That said, I'm interested in some feedback before we get too far into
it, to make sure we aren't making serious mistakes or missing issues
that people feel are unpleasant about URL-centric schemes.

Any thoughts? Notes on other failure modes in package managers? Requests
for ponies? I'd prefer specificity in response, if you can manage it. We
all know package management is "generally miserable", but if you can
describe _exactly_ how things go wrong in other systems and what we
should be trying to avoid, or aiming for, that'd be good.

Thanks,

-Graydon