[rust-dev] Update on I/O progress

[Brian Anderson]

Hi.

As you may know, I'm in the process of redesigning the I/O module on top 
of a new task scheduler. With my [latest pull request] I've built enough 
infrastructure that I'm about ready to start implementing TCP streams. 
Before I get too deep though I want to make sure that everybody who 
cares is aware of the design.

[latest pull request]: https://github.com/mozilla/rust/pull/6046

I had a grand idea that I would be able to produce a full [design doc], 
but that's not going to happen I think. What I do have is a fair bit of 
[API documentation], along with a growing list of [unresolved issues], 
some of which I'll present here.

[design doc]: https://github.com/mozilla/rust/wiki/Lib-io
[API documentation]: 
https://github.com/brson/rust/blob/io/src/libcore/rt/io/mod.rs#L11
[unresolved issues]: 
https://github.com/brson/rust/blob/io/src/libcore/rt/io/mod.rs#L220

# Scope

What I am focused on now is the design of a synchronous API for sending 
and receiving streams of bytes, along with an implementation built on an 
internal *asynchronous* I/O interface attached to the scheduler event 
loop. This includes reading and writing files, TCP, UDP, Unix sockets, 
stdio, pipes, adapters for string parsing, compression, memory buffers, etc.

There are plenty of important and related issues that I have not 
considered yet, such as file and directory traversal, application-level 
protocols like HTTP, async I/O. We need to take important use cases into 
account, but don't necessarily need to implement them now.

I am particularly interested in making sure that common operations are 
convenient and don't require a lot of boilerplate. To that end I've been 
collecting [aspirational examples] of succinctly written Rust I/O code 
using the hypothetical API. The goal will be to turn these into test 
cases. I welcome additional examples.

[aspirational examples]: 
https://github.com/brson/rust/blob/io/src/libcore/rt/io/mod.rs#L24

# Overview

The API I am proposing has a structure based on what we have now, with a 
Reader and Writer trait that define a few primitive operations, along 
with a number of types that implement them. Types that implement Reader 
and Writer are called 'streams' and automatically implement `Stream`. 
The types are influenced primarily by the existing `core::io`, .NET, Go, 
and Ruby.

I recommend reading the previously linked module documentation for the 
complete story.

# Issues

Here I'll discuss some of the issues I've encountered so far.

## Error handling

I've spent a lot of time thinking about - and talking to Patrick about - 
how I/O should deal with errors (the old `io` doesn't have a consistent 
strategy). The [error handling strategy] I'm proposing is intended to 
eliminate boilerplate and allow efficient recovery from errors, but it 
uses some uncommon patterns.

[error handling strategy]: 
https://github.com/brson/rust/blob/io/src/libcore/rt/io/mod.rs#L128

In particular, it forces all return values to be 'nullable' or 
'zeroable', which will lead to the usual logic errors under the right 
circumstances. It also uses conditions, which aren't used much yet in Rust.

## String encoding

I have not yet put much thought about how to deal with string encoding 
and decoding. The existing `io` module simply has Reader and Writer 
extension traits that add a number of methods like `read_str`, etc. I 
think this is the wrong approach because it doesn't allow any extra 
state for the encoding/decoding, e.g. there's no way to customize the 
way newline is handled. Probably we'll need some decorator types like 
`StringReader`, etc.

## Close

Do we need to have `close` methods or do we depend solely on RAII for 
closing streams?

## Constructor options

There tend to be a lot of ways to open I/O types. Some potential examples:

* File::open("diary.txt")
* File::open(Path("diary.txt"))
* File::open(Path("diary.txt"), Create, WriteOnly)
* File::open(Url("file://diary.txt"))

The one you want to put at the front of the tutorial is the simple 
`File::open("diary.txt")`, but then how do you use the other variations? 
`File::open_path(...)`, `File::open_with_options(...)`?

I am considering defining `open` more like `fn open:<S: OpenSpec>(spec: 
S)`, then implementing `OpenSpec` for e.g. `&str`, `Path`, `(Path, 
FileFlags, FileAccess), `Url`, etc.

Does this seem reasonable?

# Closing

Sorry for the barrage of random notes. Again I encourage those 
interested to read the [rt::io module docs] and the rest of the [rt::io 
module] to get a clearer idea of where I'm going with this.

[rt::io module docs]: 
https://github.com/brson/rust/blob/io/src/libcore/rt/io/mod.rs#L11
[rt::io module]: https://github.com/brson/rust/tree/io/src/libcore/rt/io

-Brian