[rust-dev] Designing long-running subtasks

Paul Nathan pnathan.software at gmail.com
Wed Jun 4 23:27:22 PDT 2014


I'm designing a system which, for the purposes of the email, is a
job/worker assignment queuing system (it's more nuanced than this, but this
is what the approach is). Also, it's not just a "toy" system, I'd like to
use it daily.

Fundamentally, the task is as follows:
The dispatcher receives a job list, sends out jobs to workers, occasionally
gets a new job list, assigns any unperformed task to free workers, and
monitors workers for being still around. Imagine if you would that the
workers are threads that do calls out to, say, java processes or SSH
subcommands - not pure Rust code.

One approach in C might be to have a list of jobs with mutexes around them,
some worker threads with mutexed state, and to have the dispatched check
through the lists and verify that everything is assigned correctly. The
workers' busyness would be indicated by a flag  on the worker struct.   So:
the worker and the dispatcher pass the job back and forth (but the
dispatcher can still look at the job state), and the dispatcher can eyeball
a worker's state. All very shared state-y (and managable with some mutex
discipline ).

This won't work in a CSP system. So....

Let's say we have our main task, which spawns worker tasks which
synchronously idle for new work to do. The dispatcher reads a new job, sets
the "working" flag to true, and sends it off to the pool. On receipt of a
do-work message, the worker begins his duties. After he's completed them,
he sends a message back to the dispatch that he is done and the results can
be collected. The job state is set to "finished", and the worker is then

Several difficulties present themselves: how does the dispatcher do its
checkups on the worker? The worker task will be presumably chugging along
and only flagging if something goes pear-shaped.

Similarily, the job state should correspond roughly on both the dispatcher
and the worker; the job should only be allocated to a free worker, which
will then essentially take write-capabilities on the job away from the

I think the shape of the problem is illustrated now.  Design ideas are
requested here.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/rust-dev/attachments/20140604/5089acc8/attachment.html>

More information about the Rust-dev mailing list