Module Parallel

A type-safe parallel library built on top of Async_rpc.

module Worker = Parallel.Make_worker(T:Worker_spec)

The Worker module can be used to spawn new workers, either locally or remotely, and run functions on these workers. T specifies which functions can be run on a Worker.t as well as the implementations for these functions. In addition, different instances of Worker.t types can be differentiated upon spawn by passing in a different init_arg and handling this argument in the init function of T.

Basic usage:

module Worker = struct module T = struct type 'worker functions = TODO: CUSTOM

type init_arg = unit with bin_io let init = return

module Functions(C:Parallel.Creator) = struct let f1 = C.create_rpc ~f:(fun i -> return (i+42)) ~bin_input:Int.bin_t ~bin_output:Int.bin_t

let functions = TODO: CUSTOM end end include Parallel.Make_worker(T) end

let main ... = ... Worker.spawn_exn ~on_failure:Error.raise () >>= fun worker -> ... Worker.run_exn worker ~f:Worker.functions.f1 ~arg:0 >>= fun res -> ...

let command = Command.async ~summary spec main

let () = Parallel.start_app command

* NOTE *: It is highly recommended for Parallel.start_app and Parallel.Make_worker calls to be top-level. But the only real requirements are:

1) The master's state is initialized before any calls to spawn. This will be achieved either by Parallel.start_app or Parallel.init_master_exn.

2) Spawned workers (runs of your executable with a certain environment variable set) must start running as a worker. This will be achieved either by Parallel.start_app or Parallel.run_as_worker_exn. With Parallel.run_as_worker_exn, you must carefully supply the necessary command arguments to ensure Parallel.run_as_worker_exn is indeed called.

3) Spawned workers must be able to find their function implementations when they start running as a worker. These implementations are gathered on the application of the Parallel.Make_worker functor.

A word on monitoring:

Exceptions in workers will always result in the worker shutting down. The master can be notified of these exceptions in multiple ways:

• If the exception occured in a function implementation before return, the exception will be returned back in the Error.t of the run (or spawn) call.

• If the exception occured after return, on_failure exn will be called.

• If redirect_stderr specifies a file, the worker will write its exception to that file before shutting down.