A type-safe parallel library built on top of Async_rpc.
module Worker = Parallel.Make_worker(T:Worker_spec)
The Worker
module can be used to spawn new workers, either locally or remotely, and
run functions on these workers. T
specifies which functions can be run on a
Worker.t
as well as the implementations for these functions. In addition, different
instances of Worker.t
types can be differentiated upon spawn
by passing in a
different init_arg
and handling this argument in the init
function of T
.
Basic usage:
module Worker = struct module T = struct type 'worker functions = TODO: CUSTOM
type init_arg = unit with bin_io let init = return
module Functions(C:Parallel.Creator) = struct let f1 = C.create_rpc ~f:(fun i -> return (i+42)) ~bin_input:Int.bin_t ~bin_output:Int.bin_t
let functions = TODO: CUSTOM end end include Parallel.Make_worker(T) end
let main ... = ... Worker.spawn_exn ~on_failure:Error.raise () >>= fun worker -> ... Worker.run_exn worker ~f:Worker.functions.f1 ~arg:0 >>= fun res -> ...
let command = Command.async ~summary spec main
let () = Parallel.start_app command
* NOTE *: It is highly recommended for Parallel.start_app
and Parallel.Make_worker
calls to be top-level. But the only real requirements are:
1) The master's state is initialized before any calls to spawn
. This will be
achieved either by Parallel.start_app
or Parallel.init_master_exn
.
2) Spawned workers (runs of your executable with a certain environment variable set)
must start running as a worker. This will be achieved either by Parallel.start_app
or Parallel.run_as_worker_exn
. With Parallel.run_as_worker_exn
, you must carefully
supply the necessary command arguments to ensure Parallel.run_as_worker_exn
is
indeed called.
3) Spawned workers must be able to find their function implementations when they start
running as a worker. These implementations are gathered on the application of the
Parallel.Make_worker
functor.
A word on monitoring:
Exceptions in workers will always result in the worker shutting down. The master can be notified of these exceptions in multiple ways:
Error.t
of the run
(or spawn
) call.on_failure exn
will be called.redirect_stderr
specifies a file, the worker will write its exception to that
file before shutting down.('worker, 'query, 'response) Function.t
is a type-safe function 'query ->
'response Deferred.t
that can only be run on a 'worker
.
start_app command
should be called from the top-level in order to start the parallel
application. command
must be constructed with a call to Command.async
so that the
Async scheduler is started. This function will parse certain environment variables and
determine whether to start as a master or a worker.
rpc_max_message_size
, rpc_heartbeat_config
, where_to_listen
specify the RPC
server that the master starts.
implementations
are additional implementations that the master will implement.
Expert
module instead of start_app
.