Module `Rpc_parallel`

A type-safe parallel library built on top of Async_rpc.

module Worker = Rpc_parallel.Make (T : Worker_spec)

The Worker module can be used to spawn new workers, either locally or remotely, and run functions on these workers. T specifies which functions can be run on a Worker.t as well as the implementations for these functions. In addition, T specifies worker states and connection states. See README for more details

module Remote_executable : sig ... end

module Executable_location : sig ... end

module Fd_redirection : sig ... end

module Managed = Parallel_managed

module Map_reduce : sig ... end: A parallel map/reduce library. See examples/add_numbers.ml and examples/number_stats.ml for examples.

include Parallel

include Rpc_parallel__.Parallel_intf.Parallel

module Function : Rpc_parallel__.Parallel_intf.Function

module type Worker : Rpc_parallel__.Parallel_intf.Worker with type ('w, 'q, 'r) _function := ('w, 'q, 'r) Function.t

module type Functions = Rpc_parallel__.Parallel_intf.Functions

module type Creator : Rpc_parallel__.Parallel_intf.Creator with type ('w, 'q, 'r) _function := ('w, 'q, 'r) Function.t and type ('w, 'q, 'r) _direct := ('w, 'q, 'r) Function.Direct_pipe.t

module type Worker_spec : Rpc_parallel__.Parallel_intf.Worker_spec with type ('w, 'q, 'r) _function := ('w, 'q, 'r) Function.t and type ('w, 'q, 'r) _direct := ('w, 'q, 'r) Function.Direct_pipe.t

module Make : functor (S : Worker_spec) -> Worker with type 'a functions := 'a S.functions and type worker_state_init_arg := S.Worker_state.init_arg and type connection_state_init_arg := S.Connection_state.init_arg: module Worker = Make(T)

val start_app : ?⁠rpc_max_message_size:int -> ?⁠rpc_handshake_timeout:Core.Time.Span.t -> ?⁠rpc_heartbeat_config:Async.Rpc.Connection.Heartbeat_config.t -> Async.Command.t -> unit

start_app command should be called from the top-level in order to start the parallel application. This function will parse certain environment variables and determine whether to start as a master or a worker.

rpc_max_message_size, rpc_handshake_timeout, rpc_heartbeat_config can be used to alter the rpc defaults. These rpc settings will be used for all connections. This can be useful if you have long async jobs.

module State : sig ... end: Use State.get to query whether the current process has been initialized as an rpc parallel master (start_app or init_master_exn has been called). We return a State.t rather than a bool so that you can require evidence at the type level. If you want to certify, as a precondition, for some function that start_app was used, require a State.t as an argument. If you don't need the State.t anymore, just pattern match on it.

module For_testing : sig ... end

module Expert : sig ... end: If you want more direct control over your executable, you can use the Expert module instead of start_app. If you use Expert, you are responsible for starting the master and worker rpc servers. worker_command_args will be the arguments sent to each spawned worker. Running your executable with these args must follow a code path that calls worker_init_before_async_exn and then start_worker_server_exn. An easy way to do this is to use worker_command.

module Std : sig ... end: Old Std style interface, which has slightly different module names.

module Parallel : sig ... end

module Parallel_managed : sig ... end: This module is primarily meant for backwards compatibility with code that used earlier versions of Rpc_parallel. Please consider using the Parallel.Make() functor instead as the semantics are more transparent and intuitive.