module Bench: Bench
Bench
Core_bench
is a micro-benchmarking library for OCaml that can measure execution
costs of operations that take 1ns to about 100ms. Core_bench
tries to measure
execution costs of such short-lived computations precisely while trying to account for
delayed GC costs and noise introduced by other activity on the system.
The easiest way to get started is using an example:
open Core.Std
open Core_bench.Std
let () =
Random.self_init ();
let x = Random.float 10.0 in
let y = Random.float 10.0 in
Command.run (Bench.make_command [
Bench.Test.create ~name:"Float add" (fun () ->
ignore (x +. y));
Bench.Test.create ~name:"Float mul" (fun () ->
ignore (x *. y));
Bench.Test.create ~name:"Float div" (fun () ->
ignore (x /. y));
])
When compiled this gives you an executable:
$ ./z.exe -ascii
Estimated testing time 30s (3 benchmarks x 10s). Change using -quota SECS.
Name Time/Run mWd/Run Percentage
----------- ---------- --------- ------------
Float add 2.50ns 2.00w 41.70%
Float mul 2.55ns 2.00w 42.52%
Float div 5.99ns 2.00w 100.00%
If any of the functions resulted in allocation on the major heap (mjWd) or promotions (Prom), columns corresponding to those would be automatically displayed. Columns that do not have significant values are not displayed by default. The most common options one would want to change are the `-q` flag which controls the time quota for testing and enabling/disabling specific columns. For example:
$ ./z.exe -ascii -q 1 cycles
Estimated testing time 3s (3 benchmarks x 1s). Change using -quota SECS.
Name Time/Run Cycls/Run mWd/Run Percentage
----------- ---------- ----------- --------- ------------
Float add 2.50ns 8.49c 2.00w 41.78%
Float mul 2.77ns 9.40c 2.00w 46.29%
Float div 5.99ns 20.31c 2.00w 100.00%
If you drop the `-ascii` flag, the output table uses extended Ascii characters. These display well on most modern terminals, but not on ocamldoc.
The simplest benchmark specification is just a unit -> unit
thunk and a name:
Bench.Test.create ~name:"Float add" (fun () -> ignore (x +. y));
One can also create indexed benchmarks, which can be helpful in understanding non-linearities in the execution profiles of functions. For example:
open Core.Std open Core_bench.Std
let () =
Command.run (Bench.make_command [
Bench.Test.create_indexed
~name:"Array.create"
~args:[1; 10; 100; 200; 300; 400]
(fun len ->
Staged.stage (fun () -> ignore(Array.create ~len 0)));
])
this produces:
$ ./z.exe -ascii -q 3
Estimated testing time 18s (6 benchmarks x 3s). Change using -quota SECS.
Name Time/Run mWd/Run mjWd/Run Percentage
------------------ ------------ --------- ---------- ------------
Array.create:1 27.23ns 2.00w 1.08%
Array.create:10 38.79ns 11.00w 1.53%
Array.create:100 124.05ns 101.00w 4.91%
Array.create:200 188.13ns 201.00w 7.44%
Array.create:300 1_887.20ns 301.00w 74.64%
Array.create:400 2_528.43ns 401.00w 100.00%
Executables produced using Bench.make_command
are self documenting (use the `-?`
flag). The documentation in the executable also closely corresponds to the
functionality exposed through the .mli and is a great way to interactively explore
what the various options do.
module Test:sig
..end
Test.t
are benchmarked by calls to bench.
module Variable:sig
..end
Variable.t
s represent variables than can be used as predictors or the responder
when specifying a regression.
module Run_config:sig
..end
Run_config.t
specifies how a benchmark should be run.
module Display_config:sig
..end
Display_config.t
specifies how the output tables should be formatted.
module Analysis_config:sig
..end
Analysis_config.t
specifies a regression run by Core_bench
.
module Measurement:sig
..end
Measurement.t
represents the result of measuring execution of a Test.t
.
val make_command : Test.t list -> Core.Std.Command.t
make_command tests
is the easiest way to generate a command-line program that runs a
list of benchmarks. Here tests : Test.t list
are the benchmarks that should be run.
This returns a Command.t
which provides a command-line interface for running the
benchmarks. See notes above for an example.val bench : ?run_config:Run_config.t ->
?analysis_configs:Analysis_config.t list ->
?display_config:Display_config.t -> Test.t list -> unit
bench tests
will run, analyze and display the specified tests
. Use this when one
needs more control over the execution parameters that what is exposed through
make_command
.val measure : ?run_config:Run_config.t ->
Test.t list -> Measurement.t list
measure
is a fragment of the functionality of bench
. measure tests
will run
the specified tests
and return the resulting measurement results.val analyze : ?analysis_configs:Analysis_config.t list ->
Measurement.t -> Analysis_result.t Core.Std.Or_error.t
analyze
is a fragment of the functionality of bench
. analyze ~analysis_configs m
will analyze the measurement m
using the regressions specified.val display : ?display_config:Display_config.t -> Analysis_result.t list -> unit
display
is a fragment of the functionality of bench
. display results
will
display a tabular summary of results
on the terminal.val make_command_ext : summary:string ->
extra_spec:('a, unit -> unit) Core.Std.Command.Spec.t ->
f:(Analysis_config.t list * Display_config.t *
[ `From_file of string list
| `Run of (Measurement.t -> string) option * Run_config.t ] ->
'a) ->
Core.Std.Command.t
make_command_ext
is useful for creating Command.t
s that have command line flags in
addition to those provided by make_command
.