Module Bench

module Bench: Bench

Please look at Bench

Core_bench is a micro-benchmarking library for OCaml that can measure execution costs of operations that take 1ns to about 100ms. Core_bench tries to measure execution costs of such short-lived computations precisely while trying to account for delayed GC costs and noise introduced by other activity on the system.

The easiest way to get started is using an example:

      open Core.Std
      open Core_bench.Std

      let () =
        Random.self_init ();
        let x = Random.float 10.0 in
        let y = Random.float 10.0 in
        Command.run (Bench.make_command [
          Bench.Test.create ~name:"Float add" (fun () ->
            ignore (x +. y));
          Bench.Test.create ~name:"Float mul" (fun () ->
            ignore (x *. y));
          Bench.Test.create ~name:"Float div" (fun () ->
            ignore (x /. y));
        ])

When compiled this gives you an executable:

      $ ./z.exe -ascii
      Estimated testing time 30s (3 benchmarks x 10s). Change using -quota SECS.

        Name        Time/Run   mWd/Run   Percentage
       ----------- ---------- --------- ------------
        Float add     2.50ns     2.00w       41.70%
        Float mul     2.55ns     2.00w       42.52%
        Float div     5.99ns     2.00w      100.00%

If any of the functions resulted in allocation on the major heap (mjWd) or promotions (Prom), columns corresponding to those would be automatically displayed. Columns that do not have significant values are not displayed by default. The most common options one would want to change are the `-q` flag which controls the time quota for testing and enabling/disabling specific columns. For example:

       $ ./z.exe -ascii -q 1 cycles
       Estimated testing time 3s (3 benchmarks x 1s). Change using -quota SECS.

         Name        Time/Run   Cycls/Run   mWd/Run   Percentage
        ----------- ---------- ----------- --------- ------------
         Float add     2.50ns       8.49c     2.00w       41.78%
         Float mul     2.77ns       9.40c     2.00w       46.29%
         Float div     5.99ns      20.31c     2.00w      100.00%

If you drop the `-ascii` flag, the output table uses extended Ascii characters. These display well on most modern terminals, but not on ocamldoc.

The simplest benchmark specification is just a unit -> unit thunk and a name:

      Bench.Test.create ~name:"Float add" (fun () -> ignore (x +. y));

One can also create indexed benchmarks, which can be helpful in understanding non-linearities in the execution profiles of functions. For example:

      open Core.Std open Core_bench.Std

      let () =
       Command.run (Bench.make_command [
         Bench.Test.create_indexed
           ~name:"Array.create"
           ~args:[1; 10; 100; 200; 300; 400]
           (fun len ->
              Staged.stage (fun () -> ignore(Array.create ~len 0)));
       ])

this produces:

      $ ./z.exe -ascii -q 3
      Estimated testing time 18s (6 benchmarks x 3s). Change using -quota SECS.

        Name                 Time/Run   mWd/Run   mjWd/Run   Percentage
       ------------------ ------------ --------- ---------- ------------
        Array.create:1        27.23ns     2.00w                   1.08%
        Array.create:10       38.79ns    11.00w                   1.53%
        Array.create:100     124.05ns   101.00w                   4.91%
        Array.create:200     188.13ns   201.00w                   7.44%
        Array.create:300   1_887.20ns              301.00w       74.64%
        Array.create:400   2_528.43ns              401.00w      100.00%

Executables produced using Bench.make_command are self documenting (use the `-?` flag). The documentation in the executable also closely corresponds to the functionality exposed through the .mli and is a great way to interactively explore what the various options do.

module Test: sig .. end

Test.t are benchmarked by calls to bench.

module Variable: sig .. end

Variable.ts represent variables than can be used as predictors or the responder when specifying a regression.

module Run_config: sig .. end

Run_config.t specifies how a benchmark should be run.

module Display_config: sig .. end

Display_config.t specifies how the output tables should be formatted.

module Analysis_config: sig .. end

Each Analysis_config.t specifies a regression run by Core_bench.

module Measurement: sig .. end

A Measurement.t represents the result of measuring execution of a Test.t.

val make_command : Test.t list -> Core.Std.Command.t

make_command tests is the easiest way to generate a command-line program that runs a list of benchmarks. Here tests : Test.t list are the benchmarks that should be run. This returns a Command.t which provides a command-line interface for running the benchmarks. See notes above for an example.

val bench : ?run_config:Run_config.t ->
       ?analysis_configs:Analysis_config.t list ->
       ?display_config:Display_config.t -> Test.t list -> unit

bench tests will run, analyze and display the specified tests. Use this when one needs more control over the execution parameters that what is exposed through make_command.

val measure : ?run_config:Run_config.t ->
       Test.t list -> Measurement.t list

measure is a fragment of the functionality of bench. measure tests will run the specified tests and return the resulting measurement results.

val analyze : ?analysis_configs:Analysis_config.t list ->
       Measurement.t -> Analysis_result.t Core.Std.Or_error.t

analyze is a fragment of the functionality of bench. analyze ~analysis_configs m will analyze the measurement m using the regressions specified.

val display : ?display_config:Display_config.t -> Analysis_result.t list -> unit

display is a fragment of the functionality of bench. display results will display a tabular summary of results on the terminal.

val make_command_ext : summary:string ->
       extra_spec:('a, unit -> unit) Core.Std.Command.Spec.t ->
       f:(Analysis_config.t list * Display_config.t *
          [ `From_file of string list
          | `Run of (Measurement.t -> string) option * Run_config.t ] ->
          'a) ->
       Core.Std.Command.t

make_command_ext is useful for creating Command.ts that have command line flags in addition to those provided by make_command.