Changelog

113.24.00

async

Keep up to date with interface changes in Async_kernel, Async_extra and Async_unix.

async_extended

Make LTL predicates comparable by tagging and id to each one. Fixes a functional comparison bug.
Switched to PPX.
Add an mli for async_extended/src/reader_ext.ml and remove a couple of unused functions (notably Reader_ext had its own version of Reader.read_char).
Add async-friendly color print
Ltl.eval should close the pipe after it is done with it.
Deleted Async_extended.Cml.
Remove Async_extended.Std.Gzip and redirect references to Async_gzip.Std.Gzip.
Update Command_rpc.Connection to check the program before exec'ing it. The filename must now be absolute, exist, and be executable. Previously errors with nonexistent or nonexecutable files would only be found out after forking.
Change Command_rpc.Command to use Versioned_rpc.Callee_converts instead of Versioned_rpc.Both_convert so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate.

Removed val rpcs from Callee_converts interfaces because nothing appears to use it, and Both_convert does not provide it. Now Both_convert.S can be supplied to satisfy Callee_converts.S.
Add simple example of Command_rpc
Add Deferred_cache.
Fixing a couple of issues noticed in Command_rpc:
- If propagate_stderr is false, the child's stderr is now drained instead of ignored.
- When connections are closed, stderr is now closed as well, which prevents a file descriptor leak if the child process is unresponsive.

async_extra

N.B. some changes happening for this release are not listed in this changelog since they appear only as a consequence of changes in core or async_kernel.

When Transfer.Writer.send* raises, send an error to the client.
Add a new rpc that enables a "push" rather than a "poll" model.
Switched to PPX.
For connected UDP sockets, expose send in the same fashion as sendto.
Tcp.Server is documented to refuse excess connections beyond max_connections + max_pending_connections, but it treats them as pending connections in our standard OS configuration. In fact, research indicates that the documented behavior is nearly impossible to obtain directly and consistently from listen.

Clarify the name and role of the backlog argument to listen and rename and update documentation for max_pending_connections to clarify what it actually does, in light of some research:
```
`listen` does not generally respect the backlog argument as an
upper limit, but as a lower limit (mod `tcp_max_syn_backlog`) and,

with `tcp_abort_on_overflow=0`, `listen` will ignore excess
connections rather than actively refusing them.

(With `syncookies=1`, this can look like an indefinite backlog.)
```
Existing, working code can substitute max_pending_connections -> backlog and move on. The behavior is not changed.

When possible, consider architecting applications so the server can simply accept and close excess connections, rather than relying on the listen backlog to return an active indication to the client that they won't be serviced. To make sure the client receives an RST rather than an orderly shutdown, you can set the linger time to 0 before closing the socket. (Added to unit tests.)

Direct Tcp.Server support for this paradigm is left for future work.
Make Rpc_low_latency_transport treat disconnections as eof, like Async_unix.Reader does.
Add an implementation of Mvars to Async
Allow custom handling of missed async_rpc heartbeats.
adds a configuration limit on the number of tokens that can be in-flight
Replace an #include <sys/errno.h> by #include <errno.h>.

Fixes janestreet/async_extra#4
Added Tcp.Server.sexp_of_t
Adds Rpc.Pipe_rpc.dispatch_iter, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: Pipe_rpc.dispatch followed by Pipe.iter_without_pushback consumes ~105 words in the steady state (i.e., when no messages are coming in) while dispatch_iter consumes ~15. I'm sure dispatch can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as dispatch_iter.

Both cases can be made smaller by making Connection.response_handler a GADT instead of a closure. I plan to do this later.

One annoying property of the interface is that the only way to cancel a subscription is to use Pipe_rpc.abort, which has a terrible interface. The logical way to improve the interface is to return a record of a Pipe_rpc.t, a Connection.t, and a Query_id.t, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage.
Added Tcp.Server.listening_on_address, so that one can get the address a server is listening on, as compared with listening_on, which just returns the port.
Marked Command.async_basic as deprecated using the appropriate ocaml attribute.

@@ocaml.deprecated

(http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec241)
Extend the interface of Persistent_rpc_client to make the "address" type - previously fixed as Host_and_port.t - abstract. This is helpful for integrating with libraries that have a different notion of an address, e.g. rpc_discovery_lib.
Typed_tcp mutated a Hashtbl while iterating over it when closing.
Added Async.Bus.first_exn, which takes a bus and a function, and returns a deferred that becomes determined when the first event is published to the bus for which the function returns Some.

This function is useful to reduce boilerplate for dealing with unsubscription.
Reduced the number of threads required by tests in:

async_extra/src/tcp.ml
Added to the error message Bus.subscribe_exn called after first write the source-code position of the caller, in case there isn't a backtrace, to make the source of the problem clearer, and to avoid confusion with other source-code positions of subscribers already in the bus.
Added to Bus.first_exn a Source_code_position.t argument, so that in the event of subscription failure, we can see who caused the subscription to the bus.
Added to Tcp.Server.close an optional argument:

?close_existing_connections : bool

This closes the sockets of all existing connections.
Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket).
Improved Async.Udp.bind's error message when it fails to mcast_join a multicast group.
Change ~callback to ~f throughout the Bus interface

async_find

Initial release.

async_kernel

N.B. some interface change in Core (notably to Hashtbl and Map) implied some interface change in this package as well, although they are not mentionned in this changelog.

Switched to ppx.
Improved the Async scheduler's to allocate a handle_fired function once, rather than every time it calls advance_clock.
Removed configurability of Monitor's try_with-ignored-exception handling, i.e. removed Monitor.try_with_rest_handling and Monitor.try_with_ignored_exn_handling.

The behavior of exceptions raised to a monitor after it returns is unchanged -- i.e. they are logged, as they have been since 112.28.

Changed Monitor.try_with's ?rest argument from:

?rest : Ignore | Raise ?rest : Log | Raise

This naming reflects the fact that subsequent exceptions are logged, not ignored.
In Async_kernel, moved Scheduler.set_execution_context from the Types.Scheduler module to its own file. Because Types is a module rec, set_execution_context hadn't been inlined and was called via caml_apply2. In its own file, it will be inlined.

This release creates a new scheduler0.ml, and moves the old scheduler0.ml to scheduler1.ml.
Fixed a space leak in Pipe due to a pipe holding pointers to its upstream_flusheds after they are closed. The leak shows up in Pipe.transfer and related functions, e.g. with:

Pipe.transfer temporary_pipe long_lived_pipe

called repeatedly, in which long_lived_pipe would accumulate a large number of upstream_flusheds.

The fix is to maintain upstream_flusheds as a Bag.t, and to remove an upstream pipe when it is closed.
Implement Pipe.of_sequence
Improved the error message when an exception is raised to a Monitor.try_with that has returned before Async has initialized Monitor.try_with_log_exn.
Improved the implementation of Monitor.get_next_error, by replacing the monitor's list of handlers:

; mutable handlers_for_next_error : (exn -> unit) list

with a single ivar:

; mutable next_error : exn Ivar.t

I think this wasn't done originally because of a dependency cycle. But now that we have types.ml, we can do the clear thing.
Improved the implementation of Monitor exception handling, i.e. detach_and_iter_errorsto make it clear that Monitor.send_exn does not run user code -- it only schedules jobs.
Fix an error message in Pipe to match the condition that led to it.
Add a new pipe constructor:

val unfold : 'b -> f:('b -> ('a * 'b) option Deferred.t) -> 'a Reader.t

unfold is more powerful than the combination of

Useful for, e.g., creating a pipe of natural numbers:

Pipe.unfold 0 ~f:(fun n -> return (Some (n, n+1)))
Add Deferred.Map.all similar to Deferred.List.all.

This does what you would expect:

val all : ('a, 'b Deferred.t, 'cmp) Map.t -> ('a, 'b, 'cmp) Map.t Deferred.t
Added some simple functions that seem missing from Deferred and Ivar.

val Ivar.peek : 'a t -> 'a option val Ivar.value_exn : 'a t -> 'a val Deferred.value_exn : 'a t -> 'a
Add Monitor.catch_error, which provides error handling for processes/subsystems intended to run forever.
Added to the Async scheduler a configurable:

min_inter_cycle_timeout : Time_ns.Span.t

When scheduler calls epoll(), it uses a timeout of at least min_inter_cycle_timeout.

min_inter_cycle_timeout can be configured via ASYNC_CONFIG or via

val Scheduler.set_min_inter_cycle_timeout : Time_ns.Span.t -> unit

This allows one to tweak the scheduler to be more fair w.r.t. threads, e.g. with:

Scheduler.set_min_inter_cycle_timeout <- Time_ns.Span.of_us 1.;
Optimized Scheduler.schedule' to avoid a closure allocation.
Removed Monitor.kill, which was unused internally. This removes the kill_index field from Execution_context.t, which saves us a word everytime we allocate or store an execution context.
Assert that Deferred.forever never returns statically, rather than dynamically.
Changed Async.Std to not include Deferred.Monad_syntax, so that one must explicitly opt in (via open Deferred.Monad_syntax) to use let%bind syntax with Async.
Add Pipe.to_sequence
Make Stream.closed s return immediately when s is already closed.

Currently the following property holds:

for any s, Deferred.peek (Stream.closed s) = None
For Pipe functions that deal with batches of elements in a queue, added an optional argument:

?max_queue_length : int (** default is Int.max_value *)

This limits the size of the queue that is used, which can improve Async fairness.

Affected functions are:

filter_map filter_map' fold' iter' map' read' read_now' transfer' transfer_id

This also obviates read_at_most and read_now_at_most, which we will deprecate in a later release.

Removed a couple helper types, iter and fold, that had been used to express commonality among functions, but were becoming unwieldy due to differences.
Changed Pipe's default max_queue_length from Int.max_value to 100.
Added to Pipe.iter_without_pushback an optional argument:

?max_iterations_per_job : int (** default is Int.max_value *)

iter_without_pushback will not make more than max_iterations_per_job calls to f in a single Async_job; this can be used to increase Async-scheduling fairness.
Added Pipe.write_if_open which does exactly what it says. This is a common pattern. Also added a pushback-oblivious variant write_without_pushback_if_open.

Call sites for these two new functions were introduced wherever I found that doing so would not introduce any side effects (even counting allocation) in the case of a closed pipe.

async_parallel

Switched to ppx.

async_rpc_kernel

When Transfer.Writer.send* raises, send an error to the client.
Added Pipe_rpc in Versioned_rpc.Both_convert.
Switched to ppx.
Expose the lower-level registration hook in Versioned_rpc.
Allow custom handling of missed async_rpc heartbeats.
Client-side State_rpc dispatch function does not behave well when the reader side of the pipe is closed.

It should gracefully abort the rpc instead of raising exceptions, or whatever it currently does.
Add Rpc.Expert.implement and Rpc.Expert.implement' with a similar interface as One_way.Expert.implement but for 2-way rpcs.

Exceptions raised by an expert implementations are handled as follow:
- if the query has already been answered, stop the server (as for one-way expert)
- if not, send a Rpc_error.Uncaught_exn (as for 2-way rpc)
Adds Rpc.Pipe_rpc.dispatch_iter, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: Pipe_rpc.dispatch followed by Pipe.iter_without_pushback consumes ~105 words in the steady state (i.e., when no messages are coming in) while dispatch_iter consumes ~15. I'm sure dispatch can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as dispatch_iter.

Both cases can be made smaller by making Connection.response_handler a GADT instead of a closure. I plan to do this later.

One annoying property of the interface is that the only way to cancel a subscription is to use Pipe_rpc.abort, which has a terrible interface. The logical way to improve the interface is to return a record of a Pipe_rpc.t, a Connection.t, and a Query_id.t, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage.
Adds Rpc.Pipe_rpc.implement_direct, which uses a "direct stream writer" to write results to the other side, rather than using a Pipe.Writer.t. This consumes much less memory, ~15 words per open query as opposed to ~225 for a regular Pipe_rpc implementation.

A large part of the diff in this feature is the addition of a module Implementation_types, which just contains copies of type definitions from Implementation and Implementations. This is necessary to handle some cyclic type definitions (both of these modules now depend on types defined in the other module).

This is the implementation-side dual of Pipe_rpc.dispatch_iter.
Change Command_rpc.Command to use Versioned_rpc.Callee_converts instead of Versioned_rpc.Both_convert so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate.

Removed val rpcs from Callee_converts interfaces because nothing appears to use it, and Both_convert does not provide it. Now Both_convert.S can be supplied to satisfy Callee_converts.S.
Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket).
Bring back val rpcs in versioned_rpc modules.

async_shell

Initial release.

async_smtp

Switched to PPX.
Follow Core & Async evolution.

async_ssl

Switched to ppx.

async_unix

In the periodic check for a Writer buffer have too old data, eliminated allocation and generally improved performance.

This eliminated a large source of allocation in a simple TCP pingpong benchmark bench/pingpong.
Removed allocation in the Async scheduler's code that detects the thread-pool being stuck. This involved switching it to use Time_ns rather than Time.

This eliminates a relatively large source of allocation in a simple TCP-pingpong benchmark bench/pingpong.
Switched to ppx.
Improved the Async scheduler's to allocate a handle_fired function once, rather than every time it calls advance_clock.
Added Fd_by_descr.find_exn, to avoid the option allocated by Fd_by_descr.find. Used it to reduce allocation in the Async scheduler.
Improved Reader.load_sexp* functions to behave better when loading from a non files, e.g. a pipe. Previously, it produced an empty error message because it mistakenly attempted to read the sexp a second time in order to determine the error position. Now it produces a good error message, but without the (impossible to obtain) error position.
In Async_unix.Syscall, added a zero-allocation syscall interface, removing sources of allocation as observed when running a simple TCP pingpong benchmark (found in bench/pingpong).
Added

val time_spent_waiting_for_io : unit -> Time_ns.Span.t

to Scheduler which returns the amount of time that the Async scheduler has spent in calls to epoll_wait (or select) since the start of the program.
Changed In_thread.Helper_thread.create from:

val create : ?priority:Priority.t -> ?name:string -> unit -> t Or_error.t

to:

val create : ?priority:Priority.t -> ?name:string -> unit -> t Deferred.t

Kept around the prior function, renamed as create_now.

Switching create to return a deferred allows it to, when there are no available threads, wait until one becomes available. This, in turn, avoids rare nondeterminstic failures in programs that make heavy use of the thread pool and create a helper thread, when the creation happens at just the wrong time, when no thread is available.

Split out Thread_safe_ivar from the internals of Thread_pool, so that it can be used in other tests, and in particular in a new unified test added by this feature.
Make Unix_syscalls.Stats bin-io-able.
Fixed a bug in Thread_safe.block_on_async*, in which the execution context wasn't properly restored before returning.
Add a function in Process that expects empty output, mirroring Shell.run.
Added Reader.read_one_iobuf_at_a_time, which is like read_one_chunk_at_a_time, except that the user-supplied handle_chunk function receives its data in an Iobuf.t, and uses the Iobuf position to communicate how much data was consumed.

This facilitates using reader in scenarios (such as with protogen) where Iobufs are expected (and presently allocated around the bigstring at each call) and the calculation of consumed bytes from the Iobuf is duplicated in few places.
Log.message used to always logs the message, even if its log level was too low. This has been fixed.
Add writer functions to schedule an iobuf to be written out.
Add Unix.Inet_addr.Stable.
Alter Async.Std.Socket.Address.Inet.Blocking_sexp to expose the polymorphic variant functions, so that you can include it in a separate polymorphic variant type.

Also, expose Async.Std.Socket.Address.Inet.__t_of_sexp__ to give a deprecation message, instead of a message about the function not existing.
Fixed a bug in Async's In_thread.Helper_thread, which wasn't finalizing helper threads, due to a bug in Thread_pool, which wasn't finalizing helper threads. The fix was to move the finalization out of Thread_pool, where we don't care about it, to In_thread.Helper_thread, where we do.

Added Scheduler.max_num_threads : unit -> int.
Make Epoll_file_descr_watcher trigger callbacks for error conditions such as closed pipes.

Testing

Three new unit tests, all validating appropriate behavior in the case that a Unix pipe is opened, then the read end is closed after reading only part of the data sent by the write end.
1. A test in writer.ml verifying that Writer.consumer_left is triggered. Before the fix to Epoll_file_descr_watcher, Writer.consumer_left would never become determined in this case.
2. A test in fd.ml verifying that Fd.ready_to is triggered for the writing fd. Before the fix to Epoll_file_descr_watcher, Fd.ready_to would never become determined in this case.
3. A test in linux_ext.ml verifying that Epoll.wait_timeout_after is triggered. This test shows that epoll reports the ERR flag for the file descriptor in this case, and therefore that Epoll_file_descr_watcher needs to pay attention to the ERR flag.
Added to Writer.write_sexp an optional ?terminate_with argument, that specifies how to terminate the string representation of the sexp. This also makes it clear that the default behavior, ~terminate_with:Space_if_needed, might append a space to the sexp you are outputting if its representation is not enclosed in either () or "" . Sexp.output_hum an Sexp.output_mach do not have this behavior, so porting non-async code to async could introduce unexpected differences in the output.
Add an Async wrapper for Core.Std.Unix.getifaddrs.

bignum

Switched to PPX.
The library used polymorphic compare, rather than Zarith.Q's compare, in a few locations. Fixed this.
Previously stable types in Bignum were defined with unstable types in the scope. Fixd this.
Update to zarith-1.4
Bignum.of_string needs to handle different formats for its input. The previous version of the code was trying to parse the common format (floats), and in case of failure, was attempting to use a different format (based on the error). This resulted in the string being parsed twice in some cases.

This version is a complete rewriting of of_string to do the parsing in one step. The new code for to_string encode an automaton and remembers the positions of the various elements of the string (depending on the format).

This feature uses a function which has been upstreamed in the new version of ZArith (1.4) which is a variant of the Zarith.of_string function to work with substrings. This variant alone is responsible for a big part of the performance improvement.

Summary of benchmarks

The new version of the code performs better than the original one in all cases. The performance improvement are variable depending on the micro benchmark. See below.

Follow ups

We also tried to implement the lexing engine using OCamllex. This makes for a much more concise description, but the performance are significantly lower. OCamllex produces code which allocates some table and some state, which is avoided in the hand written code. Also, it will allocate the sub strings matched.

Benchmark results

New version (patch for ZArith + of_substring, reimplementation of of_string)

┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ bigint\_bench.ml random │ 48_381.13ns │ 7_166.00w │ 1.24w │ 1.24w │ 45.12% │ │ bigint\_bench.ml:vs. Big\_int plus_self │ 293.96ns │ 72.00w │ │ │ 0.27% │ │ bigint\_bench.ml:vs. Big\_int plus_other │ 807.62ns │ 124.00w │ │ │ 0.75% │ │ bigint\_bench.ml:vs. Big\_int mult_self │ 353.98ns │ 91.00w │ │ │ 0.33% │ │ bigint\_bench.ml:vs. Big\_int mult_other │ 783.78ns │ 128.00w │ │ │ 0.73% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (decimal) │ 14_415.44ns │ 475.00w │ │ │ 13.44% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (scientific) │ 61_363.80ns │ 3_929.00w │ │ │ 57.23% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (fraction) │ 24_957.02ns │ 303.00w │ │ │ 23.28% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (decimal) │ 15_867.52ns │ 1_523.00w │ │ │ 14.80% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (scientific) │ 33_345.31ns │ 4_206.00w │ │ │ 31.10% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (fraction) │ 31_770.26ns │ 3_779.00w │ │ │ 29.63% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (decimal) │ 9_726.82ns │ 380.00w │ │ │ 9.07% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (scientific) │ 28_141.40ns │ 2_059.00w │ │ │ 26.25% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (fraction) │ 70_436.16ns │ 5_541.00w │ │ │ 65.69% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (decimal) │ 27_000.73ns │ 1_994.00w │ │ │ 25.18% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (scientific) │ 66_057.63ns │ 6_217.00w │ │ │ 61.61% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (fraction) │ 107_219.89ns │ 8_097.00w │ │ │ 100.00% │ │ bignum\_bench.ml:Bignum binprot roundtrip compact │ 5_997.81ns │ 581.00w │ │ │ 5.59% │ │ bignum\_bench.ml:Bignum binprot roundtrip classic │ 18_522.20ns │ 779.00w │ │ │ 17.27% │ │ bignum\_bench.ml:round round_decimal:0 │ 8_479.49ns │ 463.00w │ │ │ 7.91% │ │ bignum\_bench.ml:round round_decimal:3 │ 24_621.71ns │ 2_115.00w │ │ │ 22.96% │ │ bignum\_bench.ml:round round_decimal:6 │ 26_896.35ns │ 2_437.00w │ │ │ 25.09% │ │ bignum\_bench.ml:round round_decimal:9 │ 29_428.19ns │ 2_730.00w │ │ │ 27.45% │ │ bignum\_bench.ml:round round │ 8_452.31ns │ 459.00w │ │ │ 7.88% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘

Original version

┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ bigint\_bench.ml random │ 51_218.04ns │ 7_166.00w │ 1.25w │ 1.25w │ 43.26% │ │ bigint\_bench.ml:vs. Big\_int plus_self │ 336.84ns │ 72.00w │ │ │ 0.28% │ │ bigint\_bench.ml:vs. Big\_int plus_other │ 837.73ns │ 124.00w │ │ │ 0.71% │ │ bigint\_bench.ml:vs. Big\_int mult_self │ 411.03ns │ 91.00w │ │ │ 0.35% │ │ bigint\_bench.ml:vs. Big\_int mult_other │ 808.03ns │ 128.00w │ │ │ 0.68% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (decimal) │ 29_650.60ns │ 2_415.00w │ │ │ 25.04% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (scientific) │ 92_495.93ns │ 6_465.00w │ │ │ 78.12% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (fraction) │ 39_482.77ns │ 2_060.00w │ │ │ 33.35% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (decimal) │ 16_195.93ns │ 1_523.00w │ │ │ 13.68% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (scientific) │ 34_227.78ns │ 4_059.00w │ │ │ 28.91% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (fraction) │ 32_856.17ns │ 3_779.00w │ │ │ 27.75% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (decimal) │ 19_745.71ns │ 2_149.00w │ │ │ 16.68% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (scientific) │ 51_024.99ns │ 3_853.00w │ │ │ 43.09% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (fraction) │ 88_884.15ns │ 7_819.00w │ │ │ 75.07% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (decimal) │ 32_812.27ns │ 2_498.00w │ │ │ 27.71% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (scientific) │ 77_518.77ns │ 6_369.00w │ │ │ 65.47% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (fraction) │ 118_402.78ns │ 8_907.00w │ │ │ 100.00% │ │ bignum\_bench.ml:Bignum binprot roundtrip compact │ 8_947.02ns │ 371.00w │ │ │ 7.56% │ │ bignum\_bench.ml:Bignum binprot roundtrip classic │ 22_799.74ns │ 1_039.00w │ │ │ 19.26% │ │ bignum\_bench.ml:round round_decimal:0 │ 8_176.74ns │ 463.00w │ │ │ 6.91% │ │ bignum\_bench.ml:round round_decimal:3 │ 25_798.77ns │ 2_115.00w │ │ │ 21.79% │ │ bignum\_bench.ml:round round_decimal:6 │ 28_561.23ns │ 2_437.00w │ │ │ 24.12% │ │ bignum\_bench.ml:round round_decimal:9 │ 30_861.38ns │ 2_730.00w │ │ │ 26.06% │ │ bignum\_bench.ml:round round │ 8_237.26ns │ 459.00w │ │ │ 6.96% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘

Tentative version using OCamllex

┌─────────────────────────────────────────────────────────────────────┬──────────────┬────────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼────────────┼──────────┼──────────┼────────────┤ │ bigint\_bench.ml random │ 48_164.21ns │ 7_166.00w │ 1.25w │ 1.25w │ 39.99% │ │ bigint\_bench.ml:vs. Big\_int plus_self │ 285.84ns │ 72.00w │ │ │ 0.24% │ │ bigint\_bench.ml:vs. Big\_int plus_other │ 768.12ns │ 124.00w │ │ │ 0.64% │ │ bigint\_bench.ml:vs. Big\_int mult_self │ 343.14ns │ 91.00w │ │ │ 0.28% │ │ bigint\_bench.ml:vs. Big\_int mult_other │ 780.00ns │ 128.00w │ │ │ 0.65% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (decimal) │ 26_931.12ns │ 3_108.00w │ │ │ 22.36% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (scientific) │ 79_750.28ns │ 6_599.00w │ 0.11w │ 0.11w │ 66.21% │ │ bignum\_bench.ml:Bignum of\_string/to\_string of_string (fraction) │ 34_988.94ns │ 4_300.00w │ │ │ 29.05% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (decimal) │ 15_958.17ns │ 1_523.00w │ │ │ 13.25% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (scientific) │ 32_495.25ns │ 4_059.00w │ │ │ 26.98% │ │ bignum\_bench.ml:Bignum of\_string/to\_string to_string (fraction) │ 31_802.75ns │ 3_779.00w │ │ │ 26.40% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (decimal) │ 18_742.81ns │ 2_924.00w │ │ │ 15.56% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (scientific) │ 45_282.09ns │ 4_622.00w │ │ │ 37.60% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp of_sexp (fraction) │ 86_907.83ns │ 8_777.00w │ 0.15w │ 0.15w │ 72.16% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (decimal) │ 35_727.73ns │ 4_493.00w │ │ │ 29.66% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (scientific) │ 82_247.61ns │ 8_273.00w │ 0.13w │ 0.13w │ 68.29% │ │ bignum\_bench.ml:Bignum of\_sexp/to\_sexp to_sexp (fraction) │ 120_445.25ns │ 10_688.00w │ 0.12w │ 0.12w │ 100.00% │ │ bignum\_bench.ml:Bignum binprot roundtrip compact │ 6_734.49ns │ 371.00w │ │ │ 5.59% │ │ bignum\_bench.ml:Bignum binprot roundtrip classic │ 21_773.79ns │ 1_890.00w │ │ │ 18.08% │ │ bignum\_bench.ml:round round_decimal:0 │ 8_306.45ns │ 463.00w │ │ │ 6.90% │ │ bignum\_bench.ml:round round_decimal:3 │ 24_714.96ns │ 2_115.00w │ │ │ 20.52% │ │ bignum\_bench.ml:round round_decimal:6 │ 26_894.27ns │ 2_437.00w │ │ │ 22.33% │ │ bignum\_bench.ml:round round_decimal:9 │ 29_343.81ns │ 2_730.00w │ │ │ 24.36% │ │ bignum\_bench.ml:round round │ 8_296.05ns │ 459.00w │ │ │ 6.89% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴────────────┴──────────┴──────────┴────────────┘

bin_prot

Bin_prot can be configured to use the primitives to read/write integers from bigarrays. This was never enabled due to missing tests that selecting this code path doesn't change the format.

This version add these tests and enable the use of the fast primitives.
Add benchmarks for all exposed bin_prot read/write functions. These are intended to check performance regressions.
Remove most use of cpp in bin_prot.

Replace the pre-processor conditionals by runtime one. This make the code a little less obfuscated.
Remove big literals so that the compiler does not complain in 32bit

core

N.B. Some interface change were made which are not listed here, as they are only cascading from core_kernel. Look at core_kernel's CHANGES.md file to get a complete history of the changes made for this release.

Add Core.Command.Arg_type.Export.time_zone
Fix Command.shape to run external programs only once to get their sexp.

Introduces a new variant of Command.t called Proxy. The Exec variant represents the top-level command of an external executable that has not yet been run; the Proxy variant represents an arbitrary subcommand that has been extracted by running an external executable. Command.exec constructs an Exec variant; Command.shape of an Exec variant runs the executable and generates a tree of Proxy variants representing all the information from the generated sexp, so the executable will not need to be re-run.
A version of recvmmsg that pre-allocates and reuses the iovec record. Profiling indicates this is a non-trivial amount of our I/O loop (under very heavy load).

Since nobody is using the heavyweight features of the existing recvmmsg, replace it with the lightweight one. This leads to minor but important changes to the interfaces of Iobuf.recvmmsg_assume_fd_nonblocking and Udp.recvmmsg_loop.
Switch to ppx.
Time.set_sexp_zone affects t_of_sexp

If we're willing to read sexps without zones, we should be willing to let you control what timezone they're read with.
Sped up Llimiter.
Make Interval.Int implement Container and Binary_searchable.
Add Identifiable.S to Unix.Cidr so that it supports hash tables. Update Unix.Cidr.t to normalize values, e.g. "192.168.1.101/24" ==> "192.168.1.0/24".
Added "noalloc" attribute to Linux_ext.unsafe_timerfd_settime.
In Iobuf, made some functions take:

(> write, _) Iobuf.t

rather than:

(read_write, _) Iobuf.t

if they only need to write to the iobuf and don't need to read it.
Time.of_string_abs didn't support ISO 8601 time zone strings without colons, or those specified as locations. This version adds support for time zones without colons
Unix.Passwd.getpwents takes the lock, partially applies Exn.protect, then releases the lock, then completes the application and actually runs stuff.
Command.file-completion

Files are now completed correctly when paths contain a directory.

Previously, completion when pointed at a directory would put a space at the end. This would cause the user to hit backspace every time a directory was in the path.
Add diff_weekdays and diff_weekend_days functions to date module.
Time.to_date_ofday_precise implements a complete inverse for of_date_ofday. This is needed to give the DWIM-est semantics to Schedule.t that we can think of.
Reduce allocation in Linux_ext.Epoll
Add Time_ns.Of_day.(add_exn, sub_exn, diff).
Adding head_padded_fixed_string to Iobuf.
Move Core_extended.Std.Sys.home to Core.Std.Sys.home_directory.
Add Iobuf.{read,write,input,output} akin to the bigstring versions.
Add expert iobuf functions for extracting bigstrings and iovecs that share the iobuf's underlying storage.
Add stable Int63.t conversions for types in Time_ns.
Rename DNS-based Inet_addr sexp conversions to expose their blocking nature.
Add Unix.Inet_addr.Stable.
Add [Core.Std.Schedule].
Fixed Iobuf_packet.iter, which behaved incorrectly if the packet didn't start at the lo_min of the iobuf. It used Iobuf.rewind when it should have used Iobuf.Lo_bound.restore.

Added @@deriving compare to Iobuf.Bound.
Add ability to Time.format for a specific time zone
Make more information available via Command.Shape.t

Expose machine-readable info on anonymous arguments.
Remove unnecessary rebinding of (^/) in core_filename.ml. It had one call site and wasn't exposed. The (^/) everyone uses comes from std.ml
Add getifaddrs to Core.Std.Unix.

Handles Packet (for interfaces that do not have an address on Linux systems only), IPv4 and IPv6 address families.
Implement Time_ns.Span.to_string_hum by analogy to Time.Span.to_string_hum. The code and tests are essentially copied from "lib/core/src/span.ml".
Remove our stubs for Unix.stat.

They were upstreamed in 4.02.2.

core_bench

Switched to ppx.

core_extended

N.B. Some interface changes occured in Core which are repercuted in this package, they are not all list in this file though.

Switched to PPX.
Upgrade Interval_map.t with monad operations.
Update the interval_map_intf.ml file to try to make the documentation clearer.

This mostly constitutes splitting out the core operations into a separate module type earlier in the file, so that their documentation occurs before the various more specific module types in reading order.

Various bits of documentation have been tweaked with examples or laws.
Add underscores to color print names Improve and uniformize the behavior of colorprintf functions at the cost of changing the type slightly
Fix core_extended stubs on openbsd

Closes #7 Closes #2
Move Core_extended.Std.Sys.home to Core.Std.Sys.home_directory.
Add a module whose type 'a t acts as a container of ordered items of type 'a (morally, a 'a list) but which supports efficient append operations.

Sometimes called a Rope, or Concatenable_list.
Expose the constructors of Ascii_table.Align.t so that we can write

Column.create ~align:Left ...

instead of

Column.create ~align:Align.left
Fix sexp diffing on records The wrong comparaison was leading to huge diff as soon as one field was missing on one side

core_kernel

Add Container.Make0 for monomorphic container types.
Improved the performance of the implementation of Bounded_int_table.find.
Switched to ppx
Remove references to Core_list from Sequence.
Added functions to Bigstring and Iobuf for reading unsigned 64-bit integers.
Move Comparable.bound to Maybe_bound.t. The purpose is to break up dependencies between the two.
Doubly_linked allocated during iteration. This became a large source of allocation for simple benchmarks like TCP pingpong (async/bench/pingpong). Some unnecessary allocations have been removed.
Added Timing_wheel.next_alarm_fires_at_exn, which is useful to avoid allocation when you know the timing wheel isn't empty.
Make versions of Binary_searchable.Make* that don't require a For_test argument. This allows Binary_searchable.Make to be used for types that don't easily convert from arrays.
Add Quickcheckable interface to Core and move generators/observers into type modules.

Renames core_list.ml to core_list0.ml, then adds a new core_list.ml with quickcheck generators and observers. This allows quickcheck.ml to use core_list0.ml without a dependency cycle.

The feature also moves the contents of quickcheck.mli into quickcheck_intf.ml.
Made Core.Unpack_buffer.Unpack_one.t be a unary type rather than a binary one, by hiding its partial_unpack type under an existential.

This makes it possible to make Unpack_one into a monad because we can combine two Unpack_one.t's with different partial_unpack types into a new Unpack_one.t with a different partial_unpack type.
https://github.com/janestreet/core\_kernel/pull/20 Core.Std module is not found when compiling lib_test/pool_caml_modify_check.ml.
Added an optional argument ?key_order for specifying the order of Map.to_alist output: either Increasing orDecreasing.

The default key order is no longer left unspecified: we're now committed to the `Increasing, which was the old behavior.
Add Sexpable.Of_sexpable2 functor, for symmetry with Binable.Of_binable2. Add sexpable.mli
Added a function for sequencing computations stored in a total map: Total_map.sequence.
Added Core.Bus, a publisher/subscriber system within the memory space of the program. This is a synchronous version of Async.Bus.
Added Core_map.fold2 (fold based on the contents of two maps side-by-side).
Core.Interfaces defines the Unit module type to be sig end. Increase uniformity with its other definitions by defining it to be Unit.S instead.
Adapt Core_random.int to accept larger values than 1 lsl 30.
Mark the Sexpable.Of_* and Binable.Of_* functors as stable.
In Core_char.int_is_ok, used by of_int and of_int_exn, use int compare instead of polymorphic compare.
Fix a few files where toplevel side effects might not be running when we don't pack libraries anymore and use -no-alias-deps.
In Char.For_quickcheck, memoize construction of the filtered chars generators, since if they are used once, they are likely to be used many times, and the construction is costly compared to generating a single char.
Extend Core_map to implement quickcheckable Extend Core_set to implement quickcheckable
In Avltree.add, replace ?(replace = true) with ~replace. This both makes the behavior more explicit, and saves some allocation occasionally.
Reimplement Avltree.iter directly, rather than using fold, which requires allocating a closure. This winds up being expensive during Hashtbl.iter.
Add a function in Blang to deal with boolean expressions E representing the membership of elements in a set, given a universe U and a function projecting each atoms of E to a subset of U.

Example:

{Blang.eval_set ~universe:js_tech resolve_named_set ("(or (and has-blue-eyes has-brown-hair) (and has-brown-eyes has-blue-hair))" |> Sexp.of_string |> t_of_sexp)}
Expose more functions in univ_map interface
Made Random.self_init by default raise if used in inline tests. One can opt out by passing ~allow_in_tests:true.
In core_hashtbl.ml, maybe_resize_table allocates the same closure in each iteration of a for loop. Allocate it just once.
Hashtbl.remove_one and Hashtbl.remove_multi are the same function, written twice. Remove remove_one and replace uses with remove_multi.
Bigstring.unsafe_{get,set}-{,u}int8 used the generic bigarray access function without a type annotation. As a result the compiler generated a call to the generic C function.

Fixed this by adding type annotations.
Add new functions to map that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules.
Made Unpack_buffer.Unpack_one monadic so that users can easily compose file parsers

Added a couple simple parsers as examples and for testing.
Avoid use of polymorphic compare in Quickcheck. Make Quickcheck.Generator.bind_choice lazy: do not eagerly descend into all branches.

Reduces memory overhead by setting a threshold on the probability of choices that are remembered and discarded by Quickcheck.iter and friends.

Motivation: Currently, Quickcheck.iter and related functions guarantee never to repeat a choice from a generator. This winds up recording every choice ever made, which for a lot of generators is a prohibitive cost in space, and most of the recorded values are very unlikely to be repeated anyway.

Implementation: This feature sets a probability threshold below which choices will not be remembered. Choosing a fairly low, but still non-zero, threshold means values are still very unlikely to be repeated, but memory usage stays low.

As of this version, the benefits of "forgetting" unlikely-to-repeat values:

┌──────────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ quickcheck.ml:Quickcheck.iter remember │ 20.26ms │ 16.33Mw │ 100.85kw │ 100.85kw │ 100.00% │ │ quickcheck.ml:Quickcheck.iter forget │ 17.65ms │ 16.21Mw │ 34.83kw │ 34.83kw │ 87.10% │ └──────────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘
Optimizations to:
- various Float.t validation functions
- various functions in Validate
- List.fold_right
Made the type of Option.compare compatible with @@deriving compare.
Fixed an example code fragment in a comment in applicative_intf.ml
In Core_hashtbl, the add_worker function used a bool ref both internally and to pass to Avltree to track whether a new key is added. This was allocated on every call to add or set, and set didn't even use its contents.

This version pre-allocates the bool ref inside each Core_hashtbl.t and reuses it. It still can't be a mutable field because it does need to be passed to Avltree.

After change:

┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set no collisions │ 84.73ns │ 3.00w │ 0.83w │ 0.83w │ 0.01% │ │ Hashtbl.set w/ collisions │ 112.46ns │ │ │ │ 0.02% │ │ Hashtbl.change no collisions │ 82.74ns │ 3.50w │ 0.53w │ 0.53w │ 0.01% │ │ Hashtbl.change w/ collisions │ 191.50ns │ 4.56w │ 1.15w │ 1.15w │ 0.03% │ │ Hashtbl.merge no collisions │ 292_976.43ns │ 26_669.00w │ 15_381.62w │ 12_305.62w │ 48.52% │ │ Hashtbl.merge w/ collisions │ 603_822.86ns │ 33_001.00w │ 20_037.22w │ 16_961.22w │ 100.00% │ │ Hashtbl.add_exn no resize, no collisions │ 80_992.57ns │ 3_088.00w │ 4_102.63w │ 3_077.63w │ 13.41% │ │ Hashtbl.add_exn no resize, w/ collisions │ 178_080.05ns │ 4_621.00w │ 5_668.61w │ 4_643.61w │ 29.49% │ │ Hashtbl.add_exn w/ resize, no collisions │ 176_442.98ns │ 16_403.00w │ 9_222.64w │ 6_148.64w │ 29.22% │ │ Hashtbl.add_exn w/ resize, w/ collisions │ 297_577.29ns │ 19_472.00w │ 12_292.13w │ 9_218.13w │ 49.28% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘

Before change:

┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set no collisions │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set w/ collisions │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change no collisions │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change w/ collisions │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge no collisions │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge w/ collisions │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn no resize, no collisions │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn no resize, w/ collisions │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn w/ resize, no collisions │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn w/ resize, w/ collisions │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘
In Core_hashtbl.add_worker, removed a match that avoided calling Avltree.add, but actually did hurt performance overall.

Perhaps at some point before cross-module inlining, this was a helpful optimization. Right now it bypasses the mutation inside Avltree, so replacing a value in a non-colliding bucket (a Leaf) causes unnecessary re-allocation of the leaf.

After changes:

┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set no collisions │ 52.19ns │ 2.00w │ │ │ │ │ Hashtbl.set w/ collisions │ 112.04ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change no collisions │ 87.25ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change w/ collisions │ 195.85ns │ 5.56w │ 1.29w │ 1.29w │ 0.04% │ │ Hashtbl.merge no collisions │ 308_164.10ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.48% │ │ Hashtbl.merge w/ collisions │ 526_914.80ns │ 38_119.00w │ 20_013.81w │ 16_937.81w │ 100.00% │ │ Hashtbl.add_exn no resize, no collisions │ 76_983.60ns │ 5_135.00w │ 4_100.44w │ 3_075.44w │ 14.61% │ │ Hashtbl.add_exn no resize, w/ collisions │ 174_712.92ns │ 6_668.00w │ 5_667.47w │ 4_642.47w │ 33.16% │ │ Hashtbl.add_exn w/ resize, no collisions │ 176_681.57ns │ 19_476.00w │ 9_231.75w │ 6_157.75w │ 33.53% │ │ Hashtbl.add_exn w/ resize, w/ collisions │ 280_448.62ns │ 22_545.00w │ 12_293.32w │ 9_219.32w │ 53.22% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘

Before changes:

┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set no collisions │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set w/ collisions │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change no collisions │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change w/ collisions │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge no collisions │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge w/ collisions │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn no resize, no collisions │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn no resize, w/ collisions │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn w/ resize, no collisions │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn w/ resize, w/ collisions │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘
Add new functions to hashtbl that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules.
Add a bunch of functions to list and array that add common missing functionality and/or that make their interfaces more uniform and consistent with other container modules.
Rewrite Hashtbl.merge to be simpler and faster.

After changes:

┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge no collisions │ 172.57us │ 17.44kw │ 9.22kw │ 7.69kw │ 48.76% │ │ Hashtbl.merge w/ collisions │ 284.55us │ 20.61kw │ 11.53kw │ 9.99kw │ 80.41% │ │ Pooled_hashtbl.merge no collisions │ 260.57us │ 5.20kw │ 19.18kw │ 3.09kw │ 73.63% │ │ Pooled_hashtbl.merge w/ collisions │ 353.88us │ 5.20kw │ 19.18kw │ 3.09kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘

Before changes:

┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge no collisions │ 309.59us │ 31.79kw │ 15.38kw │ 12.30kw │ 48.91% │ │ Hashtbl.merge w/ collisions │ 526.67us │ 38.12kw │ 19.97kw │ 16.90kw │ 83.21% │ │ Pooled_hashtbl.merge no collisions │ 469.41us │ 7.32kw │ 35.29kw │ 3.12kw │ 74.16% │ │ Pooled_hashtbl.merge w/ collisions │ 632.96us │ 7.32kw │ 35.29kw │ 3.12kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘
Make Hashtbl functions raise an exception if a callback passed in as an argument mutates one of the hash tables being worked on.

Usually, though not always, this comes up for iteration functions. Once a hash table has been mutated, it is unsafe to continue operating on it, as its structure may have changed. Buckets and their contents may have been moved or resized; continuing may result in skipping key/value pairs, repeating key/value pairs, or executing unsafe operations.

This feature adds a mutation_allowed flag to hash tables. Each mutating operation first checks the flag, and raises if it is not set. Each operation with callbacks that must not mutate unsets the flag before calling the callbacks, and restores the flag's original value when it finishes.

We compared the timing of this implementation to an alternate implementation using a mutation counter, and the time and space used for this implementation was much better for iteration and within epsilon of the other for single-key operations like set.
Array function names related to zipping are all over the place. Make them match List, which has a nice uniform naming scheme.
- Rename combine -> zip_exn
- Rename split -> unzip
- (zip remains named as zip)
Add ~key and ~data labels to Hashtbl.filteri_inplace
Added Hash_set.to_hashtbl, by analogy to Set.to_map.
Since we are mutating avltrees in place, make sure the compiler sees the type parameters as invariant.

Tested that a segfaulting example doesn't compile anymore.
Add label f to Hashtbl.change, Map.change, & family.

Introduce the new function update in those modules, which enforces statically the presence of a resulting value

Example:

-|val Hashtbl.change : 'a t -> key -> ('a option -> 'a option) -> unit

+|val Hashtbl.change : 'a t -> key -> f:('a option -> 'a option) -> unit +|val Hashtbl.update : 'a t -> key -> f:('a option -> 'a) -> unit

The motivation for the introduction of update is that in an overwhelming majority of the places where Hashtbl.change is used in our codebase, it is statically known that a new value shall be computed and stored. The use of the dynamism offered by change, which can return an option, is error prone.

The addition of the label is considered acceptable in consideration to external libraries depending on core, because a missing label is just a warning, and we do not guarantee stability in the presence of -warn-error = true.
Changed Source_code_position.t from:

@@deriving bin_io, sexp

to:

@@deriving sexp_of

and made sexp_of use the human-readable format, "FILE:LINE:COL", rather than the unreadable format. Removed Source_code_position.t_hum, which is now obsolete.

If one wants a serialized source-code position, one can use Source_code_position.Stable.
Added Ref.set_temporarily, for temporarily setting a ref to a value for the duration of a thunk.

val set_temporarily : 'a t -> 'a -> f:(unit -> 'b) -> 'b
Add the function singleton : 'a -> 'a t in the stack containers. It cannot be added to Container.S directly because some container never have exactly 1 element.
Made Core.Array match Invariant.S1.
Change the interface of Make_iterable_binable* to give the control back to the user when deserializing Bin_protted data.

Improve the bin_prot deserialization of Maps and Sets. We construct a balanced tree directly instead of relying on Map.add / Set.add. This is possibile because the size of the map is known and elements are sorted.

The complexity goes down from n.log(n) to n.

In case the comparison function changes (and the invariant is not respected), there is a fallback to reconstruct the whole map from scratch.
Add a function to blit a Rope.t into a Buffer.t.
Hashtbl differs from some other core containers with idiosyncratic naming of iteration functions. Change to be consistent and to more closely match the conventions for List and Array.

Hashtbl:
- Copy iter -> iteri.
- Add a deprecation tag to iter.
Made Bag.invariant and Doubly_linked.invariant match Invariant.S1.
Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current Map.iter vs Map.map vs Map.mapi). Change to be consistent and to more closely match the conventions for List and Array.

Map:
- Copy filter -> filteri.
- Add a deprecation tag to filter.
Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current Map.iter vs Map.map vs Map.mapi). Change to be consistent and to more closely match the conventions for List and Array.

Map:
- Copy iter -> iteri.
- Add a deprecation tag to iter.
Made Core.Set_once match Invariant.S1.
Add Bigstring.concat.
For Core.Unique_id, exposed @@deriving typerep.
Expose Hashtbl.hashable, analogous to Map.comparator.
Adds a constant-time val mem_elt : 'a t -> 'a Elt.t -> bool to Doubly_linked and Bag
Add Ordering.to_int which can be useful when one is writing a comparison function. Instead of dealing with the int directly, one can return Ordering.t values and transform them later into ints.
Float.int_pow: Fast computation of x ** n when n is an integer.
Make Core_kernel.Std.Nothing.t enumerable. There's no particular reason not to.
Minor improvements to queue interface
Call Caml.Pervasives.do_at_exit before printing an exception and exiting

The default ocaml uncaught exception handler does this. It is especially useful for curses applications as the at_exit handler has a chance to put back the terminal in a good state before printing the exception and backtrace.

Do the same in Core and Async.
Removed big literals so that the compiler does not complain in 32bit
Add List.range', a generalization of List.range.
Add some functions to Map that are present in Hashtbl:
- remove_multi
- partition_tf
- partitioni_tf
- partition_map
- partition_mapi
Add a Map.nth_exn as a missing complementary function to nth
Renamed Validate.fail_sexp as fail_s, to follow our new *_s convention for Sexp.t-taking functions.
Sequence.split_n_eagerly returns a pair of sequences, but every element of the first sequence has already been evaluated by the time it returns. This feature just makes the first component of the tuple a list instead of a sequence, and renames split_n_eagerly to split_n.

Additionally, this feature adds a new chunks_exn function, which just applies split_n until the input sequence is empty.
Removed Timing_wheel's default alarm_precision, to force people to think about the precision they want when they create a timing wheel.
In Timing_wheel.Config.sexp_of_t, used @sexp_drop_default with level_bits.
Write a better-performing Array.filter_mapi function, and implement Array.filter_map, Array.filter_opt, Array.partitioni_tf, and Array.partition_tf in terms of it.

Slightly worse for zero-length input arrays, about unch'd if we're filtering out almost everything (eq_zero), better on most everything else.

┌──────────────────── │ Name │ ├──────────────────── │ core\_array.ml:filter old-filter-even:0 │ core\_array.ml:filter old-filter-even:1 │ core\_array.ml:filter old-filter-even:10 │ core\_array.ml:filter old-filter-even:100 │ core\_array.ml:filter old-filter-even:1000 │ core\_array.ml:filter old-filter-even:10000 │ core\_array.ml:filter old-filter-even:100000 │ core\_array.ml:filter new-filter-even:0 │ core\_array.ml:filter new-filter-even:1 │ core\_array.ml:filter new-filter-even:10 │ core\_array.ml:filter new-filter-even:100 │ core\_array.ml:filter new-filter-even:1000 │ core\_array.ml:filter new-filter-even:10000 │ core\_array.ml:filter new-filter-even:100000 │ core\_array.ml:filter old-filter-eq_zero:0 │ core\_array.ml:filter old-filter-eq_zero:1 │ core\_array.ml:filter old-filter-eq_zero:10 │ core\_array.ml:filter old-filter-eq_zero:100 │ core\_array.ml:filter old-filter-eq_zero:1000 │ core\_array.ml:filter old-filter-eq_zero:10000 │ core\_array.ml:filter old-filter-eq_zero:100000 │ core\_array.ml:filter new-filter-eq_zero:0 │ core\_array.ml:filter new-filter-eq_zero:1 │ core\_array.ml:filter new-filter-eq_zero:10 │ core\_array.ml:filter new-filter-eq_zero:100 │ core\_array.ml:filter new-filter-eq_zero:1000 │ core\_array.ml:filter new-filter-eq_zero:10000 │ core\_array.ml:filter new-filter-eq_zero:100000 │ core\_array.ml:filter old-filter-neq_zero:0 │ core\_array.ml:filter old-filter-neq_zero:1 │ core\_array.ml:filter old-filter-neq_zero:10 │ core\_array.ml:filter old-filter-neq_zero:100 │ core\_array.ml:filter old-filter-neq_zero:1000 │ │ core\_array.ml:filter old-filter-neq_zero:10000 │ │ core\_array.ml:filter old-filter-neq_zero:100000 │ core\_array.ml:filter new-filter-neq_zero:0 │ core\_array.ml:filter new-filter-neq_zero:1 │ core\_array.ml:filter new-filter-neq_zero:10 │ core\_array.ml:filter new-filter-neq_zero:100 │ core\_array.ml:filter new-filter-neq_zero:1000 │ │ core\_array.ml:filter new-filter-neq_zero:10000 │ │ core\_array.ml:filter new-filter-neq_zero:100000 │ core\_array.ml:filter old-filter_map-int:0 │ core\_array.ml:filter old-filter_map-int:1 │ core\_array.ml:filter old-filter_map-int:10 │ core\_array.ml:filter old-filter_map-int:100 │ core\_array.ml:filter old-filter_map-int:1000 │ core\_array.ml:filter old-filter_map-int:10000 │ core\_array.ml:filter old-filter_map-int:100000 │ core\_array.ml:filter new-filter_map-int:0 │ core\_array.ml:filter new-filter_map-int:1 │ core\_array.ml:filter new-filter_map-int:10 │ core\_array.ml:filter new-filter_map-int:100 │ core\_array.ml:filter new-filter_map-int:1000 │ core\_array.ml:filter new-filter_map-int:10000 │ core\_array.ml:filter new-filter_map-int:100000 │ core\_array.ml:filter old-filter_map-float:0 │ core\_array.ml:filter old-filter_map-float:1 │ core\_array.ml:filter old-filter_map-float:10 │ core\_array.ml:filter old-filter_map-float:100 │ │ core\_array.ml:filter old-filter_map-float:1000 │ │ core\_array.ml:filter old-filter_map-float:10000 │ │ core\_array.ml:filter old-filter_map-float:100000 │ │ core\_array.ml:filter new-filter_map-float:0 │ core\_array.ml:filter new-filter_map-float:1 │ core\_array.ml:filter new-filter_map-float:10 │ core\_array.ml:filter new-filter_map-float:100 │ │ core\_array.ml:filter new-filter_map-float:1000 │ │ core\_array.ml:filter new-filter_map-float:10000 │ │ core\_array.ml:filter new-filter_map-float:100000 │ │ core\_array.ml:filter old-filter_map-boxed:0 │ core\_array.ml:filter old-filter_map-boxed:1 │ core\_array.ml:filter old-filter_map-boxed:10 │ core\_array.ml:filter old-filter_map-boxed:100 │ │ core\_array.ml:filter old-filter_map-boxed:1000 │ │ core\_array.ml:filter old-filter_map-boxed:10000 │ │ core\_array.ml:filter old-filter_map-boxed:100000 │ │ core\_array.ml:filter new-filter_map-boxed:0 │ core\_array.ml:filter new-filter_map-boxed:1 │ core\_array.ml:filter new-filter_map-boxed:10 │ core\_array.ml:filter new-filter_map-boxed:100 │ │ core\_array.ml:filter new-filter_map-boxed:1000 │ │ core\_array.ml:filter new-filter_map-boxed:10000 │ │ core\_array.ml:filter new-filter_map-boxed:100000 │ │ core\_array.ml:filter old-partition_tf:0 │ core\_array.ml:filter old-partition_tf:1 │ core\_array.ml:filter old-partition_tf:10 │ core\_array.ml:filter old-partition_tf:100 │ core\_array.ml:filter old-partition_tf:1000 │ core\_array.ml:filter old-partition_tf:10000 │ core\_array.ml:filter old-partition_tf:100000 │ core\_array.ml:filter new-partition_tf:0 │ core\_array.ml:filter new-partition_tf:1 │ core\_array.ml:filter new-partition_tf:10 │ core\_array.ml:filter new-partition_tf:100 │ core\_array.ml:filter new-partition_tf:1000 │ core\_array.ml:filter new-partition_tf:10000 │ core\_array.ml:filter new-partition_tf:100000 └──────────────────── ────────────────────────────────┬─────────────────┬─────────────┬─────────────┬─────────────┬────────────┐ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ────────────────────────────────┼─────────────────┼─────────────┼─────────────┼─────────────┼────────────┤ │ 12.37ns │ 9.00w │ │ │ │ │ 77.44ns │ 15.00w │ │ │ │ │ 207.10ns │ 36.00w │ │ │ │ │ 1_699.41ns │ 261.00w │ │ │ │ │ 56_320.50ns │ 1_009.00w │ 2_506.01w │ 1_004.01w │ 0.30% │ │ 469_134.89ns │ 10_009.00w │ 25_007.38w │ 10_005.38w │ 2.46% │ │ 4_421_742.22ns │ 100_009.00w │ 250_130.09w │ 100_128.09w │ 23.17% │ │ 13.87ns │ 14.00w │ │ │ │ │ 57.64ns │ 18.00w │ │ │ │ │ 196.28ns │ 35.00w │ │ │ │ │ 1_361.04ns │ 215.00w │ │ │ │ │ 21_473.76ns │ 1_014.00w │ 1_001.02w │ │ 0.11% │ │ 204_033.12ns │ 10_014.00w │ 10_001.14w │ 0.14w │ 1.07% │ │ 2_058_144.47ns │ 100_014.00w │ 100_002.00w │ 1.00w │ 10.78% │ │ 12.21ns │ 9.00w │ │ │ │ │ 71.23ns │ 15.00w │ │ │ │ │ 174.80ns │ 24.00w │ │ │ │ │ 1_212.70ns │ 114.00w │ │ │ │ │ 23_347.51ns │ 13.00w │ 1_007.00w │ 6.00w │ 0.12% │ │ 210_509.83ns │ 13.00w │ 10_007.00w │ 6.00w │ 1.10% │ │ 1_912_253.91ns │ 13.00w │ 100_007.01w │ 6.01w │ 10.02% │ │ 13.70ns │ 14.00w │ │ │ │ │ 56.56ns │ 18.00w │ │ │ │ │ 179.42ns │ 27.00w │ │ │ │ │ 1_254.49ns │ 117.00w │ │ │ │ │ 20_968.06ns │ 16.00w │ 1_001.02w │ │ 0.11% │ │ 204_299.82ns │ 16.00w │ 10_001.13w │ 0.13w │ 1.07% │ │ 2_019_283.81ns │ 16.00w │ 100_001.91w │ 0.91w │ 10.58% │ │ 12.14ns │ 9.00w │ │ │ │ │ 32.72ns │ 11.00w │ │ │ │ │ 219.18ns │ 48.00w │ │ │ │ │ 1_902.76ns │ 408.00w │ 0.12w │ 0.12w │ │ 82_032.44ns │ 2_007.00w │ 3_998.20w │ 1_997.20w │ 0.43% │ 850_234.44ns │ 20_007.00w │ 40_014.86w │ 20_013.86w │ 4.46% │ │ 7_345_941.05ns │ 200_007.00w │ 400_407.82w │ 200_406.82w │ 38.49% │ │ 13.66ns │ 14.00w │ │ │ │ │ 18.26ns │ 14.00w │ │ │ │ │ 201.04ns │ 43.00w │ │ │ │ │ 1_404.33ns │ 313.00w │ │ │ │ 22_829.70ns │ 2_012.00w │ 1_001.02w │ │ 0.12% │ 218_872.52ns │ 20_012.00w │ 10_001.21w │ 0.21w │ 1.15% │ │ 2_121_340.68ns │ 200_012.00w │ 100_002.77w │ 1.77w │ 11.12% │ │ 9.58ns │ 5.00w │ │ │ │ │ 68.46ns │ 11.00w │ │ │ │ │ 191.66ns │ 32.00w │ │ │ │ │ 1_492.60ns │ 257.00w │ │ │ │ │ 57_155.42ns │ 1_005.00w │ 2_507.01w │ 1_005.01w │ 0.30% │ │ 522_177.50ns │ 10_005.00w │ 25_008.54w │ 10_006.54w │ 2.74% │ │ 5_945_405.67ns │ 100_005.00w │ 250_170.69w │ 100_168.69w │ 31.15% │ │ 12.03ns │ 10.00w │ │ │ │ │ 53.63ns │ 14.00w │ │ │ │ │ 164.16ns │ 31.00w │ │ │ │ │ 1_263.42ns │ 211.00w │ │ │ │ │ 23_113.12ns │ 1_010.00w │ 1_001.02w │ │ 0.12% │ │ 218_152.23ns │ 10_010.00w │ 10_001.15w │ 0.15w │ 1.14% │ │ 2_217_307.86ns │ 100_010.00w │ 100_002.11w │ 1.11w │ 11.62% │ │ 9.32ns │ 5.00w │ │ │ │ │ 66.68ns │ 13.00w │ │ │ │ │ 182.86ns │ 42.00w │ │ │ │ 1_496.56ns │ 357.00w │ │ │ │ 76_479.74ns │ 2_005.00w │ 3_507.02w │ 2_005.02w │ 0.40% │ 694_999.59ns │ 20_005.00w │ 35_011.08w │ 20_009.08w │ 3.64% │ 8_694_669.26ns │ 200_005.00w │ 350_476.44w │ 200_474.44w │ 45.56% │ │ 12.29ns │ 10.00w │ │ │ │ │ 58.24ns │ 16.00w │ │ │ │ │ 142.67ns │ 41.00w │ │ │ │ 1_119.41ns │ 311.00w │ │ │ │ 14_262.66ns │ 2_010.00w │ 1_001.02w │ │ 0.07% │ 136_448.05ns │ 20_010.00w │ 10_001.23w │ 0.23w │ 0.71% │ 1_282_005.01ns │ 200_010.00w │ 100_003.14w │ 2.14w │ 6.72% │ │ 9.48ns │ 5.00w │ │ │ │ │ 71.16ns │ 13.00w │ │ │ │ │ 197.40ns │ 42.00w │ │ │ │ 1_762.40ns │ 357.00w │ │ │ │ 86_220.67ns │ 2_005.00w │ 3_507.02w │ 2_005.02w │ 0.45% │ 828_291.42ns │ 20_005.00w │ 35_011.84w │ 20_009.84w │ 4.34% │ 7_955_395.61ns │ 200_005.00w │ 350_441.44w │ 200_439.44w │ 41.68% │ │ 14.43ns │ 10.00w │ │ │ │ │ 59.24ns │ 16.00w │ │ │ │ │ 198.19ns │ 41.00w │ │ │ │ 1_580.21ns │ 311.00w │ │ │ │ 52_045.31ns │ 2_010.00w │ 2_011.01w │ 1_010.01w │ 0.27% │ 479_239.44ns │ 20_010.00w │ 20_012.42w │ 10_011.42w │ 2.51% │ 4_389_392.06ns │ 200_010.00w │ 200_135.09w │ 100_134.09w │ 23.00% │ │ 16.55ns │ 16.00w │ │ │ │ │ 128.08ns │ 29.00w │ │ │ │ │ 554.15ns │ 111.00w │ │ │ │ │ 4_853.58ns │ 921.00w │ 0.46w │ 0.46w │ 0.03% │ │ 201_289.06ns │ 5_016.00w │ 9_015.21w │ 5_010.21w │ 1.05% │ │ 1_796_749.87ns │ 50_016.00w │ 90_040.96w │ 50_035.96w │ 9.41% │ │ 19_084_871.85ns │ 500_016.00w │ 902_187.67w │ 502_182.67w │ 100.00% │ │ 28.29ns │ 23.00w │ │ │ │ │ 103.78ns │ 31.00w │ │ │ │ │ 504.10ns │ 96.00w │ │ │ │ │ 3_869.52ns │ 726.00w │ 0.23w │ 0.23w │ 0.02% │ │ 122_807.29ns │ 4_023.00w │ 5_013.04w │ 2_010.04w │ 0.64% │ │ 1_197_596.39ns │ 40_023.00w │ 50_020.05w │ 20_017.05w │ 6.28% │ │ 10_458_344.09ns │ 400_023.00w │ 500_590.94w │ 200_587.94w │ 54.80% │ ────────────────────────────────┴─────────────────┴─────────────┴─────────────┴─────────────┴────────────┘
Added Binable.Of_sexpable functor.
Install the sexp exception printer sooner so that we can get proper %test_result ... errors in things that come before core_kernel.
In Stable_unit_test.Make functors, include all test failures rather than just the first. This is useful for updating batches of expected bin_io results when stabilizing a module.
Remove an unnecessary cast in or_error.ml

core_profiler

Switched to ppx.
Minor adjustments to the command line of profiler_tool.exe:
- Make '-%' an alias for '-percentile'
- Make '-percentile' accept a comma-separated list of numbers
- Add '-median' argument that is equivalent to '-percentile 50'

email_message

Bugfixes and minor API improvements.

incremental

Add README.org to Incremental.
Added some type annotations based on comments by @def-lkb about lack of principality.
Switched to ppx.

jenga

Restructure the code in a way that allows to build binaries that statically link jenga with the rules. This is useful because some debugging/profiling tools don't work in the presence of dynamically loaded code very well.
Switch to PPX.
Change the gc info output by jenga so it shows heap size and top heap size, instead of live and heap size. The live part is not super useful given how random it is. I have seen cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at the end so the top heap size avoids being tricked.
First half of the fixes no packing: sharing the structures of dependencies, so they take less space on disk (and in memory as well, when they are loaded from disk, but not really when building from scratch given the way we will use them).

The sexp format also has sharing, because it would also blow up in size otherwise (this is different from the interning of paths, where the interning saves a constant factor). And of course, it makes it possible to see the actual on-disk representation which is nice.

Also fix unhelpful error (contains no information) when the db can't be loaded.

Break the thing that avoids rerunning rules when the set of dependencies decreases. I think it was never useful anyway.
Better error on duplicate targets in the same rule.
To prevent running more than one jenga in a repository, use a local lock rather than an nfs one. We need a transition period though, so for now we use both kinds of locks. Building on nfs is slow, so I don't think there's any downside is not supporting nfs this way. And maybe inotify doesn't work. The upside is that we don't step into Lock.Nfs bugs where if a process is interrupted at the wrong time (when the two lock files are empty) the locks won't be cleaned up automatically, forcing someone to get rid of the lock files manually.

Also rename .jenga/.jenga.* to .jenga/*, because all these prefixes are annoying.
Added a couple of options to turn off some part of jenga, which I used to check how they impacted performance, and could still be handy later.
Optionally display additional information about much allocation was done, at the end of builds. Used it to try to improve memory usage of full tree builds without actually doing full tree builds.
Make stat'ing faster. Hash cons some tenacious that build mtimes map to avoid a huge increase of memory use.
Got rid of noise when stopping jenga.
Some changes to the implementation of Tenacious to improve memory efficiency.

Includes the following changes:
- Remove strong_refs field of Heart.fragile type and instead insert links from the clients Ring.t to its parent using Ring.keep_alive.
- Replace the Ring.t in the triggers field of Hearts.fragile with an Ivar.t since all uses of triggers were producing their own equivalent IVar.ts. Remove the functions broken by this because they were unused.
- Make the Tenacious.t type a concrete datatype, and optimize pure computations by partially evaluating this datatype directly in the pure case.
- Rather than building separate Heart.ts for cancellation and the result, split the cancellation heart into two hearts cancel and dep and then use dep as the result heart. This means that a tenacious is cancelled if either cancel or dep is broken, and it must return a heart representing the validity of its result combined with dep.
Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution time when building the lib directory from scratch.
Adding some sexp_of functions, since they're always missing and it's a pain when debugging.
Adding direct support for Dep.map. Even now that Tenacious is smarter, this still creates less binds. Doesn't seem to make much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if nothing else, it's much less surprising to think that Dep.Map becomes Tenacious.Map.
Added Dep.List.concat.
Make it possible to turn off the behavior where jenga rejects commands that output on stderr.

It increases slightly the footprint of the in memory db, but the difference is tiny compared to the rest of the memory usage.
Added a few tests about Jenga_lib.Api.Reflect

ocaml_plugin

Switch to ppx.
Allow ppx-style code to be loaded by plugin-applications build using ocaml_plugin.
Follow Core & Async evolution.

patdiff

patdiff -location-style omake should print the line number of the first difference in each hunk, skipping context lines.
Switched to PPX.
Added binding in patdiff to use the newly minted colors of Ansi_terminal. This will be used notably by patdiff4 to produce better ddiff.

Also, have the module Color and Style implement and export Comparable.S. This is useful for example to dedup styles from a list of styles without relying on the polymorphic equality.
Make it so that if you pass -warn-if-no-trailing-newline-in-both false then you get the warning only when one file has a trailing newline and the other file does not.

If you pass -warn-if-no-trailing-newline-in-both true or omit this flag, then you get the current behavior of warning for each file independently.
Patdiff's unified-tests currently render colors codes in angle brackets. Change them to square brackets. Square brackets are word boundaries, so we'll get more legible diffs when tests fail.
Simple code change in patdiff to prepare more changes in patdiff4. This change is a pure refactoring and has zero runtime change. Just moving some functions around.
patdiff_core.ml is a very long module. start extracting module from it. start with format. in the process, expose in a private fashion the record Rule.t.
Continue on splitting the file patdiff_core.ml into smaller pieces. In this version, we extract each output mode into its own file.
Kill the generation of html diffs in patdiff. There are good third party tools that can convert efficiently ansi texts to html directly. We plan on simplifying a bit the patdiff source code to increase its maintainability, and dropping the requirement of producing html output seems a step in the right direction.

Some pointers:

http://www.pixelbeat.org/scripts/ansi2html.sh jane: app/ralloc/commander/ansi2html.ml

patience_diff

Switch to PPX.

ppx_assert

Update to follow evolution of Ppx_core.

ppx_bench

Update to follow Ppx_core evolution.
Mark attributes as handled inside explicitly dropped pieces of code.

So that a @@deriving inside a let%test dropped by ppx_inline_test_drop doesn't cause a failure.

ppx_bin_prot

Minor changes, nothing worth mentionning.

ppx_compare

Follow evolution of Ppx_core and Type_conv.

ppx_core

Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.
Merlin uses @merlin.* ... attributes in different places. Which ppx_driver reports as unused.

Introduce the concept of reserved namespaces. When one declares the namespace "foo" as reserved then:
- foo.* will never get reported as unused
- it is impossible to Attribute.declare "foo.*"
Mark the "merlin" namespace as reserved by default.
Don't print:

Extension `foo' was not translated. Hint: Did you mean foo?
OCaml makes no distinctions between "foo" and {whatever|foo|whatever}. The delimiter choice is simply left to the user.

Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept {whatever|foo|whatever}.
Avoid stupid hints like this one:

Attribute default' was not used. Hint:default' is available for label declarations but is used here in the context of a label declaration. Did you put it at the wrong level?
Update the API for the common case of extension point expanders.

Make it simpler to define ppx rewriters that locally expand extension points, which is the majority of our non-type-conv rewriters.

Such expanders are run inside the same Ast_traverse.map in a top-down manner which:
- probably improve speed
- help with rewriters that capture a pretty-print of their payload
- help with rewriter that interpret some extension points in a special way inside their payload
Fix the order in which errors are reported by ppx rewriters. Make them be reported in the same order as they appear.
Mark attributes as handled inside explicitly dropped pieces of code.

So that a @@deriving inside a let%test dropped by ppx_inline_test_drop doesn't cause a failure.

ppx_csv_conv

ppx

An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml
ppx/changes-for-public-release

Changes required for the first public release of our ppx rewriters

Improved the type_conv export to ppx_deriving
- improved how it works, to enable it we just need:
  
  Type_conv.Ppx_deriving_exporter.set (module Ppx_deriving)
- Type_conv.add_alias now takes as argument registered type_conv deriviers. This ensure that we can resolve aliases right from the registration time. This simplify the export to ppx_deriving
Split some ppx rewirters

Split some functions out of their main ppx_XXX library that does the registration with ppx_driver or ppx_type_conv.

For instance:
- ppx_sexp_conv defines sexp_of_quote that is used by ppx_assert
- ppx_here defines ast_of_pos that is used by ppx_fail and ppx_assert
Making ppx_assert depends on ppx_sexp_conv and ppx_here is not good as it enables @@deriving sexp and %sexp_of: ty even if the user only writes ppx_assert in the jbuild. This is especially problematic for the public release as it makes ppx_assert incompatible with ppx_deriving.

This feature moves these functions into libraries called ppx_XXX_expander (and ppx_inline_test_libname for ppx_inline_test).

To help reviewing the changes, you can run:

patdiff <(hg cat -r fe show -base ppx/ppx_compare/src/ppx_compare.ml) <(hg cat -r fe show -tip ppx/ppx_compare/expander/ppx_compare_expander.ml) patdiff <(hg cat -r fe show -base ppx/ppx_sexp_conv/src/ppx_sexp_conv.ml) <(hg cat -r fe show -tip ppx/ppx_sexp_conv/expander/ppx_sexp_conv_expander.ml)

Move Ppx_type_conv.Std.Type_conv_path to Ppx_core

It's not type-conv specific anymore and is needed by libraries that don't use type_conv otherwise.

Rename some old runtime libraries

pa_test_lib --> ppx_assert_lib pa_bench_lib --> ppx_bench_lib
Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml
ppx

An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml
ppx/delete-make-at-the-end

Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply.

For users

This feature moves a few values up in signatures. For instance in this interface sexp_of_t is now the first value instead of the second one as before this feature:
```
type t `@@deriving sexp_of`
val x : int
```
In some cases this caused the OCaml compiler to complain about items in the signature being re-ordered. In these cases the signature was adapted to match the expected ordering.
Affected files

ppx/ppx_csv_conv/src/ppx_csv_conv.ml
ppx

An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
Affected files

ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv
ppx/ppx_csv_conv-example

Make ppx_csv_conv example do something.

Testing

Added inline test.
Affected files

ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv

ppx_custom_printf

OCaml makes no distinctions between "foo" and {whatever|foo|whatever}. The delimiter choice is simply left to the user.

Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept {whatever|foo|whatever}.
Fix missing location in errors for broken custom printf example like:

printf !"%{sexp: int" 3;;
Update to follow Ppx_core evolution.

ppx_driver

Disable safety check when code transformations are used as standard "-ppx" rewriters.
Introduce reserved namespaces, see Ppx_core's changelog.

Pass errors as attribute with -dparsetree to avoid "Error while running external preprocessor".
Update to follow Ppx_core evolution.

ppx_enumerate

Update to follow type_conv evolution.

ppx_expect

Initial release.

ppx_fail

Added a README.md
Update to follow Ppx_core evolution.

ppx_fields_conv

The iter function generated by ppx_variants_conv and ppx_fields_conv allowed one to give function which returned values of arbitrary types as iter function. This release constraint these functions to return unit.

N.B. the signature generated by the use of @@deriving variants (resp. fields) in interface already constrained the type to unit.
Update to follow type_conv's evolution.
Add Fields.make_creator to ppx_fields_conv's readme, since it appears to not be all that deprecated.

ppx_here

Make ppx_here translate [%here] instead of _here_.
Update to follow Ppx_core evolution.

ppx_inline_test

Support literate-style .ml files that allow ocaml code interleaved with expected output annotations. Compiling with the ppx_expect_test generates a program that outputs the original source file, but with the actual output substituted for the expected-output annotations. Then we can pat-diff the original file against the output file.

Testing

Examples in the test/ and example/ folders.
Expect-tests can now be written inline in libraries by using let%expect_test.

The runtime library has been split into two components: the test runner, which collects the output of the test body, and registers enough information to construct the *.ml.corrected file from the input; and the test evaluator, which compares the test output against the expected output and generates the output files.
Update to follow Ppx_core evolution.
When an exception is raised inside a let%test_module, display the position and name of the TEST_MODULE, same as for the let%test.
Mark attributes as handled inside explicitly dropped pieces of code.

So that a @@deriving inside a let%test dropped by ppx_inline_test_drop doesn't cause a failure.

ppx_optcomp

Change the way optcomp resolve filenames in #import directives

Do the same as cpp, i.e. for relative filenames, consider they are relative to the directory of the file being parsed. This doesn't matter internally as build commands are always executed from the current directory, but it matters for the public release as everything is executed from the root.

ppx_sexp_conv

Trying to improve the tests in ppx_sexp_conv because they are a mess. At least all tests are automatic now. And more things are tested like the sexpification of exceptions.
Update to follow Type_conv and Ppx_core evolution.
Make ppx_sexp_conv correctly handle aliases to polymorphic variants:

type t = A @@deriving sexptype u = t@@deriving sexptype v = u | B @@deriving sexp

Before, v_of_sexp would never manage to read B. This problem is now fixed if you usesexp_polyonuinstead ofsexp, and if you don't, you get an "unbound value __u_of_sexp__". People should usesexp_polywhen they have a polymorphic variant type that is not syntactically a polymorphic variant, but in practice it's simpler to replacesexpbysexp_poly` when faced with the error above.

The need for sexp_poly should happen only in one new case: an implementation says type u = t@@deriving sexpbut the interface says `type u =A@@deriving sexp. (the old case where it was already needed is when you have an interface that says `type u = t `@@deriving sexp and in some other implementation you try to say type t = That_module.t | A @@deriving sexp`).

ppx_type_conv

Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.
Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply.
Update to follow Ppx_core evolution.

ppx_typerep_conv

Update following Ppx_core and Type_conv evolution.
Add a README.

ppx_variants_conv

The iter function generated by ppx_variants_conv and ppx_fields_conv allowed one to give function which returned values of arbitrary types as iter function. This feature constraint these functions to return unit.

N.B. the signature generated by the use of @@deriving variants (resp. fields) in interface already constrained the type to unit.
Update to follow Type_conv evolution.

re2

Switched to PPX.
Add Re2.Parser.any_string combinator.

There are no tests because any_string is constructed only from the tested API and there's almost no interesting properties of it that can be verified.

rpc_parallel

Switched to PPX.
Expose the connection_timeout argument in rpc_parallel. This argument exists in Rpc_parallel_core.Parallel, but it is not exposed in Rpc_parallel.Parallel.
Allow custom handling of missed async_rpc heartbeats.
Give a better error message when redirecting output on a remote box to a file path that does not exist.
remove unncessary chmod 700 call on the remote executable
Give a clear error message for the common mistake of not making the Parallel.Make_worker() functor application top-level
Make errors/exceptions in Rpc_parallel more observable
- Make stderr and stdout redirection mandatory in order to encourage logging stderr
- Clean up the use of monitors across Rpc_parallel
- Fix bug with exceptions that are sent directly to Monitor.main (e.g. Async_log does this)
Add the ability to explicitly initialize as a master and use some subcommand for the worker. This would allow writing programs with complex command structures that don't have to invoke a bunch of Rpc_parallel logic and start RPC servers for every command.
Add the ability to get log messages from a worker sent back to the master. In fact, any worker can register for the log messages of any other workers.

sexplib

Switch code in lib subdir to ppx-style.

textutils

Switched to PPX.
Fixed a bug where the computation of cell heights could cause division by zero in some cases.
Expose the constructors of Ascii_table.Align.t so that we can write

Column.create ~align:Left ...

instead of

Column.create ~align:Align.left

typerep

Add whether record fields are mutable.

typerep_extended

Switched to ppx.
Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.

Changelog

113.24.00

async

async_extended

async_extra

async_find

async_kernel

async_parallel

async_rpc_kernel

async_shell

async_smtp

async_ssl

async_unix

Testing

bignum

Summary of benchmarks

Follow ups

Benchmark results

bin_prot

core

core_bench

core_extended

core_kernel

Example:

core_profiler

email_message

incremental

jenga

ocaml_plugin

patdiff

patience_diff

ppx_assert

ppx_bench

ppx_bin_prot

ppx_compare

ppx_core

ppx_csv_conv

Improved the type_conv export to ppx_deriving

Split some ppx rewirters

Move Ppx_type_conv.Std.Type_conv_path to Ppx_core

Rename some old runtime libraries

For users

Testing

ppx_custom_printf

ppx_driver

ppx_enumerate

ppx_expect

ppx_fail

ppx_fields_conv

ppx_here

ppx_inline_test

Testing

ppx_optcomp

ppx_sexp_conv

ppx_type_conv

ppx_typerep_conv

ppx_variants_conv

re2

rpc_parallel

sexplib

textutils

typerep

typerep_extended