## 113.24.00 ### async Keep up to date with interface changes in `Async_kernel`, `Async_extra` and `Async_unix`. ### async_extended - Make LTL predicates comparable by tagging and id to each one. Fixes a functional comparison bug. - Switched to PPX. - Add an mli for `async_extended/src/reader_ext.ml` and remove a couple of unused functions (notably `Reader_ext` had its own version of `Reader.read_char`). - Add async-friendly color print - `Ltl.eval` should close the pipe after it is done with it. - Deleted `Async_extended.Cml`. - Remove `Async_extended.Std.Gzip` and redirect references to `Async_gzip.Std.Gzip`. - Update `Command_rpc.Connection` to check the program before exec'ing it. The filename must now be absolute, exist, and be executable. Previously errors with nonexistent or nonexecutable files would only be found out after forking. - Change `Command_rpc.Command` to use `Versioned_rpc.Callee_converts` instead of `Versioned_rpc.Both_convert` so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate. Removed `val rpcs` from `Callee_converts` interfaces because nothing appears to use it, and `Both_convert` does not provide it. Now `Both_convert.S` can be supplied to satisfy `Callee_converts.S`. - Add simple example of Command_rpc - Add `Deferred_cache`. - Fixing a couple of issues noticed in `Command_rpc`: - If `propagate_stderr` is false, the child's stderr is now drained instead of ignored. - When connections are closed, stderr is now closed as well, which prevents a file descriptor leak if the child process is unresponsive. ### async_extra N.B. some changes happening for this release are not listed in this changelog since they appear only as a consequence of changes in core or async\_kernel. - When `Transfer.Writer.send*` raises, send an error to the client. - Add a new rpc that enables a "push" rather than a "poll" model. - Switched to PPX. - For connected UDP sockets, expose `send` in the same fashion as `sendto`. - `Tcp.Server` is documented to refuse excess connections beyond `max_connections + max_pending_connections`, but it treats them as pending connections in our standard OS configuration. In fact, research indicates that the documented behavior is nearly impossible to obtain directly and consistently from `listen`. Clarify the name and role of the `backlog` argument to `listen` and rename and update documentation for `max_pending_connections` to clarify what it actually does, in light of some research: `listen` does not generally respect the backlog argument as an upper limit, but as a lower limit (mod `tcp_max_syn_backlog`) and, with `tcp_abort_on_overflow=0`, `listen` will ignore excess connections rather than actively refusing them. (With `syncookies=1`, this can look like an indefinite backlog.) Existing, working code can substitute `max_pending_connections -> backlog` and move on. The behavior is not changed. When possible, consider architecting applications so the server can simply accept and close excess connections, rather than relying on the `listen` backlog to return an active indication to the client that they won't be serviced. To make sure the client receives an RST rather than an orderly shutdown, you can set the linger time to 0 before closing the socket. (Added to unit tests.) Direct `Tcp.Server` support for this paradigm is left for future work. - Make `Rpc_low_latency_transport` treat disconnections as eof, like `Async_unix.Reader` does. - Add an implementation of Mvars to Async - Allow custom handling of missed async_rpc heartbeats. - adds a configuration limit on the number of tokens that can be in-flight - Replace an `#include ` by `#include `. Fixes janestreet/async\_extra#4 - Added `Tcp.Server.sexp_of_t` - Adds `Rpc.Pipe_rpc.dispatch_iter`, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: `Pipe_rpc.dispatch` followed by `Pipe.iter_without_pushback` consumes ~105 words in the steady state (i.e., when no messages are coming in) while `dispatch_iter` consumes ~15. I'm sure `dispatch` can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as `dispatch_iter`. Both cases can be made smaller by making `Connection.response_handler` a GADT instead of a closure. I plan to do this later. One annoying property of the interface is that the only way to cancel a subscription is to use `Pipe_rpc.abort`, which has a terrible interface. The logical way to improve the interface is to return a record of a `Pipe_rpc.t`, a `Connection.t`, and a `Query_id.t`, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. - Added `Tcp.Server.listening_on_address`, so that one can get the address a server is listening on, as compared with `listening_on`, which just returns the port. - Marked Command.async_basic as deprecated using the appropriate ocaml attribute. `@@ocaml.deprecated` (http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec241) - Extend the interface of `Persistent_rpc_client` to make the "address" type - previously fixed as `Host_and_port.t` - abstract. This is helpful for integrating with libraries that have a different notion of an address, e.g. `rpc_discovery_lib`. - `Typed_tcp` mutated a Hashtbl while iterating over it when closing. - Added `Async.Bus.first_exn`, which takes a bus and a function, and returns a deferred that becomes determined when the first event is published to the bus for which the function returns `Some`. This function is useful to reduce boilerplate for dealing with unsubscription. - Reduced the number of threads required by tests in: async_extra/src/tcp.ml - Added to the error message `Bus.subscribe_exn called after first write` the source-code position of the caller, in case there isn't a backtrace, to make the source of the problem clearer, and to avoid confusion with other source-code positions of subscribers already in the bus. - Added to `Bus.first_exn` a `Source_code_position.t` argument, so that in the event of subscription failure, we can see who caused the subscription to the bus. - Added to `Tcp.Server.close` an optional argument: ?close_existing_connections : bool This closes the sockets of all existing connections. - Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket). - Improved `Async.Udp.bind`'s error message when it fails to `mcast_join` a multicast group. - Change `~callback` to `~f` throughout the `Bus` interface ### async_find Initial release. ### async_kernel N.B. some interface change in Core (notably to `Hashtbl` and `Map`) implied some interface change in this package as well, although they are not mentionned in this changelog. - Switched to ppx. - Improved the Async scheduler's to allocate a `handle_fired` function once, rather than every time it calls `advance_clock`. - Removed configurability of Monitor's `try_with`-ignored-exception handling, i.e. removed `Monitor.try_with_rest_handling` and `Monitor.try_with_ignored_exn_handling`. The behavior of exceptions raised to a monitor after it returns is unchanged -- i.e. they are logged, as they have been since 112.28. Changed `Monitor.try_with`'s `?rest` argument from: ?rest : ` `Ignore | `Raise ` ?rest : ` `Log | `Raise ` This naming reflects the fact that subsequent exceptions are logged, not ignored. - In `Async_kernel`, moved `Scheduler.set_execution_context` from the `Types.Scheduler` module to its own file. Because `Types` is a `module rec`, `set_execution_context` hadn't been inlined and was called via `caml_apply2`. In its own file, it will be inlined. This release creates a new scheduler0.ml, and moves the old scheduler0.ml to scheduler1.ml. - Fixed a space leak in `Pipe` due to a pipe holding pointers to its `upstream_flusheds` after they are closed. The leak shows up in `Pipe.transfer` and related functions, e.g. with: Pipe.transfer temporary_pipe long_lived_pipe called repeatedly, in which `long_lived_pipe` would accumulate a large number of `upstream_flusheds`. The fix is to maintain `upstream_flusheds` as a `Bag.t`, and to remove an upstream pipe when it is closed. - Implement `Pipe.of_sequence` - Improved the error message when an exception is raised to a `Monitor.try_with` that has returned before Async has initialized `Monitor.try_with_log_exn`. - Improved the implementation of `Monitor.get_next_error`, by replacing the monitor's list of handlers: ; mutable handlers_for_next_error : (exn -> unit) list with a single ivar: ; mutable next_error : exn Ivar.t I think this wasn't done originally because of a dependency cycle. But now that we have types.ml, we can do the clear thing. - Improved the implementation of Monitor exception handling, i.e. `detach_and_iter_errors`to make it clear that `Monitor.send_exn` does not run user code -- it only schedules jobs. - Fix an error message in `Pipe` to match the condition that led to it. - Add a new pipe constructor: val unfold : 'b -> f:('b -> ('a * 'b) option Deferred.t) -> 'a Reader.t `unfold` is more powerful than the combination of Useful for, e.g., creating a pipe of natural numbers: Pipe.unfold 0 ~f:(fun n -> return (Some (n, n+1))) - Add `Deferred.Map.all` similar to `Deferred.List.all`. This does what you would expect: val all : ('a, 'b Deferred.t, 'cmp) Map.t -> ('a, 'b, 'cmp) Map.t Deferred.t - Added some simple functions that seem missing from `Deferred` and `Ivar`. val Ivar.peek : 'a t -> 'a option val Ivar.value_exn : 'a t -> 'a val Deferred.value_exn : 'a t -> 'a - Add `Monitor.catch_error`, which provides error handling for processes/subsystems intended to run forever. - Added to the Async scheduler a configurable: min_inter_cycle_timeout : Time_ns.Span.t When scheduler calls epoll(), it uses a timeout of at least `min_inter_cycle_timeout`. `min_inter_cycle_timeout` can be configured via `ASYNC_CONFIG` or via val Scheduler.set_min_inter_cycle_timeout : Time_ns.Span.t -> unit This allows one to tweak the scheduler to be more fair w.r.t. threads, e.g. with: Scheduler.set_min_inter_cycle_timeout <- Time_ns.Span.of_us 1.; - Optimized `Scheduler.schedule'` to avoid a closure allocation. - Removed `Monitor.kill`, which was unused internally. This removes the `kill_index` field from `Execution_context.t`, which saves us a word everytime we allocate or store an execution context. - Assert that `Deferred.forever` never returns statically, rather than dynamically. - Changed `Async.Std` to not include `Deferred.Monad_syntax`, so that one must explicitly opt in (via `open Deferred.Monad_syntax`) to use `let%bind` syntax with Async. - Add `Pipe.to_sequence` - Make `Stream.closed s` return immediately when `s` is already closed. Currently the following property holds: for any s, Deferred.peek (Stream.closed s) = None - For `Pipe` functions that deal with batches of elements in a queue, added an optional argument: ?max_queue_length : int (** default is `Int.max_value` *) This limits the size of the queue that is used, which can improve Async fairness. Affected functions are: filter_map filter_map' fold' iter' map' read' read_now' transfer' transfer_id This also obviates `read_at_most` and `read_now_at_most`, which we will deprecate in a later release. Removed a couple helper types, `iter` and `fold`, that had been used to express commonality among functions, but were becoming unwieldy due to differences. - Changed `Pipe`'s default `max_queue_length` from `Int.max_value` to 100. - Added to `Pipe.iter_without_pushback` an optional argument: ?max_iterations_per_job : int (** default is `Int.max_value` *) `iter_without_pushback` will not make more than `max_iterations_per_job` calls to `f` in a single Async_job; this can be used to increase Async-scheduling fairness. - Added `Pipe.write_if_open` which does exactly what it says. This is a common pattern. Also added a pushback-oblivious variant `write_without_pushback_if_open`. Call sites for these two new functions were introduced wherever I found that doing so would not introduce any side effects (even counting allocation) in the case of a closed pipe. ### async_parallel - Switched to ppx. ### async_rpc_kernel - When `Transfer.Writer.send*` raises, send an error to the client. - Added `Pipe_rpc` in `Versioned_rpc.Both_convert`. - Switched to ppx. - Expose the lower-level registration hook in `Versioned_rpc`. - Allow custom handling of missed async\_rpc heartbeats. - Client-side State\_rpc `dispatch` function does not behave well when the reader side of the pipe is closed. It should gracefully abort the rpc instead of raising exceptions, or whatever it currently does. - Add `Rpc.Expert.implement` and `Rpc.Expert.implement'` with a similar interface as `One_way.Expert.implement` but for 2-way rpcs. Exceptions raised by an expert implementations are handled as follow: - if the query has already been answered, stop the server (as for one-way expert) - if not, send a `Rpc_error.Uncaught_exn` (as for 2-way rpc) - Adds `Rpc.Pipe_rpc.dispatch_iter`, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: `Pipe_rpc.dispatch` followed by `Pipe.iter_without_pushback` consumes ~105 words in the steady state (i.e., when no messages are coming in) while `dispatch_iter` consumes ~15. I'm sure `dispatch` can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as `dispatch_iter`. Both cases can be made smaller by making `Connection.response_handler` a GADT instead of a closure. I plan to do this later. One annoying property of the interface is that the only way to cancel a subscription is to use `Pipe_rpc.abort`, which has a terrible interface. The logical way to improve the interface is to return a record of a `Pipe_rpc.t`, a `Connection.t`, and a `Query_id.t`, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. - Adds `Rpc.Pipe_rpc.implement_direct`, which uses a "direct stream writer" to write results to the other side, rather than using a `Pipe.Writer.t`. This consumes much less memory, ~15 words per open query as opposed to ~225 for a regular `Pipe_rpc` implementation. A large part of the diff in this feature is the addition of a module `Implementation_types`, which just contains copies of type definitions from `Implementation` and `Implementations`. This is necessary to handle some cyclic type definitions (both of these modules now depend on types defined in the other module). This is the implementation-side dual of `Pipe_rpc.dispatch_iter`. - Change `Command_rpc.Command` to use `Versioned_rpc.Callee_converts` instead of `Versioned_rpc.Both_convert` so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate. Removed `val rpcs` from `Callee_converts` interfaces because nothing appears to use it, and `Both_convert` does not provide it. Now `Both_convert.S` can be supplied to satisfy `Callee_converts.S`. - Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket). - Bring back `val rpcs` in versioned\_rpc modules. ### async_shell Initial release. ### async_smtp - Switched to PPX. - Follow Core & Async evolution. ### async_ssl - Switched to ppx. ### async_unix - In the periodic check for a Writer buffer have too old data, eliminated allocation and generally improved performance. This eliminated a large source of allocation in a simple TCP pingpong benchmark `bench/pingpong`. - Removed allocation in the Async scheduler's code that detects the thread-pool being stuck. This involved switching it to use `Time_ns` rather than `Time`. This eliminates a relatively large source of allocation in a simple TCP-pingpong benchmark `bench/pingpong`. - Switched to ppx. - Improved the Async scheduler's to allocate a `handle_fired` function once, rather than every time it calls `advance_clock`. - Added `Fd_by_descr.find_exn`, to avoid the option allocated by `Fd_by_descr.find`. Used it to reduce allocation in the Async scheduler. - Improved `Reader.load_sexp*` functions to behave better when loading from a non files, e.g. a pipe. Previously, it produced an empty error message because it mistakenly attempted to read the sexp a second time in order to determine the error position. Now it produces a good error message, but without the (impossible to obtain) error position. - In `Async_unix.Syscall`, added a zero-allocation syscall interface, removing sources of allocation as observed when running a simple TCP pingpong benchmark (found in `bench/pingpong`). - Added val time_spent_waiting_for_io : unit -> Time_ns.Span.t to `Scheduler` which returns the amount of time that the Async scheduler has spent in calls to `epoll_wait` (or `select`) since the start of the program. - Changed `In_thread.Helper_thread.create` from: val create : ?priority:Priority.t -> ?name:string -> unit -> t Or_error.t to: val create : ?priority:Priority.t -> ?name:string -> unit -> t Deferred.t Kept around the prior function, renamed as `create_now`. Switching `create` to return a deferred allows it to, when there are no available threads, wait until one becomes available. This, in turn, avoids rare nondeterminstic failures in programs that make heavy use of the thread pool and create a helper thread, when the creation happens at just the wrong time, when no thread is available. Split out `Thread_safe_ivar` from the internals of `Thread_pool`, so that it can be used in other tests, and in particular in a new unified test added by this feature. - Make `Unix_syscalls.Stats` bin-io-able. - Fixed a bug in `Thread_safe.block_on_async*`, in which the execution context wasn't properly restored before returning. - Add a function in `Process` that expects empty output, mirroring `Shell.run`. - Added `Reader.read_one_iobuf_at_a_time`, which is like `read_one_chunk_at_a_time`, except that the user-supplied `handle_chunk` function receives its data in an `Iobuf.t`, and uses the `Iobuf` position to communicate how much data was consumed. This facilitates using reader in scenarios (such as with protogen) where `Iobuf`s are expected (and presently allocated around the bigstring at each call) and the calculation of consumed bytes from the `Iobuf` is duplicated in few places. - `Log.message` used to always logs the message, even if its log level was too low. This has been fixed. - Add writer functions to schedule an iobuf to be written out. - Add `Unix.Inet_addr.Stable`. - Alter `Async.Std.Socket.Address.Inet.Blocking_sexp` to expose the polymorphic variant functions, so that you can include it in a separate polymorphic variant type. Also, expose `Async.Std.Socket.Address.Inet.__t_of_sexp__` to give a deprecation message, instead of a message about the function not existing. - Fixed a bug in Async's `In_thread.Helper_thread`, which wasn't finalizing helper threads, due to a bug in `Thread_pool`, which wasn't finalizing helper threads. The fix was to move the finalization out of `Thread_pool`, where we don't care about it, to `In_thread.Helper_thread`, where we do. Added `Scheduler.max_num_threads : unit -> int`. - Make `Epoll_file_descr_watcher` trigger callbacks for error conditions such as closed pipes. Testing ------- Three new unit tests, all validating appropriate behavior in the case that a Unix pipe is opened, then the read end is closed after reading only part of the data sent by the write end. 1. A test in writer.ml verifying that `Writer.consumer_left` is triggered. Before the fix to `Epoll_file_descr_watcher`, `Writer.consumer_left` would never become determined in this case. 2. A test in fd.ml verifying that `Fd.ready_to` is triggered for the writing fd. Before the fix to `Epoll_file_descr_watcher`, `Fd.ready_to` would never become determined in this case. 3. A test in linux_ext.ml verifying that `Epoll.wait_timeout_after` is triggered. This test shows that epoll reports the ERR flag for the file descriptor in this case, and therefore that `Epoll_file_descr_watcher` needs to pay attention to the ERR flag. - Added to `Writer.write_sexp` an optional `?terminate_with` argument, that specifies how to terminate the string representation of the sexp. This also makes it clear that the default behavior, `~terminate_with:Space_if_needed`, might append a space to the sexp you are outputting if its representation is not enclosed in either () or "" . `Sexp.output_hum` an `Sexp.output_mach` do not have this behavior, so porting non-async code to async could introduce unexpected differences in the output. - Add an Async wrapper for `Core.Std.Unix.getifaddrs`. ### bignum - Switched to PPX. - The library used polymorphic compare, rather than `Zarith.Q`'s compare, in a few locations. Fixed this. - Previously stable types in Bignum were defined with unstable types in the scope. Fixd this. - Update to zarith-1.4 - `Bignum.of_string` needs to handle different formats for its input. The previous version of the code was trying to parse the common format (floats), and in case of failure, was attempting to use a different format (based on the error). This resulted in the string being parsed twice in some cases. This version is a complete rewriting of `of_string` to do the parsing in one step. The new code for `to_string` encode an automaton and remembers the positions of the various elements of the string (depending on the format). This feature uses a function which has been upstreamed in the new version of ZArith (1.4) which is a variant of the `Zarith.of_string` function to work with substrings. This variant alone is responsible for a big part of the performance improvement. Summary of benchmarks ----------------- The new version of the code performs better than the original one in all cases. The performance improvement are variable depending on the micro benchmark. See below. Follow ups ---------- We also tried to implement the lexing engine using OCamllex. This makes for a much more concise description, but the performance are significantly lower. OCamllex produces code which allocates some table and some state, which is avoided in the hand written code. Also, it will allocate the sub strings matched. Benchmark results ----------------- New version (patch for ZArith + of\_substring, reimplementation of of\_string) ┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 48\_381.13ns │ 7\_166.00w │ 1.24w │ 1.24w │ 45.12% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 293.96ns │ 72.00w │ │ │ 0.27% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 807.62ns │ 124.00w │ │ │ 0.75% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 353.98ns │ 91.00w │ │ │ 0.33% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 783.78ns │ 128.00w │ │ │ 0.73% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 14\_415.44ns │ 475.00w │ │ │ 13.44% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 61\_363.80ns │ 3\_929.00w │ │ │ 57.23% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 24\_957.02ns │ 303.00w │ │ │ 23.28% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 15\_867.52ns │ 1\_523.00w │ │ │ 14.80% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 33\_345.31ns │ 4\_206.00w │ │ │ 31.10% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 31\_770.26ns │ 3\_779.00w │ │ │ 29.63% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 9\_726.82ns │ 380.00w │ │ │ 9.07% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 28\_141.40ns │ 2\_059.00w │ │ │ 26.25% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 70\_436.16ns │ 5\_541.00w │ │ │ 65.69% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 27\_000.73ns │ 1\_994.00w │ │ │ 25.18% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 66\_057.63ns │ 6\_217.00w │ │ │ 61.61% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 107\_219.89ns │ 8\_097.00w │ │ │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 5\_997.81ns │ 581.00w │ │ │ 5.59% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 18\_522.20ns │ 779.00w │ │ │ 17.27% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_479.49ns │ 463.00w │ │ │ 7.91% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 24\_621.71ns │ 2\_115.00w │ │ │ 22.96% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 26\_896.35ns │ 2\_437.00w │ │ │ 25.09% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 29\_428.19ns │ 2\_730.00w │ │ │ 27.45% │ │ `bignum\_bench.ml:round` round │ 8\_452.31ns │ 459.00w │ │ │ 7.88% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘ Original version ┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 51\_218.04ns │ 7\_166.00w │ 1.25w │ 1.25w │ 43.26% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 336.84ns │ 72.00w │ │ │ 0.28% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 837.73ns │ 124.00w │ │ │ 0.71% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 411.03ns │ 91.00w │ │ │ 0.35% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 808.03ns │ 128.00w │ │ │ 0.68% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 29\_650.60ns │ 2\_415.00w │ │ │ 25.04% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 92\_495.93ns │ 6\_465.00w │ │ │ 78.12% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 39\_482.77ns │ 2\_060.00w │ │ │ 33.35% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 16\_195.93ns │ 1\_523.00w │ │ │ 13.68% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 34\_227.78ns │ 4\_059.00w │ │ │ 28.91% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 32\_856.17ns │ 3\_779.00w │ │ │ 27.75% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 19\_745.71ns │ 2\_149.00w │ │ │ 16.68% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 51\_024.99ns │ 3\_853.00w │ │ │ 43.09% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 88\_884.15ns │ 7\_819.00w │ │ │ 75.07% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 32\_812.27ns │ 2\_498.00w │ │ │ 27.71% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 77\_518.77ns │ 6\_369.00w │ │ │ 65.47% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 118\_402.78ns │ 8\_907.00w │ │ │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 8\_947.02ns │ 371.00w │ │ │ 7.56% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 22\_799.74ns │ 1\_039.00w │ │ │ 19.26% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_176.74ns │ 463.00w │ │ │ 6.91% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 25\_798.77ns │ 2\_115.00w │ │ │ 21.79% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 28\_561.23ns │ 2\_437.00w │ │ │ 24.12% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 30\_861.38ns │ 2\_730.00w │ │ │ 26.06% │ │ `bignum\_bench.ml:round` round │ 8\_237.26ns │ 459.00w │ │ │ 6.96% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘ Tentative version using OCamllex ┌─────────────────────────────────────────────────────────────────────┬──────────────┬────────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼────────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 48\_164.21ns │ 7\_166.00w │ 1.25w │ 1.25w │ 39.99% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 285.84ns │ 72.00w │ │ │ 0.24% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 768.12ns │ 124.00w │ │ │ 0.64% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 343.14ns │ 91.00w │ │ │ 0.28% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 780.00ns │ 128.00w │ │ │ 0.65% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 26\_931.12ns │ 3\_108.00w │ │ │ 22.36% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 79\_750.28ns │ 6\_599.00w │ 0.11w │ 0.11w │ 66.21% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 34\_988.94ns │ 4\_300.00w │ │ │ 29.05% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 15\_958.17ns │ 1\_523.00w │ │ │ 13.25% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 32\_495.25ns │ 4\_059.00w │ │ │ 26.98% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 31\_802.75ns │ 3\_779.00w │ │ │ 26.40% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 18\_742.81ns │ 2\_924.00w │ │ │ 15.56% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 45\_282.09ns │ 4\_622.00w │ │ │ 37.60% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 86\_907.83ns │ 8\_777.00w │ 0.15w │ 0.15w │ 72.16% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 35\_727.73ns │ 4\_493.00w │ │ │ 29.66% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 82\_247.61ns │ 8\_273.00w │ 0.13w │ 0.13w │ 68.29% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 120\_445.25ns │ 10\_688.00w │ 0.12w │ 0.12w │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 6\_734.49ns │ 371.00w │ │ │ 5.59% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 21\_773.79ns │ 1\_890.00w │ │ │ 18.08% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_306.45ns │ 463.00w │ │ │ 6.90% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 24\_714.96ns │ 2\_115.00w │ │ │ 20.52% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 26\_894.27ns │ 2\_437.00w │ │ │ 22.33% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 29\_343.81ns │ 2\_730.00w │ │ │ 24.36% │ │ `bignum\_bench.ml:round` round │ 8\_296.05ns │ 459.00w │ │ │ 6.89% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴────────────┴──────────┴──────────┴────────────┘ ### bin_prot - Bin\_prot can be configured to use the primitives to read/write integers from bigarrays. This was never enabled due to missing tests that selecting this code path doesn't change the format. This version add these tests and enable the use of the fast primitives. - Add benchmarks for all exposed bin\_prot read/write functions. These are intended to check performance regressions. - Remove most use of cpp in bin\_prot. Replace the pre-processor conditionals by runtime one. This make the code a little less obfuscated. - Remove big literals so that the compiler does not complain in 32bit ### core N.B. Some interface change were made which are not listed here, as they are only cascading from `core_kernel`. Look at core\_kernel's CHANGES.md file to get a complete history of the changes made for this release. - Add `Core.Command.Arg_type.Export.time_zone` - Fix `Command.shape` to run external programs only once to get their sexp. Introduces a new variant of Command.t called Proxy. The Exec variant represents the top-level command of an external executable that has not yet been run; the Proxy variant represents an arbitrary subcommand that has been extracted by running an external executable. Command.exec constructs an Exec variant; Command.shape of an Exec variant runs the executable and generates a tree of Proxy variants representing all the information from the generated sexp, so the executable will not need to be re-run. - A version of recvmmsg that pre-allocates and reuses the iovec record. Profiling indicates this is a non-trivial amount of our I/O loop (under very heavy load). Since nobody is using the heavyweight features of the existing recvmmsg, replace it with the lightweight one. This leads to minor but important changes to the interfaces of `Iobuf.recvmmsg_assume_fd_nonblocking` and `Udp.recvmmsg_loop`. - Switch to ppx. - `Time.set_sexp_zone` affects `t_of_sexp` If we're willing to read sexps without zones, we should be willing to let you control what timezone they're read with. - Sped up `Llimiter`. - Make `Interval.Int` implement `Container` and `Binary_searchable`. - Add `Identifiable.S` to `Unix.Cidr` so that it supports hash tables. Update `Unix.Cidr.t` to normalize values, e.g. "192.168.1.101/24" ==> "192.168.1.0/24". - Added "noalloc" attribute to `Linux_ext.unsafe_timerfd_settime`. - In Iobuf, made some functions take: (`> write`, _) Iobuf.t rather than: (read_write, _) Iobuf.t if they only need to write to the iobuf and don't need to read it. - `Time.of_string_abs` didn't support ISO 8601 time zone strings without colons, or those specified as locations. This version adds support for time zones without colons - `Unix.Passwd.getpwents` takes the lock, partially applies `Exn.protect`, then releases the lock, then completes the application and actually runs stuff. - Command.file-completion Files are now completed correctly when paths contain a directory. Previously, completion when pointed at a directory would put a space at the end. This would cause the user to hit backspace every time a directory was in the path. - Add `diff_weekdays` and `diff_weekend_days` functions to date module. - `Time.to_date_ofday_precise` implements a complete inverse for `of_date_ofday`. This is needed to give the DWIM-est semantics to Schedule.t that we can think of. - `Reduce allocation in Linux_ext.Epoll` - Add `Time_ns.Of_day.(add_exn, sub_exn, diff)`. - Adding `head_padded_fixed_string` to Iobuf. - Move `Core_extended.Std.Sys.home` to `Core.Std.Sys.home_directory`. - Add `Iobuf.{read,write,input,output}` akin to the bigstring versions. - Add expert iobuf functions for extracting bigstrings and iovecs that share the iobuf's underlying storage. - Add stable `Int63.t` conversions for types in `Time_ns`. - Rename DNS-based `Inet_addr` sexp conversions to expose their blocking nature. - Add `Unix.Inet_addr.Stable`. - Add [Core.Std.Schedule]. - Fixed `Iobuf_packet.iter`, which behaved incorrectly if the packet didn't start at the lo_min of the iobuf. It used `Iobuf.rewind` when it should have used `Iobuf.Lo_bound.restore`. Added `@@deriving compare` to `Iobuf.Bound`. - Add ability to `Time.format` for a specific time zone - Make more information available via Command.Shape.t Expose machine-readable info on anonymous arguments. - Remove unnecessary rebinding of `(^/)` in `core_filename.ml`. It had one call site and wasn't exposed. The `(^/)` everyone uses comes from std.ml - Add `getifaddrs` to `Core.Std.Unix`. Handles Packet (for interfaces that do not have an address on Linux systems only), IPv4 and IPv6 address families. - Implement `Time_ns.Span.to_string_hum` by analogy to `Time.Span.to_string_hum`. The code and tests are essentially copied from "lib/core/src/span.ml". - Remove our stubs for `Unix.stat`. They were upstreamed in 4.02.2. ### core_bench - Switched to ppx. ### core_extended N.B. Some interface changes occured in Core which are repercuted in this package, they are not all list in this file though. - Switched to PPX. - Upgrade `Interval_map.t` with monad operations. - Update the `interval_map_intf.ml` file to try to make the documentation clearer. This mostly constitutes splitting out the core operations into a separate module type earlier in the file, so that their documentation occurs before the various more specific module types in reading order. Various bits of documentation have been tweaked with examples or laws. - Add underscores to color print names Improve and uniformize the behavior of colorprintf functions at the cost of changing the type slightly - Fix core_extended stubs on openbsd Closes #7 Closes #2 - Move `Core_extended.Std.Sys.home` to `Core.Std.Sys.home_directory`. - Add a module whose type `'a t` acts as a container of ordered items of type 'a (morally, a `'a list`) but which supports efficient append operations. Sometimes called a Rope, or Concatenable_list. - Expose the constructors of `Ascii_table.Align.t` so that we can write Column.create ~align:Left ... instead of Column.create ~align:Align.left - Fix sexp diffing on records The wrong comparaison was leading to huge diff as soon as one field was missing on one side ### core_kernel - Add `Container.Make0` for monomorphic container types. - Improved the performance of the implementation of `Bounded_int_table.find`. - Switched to ppx - Remove references to `Core_list` from `Sequence`. - Added functions to `Bigstring` and `Iobuf` for reading unsigned 64-bit integers. - Move `Comparable.bound` to `Maybe_bound.t`. The purpose is to break up dependencies between the two. - `Doubly_linked` allocated during iteration. This became a large source of allocation for simple benchmarks like TCP pingpong (`async/bench/pingpong`). Some unnecessary allocations have been removed. - Added `Timing_wheel.next_alarm_fires_at_exn`, which is useful to avoid allocation when you know the timing wheel isn't empty. - Make versions of `Binary_searchable.Make*` that don't require a `For_test` argument. This allows `Binary_searchable.Make` to be used for types that don't easily convert from arrays. - Add `Quickcheckable` interface to Core and move generators/observers into type modules. Renames core\_list.ml to core\_list0.ml, then adds a new core\_list.ml with quickcheck generators and observers. This allows quickcheck.ml to use core\_list0.ml without a dependency cycle. The feature also moves the contents of quickcheck.mli into quickcheck\_intf.ml. - Made `Core.Unpack_buffer.Unpack_one.t` be a unary type rather than a binary one, by hiding its `partial_unpack` type under an existential. This makes it possible to make `Unpack_one` into a monad because we can combine two `Unpack_one.t`'s with different `partial_unpack` types into a new `Unpack_one.t` with a different `partial_unpack` type. - https://github.com/janestreet/core\_kernel/pull/20 Core.Std module is not found when compiling lib\_test/pool\_caml\_modify\_check.ml. - Added an optional argument `?key_order` for specifying the order of Map.to\_alist output: either `Increasing or `Decreasing. The default key order is no longer left unspecified: we're now committed to the \`Increasing, which was the old behavior. - Add Sexpable.Of\_sexpable2 functor, for symmetry with Binable.Of\_binable2. Add sexpable.mli - Added a function for sequencing computations stored in a total map: `Total_map.sequence`. - Added `Core.Bus`, a publisher/subscriber system within the memory space of the program. This is a synchronous version of `Async.Bus`. - Added `Core_map.fold2` (fold based on the contents of two maps side-by-side). - `Core.Interfaces` defines the `Unit` module type to be `sig end`. Increase uniformity with its other definitions by defining it to be `Unit.S` instead. - Adapt `Core_random.int` to accept larger values than `1 lsl 30`. - Mark the `Sexpable.Of_*` and `Binable.Of_*` functors as stable. - In `Core_char.int_is_ok`, used by `of_int` and `of_int_exn`, use int compare instead of polymorphic compare. - Fix a few files where toplevel side effects might not be running when we don't pack libraries anymore and use -no-alias-deps. - In `Char.For_quickcheck`, memoize construction of the filtered chars generators, since if they are used once, they are likely to be used many times, and the construction is costly compared to generating a single char. - Extend `Core_map` to implement quickcheckable Extend `Core_set` to implement quickcheckable - In `Avltree.add`, replace `?(replace = true)` with `~replace`. This both makes the behavior more explicit, and saves some allocation occasionally. - Reimplement `Avltree.iter` directly, rather than using `fold`, which requires allocating a closure. This winds up being expensive during `Hashtbl.iter`. - Add a function in Blang to deal with boolean expressions E representing the membership of elements in a set, given a universe U and a function projecting each atoms of E to a subset of U. Example: -------- {` Blang.eval_set ~universe:js_tech resolve_named_set ("(or (and has-blue-eyes has-brown-hair) (and has-brown-eyes has-blue-hair))" |> Sexp.of_string |> t_of_sexp) `} - Expose more functions in univ_map interface - Made `Random.self_init` by default raise if used in inline tests. One can opt out by passing `~allow_in_tests:true`. - In core\_hashtbl.ml, `maybe_resize_table` allocates the same closure in each iteration of a for loop. Allocate it just once. - `Hashtbl.remove_one` and `Hashtbl.remove_multi` are the same function, written twice. Remove `remove_one` and replace uses with `remove_multi`. - `Bigstring.unsafe_{get,set}-{,u}int8` used the generic bigarray access function without a type annotation. As a result the compiler generated a call to the generic C function. Fixed this by adding type annotations. - Add new functions to map that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules. - Made `Unpack_buffer.Unpack_one` monadic so that users can easily compose file parsers Added a couple simple parsers as examples and for testing. - Avoid use of polymorphic compare in Quickcheck. Make `Quickcheck.Generator.bind_choice` lazy: do not eagerly descend into all branches. Reduces memory overhead by setting a threshold on the probability of choices that are remembered and discarded by `Quickcheck.iter` and friends. Motivation: Currently, `Quickcheck.iter` and related functions guarantee never to repeat a choice from a generator. This winds up recording every choice ever made, which for a lot of generators is a prohibitive cost in space, and most of the recorded values are very unlikely to be repeated anyway. Implementation: This feature sets a probability threshold below which choices will not be remembered. Choosing a fairly low, but still non-zero, threshold means values are still very unlikely to be repeated, but memory usage stays low. As of this version, the benefits of "forgetting" unlikely-to-repeat values: ┌──────────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ `quickcheck.ml:Quickcheck.iter` remember │ 20.26ms │ 16.33Mw │ 100.85kw │ 100.85kw │ 100.00% │ │ `quickcheck.ml:Quickcheck.iter` forget │ 17.65ms │ 16.21Mw │ 34.83kw │ 34.83kw │ 87.10% │ └──────────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ - Optimizations to: + various `Float.t` validation functions + various functions in `Validate` + `List.fold_right` - Made the type of `Option.compare` compatible with `@@deriving compare`. - Fixed an example code fragment in a comment in applicative_intf.ml - In `Core_hashtbl`, the `add_worker` function used a `bool ref` both internally and to pass to `Avltree` to track whether a new key is added. This was allocated on every call to `add` or `set`, and `set` didn't even use its contents. This version pre-allocates the `bool ref` inside each `Core_hashtbl.t` and reuses it. It still can't be a `mutable` field because it does need to be passed to `Avltree`. After change: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 84.73ns │ 3.00w │ 0.83w │ 0.83w │ 0.01% │ │ Hashtbl.set `w/ collisions` │ 112.46ns │ │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 82.74ns │ 3.50w │ 0.53w │ 0.53w │ 0.01% │ │ Hashtbl.change `w/ collisions` │ 191.50ns │ 4.56w │ 1.15w │ 1.15w │ 0.03% │ │ Hashtbl.merge `no collisions` │ 292_976.43ns │ 26_669.00w │ 15_381.62w │ 12_305.62w │ 48.52% │ │ Hashtbl.merge `w/ collisions` │ 603_822.86ns │ 33_001.00w │ 20_037.22w │ 16_961.22w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 80_992.57ns │ 3_088.00w │ 4_102.63w │ 3_077.63w │ 13.41% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 178_080.05ns │ 4_621.00w │ 5_668.61w │ 4_643.61w │ 29.49% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 176_442.98ns │ 16_403.00w │ 9_222.64w │ 6_148.64w │ 29.22% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 297_577.29ns │ 19_472.00w │ 12_292.13w │ 9_218.13w │ 49.28% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ Before change: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set `w/ collisions` │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge `w/ collisions` │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ - In `Core_hashtbl.add_worker`, removed a `match` that avoided calling `Avltree.add`, but actually did hurt performance overall. Perhaps at some point before cross-module inlining, this was a helpful optimization. Right now it bypasses the mutation inside `Avltree`, so replacing a value in a non-colliding bucket (a `Leaf`) causes unnecessary re-allocation of the leaf. After changes: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 52.19ns │ 2.00w │ │ │ │ │ Hashtbl.set `w/ collisions` │ 112.04ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 87.25ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 195.85ns │ 5.56w │ 1.29w │ 1.29w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 308_164.10ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.48% │ │ Hashtbl.merge `w/ collisions` │ 526_914.80ns │ 38_119.00w │ 20_013.81w │ 16_937.81w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 76_983.60ns │ 5_135.00w │ 4_100.44w │ 3_075.44w │ 14.61% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 174_712.92ns │ 6_668.00w │ 5_667.47w │ 4_642.47w │ 33.16% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 176_681.57ns │ 19_476.00w │ 9_231.75w │ 6_157.75w │ 33.53% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 280_448.62ns │ 22_545.00w │ 12_293.32w │ 9_219.32w │ 53.22% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ Before changes: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set `w/ collisions` │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge `w/ collisions` │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ - Add new functions to hashtbl that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules. - Add a bunch of functions to list and array that add common missing functionality and/or that make their interfaces more uniform and consistent with other container modules. - Rewrite `Hashtbl.merge` to be simpler and faster. After changes: ┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge `no collisions` │ 172.57us │ 17.44kw │ 9.22kw │ 7.69kw │ 48.76% │ │ Hashtbl.merge `w/ collisions` │ 284.55us │ 20.61kw │ 11.53kw │ 9.99kw │ 80.41% │ │ Pooled_hashtbl.merge `no collisions` │ 260.57us │ 5.20kw │ 19.18kw │ 3.09kw │ 73.63% │ │ Pooled_hashtbl.merge `w/ collisions` │ 353.88us │ 5.20kw │ 19.18kw │ 3.09kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ Before changes: ┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge `no collisions` │ 309.59us │ 31.79kw │ 15.38kw │ 12.30kw │ 48.91% │ │ Hashtbl.merge `w/ collisions` │ 526.67us │ 38.12kw │ 19.97kw │ 16.90kw │ 83.21% │ │ Pooled_hashtbl.merge `no collisions` │ 469.41us │ 7.32kw │ 35.29kw │ 3.12kw │ 74.16% │ │ Pooled_hashtbl.merge `w/ collisions` │ 632.96us │ 7.32kw │ 35.29kw │ 3.12kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ - Make `Hashtbl` functions raise an exception if a callback passed in as an argument mutates one of the hash tables being worked on. Usually, though not always, this comes up for iteration functions. Once a hash table has been mutated, it is unsafe to continue operating on it, as its structure may have changed. Buckets and their contents may have been moved or resized; continuing may result in skipping key/value pairs, repeating key/value pairs, or executing unsafe operations. This feature adds a `mutation_allowed` flag to hash tables. Each mutating operation first checks the flag, and raises if it is not set. Each operation with callbacks that must not mutate unsets the flag before calling the callbacks, and restores the flag's original value when it finishes. We compared the timing of this implementation to an alternate implementation using a mutation counter, and the time and space used for this implementation was much better for iteration and within epsilon of the other for single-key operations like `set`. - Array function names related to zipping are all over the place. Make them match List, which has a nice uniform naming scheme. * Rename `combine` -> `zip_exn` * Rename `split` -> `unzip` * (`zip` remains named as `zip`) - Add `~key` and `~data` labels to `Hashtbl.filteri_inplace` - Added `Hash_set.to_hashtbl`, by analogy to `Set.to_map`. - Since we are mutating avltrees in place, make sure the compiler sees the type parameters as invariant. Tested that a segfaulting example doesn't compile anymore. - Add label `f` to Hashtbl.change, Map.change, & family. Introduce the new function `update` in those modules, which enforces statically the presence of a resulting value Example: -|val Hashtbl.change : 'a t -> key -> ('a option -> 'a option) -> unit +|val Hashtbl.change : 'a t -> key -> f:('a option -> 'a option) -> unit +|val Hashtbl.update : 'a t -> key -> f:('a option -> 'a) -> unit The motivation for the introduction of `update` is that in an overwhelming majority of the places where `Hashtbl.change` is used in our codebase, it is statically known that a new value shall be computed and stored. The use of the dynamism offered by `change`, which can return an option, is error prone. The addition of the label is considered acceptable in consideration to external libraries depending on core, because a missing label is just a warning, and we do not guarantee stability in the presence of -warn-error = true. - Changed `Source_code_position.t` from: `@@deriving bin_io, sexp` to: `@@deriving sexp_of` and made `sexp_of` use the human-readable format, `"FILE:LINE:COL"`, rather than the unreadable format. Removed `Source_code_position.t_hum`, which is now obsolete. If one wants a serialized source-code position, one can use `Source_code_position.Stable`. - Added `Ref.set_temporarily`, for temporarily setting a ref to a value for the duration of a thunk. val set_temporarily : 'a t -> 'a -> f:(unit -> 'b) -> 'b - Add the function `singleton : 'a -> 'a t` in the stack containers. It cannot be added to `Container.S` directly because some container never have exactly 1 element. - Made `Core.Array` match `Invariant.S1`. - Change the interface of `Make_iterable_binable*` to give the control back to the user when deserializing Bin\_protted data. Improve the bin\_prot deserialization of `Map`s and `Set`s. We construct a balanced tree directly instead of relying on `Map.add` / `Set.add`. This is possibile because the size of the map is known and elements are sorted. The complexity goes down from n.log(n) to n. In case the comparison function changes (and the invariant is not respected), there is a fallback to reconstruct the whole map from scratch. - Add a function to blit a `Rope.t` into a `Buffer.t`. - Hashtbl differs from some other core containers with idiosyncratic naming of iteration functions. Change to be consistent and to more closely match the conventions for List and Array. Hashtbl: * Copy `iter` -> `iteri`. * Add a deprecation tag to `iter`. - Made `Bag.invariant` and `Doubly_linked.invariant` match `Invariant.S1`. - Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current `Map.iter` vs `Map.map` vs `Map.mapi`). Change to be consistent and to more closely match the conventions for List and Array. Map: * Copy `filter` -> `filteri`. * Add a deprecation tag to `filter`. - Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current `Map.iter` vs `Map.map` vs `Map.mapi`). Change to be consistent and to more closely match the conventions for List and Array. Map: * Copy `iter` -> `iteri`. * Add a deprecation tag to `iter`. - Made `Core.Set_once` match `Invariant.S1`. - Add `Bigstring.concat`. - For `Core.Unique_id`, exposed `@@deriving typerep`. - Expose Hashtbl.hashable, analogous to Map.comparator. - Adds a constant-time `val mem_elt : 'a t -> 'a Elt.t -> bool` to Doubly_linked and Bag - Add `Ordering.to_int` which can be useful when one is writing a comparison function. Instead of dealing with the int directly, one can return Ordering.t values and transform them later into ints. - `Float.int_pow`: Fast computation of `x ** n` when n is an integer. - Make `Core_kernel.Std.Nothing.t` enumerable. There's no particular reason not to. - Minor improvements to queue interface - Call `Caml.Pervasives.do_at_exit` before printing an exception and exiting The default ocaml uncaught exception handler does this. It is especially useful for curses applications as the `at_exit` handler has a chance to put back the terminal in a good state before printing the exception and backtrace. Do the same in Core and Async. - Removed big literals so that the compiler does not complain in 32bit - Add `List.range'`, a generalization of `List.range`. - Add some functions to `Map` that are present in `Hashtbl`: - `remove_multi` - `partition_tf` - `partitioni_tf` - `partition_map` - `partition_mapi` - Add a `Map.nth_exn` as a missing complementary function to nth - Renamed `Validate.fail_sexp` as `fail_s`, to follow our new `*_s` convention for `Sexp.t`-taking functions. - `Sequence.split_n_eagerly` returns a pair of sequences, but every element of the first sequence has already been evaluated by the time it returns. This feature just makes the first component of the tuple a list instead of a sequence, and renames `split_n_eagerly` to `split_n`. Additionally, this feature adds a new `chunks_exn` function, which just applies `split_n` until the input sequence is empty. - Removed `Timing_wheel`'s default `alarm_precision`, to force people to think about the precision they want when they create a timing wheel. - In `Timing_wheel.Config.sexp_of_t`, used `@sexp_drop_default` with `level_bits`. - Write a better-performing `Array.filter_mapi` function, and implement `Array.filter_map`, `Array.filter_opt`, `Array.partitioni_tf`, and `Array.partition_tf` in terms of it. Slightly worse for zero-length input arrays, about unch'd if we're filtering out almost everything (`eq_zero`), better on most everything else. ┌────────────────────────────────────────────────────┬─────────────────┬─────────────┬─────────────┬─────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├────────────────────────────────────────────────────┼─────────────────┼─────────────┼─────────────┼─────────────┼────────────┤ │ `core\_array.ml:filter` old-filter-even:0 │ 12.37ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:1 │ 77.44ns │ 15.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:10 │ 207.10ns │ 36.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:100 │ 1\_699.41ns │ 261.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:1000 │ 56\_320.50ns │ 1\_009.00w │ 2\_506.01w │ 1\_004.01w │ 0.30% │ │ `core\_array.ml:filter` old-filter-even:10000 │ 469\_134.89ns │ 10\_009.00w │ 25\_007.38w │ 10\_005.38w │ 2.46% │ │ `core\_array.ml:filter` old-filter-even:100000 │ 4\_421\_742.22ns │ 100\_009.00w │ 250\_130.09w │ 100\_128.09w │ 23.17% │ │ `core\_array.ml:filter` new-filter-even:0 │ 13.87ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:1 │ 57.64ns │ 18.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:10 │ 196.28ns │ 35.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:100 │ 1\_361.04ns │ 215.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:1000 │ 21\_473.76ns │ 1\_014.00w │ 1\_001.02w │ │ 0.11% │ │ `core\_array.ml:filter` new-filter-even:10000 │ 204\_033.12ns │ 10\_014.00w │ 10\_001.14w │ 0.14w │ 1.07% │ │ `core\_array.ml:filter` new-filter-even:100000 │ 2\_058\_144.47ns │ 100\_014.00w │ 100\_002.00w │ 1.00w │ 10.78% │ │ `core\_array.ml:filter` old-filter-eq\_zero:0 │ 12.21ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:1 │ 71.23ns │ 15.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:10 │ 174.80ns │ 24.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:100 │ 1\_212.70ns │ 114.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:1000 │ 23\_347.51ns │ 13.00w │ 1\_007.00w │ 6.00w │ 0.12% │ │ `core\_array.ml:filter` old-filter-eq\_zero:10000 │ 210\_509.83ns │ 13.00w │ 10\_007.00w │ 6.00w │ 1.10% │ │ `core\_array.ml:filter` old-filter-eq\_zero:100000 │ 1\_912\_253.91ns │ 13.00w │ 100\_007.01w │ 6.01w │ 10.02% │ │ `core\_array.ml:filter` new-filter-eq\_zero:0 │ 13.70ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:1 │ 56.56ns │ 18.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:10 │ 179.42ns │ 27.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:100 │ 1\_254.49ns │ 117.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:1000 │ 20\_968.06ns │ 16.00w │ 1\_001.02w │ │ 0.11% │ │ `core\_array.ml:filter` new-filter-eq\_zero:10000 │ 204\_299.82ns │ 16.00w │ 10\_001.13w │ 0.13w │ 1.07% │ │ `core\_array.ml:filter` new-filter-eq\_zero:100000 │ 2\_019\_283.81ns │ 16.00w │ 100\_001.91w │ 0.91w │ 10.58% │ │ `core\_array.ml:filter` old-filter-neq\_zero:0 │ 12.14ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:1 │ 32.72ns │ 11.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:10 │ 219.18ns │ 48.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:100 │ 1\_902.76ns │ 408.00w │ 0.12w │ 0.12w │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:1000 │ 82\_032.44ns │ 2\_007.00w │ 3\_998.20w │ 1\_997.20w │ 0.43% │ │ `core\_array.ml:filter` old-filter-neq\_zero:10000 │ 850\_234.44ns │ 20\_007.00w │ 40\_014.86w │ 20\_013.86w │ 4.46% │ │ `core\_array.ml:filter` old-filter-neq\_zero:100000 │ 7\_345\_941.05ns │ 200\_007.00w │ 400\_407.82w │ 200\_406.82w │ 38.49% │ │ `core\_array.ml:filter` new-filter-neq\_zero:0 │ 13.66ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:1 │ 18.26ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:10 │ 201.04ns │ 43.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:100 │ 1\_404.33ns │ 313.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:1000 │ 22\_829.70ns │ 2\_012.00w │ 1\_001.02w │ │ 0.12% │ │ `core\_array.ml:filter` new-filter-neq\_zero:10000 │ 218\_872.52ns │ 20\_012.00w │ 10\_001.21w │ 0.21w │ 1.15% │ │ `core\_array.ml:filter` new-filter-neq\_zero:100000 │ 2\_121\_340.68ns │ 200\_012.00w │ 100\_002.77w │ 1.77w │ 11.12% │ │ `core\_array.ml:filter` old-filter\_map-int:0 │ 9.58ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:1 │ 68.46ns │ 11.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:10 │ 191.66ns │ 32.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:100 │ 1\_492.60ns │ 257.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:1000 │ 57\_155.42ns │ 1\_005.00w │ 2\_507.01w │ 1\_005.01w │ 0.30% │ │ `core\_array.ml:filter` old-filter\_map-int:10000 │ 522\_177.50ns │ 10\_005.00w │ 25\_008.54w │ 10\_006.54w │ 2.74% │ │ `core\_array.ml:filter` old-filter\_map-int:100000 │ 5\_945\_405.67ns │ 100\_005.00w │ 250\_170.69w │ 100\_168.69w │ 31.15% │ │ `core\_array.ml:filter` new-filter\_map-int:0 │ 12.03ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:1 │ 53.63ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:10 │ 164.16ns │ 31.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:100 │ 1\_263.42ns │ 211.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:1000 │ 23\_113.12ns │ 1\_010.00w │ 1\_001.02w │ │ 0.12% │ │ `core\_array.ml:filter` new-filter\_map-int:10000 │ 218\_152.23ns │ 10\_010.00w │ 10\_001.15w │ 0.15w │ 1.14% │ │ `core\_array.ml:filter` new-filter\_map-int:100000 │ 2\_217\_307.86ns │ 100\_010.00w │ 100\_002.11w │ 1.11w │ 11.62% │ │ `core\_array.ml:filter` old-filter\_map-float:0 │ 9.32ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:1 │ 66.68ns │ 13.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:10 │ 182.86ns │ 42.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:100 │ 1\_496.56ns │ 357.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:1000 │ 76\_479.74ns │ 2\_005.00w │ 3\_507.02w │ 2\_005.02w │ 0.40% │ │ `core\_array.ml:filter` old-filter\_map-float:10000 │ 694\_999.59ns │ 20\_005.00w │ 35\_011.08w │ 20\_009.08w │ 3.64% │ │ `core\_array.ml:filter` old-filter\_map-float:100000 │ 8\_694\_669.26ns │ 200\_005.00w │ 350\_476.44w │ 200\_474.44w │ 45.56% │ │ `core\_array.ml:filter` new-filter\_map-float:0 │ 12.29ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:1 │ 58.24ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:10 │ 142.67ns │ 41.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:100 │ 1\_119.41ns │ 311.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:1000 │ 14\_262.66ns │ 2\_010.00w │ 1\_001.02w │ │ 0.07% │ │ `core\_array.ml:filter` new-filter\_map-float:10000 │ 136\_448.05ns │ 20\_010.00w │ 10\_001.23w │ 0.23w │ 0.71% │ │ `core\_array.ml:filter` new-filter\_map-float:100000 │ 1\_282\_005.01ns │ 200\_010.00w │ 100\_003.14w │ 2.14w │ 6.72% │ │ `core\_array.ml:filter` old-filter\_map-boxed:0 │ 9.48ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:1 │ 71.16ns │ 13.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:10 │ 197.40ns │ 42.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:100 │ 1\_762.40ns │ 357.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:1000 │ 86\_220.67ns │ 2\_005.00w │ 3\_507.02w │ 2\_005.02w │ 0.45% │ │ `core\_array.ml:filter` old-filter\_map-boxed:10000 │ 828\_291.42ns │ 20\_005.00w │ 35\_011.84w │ 20\_009.84w │ 4.34% │ │ `core\_array.ml:filter` old-filter\_map-boxed:100000 │ 7\_955\_395.61ns │ 200\_005.00w │ 350\_441.44w │ 200\_439.44w │ 41.68% │ │ `core\_array.ml:filter` new-filter\_map-boxed:0 │ 14.43ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:1 │ 59.24ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:10 │ 198.19ns │ 41.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:100 │ 1\_580.21ns │ 311.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:1000 │ 52\_045.31ns │ 2\_010.00w │ 2\_011.01w │ 1\_010.01w │ 0.27% │ │ `core\_array.ml:filter` new-filter\_map-boxed:10000 │ 479\_239.44ns │ 20\_010.00w │ 20\_012.42w │ 10\_011.42w │ 2.51% │ │ `core\_array.ml:filter` new-filter\_map-boxed:100000 │ 4\_389\_392.06ns │ 200\_010.00w │ 200\_135.09w │ 100\_134.09w │ 23.00% │ │ `core\_array.ml:filter` old-partition\_tf:0 │ 16.55ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:1 │ 128.08ns │ 29.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:10 │ 554.15ns │ 111.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:100 │ 4\_853.58ns │ 921.00w │ 0.46w │ 0.46w │ 0.03% │ │ `core\_array.ml:filter` old-partition\_tf:1000 │ 201\_289.06ns │ 5\_016.00w │ 9\_015.21w │ 5\_010.21w │ 1.05% │ │ `core\_array.ml:filter` old-partition\_tf:10000 │ 1\_796\_749.87ns │ 50\_016.00w │ 90\_040.96w │ 50\_035.96w │ 9.41% │ │ `core\_array.ml:filter` old-partition\_tf:100000 │ 19\_084\_871.85ns │ 500\_016.00w │ 902\_187.67w │ 502\_182.67w │ 100.00% │ │ `core\_array.ml:filter` new-partition\_tf:0 │ 28.29ns │ 23.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:1 │ 103.78ns │ 31.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:10 │ 504.10ns │ 96.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:100 │ 3\_869.52ns │ 726.00w │ 0.23w │ 0.23w │ 0.02% │ │ `core\_array.ml:filter` new-partition\_tf:1000 │ 122\_807.29ns │ 4\_023.00w │ 5\_013.04w │ 2\_010.04w │ 0.64% │ │ `core\_array.ml:filter` new-partition\_tf:10000 │ 1\_197\_596.39ns │ 40\_023.00w │ 50\_020.05w │ 20\_017.05w │ 6.28% │ │ `core\_array.ml:filter` new-partition\_tf:100000 │ 10\_458\_344.09ns │ 400\_023.00w │ 500\_590.94w │ 200\_587.94w │ 54.80% │ └────────────────────────────────────────────────────┴─────────────────┴─────────────┴─────────────┴─────────────┴────────────┘ - Added `Binable.Of_sexpable` functor. - Install the sexp exception printer sooner so that we can get proper `%test_result ...` errors in things that come before `core_kernel`. - In `Stable_unit_test.Make` functors, include all test failures rather than just the first. This is useful for updating batches of expected `bin_io` results when stabilizing a module. - Remove an unnecessary cast in or_error.ml ### core_profiler - Switched to ppx. - Minor adjustments to the command line of profiler_tool.exe: - Make '-%' an alias for '-percentile' - Make '-percentile' accept a comma-separated list of numbers - Add '-median' argument that is equivalent to '-percentile 50' ### email_message - Bugfixes and minor API improvements. ### incremental - Add README.org to Incremental. - Added some type annotations based on comments by @def-lkb about lack of principality. - Switched to ppx. ### jenga - Restructure the code in a way that allows to build binaries that statically link jenga with the rules. This is useful because some debugging/profiling tools don't work in the presence of dynamically loaded code very well. - Switch to PPX. - Change the gc info output by jenga so it shows heap size and top heap size, instead of live and heap size. The live part is not super useful given how random it is. I have seen cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at the end so the top heap size avoids being tricked. - First half of the fixes no packing: sharing the structures of dependencies, so they take less space on disk (and in memory as well, when they are loaded from disk, but not really when building from scratch given the way we will use them). The sexp format also has sharing, because it would also blow up in size otherwise (this is different from the interning of paths, where the interning saves a constant factor). And of course, it makes it possible to see the actual on-disk representation which is nice. Also fix unhelpful error (contains no information) when the db can't be loaded. Break the thing that avoids rerunning rules when the set of dependencies decreases. I think it was never useful anyway. - Better error on duplicate targets in the same rule. - To prevent running more than one jenga in a repository, use a local lock rather than an nfs one. We need a transition period though, so for now we use both kinds of locks. Building on nfs is slow, so I don't think there's any downside is not supporting nfs this way. And maybe inotify doesn't work. The upside is that we don't step into Lock.Nfs bugs where if a process is interrupted at the wrong time (when the two lock files are empty) the locks won't be cleaned up automatically, forcing someone to get rid of the lock files manually. Also rename `.jenga/.jenga.*` to `.jenga/*`, because all these prefixes are annoying. - Added a couple of options to turn off some part of jenga, which I used to check how they impacted performance, and could still be handy later. - Optionally display additional information about much allocation was done, at the end of builds. Used it to try to improve memory usage of full tree builds without actually doing full tree builds. - Make stat'ing faster. Hash cons some tenacious that build mtimes map to avoid a huge increase of memory use. - Got rid of noise when stopping jenga. - Some changes to the implementation of Tenacious to improve memory efficiency. Includes the following changes: - Remove `strong_refs` field of `Heart.fragile` type and instead insert links from the `clients` `Ring.t` to its parent using `Ring.keep_alive`. - Replace the `Ring.t` in the `triggers` field of `Hearts.fragile` with an `Ivar.t` since all uses of `triggers` were producing their own equivalent `IVar.t`s. Remove the functions broken by this because they were unused. - Make the `Tenacious.t` type a concrete datatype, and optimize pure computations by partially evaluating this datatype directly in the pure case. - Rather than building separate `Heart.t`s for cancellation and the result, split the cancellation heart into two hearts `cancel` and `dep` and then use `dep` as the result heart. This means that a tenacious is cancelled if either `cancel` or `dep` is broken, and it must return a heart representing the validity of its result combined with `dep`. Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution time when building the lib directory from scratch. - Adding some `sexp_of` functions, since they're always missing and it's a pain when debugging. - Adding direct support for `Dep.map`. Even now that `Tenacious` is smarter, this still creates less `bind`s. Doesn't seem to make much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if nothing else, it's much less surprising to think that `Dep.Map` becomes `Tenacious.Map`. - Added Dep.List.concat. - Make it possible to turn off the behavior where jenga rejects commands that output on stderr. It increases slightly the footprint of the in memory db, but the difference is tiny compared to the rest of the memory usage. - Added a few tests about `Jenga_lib.Api.Reflect` ### ocaml_plugin - Switch to ppx. - Allow ppx-style code to be loaded by plugin-applications build using ocaml\_plugin. - Follow Core & Async evolution. ### patdiff - `patdiff -location-style omake` should print the line number of the first difference in each hunk, skipping context lines. - Switched to PPX. - Added binding in patdiff to use the newly minted colors of Ansi\_terminal. This will be used notably by patdiff4 to produce better ddiff. Also, have the module `Color` and `Style` implement and export `Comparable.S`. This is useful for example to dedup styles from a list of styles without relying on the polymorphic equality. - Make it so that if you pass `-warn-if-no-trailing-newline-in-both false` then you get the warning only when one file has a trailing newline and the other file does not. If you pass `-warn-if-no-trailing-newline-in-both true` or omit this flag, then you get the current behavior of warning for each file independently. - Patdiff's unified-tests currently render colors codes in angle brackets. Change them to square brackets. Square brackets are word boundaries, so we'll get more legible diffs when tests fail. - Simple code change in patdiff to prepare more changes in patdiff4. This change is a pure refactoring and has zero runtime change. Just moving some functions around. - patdiff_core.ml is a very long module. start extracting module from it. start with format. in the process, expose in a private fashion the record `Rule.t`. - Continue on splitting the file patdiff_core.ml into smaller pieces. In this version, we extract each output mode into its own file. - Kill the generation of html diffs in patdiff. There are good third party tools that can convert efficiently ansi texts to html directly. We plan on simplifying a bit the patdiff source code to increase its maintainability, and dropping the requirement of producing html output seems a step in the right direction. Some pointers: http://www.pixelbeat.org/scripts/ansi2html.sh jane: app/ralloc/commander/ansi2html.ml ### patience_diff - Switch to PPX. ### ppx_assert - Update to follow evolution of `Ppx_core`. ### ppx_bench - Update to follow `Ppx_core` evolution. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by ppx\_inline\_test\_drop doesn't cause a failure. ### ppx_bin_prot - Minor changes, nothing worth mentionning. ### ppx_compare - Follow evolution of `Ppx_core` and `Type_conv`. ### ppx_core - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code. - Merlin uses `@merlin.* ...` attributes in different places. Which ppx\_driver reports as unused. Introduce the concept of reserved namespaces. When one declares the namespace "foo" as reserved then: - `foo.*` will never get reported as unused - it is impossible to `Attribute.declare "foo.*"` Mark the "merlin" namespace as reserved by default. - Don't print: Extension `foo' was not translated. Hint: Did you mean foo? - OCaml makes no distinctions between "foo" and `{whatever|foo|whatever}`. The delimiter choice is simply left to the user. Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept {whatever|foo|whatever}. - Avoid stupid hints like this one: Attribute `default' was not used. Hint: `default' is available for label declarations but is used here in the context of a label declaration. Did you put it at the wrong level? - Update the API for the common case of extension point expanders. Make it simpler to define ppx rewriters that locally expand extension points, which is the majority of our non-type-conv rewriters. Such expanders are run inside the same `Ast_traverse.map` in a top-down manner which: - probably improve speed - help with rewriters that capture a pretty-print of their payload - help with rewriter that interpret some extension points in a special way inside their payload - Fix the order in which errors are reported by ppx rewriters. Make them be reported in the same order as they appear. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by `ppx_inline_test_drop` doesn't cause a failure. ### ppx_csv_conv - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx/changes-for-public-release Changes required for the first public release of our ppx rewriters Improved the type_conv export to ppx_deriving --------------------------------------------- - improved how it works, to enable it we just need: Type_conv.Ppx_deriving_exporter.set (module Ppx_deriving) - Type_conv.add_alias now takes as argument registered type_conv deriviers. This ensure that we can resolve aliases right from the registration time. This simplify the export to ppx_deriving Split some ppx rewirters ------------------------ Split some functions out of their main ppx_XXX library that does the registration with ppx_driver or ppx_type_conv. For instance: - ppx_sexp_conv defines sexp_of_quote that is used by ppx_assert - ppx_here defines ast_of_pos that is used by ppx_fail and ppx_assert Making ppx_assert depends on ppx_sexp_conv and ppx_here is not good as it enables `@@deriving sexp` and `%sexp_of: ty` even if the user only writes ppx_assert in the jbuild. This is especially problematic for the public release as it makes ppx_assert incompatible with ppx_deriving. This feature moves these functions into libraries called ppx_XXX_expander (and ppx_inline_test_libname for ppx_inline_test). To help reviewing the changes, you can run: patdiff <(hg cat -r `fe show -base` ppx/ppx_compare/src/ppx_compare.ml) <(hg cat -r `fe show -tip` ppx/ppx_compare/expander/ppx_compare_expander.ml) patdiff <(hg cat -r `fe show -base` ppx/ppx_sexp_conv/src/ppx_sexp_conv.ml) <(hg cat -r `fe show -tip` ppx/ppx_sexp_conv/expander/ppx_sexp_conv_expander.ml) Move Ppx_type_conv.Std.Type_conv_path to Ppx_core ------------------------------------------------- It's not type-conv specific anymore and is needed by libraries that don't use type_conv otherwise. Rename some old runtime libraries --------------------------------- pa_test_lib --> ppx_assert_lib pa_bench_lib --> ppx_bench_lib * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx/delete-make-at-the-end Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply. For users ========= This feature moves a few values up in signatures. For instance in this interface `sexp_of_t` is now the first value instead of the second one as before this feature: type t `@@deriving sexp_of` val x : int In some cases this caused the OCaml compiler to complain about items in the signature being re-ordered. In these cases the signature was adapted to match the expected ordering. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv - ppx/ppx_csv_conv-example Make ppx_csv_conv example do something. Testing ------- Added inline test. * Affected files ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv ### ppx_custom_printf - OCaml makes no distinctions between "foo" and `{whatever|foo|whatever}`. The delimiter choice is simply left to the user. Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept `{whatever|foo|whatever}`. - Fix missing location in errors for broken custom printf example like: printf !"%{sexp: int" 3;; - Update to follow `Ppx_core` evolution. ### ppx_driver - Disable safety check when code transformations are used as standard "-ppx" rewriters. - Introduce reserved namespaces, see `Ppx_core`'s changelog. Pass errors as attribute with -dparsetree to avoid "Error while running external preprocessor". - Update to follow `Ppx_core` evolution. ### ppx_enumerate - Update to follow type\_conv evolution. ### ppx_expect Initial release. ### ppx_fail - Added a README.md - Update to follow `Ppx_core` evolution. ### ppx_fields_conv - The `iter` function generated by ppx\_variants\_conv and ppx\_fields\_conv allowed one to give function which returned values of arbitrary types as iter function. This release constraint these functions to return unit. N.B. the signature generated by the use of `@@deriving variants` (resp. fields) in interface already constrained the type to unit. - Update to follow type\_conv's evolution. - Add `Fields.make_creator` to ppx\_fields\_conv's readme, since it appears to not be all that deprecated. ### ppx_here - Make ppx\_here translate `[%here]` instead of `_here_`. - Update to follow `Ppx_core` evolution. ### ppx_inline_test - Support literate-style .ml files that allow ocaml code interleaved with expected output annotations. Compiling with the `ppx_expect_test` generates a program that outputs the original source file, but with the actual output substituted for the expected-output annotations. Then we can pat-diff the original file against the output file. Testing ------- Examples in the test/ and example/ folders. - Expect-tests can now be written inline in libraries by using `let%expect_test`. The runtime library has been split into two components: the test runner, which collects the output of the test body, and registers enough information to construct the `*.ml.corrected` file from the input; and the test evaluator, which compares the test output against the expected output and generates the output files. - Update to follow `Ppx_core` evolution. - When an exception is raised inside a `let%test_module`, display the position and name of the TEST\_MODULE, same as for the `let%test`. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by `ppx_inline_test_drop` doesn't cause a failure. ### ppx_optcomp - Change the way optcomp resolve filenames in #import directives Do the same as cpp, i.e. for relative filenames, consider they are relative to the directory of the file being parsed. This doesn't matter internally as build commands are always executed from the current directory, but it matters for the public release as everything is executed from the root. ### ppx_sexp_conv - Trying to improve the tests in ppx\_sexp\_conv because they are a mess. At least all tests are automatic now. And more things are tested like the sexpification of exceptions. - Update to follow `Type_conv` and `Ppx_core` evolution. - Make ppx\_sexp\_conv correctly handle aliases to polymorphic variants: type t = ` `A ` `@@deriving sexp` type u = t `@@deriving sexp` type v = ` u | `B ` `@@deriving sexp` Before, `v_of_sexp` would never manage to read `B. This problem is now fixed if you use `sexp_poly` on `u` instead of `sexp`, and if you don't, you get an "unbound value __u_of_sexp__". People should use `sexp_poly` when they have a polymorphic variant type that is not syntactically a polymorphic variant, but in practice it's simpler to replace `sexp` by `sexp_poly` when faced with the error above. The need for `sexp_poly` should happen only in one new case: an implementation says `type u = t `@@deriving sexp`` but the interface says `type u = ``A` `@@deriving sexp``. (the old case where it was already needed is when you have an interface that says `type u = t `@@deriving sexp`` and in some other implementation you try to say `type t = ` That_module.t | `A ` `@@deriving sexp``). ### ppx_type_conv - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code. - Cleanup in type\_conv: remove `Type_conv.Generator_result.make_at_the_end`, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply. - Update to follow `Ppx_core` evolution. ### ppx_typerep_conv - Update following `Ppx_core` and `Type_conv` evolution. - Add a README. ### ppx_variants_conv - The `iter` function generated by ppx\_variants\_conv and ppx\_fields\_conv allowed one to give function which returned values of arbitrary types as iter function. This feature constraint these functions to return unit. N.B. the signature generated by the use of `@@deriving variants` (resp. fields) in interface already constrained the type to unit. - Update to follow `Type_conv` evolution. ### re2 - Switched to PPX. - Add `Re2.Parser.any_string` combinator. There are no tests because `any_string` is constructed only from the tested API and there's almost no interesting properties of it that can be verified. ### rpc_parallel - Switched to PPX. - Expose the `connection_timeout` argument in rpc\_parallel. This argument exists in `Rpc_parallel_core.Parallel`, but it is not exposed in `Rpc_parallel.Parallel`. - Allow custom handling of missed async\_rpc heartbeats. - Give a better error message when redirecting output on a remote box to a file path that does not exist. - remove unncessary chmod 700 call on the remote executable - Give a clear error message for the common mistake of not making the `Parallel.Make_worker()` functor application top-level - Make errors/exceptions in `Rpc_parallel` more observable - Make stderr and stdout redirection mandatory in order to encourage logging stderr - Clean up the use of monitors across `Rpc_parallel` - Fix bug with exceptions that are sent directly to `Monitor.main` (e.g. `Async_log` does this) - Add the ability to explicitly initialize as a master and use some subcommand for the worker. This would allow writing programs with complex command structures that don't have to invoke a bunch of `Rpc_parallel` logic and start RPC servers for every command. - Add the ability to get log messages from a worker sent back to the master. In fact, any worker can register for the log messages of any other workers. ### sexplib - Switch code in `lib` subdir to ppx-style. ### textutils - Switched to PPX. - Fixed a bug where the computation of cell heights could cause division by zero in some cases. - Expose the constructors of `Ascii_table.Align.t` so that we can write Column.create ~align:Left ... instead of Column.create ~align:Align.left ### typerep - Add whether record fields are mutable. ### typerep_extended - Switched to ppx. - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.