## 113.24.00 ### async Keep up to date with interface changes in `Async_kernel`, `Async_extra` and `Async_unix`. ### async_extended - Make LTL predicates comparable by tagging and id to each one. Fixes a functional comparison bug. - Switched to PPX. - Add an mli for `async_extended/src/reader_ext.ml` and remove a couple of unused functions (notably `Reader_ext` had its own version of `Reader.read_char`). - Add async-friendly color print - `Ltl.eval` should close the pipe after it is done with it. - Deleted `Async_extended.Cml`. - Remove `Async_extended.Std.Gzip` and redirect references to `Async_gzip.Std.Gzip`. - Update `Command_rpc.Connection` to check the program before exec'ing it. The filename must now be absolute, exist, and be executable. Previously errors with nonexistent or nonexecutable files would only be found out after forking. - Change `Command_rpc.Command` to use `Versioned_rpc.Callee_converts` instead of `Versioned_rpc.Both_convert` so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate. Removed `val rpcs` from `Callee_converts` interfaces because nothing appears to use it, and `Both_convert` does not provide it. Now `Both_convert.S` can be supplied to satisfy `Callee_converts.S`. - Add simple example of Command_rpc - Add `Deferred_cache`. - Fixing a couple of issues noticed in `Command_rpc`: - If `propagate_stderr` is false, the child's stderr is now drained instead of ignored. - When connections are closed, stderr is now closed as well, which prevents a file descriptor leak if the child process is unresponsive. ### async_extra N.B. some changes happening for this release are not listed in this changelog since they appear only as a consequence of changes in core or async\_kernel. - When `Transfer.Writer.send*` raises, send an error to the client. - Add a new rpc that enables a "push" rather than a "poll" model. - Switched to PPX. - For connected UDP sockets, expose `send` in the same fashion as `sendto`. - `Tcp.Server` is documented to refuse excess connections beyond `max_connections + max_pending_connections`, but it treats them as pending connections in our standard OS configuration. In fact, research indicates that the documented behavior is nearly impossible to obtain directly and consistently from `listen`. Clarify the name and role of the `backlog` argument to `listen` and rename and update documentation for `max_pending_connections` to clarify what it actually does, in light of some research: `listen` does not generally respect the backlog argument as an upper limit, but as a lower limit (mod `tcp_max_syn_backlog`) and, with `tcp_abort_on_overflow=0`, `listen` will ignore excess connections rather than actively refusing them. (With `syncookies=1`, this can look like an indefinite backlog.) Existing, working code can substitute `max_pending_connections -> backlog` and move on. The behavior is not changed. When possible, consider architecting applications so the server can simply accept and close excess connections, rather than relying on the `listen` backlog to return an active indication to the client that they won't be serviced. To make sure the client receives an RST rather than an orderly shutdown, you can set the linger time to 0 before closing the socket. (Added to unit tests.) Direct `Tcp.Server` support for this paradigm is left for future work. - Make `Rpc_low_latency_transport` treat disconnections as eof, like `Async_unix.Reader` does. - Add an implementation of Mvars to Async - Allow custom handling of missed async_rpc heartbeats. - adds a configuration limit on the number of tokens that can be in-flight - Replace an `#include ` by `#include `. Fixes janestreet/async\_extra#4 - Added `Tcp.Server.sexp_of_t` - Adds `Rpc.Pipe_rpc.dispatch_iter`, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: `Pipe_rpc.dispatch` followed by `Pipe.iter_without_pushback` consumes ~105 words in the steady state (i.e., when no messages are coming in) while `dispatch_iter` consumes ~15. I'm sure `dispatch` can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as `dispatch_iter`. Both cases can be made smaller by making `Connection.response_handler` a GADT instead of a closure. I plan to do this later. One annoying property of the interface is that the only way to cancel a subscription is to use `Pipe_rpc.abort`, which has a terrible interface. The logical way to improve the interface is to return a record of a `Pipe_rpc.t`, a `Connection.t`, and a `Query_id.t`, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. - Added `Tcp.Server.listening_on_address`, so that one can get the address a server is listening on, as compared with `listening_on`, which just returns the port. - Marked Command.async_basic as deprecated using the appropriate ocaml attribute. `@@ocaml.deprecated` (http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec241) - Extend the interface of `Persistent_rpc_client` to make the "address" type - previously fixed as `Host_and_port.t` - abstract. This is helpful for integrating with libraries that have a different notion of an address, e.g. `rpc_discovery_lib`. - `Typed_tcp` mutated a Hashtbl while iterating over it when closing. - Added `Async.Bus.first_exn`, which takes a bus and a function, and returns a deferred that becomes determined when the first event is published to the bus for which the function returns `Some`. This function is useful to reduce boilerplate for dealing with unsubscription. - Reduced the number of threads required by tests in: async_extra/src/tcp.ml - Added to the error message `Bus.subscribe_exn called after first write` the source-code position of the caller, in case there isn't a backtrace, to make the source of the problem clearer, and to avoid confusion with other source-code positions of subscribers already in the bus. - Added to `Bus.first_exn` a `Source_code_position.t` argument, so that in the event of subscription failure, we can see who caused the subscription to the bus. - Added to `Tcp.Server.close` an optional argument: ?close_existing_connections : bool This closes the sockets of all existing connections. - Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket). - Improved `Async.Udp.bind`'s error message when it fails to `mcast_join` a multicast group. - Change `~callback` to `~f` throughout the `Bus` interface ### async_find Initial release. ### async_kernel N.B. some interface change in Core (notably to `Hashtbl` and `Map`) implied some interface change in this package as well, although they are not mentionned in this changelog. - Switched to ppx. - Improved the Async scheduler's to allocate a `handle_fired` function once, rather than every time it calls `advance_clock`. - Removed configurability of Monitor's `try_with`-ignored-exception handling, i.e. removed `Monitor.try_with_rest_handling` and `Monitor.try_with_ignored_exn_handling`. The behavior of exceptions raised to a monitor after it returns is unchanged -- i.e. they are logged, as they have been since 112.28. Changed `Monitor.try_with`'s `?rest` argument from: ?rest : ` `Ignore | `Raise ` ?rest : ` `Log | `Raise ` This naming reflects the fact that subsequent exceptions are logged, not ignored. - In `Async_kernel`, moved `Scheduler.set_execution_context` from the `Types.Scheduler` module to its own file. Because `Types` is a `module rec`, `set_execution_context` hadn't been inlined and was called via `caml_apply2`. In its own file, it will be inlined. This release creates a new scheduler0.ml, and moves the old scheduler0.ml to scheduler1.ml. - Fixed a space leak in `Pipe` due to a pipe holding pointers to its `upstream_flusheds` after they are closed. The leak shows up in `Pipe.transfer` and related functions, e.g. with: Pipe.transfer temporary_pipe long_lived_pipe called repeatedly, in which `long_lived_pipe` would accumulate a large number of `upstream_flusheds`. The fix is to maintain `upstream_flusheds` as a `Bag.t`, and to remove an upstream pipe when it is closed. - Implement `Pipe.of_sequence` - Improved the error message when an exception is raised to a `Monitor.try_with` that has returned before Async has initialized `Monitor.try_with_log_exn`. - Improved the implementation of `Monitor.get_next_error`, by replacing the monitor's list of handlers: ; mutable handlers_for_next_error : (exn -> unit) list with a single ivar: ; mutable next_error : exn Ivar.t I think this wasn't done originally because of a dependency cycle. But now that we have types.ml, we can do the clear thing. - Improved the implementation of Monitor exception handling, i.e. `detach_and_iter_errors`to make it clear that `Monitor.send_exn` does not run user code -- it only schedules jobs. - Fix an error message in `Pipe` to match the condition that led to it. - Add a new pipe constructor: val unfold : 'b -> f:('b -> ('a * 'b) option Deferred.t) -> 'a Reader.t `unfold` is more powerful than the combination of Useful for, e.g., creating a pipe of natural numbers: Pipe.unfold 0 ~f:(fun n -> return (Some (n, n+1))) - Add `Deferred.Map.all` similar to `Deferred.List.all`. This does what you would expect: val all : ('a, 'b Deferred.t, 'cmp) Map.t -> ('a, 'b, 'cmp) Map.t Deferred.t - Added some simple functions that seem missing from `Deferred` and `Ivar`. val Ivar.peek : 'a t -> 'a option val Ivar.value_exn : 'a t -> 'a val Deferred.value_exn : 'a t -> 'a - Add `Monitor.catch_error`, which provides error handling for processes/subsystems intended to run forever. - Added to the Async scheduler a configurable: min_inter_cycle_timeout : Time_ns.Span.t When scheduler calls epoll(), it uses a timeout of at least `min_inter_cycle_timeout`. `min_inter_cycle_timeout` can be configured via `ASYNC_CONFIG` or via val Scheduler.set_min_inter_cycle_timeout : Time_ns.Span.t -> unit This allows one to tweak the scheduler to be more fair w.r.t. threads, e.g. with: Scheduler.set_min_inter_cycle_timeout <- Time_ns.Span.of_us 1.; - Optimized `Scheduler.schedule'` to avoid a closure allocation. - Removed `Monitor.kill`, which was unused internally. This removes the `kill_index` field from `Execution_context.t`, which saves us a word everytime we allocate or store an execution context. - Assert that `Deferred.forever` never returns statically, rather than dynamically. - Changed `Async.Std` to not include `Deferred.Monad_syntax`, so that one must explicitly opt in (via `open Deferred.Monad_syntax`) to use `let%bind` syntax with Async. - Add `Pipe.to_sequence` - Make `Stream.closed s` return immediately when `s` is already closed. Currently the following property holds: for any s, Deferred.peek (Stream.closed s) = None - For `Pipe` functions that deal with batches of elements in a queue, added an optional argument: ?max_queue_length : int (** default is `Int.max_value` *) This limits the size of the queue that is used, which can improve Async fairness. Affected functions are: filter_map filter_map' fold' iter' map' read' read_now' transfer' transfer_id This also obviates `read_at_most` and `read_now_at_most`, which we will deprecate in a later release. Removed a couple helper types, `iter` and `fold`, that had been used to express commonality among functions, but were becoming unwieldy due to differences. - Changed `Pipe`'s default `max_queue_length` from `Int.max_value` to 100. - Added to `Pipe.iter_without_pushback` an optional argument: ?max_iterations_per_job : int (** default is `Int.max_value` *) `iter_without_pushback` will not make more than `max_iterations_per_job` calls to `f` in a single Async_job; this can be used to increase Async-scheduling fairness. - Added `Pipe.write_if_open` which does exactly what it says. This is a common pattern. Also added a pushback-oblivious variant `write_without_pushback_if_open`. Call sites for these two new functions were introduced wherever I found that doing so would not introduce any side effects (even counting allocation) in the case of a closed pipe. ### async_parallel - Switched to ppx. ### async_rpc_kernel - When `Transfer.Writer.send*` raises, send an error to the client. - Added `Pipe_rpc` in `Versioned_rpc.Both_convert`. - Switched to ppx. - Expose the lower-level registration hook in `Versioned_rpc`. - Allow custom handling of missed async\_rpc heartbeats. - Client-side State\_rpc `dispatch` function does not behave well when the reader side of the pipe is closed. It should gracefully abort the rpc instead of raising exceptions, or whatever it currently does. - Add `Rpc.Expert.implement` and `Rpc.Expert.implement'` with a similar interface as `One_way.Expert.implement` but for 2-way rpcs. Exceptions raised by an expert implementations are handled as follow: - if the query has already been answered, stop the server (as for one-way expert) - if not, send a `Rpc_error.Uncaught_exn` (as for 2-way rpc) - Adds `Rpc.Pipe_rpc.dispatch_iter`, plus a bunch of additional types to support it. The main reason for this is to reduce space usage: `Pipe_rpc.dispatch` followed by `Pipe.iter_without_pushback` consumes ~105 words in the steady state (i.e., when no messages are coming in) while `dispatch_iter` consumes ~15. I'm sure `dispatch` can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small as `dispatch_iter`. Both cases can be made smaller by making `Connection.response_handler` a GADT instead of a closure. I plan to do this later. One annoying property of the interface is that the only way to cancel a subscription is to use `Pipe_rpc.abort`, which has a terrible interface. The logical way to improve the interface is to return a record of a `Pipe_rpc.t`, a `Connection.t`, and a `Query_id.t`, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. - Adds `Rpc.Pipe_rpc.implement_direct`, which uses a "direct stream writer" to write results to the other side, rather than using a `Pipe.Writer.t`. This consumes much less memory, ~15 words per open query as opposed to ~225 for a regular `Pipe_rpc` implementation. A large part of the diff in this feature is the addition of a module `Implementation_types`, which just contains copies of type definitions from `Implementation` and `Implementations`. This is necessary to handle some cyclic type definitions (both of these modules now depend on types defined in the other module). This is the implementation-side dual of `Pipe_rpc.dispatch_iter`. - Change `Command_rpc.Command` to use `Versioned_rpc.Callee_converts` instead of `Versioned_rpc.Both_convert` so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate. Removed `val rpcs` from `Callee_converts` interfaces because nothing appears to use it, and `Both_convert` does not provide it. Now `Both_convert.S` can be supplied to satisfy `Callee_converts.S`. - Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket). - Bring back `val rpcs` in versioned\_rpc modules. ### async_shell Initial release. ### async_smtp - Switched to PPX. - Follow Core & Async evolution. ### async_ssl - Switched to ppx. ### async_unix - In the periodic check for a Writer buffer have too old data, eliminated allocation and generally improved performance. This eliminated a large source of allocation in a simple TCP pingpong benchmark `bench/pingpong`. - Removed allocation in the Async scheduler's code that detects the thread-pool being stuck. This involved switching it to use `Time_ns` rather than `Time`. This eliminates a relatively large source of allocation in a simple TCP-pingpong benchmark `bench/pingpong`. - Switched to ppx. - Improved the Async scheduler's to allocate a `handle_fired` function once, rather than every time it calls `advance_clock`. - Added `Fd_by_descr.find_exn`, to avoid the option allocated by `Fd_by_descr.find`. Used it to reduce allocation in the Async scheduler. - Improved `Reader.load_sexp*` functions to behave better when loading from a non files, e.g. a pipe. Previously, it produced an empty error message because it mistakenly attempted to read the sexp a second time in order to determine the error position. Now it produces a good error message, but without the (impossible to obtain) error position. - In `Async_unix.Syscall`, added a zero-allocation syscall interface, removing sources of allocation as observed when running a simple TCP pingpong benchmark (found in `bench/pingpong`). - Added val time_spent_waiting_for_io : unit -> Time_ns.Span.t to `Scheduler` which returns the amount of time that the Async scheduler has spent in calls to `epoll_wait` (or `select`) since the start of the program. - Changed `In_thread.Helper_thread.create` from: val create : ?priority:Priority.t -> ?name:string -> unit -> t Or_error.t to: val create : ?priority:Priority.t -> ?name:string -> unit -> t Deferred.t Kept around the prior function, renamed as `create_now`. Switching `create` to return a deferred allows it to, when there are no available threads, wait until one becomes available. This, in turn, avoids rare nondeterminstic failures in programs that make heavy use of the thread pool and create a helper thread, when the creation happens at just the wrong time, when no thread is available. Split out `Thread_safe_ivar` from the internals of `Thread_pool`, so that it can be used in other tests, and in particular in a new unified test added by this feature. - Make `Unix_syscalls.Stats` bin-io-able. - Fixed a bug in `Thread_safe.block_on_async*`, in which the execution context wasn't properly restored before returning. - Add a function in `Process` that expects empty output, mirroring `Shell.run`. - Added `Reader.read_one_iobuf_at_a_time`, which is like `read_one_chunk_at_a_time`, except that the user-supplied `handle_chunk` function receives its data in an `Iobuf.t`, and uses the `Iobuf` position to communicate how much data was consumed. This facilitates using reader in scenarios (such as with protogen) where `Iobuf`s are expected (and presently allocated around the bigstring at each call) and the calculation of consumed bytes from the `Iobuf` is duplicated in few places. - `Log.message` used to always logs the message, even if its log level was too low. This has been fixed. - Add writer functions to schedule an iobuf to be written out. - Add `Unix.Inet_addr.Stable`. - Alter `Async.Std.Socket.Address.Inet.Blocking_sexp` to expose the polymorphic variant functions, so that you can include it in a separate polymorphic variant type. Also, expose `Async.Std.Socket.Address.Inet.__t_of_sexp__` to give a deprecation message, instead of a message about the function not existing. - Fixed a bug in Async's `In_thread.Helper_thread`, which wasn't finalizing helper threads, due to a bug in `Thread_pool`, which wasn't finalizing helper threads. The fix was to move the finalization out of `Thread_pool`, where we don't care about it, to `In_thread.Helper_thread`, where we do. Added `Scheduler.max_num_threads : unit -> int`. - Make `Epoll_file_descr_watcher` trigger callbacks for error conditions such as closed pipes. Testing ------- Three new unit tests, all validating appropriate behavior in the case that a Unix pipe is opened, then the read end is closed after reading only part of the data sent by the write end. 1. A test in writer.ml verifying that `Writer.consumer_left` is triggered. Before the fix to `Epoll_file_descr_watcher`, `Writer.consumer_left` would never become determined in this case. 2. A test in fd.ml verifying that `Fd.ready_to` is triggered for the writing fd. Before the fix to `Epoll_file_descr_watcher`, `Fd.ready_to` would never become determined in this case. 3. A test in linux_ext.ml verifying that `Epoll.wait_timeout_after` is triggered. This test shows that epoll reports the ERR flag for the file descriptor in this case, and therefore that `Epoll_file_descr_watcher` needs to pay attention to the ERR flag. - Added to `Writer.write_sexp` an optional `?terminate_with` argument, that specifies how to terminate the string representation of the sexp. This also makes it clear that the default behavior, `~terminate_with:Space_if_needed`, might append a space to the sexp you are outputting if its representation is not enclosed in either () or "" . `Sexp.output_hum` an `Sexp.output_mach` do not have this behavior, so porting non-async code to async could introduce unexpected differences in the output. - Add an Async wrapper for `Core.Std.Unix.getifaddrs`. ### bignum - Switched to PPX. - The library used polymorphic compare, rather than `Zarith.Q`'s compare, in a few locations. Fixed this. - Previously stable types in Bignum were defined with unstable types in the scope. Fixd this. - Update to zarith-1.4 - `Bignum.of_string` needs to handle different formats for its input. The previous version of the code was trying to parse the common format (floats), and in case of failure, was attempting to use a different format (based on the error). This resulted in the string being parsed twice in some cases. This version is a complete rewriting of `of_string` to do the parsing in one step. The new code for `to_string` encode an automaton and remembers the positions of the various elements of the string (depending on the format). This feature uses a function which has been upstreamed in the new version of ZArith (1.4) which is a variant of the `Zarith.of_string` function to work with substrings. This variant alone is responsible for a big part of the performance improvement. Summary of benchmarks ----------------- The new version of the code performs better than the original one in all cases. The performance improvement are variable depending on the micro benchmark. See below. Follow ups ---------- We also tried to implement the lexing engine using OCamllex. This makes for a much more concise description, but the performance are significantly lower. OCamllex produces code which allocates some table and some state, which is avoided in the hand written code. Also, it will allocate the sub strings matched. Benchmark results ----------------- New version (patch for ZArith + of\_substring, reimplementation of of\_string) ┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 48\_381.13ns │ 7\_166.00w │ 1.24w │ 1.24w │ 45.12% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 293.96ns │ 72.00w │ │ │ 0.27% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 807.62ns │ 124.00w │ │ │ 0.75% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 353.98ns │ 91.00w │ │ │ 0.33% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 783.78ns │ 128.00w │ │ │ 0.73% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 14\_415.44ns │ 475.00w │ │ │ 13.44% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 61\_363.80ns │ 3\_929.00w │ │ │ 57.23% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 24\_957.02ns │ 303.00w │ │ │ 23.28% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 15\_867.52ns │ 1\_523.00w │ │ │ 14.80% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 33\_345.31ns │ 4\_206.00w │ │ │ 31.10% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 31\_770.26ns │ 3\_779.00w │ │ │ 29.63% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 9\_726.82ns │ 380.00w │ │ │ 9.07% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 28\_141.40ns │ 2\_059.00w │ │ │ 26.25% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 70\_436.16ns │ 5\_541.00w │ │ │ 65.69% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 27\_000.73ns │ 1\_994.00w │ │ │ 25.18% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 66\_057.63ns │ 6\_217.00w │ │ │ 61.61% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 107\_219.89ns │ 8\_097.00w │ │ │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 5\_997.81ns │ 581.00w │ │ │ 5.59% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 18\_522.20ns │ 779.00w │ │ │ 17.27% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_479.49ns │ 463.00w │ │ │ 7.91% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 24\_621.71ns │ 2\_115.00w │ │ │ 22.96% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 26\_896.35ns │ 2\_437.00w │ │ │ 25.09% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 29\_428.19ns │ 2\_730.00w │ │ │ 27.45% │ │ `bignum\_bench.ml:round` round │ 8\_452.31ns │ 459.00w │ │ │ 7.88% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘ Original version ┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 51\_218.04ns │ 7\_166.00w │ 1.25w │ 1.25w │ 43.26% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 336.84ns │ 72.00w │ │ │ 0.28% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 837.73ns │ 124.00w │ │ │ 0.71% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 411.03ns │ 91.00w │ │ │ 0.35% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 808.03ns │ 128.00w │ │ │ 0.68% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 29\_650.60ns │ 2\_415.00w │ │ │ 25.04% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 92\_495.93ns │ 6\_465.00w │ │ │ 78.12% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 39\_482.77ns │ 2\_060.00w │ │ │ 33.35% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 16\_195.93ns │ 1\_523.00w │ │ │ 13.68% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 34\_227.78ns │ 4\_059.00w │ │ │ 28.91% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 32\_856.17ns │ 3\_779.00w │ │ │ 27.75% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 19\_745.71ns │ 2\_149.00w │ │ │ 16.68% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 51\_024.99ns │ 3\_853.00w │ │ │ 43.09% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 88\_884.15ns │ 7\_819.00w │ │ │ 75.07% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 32\_812.27ns │ 2\_498.00w │ │ │ 27.71% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 77\_518.77ns │ 6\_369.00w │ │ │ 65.47% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 118\_402.78ns │ 8\_907.00w │ │ │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 8\_947.02ns │ 371.00w │ │ │ 7.56% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 22\_799.74ns │ 1\_039.00w │ │ │ 19.26% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_176.74ns │ 463.00w │ │ │ 6.91% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 25\_798.77ns │ 2\_115.00w │ │ │ 21.79% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 28\_561.23ns │ 2\_437.00w │ │ │ 24.12% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 30\_861.38ns │ 2\_730.00w │ │ │ 26.06% │ │ `bignum\_bench.ml:round` round │ 8\_237.26ns │ 459.00w │ │ │ 6.96% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘ Tentative version using OCamllex ┌─────────────────────────────────────────────────────────────────────┬──────────────┬────────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼────────────┼──────────┼──────────┼────────────┤ │ `bigint\_bench.ml` random │ 48\_164.21ns │ 7\_166.00w │ 1.25w │ 1.25w │ 39.99% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_self │ 285.84ns │ 72.00w │ │ │ 0.24% │ │ `bigint\_bench.ml:vs. Big\_int` plus\_other │ 768.12ns │ 124.00w │ │ │ 0.64% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_self │ 343.14ns │ 91.00w │ │ │ 0.28% │ │ `bigint\_bench.ml:vs. Big\_int` mult\_other │ 780.00ns │ 128.00w │ │ │ 0.65% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (decimal) │ 26\_931.12ns │ 3\_108.00w │ │ │ 22.36% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (scientific) │ 79\_750.28ns │ 6\_599.00w │ 0.11w │ 0.11w │ 66.21% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` of\_string (fraction) │ 34\_988.94ns │ 4\_300.00w │ │ │ 29.05% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (decimal) │ 15\_958.17ns │ 1\_523.00w │ │ │ 13.25% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (scientific) │ 32\_495.25ns │ 4\_059.00w │ │ │ 26.98% │ │ `bignum\_bench.ml:Bignum of\_string/to\_string` to\_string (fraction) │ 31\_802.75ns │ 3\_779.00w │ │ │ 26.40% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (decimal) │ 18\_742.81ns │ 2\_924.00w │ │ │ 15.56% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (scientific) │ 45\_282.09ns │ 4\_622.00w │ │ │ 37.60% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` of\_sexp (fraction) │ 86\_907.83ns │ 8\_777.00w │ 0.15w │ 0.15w │ 72.16% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (decimal) │ 35\_727.73ns │ 4\_493.00w │ │ │ 29.66% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (scientific) │ 82\_247.61ns │ 8\_273.00w │ 0.13w │ 0.13w │ 68.29% │ │ `bignum\_bench.ml:Bignum of\_sexp/to\_sexp` to\_sexp (fraction) │ 120\_445.25ns │ 10\_688.00w │ 0.12w │ 0.12w │ 100.00% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip compact │ 6\_734.49ns │ 371.00w │ │ │ 5.59% │ │ `bignum\_bench.ml:Bignum binprot` roundtrip classic │ 21\_773.79ns │ 1\_890.00w │ │ │ 18.08% │ │ `bignum\_bench.ml:round` round\_decimal:0 │ 8\_306.45ns │ 463.00w │ │ │ 6.90% │ │ `bignum\_bench.ml:round` round\_decimal:3 │ 24\_714.96ns │ 2\_115.00w │ │ │ 20.52% │ │ `bignum\_bench.ml:round` round\_decimal:6 │ 26\_894.27ns │ 2\_437.00w │ │ │ 22.33% │ │ `bignum\_bench.ml:round` round\_decimal:9 │ 29\_343.81ns │ 2\_730.00w │ │ │ 24.36% │ │ `bignum\_bench.ml:round` round │ 8\_296.05ns │ 459.00w │ │ │ 6.89% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴────────────┴──────────┴──────────┴────────────┘ ### bin_prot - Bin\_prot can be configured to use the primitives to read/write integers from bigarrays. This was never enabled due to missing tests that selecting this code path doesn't change the format. This version add these tests and enable the use of the fast primitives. - Add benchmarks for all exposed bin\_prot read/write functions. These are intended to check performance regressions. - Remove most use of cpp in bin\_prot. Replace the pre-processor conditionals by runtime one. This make the code a little less obfuscated. - Remove big literals so that the compiler does not complain in 32bit ### core N.B. Some interface change were made which are not listed here, as they are only cascading from `core_kernel`. Look at core\_kernel's CHANGES.md file to get a complete history of the changes made for this release. - Add `Core.Command.Arg_type.Export.time_zone` - Fix `Command.shape` to run external programs only once to get their sexp. Introduces a new variant of Command.t called Proxy. The Exec variant represents the top-level command of an external executable that has not yet been run; the Proxy variant represents an arbitrary subcommand that has been extracted by running an external executable. Command.exec constructs an Exec variant; Command.shape of an Exec variant runs the executable and generates a tree of Proxy variants representing all the information from the generated sexp, so the executable will not need to be re-run. - A version of recvmmsg that pre-allocates and reuses the iovec record. Profiling indicates this is a non-trivial amount of our I/O loop (under very heavy load). Since nobody is using the heavyweight features of the existing recvmmsg, replace it with the lightweight one. This leads to minor but important changes to the interfaces of `Iobuf.recvmmsg_assume_fd_nonblocking` and `Udp.recvmmsg_loop`. - Switch to ppx. - `Time.set_sexp_zone` affects `t_of_sexp` If we're willing to read sexps without zones, we should be willing to let you control what timezone they're read with. - Sped up `Llimiter`. - Make `Interval.Int` implement `Container` and `Binary_searchable`. - Add `Identifiable.S` to `Unix.Cidr` so that it supports hash tables. Update `Unix.Cidr.t` to normalize values, e.g. "192.168.1.101/24" ==> "192.168.1.0/24". - Added "noalloc" attribute to `Linux_ext.unsafe_timerfd_settime`. - In Iobuf, made some functions take: (`> write`, _) Iobuf.t rather than: (read_write, _) Iobuf.t if they only need to write to the iobuf and don't need to read it. - `Time.of_string_abs` didn't support ISO 8601 time zone strings without colons, or those specified as locations. This version adds support for time zones without colons - `Unix.Passwd.getpwents` takes the lock, partially applies `Exn.protect`, then releases the lock, then completes the application and actually runs stuff. - Command.file-completion Files are now completed correctly when paths contain a directory. Previously, completion when pointed at a directory would put a space at the end. This would cause the user to hit backspace every time a directory was in the path. - Add `diff_weekdays` and `diff_weekend_days` functions to date module. - `Time.to_date_ofday_precise` implements a complete inverse for `of_date_ofday`. This is needed to give the DWIM-est semantics to Schedule.t that we can think of. - `Reduce allocation in Linux_ext.Epoll` - Add `Time_ns.Of_day.(add_exn, sub_exn, diff)`. - Adding `head_padded_fixed_string` to Iobuf. - Move `Core_extended.Std.Sys.home` to `Core.Std.Sys.home_directory`. - Add `Iobuf.{read,write,input,output}` akin to the bigstring versions. - Add expert iobuf functions for extracting bigstrings and iovecs that share the iobuf's underlying storage. - Add stable `Int63.t` conversions for types in `Time_ns`. - Rename DNS-based `Inet_addr` sexp conversions to expose their blocking nature. - Add `Unix.Inet_addr.Stable`. - Add [Core.Std.Schedule]. - Fixed `Iobuf_packet.iter`, which behaved incorrectly if the packet didn't start at the lo_min of the iobuf. It used `Iobuf.rewind` when it should have used `Iobuf.Lo_bound.restore`. Added `@@deriving compare` to `Iobuf.Bound`. - Add ability to `Time.format` for a specific time zone - Make more information available via Command.Shape.t Expose machine-readable info on anonymous arguments. - Remove unnecessary rebinding of `(^/)` in `core_filename.ml`. It had one call site and wasn't exposed. The `(^/)` everyone uses comes from std.ml - Add `getifaddrs` to `Core.Std.Unix`. Handles Packet (for interfaces that do not have an address on Linux systems only), IPv4 and IPv6 address families. - Implement `Time_ns.Span.to_string_hum` by analogy to `Time.Span.to_string_hum`. The code and tests are essentially copied from "lib/core/src/span.ml". - Remove our stubs for `Unix.stat`. They were upstreamed in 4.02.2. ### core_bench - Switched to ppx. ### core_extended N.B. Some interface changes occured in Core which are repercuted in this package, they are not all list in this file though. - Switched to PPX. - Upgrade `Interval_map.t` with monad operations. - Update the `interval_map_intf.ml` file to try to make the documentation clearer. This mostly constitutes splitting out the core operations into a separate module type earlier in the file, so that their documentation occurs before the various more specific module types in reading order. Various bits of documentation have been tweaked with examples or laws. - Add underscores to color print names Improve and uniformize the behavior of colorprintf functions at the cost of changing the type slightly - Fix core_extended stubs on openbsd Closes #7 Closes #2 - Move `Core_extended.Std.Sys.home` to `Core.Std.Sys.home_directory`. - Add a module whose type `'a t` acts as a container of ordered items of type 'a (morally, a `'a list`) but which supports efficient append operations. Sometimes called a Rope, or Concatenable_list. - Expose the constructors of `Ascii_table.Align.t` so that we can write Column.create ~align:Left ... instead of Column.create ~align:Align.left - Fix sexp diffing on records The wrong comparaison was leading to huge diff as soon as one field was missing on one side ### core_kernel - Add `Container.Make0` for monomorphic container types. - Improved the performance of the implementation of `Bounded_int_table.find`. - Switched to ppx - Remove references to `Core_list` from `Sequence`. - Added functions to `Bigstring` and `Iobuf` for reading unsigned 64-bit integers. - Move `Comparable.bound` to `Maybe_bound.t`. The purpose is to break up dependencies between the two. - `Doubly_linked` allocated during iteration. This became a large source of allocation for simple benchmarks like TCP pingpong (`async/bench/pingpong`). Some unnecessary allocations have been removed. - Added `Timing_wheel.next_alarm_fires_at_exn`, which is useful to avoid allocation when you know the timing wheel isn't empty. - Make versions of `Binary_searchable.Make*` that don't require a `For_test` argument. This allows `Binary_searchable.Make` to be used for types that don't easily convert from arrays. - Add `Quickcheckable` interface to Core and move generators/observers into type modules. Renames core\_list.ml to core\_list0.ml, then adds a new core\_list.ml with quickcheck generators and observers. This allows quickcheck.ml to use core\_list0.ml without a dependency cycle. The feature also moves the contents of quickcheck.mli into quickcheck\_intf.ml. - Made `Core.Unpack_buffer.Unpack_one.t` be a unary type rather than a binary one, by hiding its `partial_unpack` type under an existential. This makes it possible to make `Unpack_one` into a monad because we can combine two `Unpack_one.t`'s with different `partial_unpack` types into a new `Unpack_one.t` with a different `partial_unpack` type. - https://github.com/janestreet/core\_kernel/pull/20 Core.Std module is not found when compiling lib\_test/pool\_caml\_modify\_check.ml. - Added an optional argument `?key_order` for specifying the order of Map.to\_alist output: either `Increasing or `Decreasing. The default key order is no longer left unspecified: we're now committed to the \`Increasing, which was the old behavior. - Add Sexpable.Of\_sexpable2 functor, for symmetry with Binable.Of\_binable2. Add sexpable.mli - Added a function for sequencing computations stored in a total map: `Total_map.sequence`. - Added `Core.Bus`, a publisher/subscriber system within the memory space of the program. This is a synchronous version of `Async.Bus`. - Added `Core_map.fold2` (fold based on the contents of two maps side-by-side). - `Core.Interfaces` defines the `Unit` module type to be `sig end`. Increase uniformity with its other definitions by defining it to be `Unit.S` instead. - Adapt `Core_random.int` to accept larger values than `1 lsl 30`. - Mark the `Sexpable.Of_*` and `Binable.Of_*` functors as stable. - In `Core_char.int_is_ok`, used by `of_int` and `of_int_exn`, use int compare instead of polymorphic compare. - Fix a few files where toplevel side effects might not be running when we don't pack libraries anymore and use -no-alias-deps. - In `Char.For_quickcheck`, memoize construction of the filtered chars generators, since if they are used once, they are likely to be used many times, and the construction is costly compared to generating a single char. - Extend `Core_map` to implement quickcheckable Extend `Core_set` to implement quickcheckable - In `Avltree.add`, replace `?(replace = true)` with `~replace`. This both makes the behavior more explicit, and saves some allocation occasionally. - Reimplement `Avltree.iter` directly, rather than using `fold`, which requires allocating a closure. This winds up being expensive during `Hashtbl.iter`. - Add a function in Blang to deal with boolean expressions E representing the membership of elements in a set, given a universe U and a function projecting each atoms of E to a subset of U. Example: -------- {` Blang.eval_set ~universe:js_tech resolve_named_set ("(or (and has-blue-eyes has-brown-hair) (and has-brown-eyes has-blue-hair))" |> Sexp.of_string |> t_of_sexp) `} - Expose more functions in univ_map interface - Made `Random.self_init` by default raise if used in inline tests. One can opt out by passing `~allow_in_tests:true`. - In core\_hashtbl.ml, `maybe_resize_table` allocates the same closure in each iteration of a for loop. Allocate it just once. - `Hashtbl.remove_one` and `Hashtbl.remove_multi` are the same function, written twice. Remove `remove_one` and replace uses with `remove_multi`. - `Bigstring.unsafe_{get,set}-{,u}int8` used the generic bigarray access function without a type annotation. As a result the compiler generated a call to the generic C function. Fixed this by adding type annotations. - Add new functions to map that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules. - Made `Unpack_buffer.Unpack_one` monadic so that users can easily compose file parsers Added a couple simple parsers as examples and for testing. - Avoid use of polymorphic compare in Quickcheck. Make `Quickcheck.Generator.bind_choice` lazy: do not eagerly descend into all branches. Reduces memory overhead by setting a threshold on the probability of choices that are remembered and discarded by `Quickcheck.iter` and friends. Motivation: Currently, `Quickcheck.iter` and related functions guarantee never to repeat a choice from a generator. This winds up recording every choice ever made, which for a lot of generators is a prohibitive cost in space, and most of the recorded values are very unlikely to be repeated anyway. Implementation: This feature sets a probability threshold below which choices will not be remembered. Choosing a fairly low, but still non-zero, threshold means values are still very unlikely to be repeated, but memory usage stays low. As of this version, the benefits of "forgetting" unlikely-to-repeat values: ┌──────────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ `quickcheck.ml:Quickcheck.iter` remember │ 20.26ms │ 16.33Mw │ 100.85kw │ 100.85kw │ 100.00% │ │ `quickcheck.ml:Quickcheck.iter` forget │ 17.65ms │ 16.21Mw │ 34.83kw │ 34.83kw │ 87.10% │ └──────────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ - Optimizations to: + various `Float.t` validation functions + various functions in `Validate` + `List.fold_right` - Made the type of `Option.compare` compatible with `@@deriving compare`. - Fixed an example code fragment in a comment in applicative_intf.ml - In `Core_hashtbl`, the `add_worker` function used a `bool ref` both internally and to pass to `Avltree` to track whether a new key is added. This was allocated on every call to `add` or `set`, and `set` didn't even use its contents. This version pre-allocates the `bool ref` inside each `Core_hashtbl.t` and reuses it. It still can't be a `mutable` field because it does need to be passed to `Avltree`. After change: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 84.73ns │ 3.00w │ 0.83w │ 0.83w │ 0.01% │ │ Hashtbl.set `w/ collisions` │ 112.46ns │ │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 82.74ns │ 3.50w │ 0.53w │ 0.53w │ 0.01% │ │ Hashtbl.change `w/ collisions` │ 191.50ns │ 4.56w │ 1.15w │ 1.15w │ 0.03% │ │ Hashtbl.merge `no collisions` │ 292_976.43ns │ 26_669.00w │ 15_381.62w │ 12_305.62w │ 48.52% │ │ Hashtbl.merge `w/ collisions` │ 603_822.86ns │ 33_001.00w │ 20_037.22w │ 16_961.22w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 80_992.57ns │ 3_088.00w │ 4_102.63w │ 3_077.63w │ 13.41% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 178_080.05ns │ 4_621.00w │ 5_668.61w │ 4_643.61w │ 29.49% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 176_442.98ns │ 16_403.00w │ 9_222.64w │ 6_148.64w │ 29.22% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 297_577.29ns │ 19_472.00w │ 12_292.13w │ 9_218.13w │ 49.28% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ Before change: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set `w/ collisions` │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge `w/ collisions` │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ - In `Core_hashtbl.add_worker`, removed a `match` that avoided calling `Avltree.add`, but actually did hurt performance overall. Perhaps at some point before cross-module inlining, this was a helpful optimization. Right now it bypasses the mutation inside `Avltree`, so replacing a value in a non-colliding bucket (a `Leaf`) causes unnecessary re-allocation of the leaf. After changes: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 52.19ns │ 2.00w │ │ │ │ │ Hashtbl.set `w/ collisions` │ 112.04ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 87.25ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 195.85ns │ 5.56w │ 1.29w │ 1.29w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 308_164.10ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.48% │ │ Hashtbl.merge `w/ collisions` │ 526_914.80ns │ 38_119.00w │ 20_013.81w │ 16_937.81w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 76_983.60ns │ 5_135.00w │ 4_100.44w │ 3_075.44w │ 14.61% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 174_712.92ns │ 6_668.00w │ 5_667.47w │ 4_642.47w │ 33.16% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 176_681.57ns │ 19_476.00w │ 9_231.75w │ 6_157.75w │ 33.53% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 280_448.62ns │ 22_545.00w │ 12_293.32w │ 9_219.32w │ 53.22% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ Before changes: ┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set `no collisions` │ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.set `w/ collisions` │ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.change `no collisions` │ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.change `w/ collisions` │ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.merge `no collisions` │ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.merge `w/ collisions` │ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exn `no resize, no collisions` │ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exn `no resize, w/ collisions` │ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exn `w/ resize, no collisions` │ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exn `w/ resize, w/ collisions` │ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ - Add new functions to hashtbl that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules. - Add a bunch of functions to list and array that add common missing functionality and/or that make their interfaces more uniform and consistent with other container modules. - Rewrite `Hashtbl.merge` to be simpler and faster. After changes: ┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge `no collisions` │ 172.57us │ 17.44kw │ 9.22kw │ 7.69kw │ 48.76% │ │ Hashtbl.merge `w/ collisions` │ 284.55us │ 20.61kw │ 11.53kw │ 9.99kw │ 80.41% │ │ Pooled_hashtbl.merge `no collisions` │ 260.57us │ 5.20kw │ 19.18kw │ 3.09kw │ 73.63% │ │ Pooled_hashtbl.merge `w/ collisions` │ 353.88us │ 5.20kw │ 19.18kw │ 3.09kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ Before changes: ┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge `no collisions` │ 309.59us │ 31.79kw │ 15.38kw │ 12.30kw │ 48.91% │ │ Hashtbl.merge `w/ collisions` │ 526.67us │ 38.12kw │ 19.97kw │ 16.90kw │ 83.21% │ │ Pooled_hashtbl.merge `no collisions` │ 469.41us │ 7.32kw │ 35.29kw │ 3.12kw │ 74.16% │ │ Pooled_hashtbl.merge `w/ collisions` │ 632.96us │ 7.32kw │ 35.29kw │ 3.12kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ - Make `Hashtbl` functions raise an exception if a callback passed in as an argument mutates one of the hash tables being worked on. Usually, though not always, this comes up for iteration functions. Once a hash table has been mutated, it is unsafe to continue operating on it, as its structure may have changed. Buckets and their contents may have been moved or resized; continuing may result in skipping key/value pairs, repeating key/value pairs, or executing unsafe operations. This feature adds a `mutation_allowed` flag to hash tables. Each mutating operation first checks the flag, and raises if it is not set. Each operation with callbacks that must not mutate unsets the flag before calling the callbacks, and restores the flag's original value when it finishes. We compared the timing of this implementation to an alternate implementation using a mutation counter, and the time and space used for this implementation was much better for iteration and within epsilon of the other for single-key operations like `set`. - Array function names related to zipping are all over the place. Make them match List, which has a nice uniform naming scheme. * Rename `combine` -> `zip_exn` * Rename `split` -> `unzip` * (`zip` remains named as `zip`) - Add `~key` and `~data` labels to `Hashtbl.filteri_inplace` - Added `Hash_set.to_hashtbl`, by analogy to `Set.to_map`. - Since we are mutating avltrees in place, make sure the compiler sees the type parameters as invariant. Tested that a segfaulting example doesn't compile anymore. - Add label `f` to Hashtbl.change, Map.change, & family. Introduce the new function `update` in those modules, which enforces statically the presence of a resulting value Example: -|val Hashtbl.change : 'a t -> key -> ('a option -> 'a option) -> unit +|val Hashtbl.change : 'a t -> key -> f:('a option -> 'a option) -> unit +|val Hashtbl.update : 'a t -> key -> f:('a option -> 'a) -> unit The motivation for the introduction of `update` is that in an overwhelming majority of the places where `Hashtbl.change` is used in our codebase, it is statically known that a new value shall be computed and stored. The use of the dynamism offered by `change`, which can return an option, is error prone. The addition of the label is considered acceptable in consideration to external libraries depending on core, because a missing label is just a warning, and we do not guarantee stability in the presence of -warn-error = true. - Changed `Source_code_position.t` from: `@@deriving bin_io, sexp` to: `@@deriving sexp_of` and made `sexp_of` use the human-readable format, `"FILE:LINE:COL"`, rather than the unreadable format. Removed `Source_code_position.t_hum`, which is now obsolete. If one wants a serialized source-code position, one can use `Source_code_position.Stable`. - Added `Ref.set_temporarily`, for temporarily setting a ref to a value for the duration of a thunk. val set_temporarily : 'a t -> 'a -> f:(unit -> 'b) -> 'b - Add the function `singleton : 'a -> 'a t` in the stack containers. It cannot be added to `Container.S` directly because some container never have exactly 1 element. - Made `Core.Array` match `Invariant.S1`. - Change the interface of `Make_iterable_binable*` to give the control back to the user when deserializing Bin\_protted data. Improve the bin\_prot deserialization of `Map`s and `Set`s. We construct a balanced tree directly instead of relying on `Map.add` / `Set.add`. This is possibile because the size of the map is known and elements are sorted. The complexity goes down from n.log(n) to n. In case the comparison function changes (and the invariant is not respected), there is a fallback to reconstruct the whole map from scratch. - Add a function to blit a `Rope.t` into a `Buffer.t`. - Hashtbl differs from some other core containers with idiosyncratic naming of iteration functions. Change to be consistent and to more closely match the conventions for List and Array. Hashtbl: * Copy `iter` -> `iteri`. * Add a deprecation tag to `iter`. - Made `Bag.invariant` and `Doubly_linked.invariant` match `Invariant.S1`. - Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current `Map.iter` vs `Map.map` vs `Map.mapi`). Change to be consistent and to more closely match the conventions for List and Array. Map: * Copy `filter` -> `filteri`. * Add a deprecation tag to `filter`. - Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current `Map.iter` vs `Map.map` vs `Map.mapi`). Change to be consistent and to more closely match the conventions for List and Array. Map: * Copy `iter` -> `iteri`. * Add a deprecation tag to `iter`. - Made `Core.Set_once` match `Invariant.S1`. - Add `Bigstring.concat`. - For `Core.Unique_id`, exposed `@@deriving typerep`. - Expose Hashtbl.hashable, analogous to Map.comparator. - Adds a constant-time `val mem_elt : 'a t -> 'a Elt.t -> bool` to Doubly_linked and Bag - Add `Ordering.to_int` which can be useful when one is writing a comparison function. Instead of dealing with the int directly, one can return Ordering.t values and transform them later into ints. - `Float.int_pow`: Fast computation of `x ** n` when n is an integer. - Make `Core_kernel.Std.Nothing.t` enumerable. There's no particular reason not to. - Minor improvements to queue interface - Call `Caml.Pervasives.do_at_exit` before printing an exception and exiting The default ocaml uncaught exception handler does this. It is especially useful for curses applications as the `at_exit` handler has a chance to put back the terminal in a good state before printing the exception and backtrace. Do the same in Core and Async. - Removed big literals so that the compiler does not complain in 32bit - Add `List.range'`, a generalization of `List.range`. - Add some functions to `Map` that are present in `Hashtbl`: - `remove_multi` - `partition_tf` - `partitioni_tf` - `partition_map` - `partition_mapi` - Add a `Map.nth_exn` as a missing complementary function to nth - Renamed `Validate.fail_sexp` as `fail_s`, to follow our new `*_s` convention for `Sexp.t`-taking functions. - `Sequence.split_n_eagerly` returns a pair of sequences, but every element of the first sequence has already been evaluated by the time it returns. This feature just makes the first component of the tuple a list instead of a sequence, and renames `split_n_eagerly` to `split_n`. Additionally, this feature adds a new `chunks_exn` function, which just applies `split_n` until the input sequence is empty. - Removed `Timing_wheel`'s default `alarm_precision`, to force people to think about the precision they want when they create a timing wheel. - In `Timing_wheel.Config.sexp_of_t`, used `@sexp_drop_default` with `level_bits`. - Write a better-performing `Array.filter_mapi` function, and implement `Array.filter_map`, `Array.filter_opt`, `Array.partitioni_tf`, and `Array.partition_tf` in terms of it. Slightly worse for zero-length input arrays, about unch'd if we're filtering out almost everything (`eq_zero`), better on most everything else. ┌────────────────────────────────────────────────────┬─────────────────┬─────────────┬─────────────┬─────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├────────────────────────────────────────────────────┼─────────────────┼─────────────┼─────────────┼─────────────┼────────────┤ │ `core\_array.ml:filter` old-filter-even:0 │ 12.37ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:1 │ 77.44ns │ 15.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:10 │ 207.10ns │ 36.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:100 │ 1\_699.41ns │ 261.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-even:1000 │ 56\_320.50ns │ 1\_009.00w │ 2\_506.01w │ 1\_004.01w │ 0.30% │ │ `core\_array.ml:filter` old-filter-even:10000 │ 469\_134.89ns │ 10\_009.00w │ 25\_007.38w │ 10\_005.38w │ 2.46% │ │ `core\_array.ml:filter` old-filter-even:100000 │ 4\_421\_742.22ns │ 100\_009.00w │ 250\_130.09w │ 100\_128.09w │ 23.17% │ │ `core\_array.ml:filter` new-filter-even:0 │ 13.87ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:1 │ 57.64ns │ 18.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:10 │ 196.28ns │ 35.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:100 │ 1\_361.04ns │ 215.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-even:1000 │ 21\_473.76ns │ 1\_014.00w │ 1\_001.02w │ │ 0.11% │ │ `core\_array.ml:filter` new-filter-even:10000 │ 204\_033.12ns │ 10\_014.00w │ 10\_001.14w │ 0.14w │ 1.07% │ │ `core\_array.ml:filter` new-filter-even:100000 │ 2\_058\_144.47ns │ 100\_014.00w │ 100\_002.00w │ 1.00w │ 10.78% │ │ `core\_array.ml:filter` old-filter-eq\_zero:0 │ 12.21ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:1 │ 71.23ns │ 15.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:10 │ 174.80ns │ 24.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:100 │ 1\_212.70ns │ 114.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-eq\_zero:1000 │ 23\_347.51ns │ 13.00w │ 1\_007.00w │ 6.00w │ 0.12% │ │ `core\_array.ml:filter` old-filter-eq\_zero:10000 │ 210\_509.83ns │ 13.00w │ 10\_007.00w │ 6.00w │ 1.10% │ │ `core\_array.ml:filter` old-filter-eq\_zero:100000 │ 1\_912\_253.91ns │ 13.00w │ 100\_007.01w │ 6.01w │ 10.02% │ │ `core\_array.ml:filter` new-filter-eq\_zero:0 │ 13.70ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:1 │ 56.56ns │ 18.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:10 │ 179.42ns │ 27.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:100 │ 1\_254.49ns │ 117.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-eq\_zero:1000 │ 20\_968.06ns │ 16.00w │ 1\_001.02w │ │ 0.11% │ │ `core\_array.ml:filter` new-filter-eq\_zero:10000 │ 204\_299.82ns │ 16.00w │ 10\_001.13w │ 0.13w │ 1.07% │ │ `core\_array.ml:filter` new-filter-eq\_zero:100000 │ 2\_019\_283.81ns │ 16.00w │ 100\_001.91w │ 0.91w │ 10.58% │ │ `core\_array.ml:filter` old-filter-neq\_zero:0 │ 12.14ns │ 9.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:1 │ 32.72ns │ 11.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:10 │ 219.18ns │ 48.00w │ │ │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:100 │ 1\_902.76ns │ 408.00w │ 0.12w │ 0.12w │ │ │ `core\_array.ml:filter` old-filter-neq\_zero:1000 │ 82\_032.44ns │ 2\_007.00w │ 3\_998.20w │ 1\_997.20w │ 0.43% │ │ `core\_array.ml:filter` old-filter-neq\_zero:10000 │ 850\_234.44ns │ 20\_007.00w │ 40\_014.86w │ 20\_013.86w │ 4.46% │ │ `core\_array.ml:filter` old-filter-neq\_zero:100000 │ 7\_345\_941.05ns │ 200\_007.00w │ 400\_407.82w │ 200\_406.82w │ 38.49% │ │ `core\_array.ml:filter` new-filter-neq\_zero:0 │ 13.66ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:1 │ 18.26ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:10 │ 201.04ns │ 43.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:100 │ 1\_404.33ns │ 313.00w │ │ │ │ │ `core\_array.ml:filter` new-filter-neq\_zero:1000 │ 22\_829.70ns │ 2\_012.00w │ 1\_001.02w │ │ 0.12% │ │ `core\_array.ml:filter` new-filter-neq\_zero:10000 │ 218\_872.52ns │ 20\_012.00w │ 10\_001.21w │ 0.21w │ 1.15% │ │ `core\_array.ml:filter` new-filter-neq\_zero:100000 │ 2\_121\_340.68ns │ 200\_012.00w │ 100\_002.77w │ 1.77w │ 11.12% │ │ `core\_array.ml:filter` old-filter\_map-int:0 │ 9.58ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:1 │ 68.46ns │ 11.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:10 │ 191.66ns │ 32.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:100 │ 1\_492.60ns │ 257.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-int:1000 │ 57\_155.42ns │ 1\_005.00w │ 2\_507.01w │ 1\_005.01w │ 0.30% │ │ `core\_array.ml:filter` old-filter\_map-int:10000 │ 522\_177.50ns │ 10\_005.00w │ 25\_008.54w │ 10\_006.54w │ 2.74% │ │ `core\_array.ml:filter` old-filter\_map-int:100000 │ 5\_945\_405.67ns │ 100\_005.00w │ 250\_170.69w │ 100\_168.69w │ 31.15% │ │ `core\_array.ml:filter` new-filter\_map-int:0 │ 12.03ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:1 │ 53.63ns │ 14.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:10 │ 164.16ns │ 31.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:100 │ 1\_263.42ns │ 211.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-int:1000 │ 23\_113.12ns │ 1\_010.00w │ 1\_001.02w │ │ 0.12% │ │ `core\_array.ml:filter` new-filter\_map-int:10000 │ 218\_152.23ns │ 10\_010.00w │ 10\_001.15w │ 0.15w │ 1.14% │ │ `core\_array.ml:filter` new-filter\_map-int:100000 │ 2\_217\_307.86ns │ 100\_010.00w │ 100\_002.11w │ 1.11w │ 11.62% │ │ `core\_array.ml:filter` old-filter\_map-float:0 │ 9.32ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:1 │ 66.68ns │ 13.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:10 │ 182.86ns │ 42.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:100 │ 1\_496.56ns │ 357.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-float:1000 │ 76\_479.74ns │ 2\_005.00w │ 3\_507.02w │ 2\_005.02w │ 0.40% │ │ `core\_array.ml:filter` old-filter\_map-float:10000 │ 694\_999.59ns │ 20\_005.00w │ 35\_011.08w │ 20\_009.08w │ 3.64% │ │ `core\_array.ml:filter` old-filter\_map-float:100000 │ 8\_694\_669.26ns │ 200\_005.00w │ 350\_476.44w │ 200\_474.44w │ 45.56% │ │ `core\_array.ml:filter` new-filter\_map-float:0 │ 12.29ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:1 │ 58.24ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:10 │ 142.67ns │ 41.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:100 │ 1\_119.41ns │ 311.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-float:1000 │ 14\_262.66ns │ 2\_010.00w │ 1\_001.02w │ │ 0.07% │ │ `core\_array.ml:filter` new-filter\_map-float:10000 │ 136\_448.05ns │ 20\_010.00w │ 10\_001.23w │ 0.23w │ 0.71% │ │ `core\_array.ml:filter` new-filter\_map-float:100000 │ 1\_282\_005.01ns │ 200\_010.00w │ 100\_003.14w │ 2.14w │ 6.72% │ │ `core\_array.ml:filter` old-filter\_map-boxed:0 │ 9.48ns │ 5.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:1 │ 71.16ns │ 13.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:10 │ 197.40ns │ 42.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:100 │ 1\_762.40ns │ 357.00w │ │ │ │ │ `core\_array.ml:filter` old-filter\_map-boxed:1000 │ 86\_220.67ns │ 2\_005.00w │ 3\_507.02w │ 2\_005.02w │ 0.45% │ │ `core\_array.ml:filter` old-filter\_map-boxed:10000 │ 828\_291.42ns │ 20\_005.00w │ 35\_011.84w │ 20\_009.84w │ 4.34% │ │ `core\_array.ml:filter` old-filter\_map-boxed:100000 │ 7\_955\_395.61ns │ 200\_005.00w │ 350\_441.44w │ 200\_439.44w │ 41.68% │ │ `core\_array.ml:filter` new-filter\_map-boxed:0 │ 14.43ns │ 10.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:1 │ 59.24ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:10 │ 198.19ns │ 41.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:100 │ 1\_580.21ns │ 311.00w │ │ │ │ │ `core\_array.ml:filter` new-filter\_map-boxed:1000 │ 52\_045.31ns │ 2\_010.00w │ 2\_011.01w │ 1\_010.01w │ 0.27% │ │ `core\_array.ml:filter` new-filter\_map-boxed:10000 │ 479\_239.44ns │ 20\_010.00w │ 20\_012.42w │ 10\_011.42w │ 2.51% │ │ `core\_array.ml:filter` new-filter\_map-boxed:100000 │ 4\_389\_392.06ns │ 200\_010.00w │ 200\_135.09w │ 100\_134.09w │ 23.00% │ │ `core\_array.ml:filter` old-partition\_tf:0 │ 16.55ns │ 16.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:1 │ 128.08ns │ 29.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:10 │ 554.15ns │ 111.00w │ │ │ │ │ `core\_array.ml:filter` old-partition\_tf:100 │ 4\_853.58ns │ 921.00w │ 0.46w │ 0.46w │ 0.03% │ │ `core\_array.ml:filter` old-partition\_tf:1000 │ 201\_289.06ns │ 5\_016.00w │ 9\_015.21w │ 5\_010.21w │ 1.05% │ │ `core\_array.ml:filter` old-partition\_tf:10000 │ 1\_796\_749.87ns │ 50\_016.00w │ 90\_040.96w │ 50\_035.96w │ 9.41% │ │ `core\_array.ml:filter` old-partition\_tf:100000 │ 19\_084\_871.85ns │ 500\_016.00w │ 902\_187.67w │ 502\_182.67w │ 100.00% │ │ `core\_array.ml:filter` new-partition\_tf:0 │ 28.29ns │ 23.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:1 │ 103.78ns │ 31.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:10 │ 504.10ns │ 96.00w │ │ │ │ │ `core\_array.ml:filter` new-partition\_tf:100 │ 3\_869.52ns │ 726.00w │ 0.23w │ 0.23w │ 0.02% │ │ `core\_array.ml:filter` new-partition\_tf:1000 │ 122\_807.29ns │ 4\_023.00w │ 5\_013.04w │ 2\_010.04w │ 0.64% │ │ `core\_array.ml:filter` new-partition\_tf:10000 │ 1\_197\_596.39ns │ 40\_023.00w │ 50\_020.05w │ 20\_017.05w │ 6.28% │ │ `core\_array.ml:filter` new-partition\_tf:100000 │ 10\_458\_344.09ns │ 400\_023.00w │ 500\_590.94w │ 200\_587.94w │ 54.80% │ └────────────────────────────────────────────────────┴─────────────────┴─────────────┴─────────────┴─────────────┴────────────┘ - Added `Binable.Of_sexpable` functor. - Install the sexp exception printer sooner so that we can get proper `%test_result ...` errors in things that come before `core_kernel`. - In `Stable_unit_test.Make` functors, include all test failures rather than just the first. This is useful for updating batches of expected `bin_io` results when stabilizing a module. - Remove an unnecessary cast in or_error.ml ### core_profiler - Switched to ppx. - Minor adjustments to the command line of profiler_tool.exe: - Make '-%' an alias for '-percentile' - Make '-percentile' accept a comma-separated list of numbers - Add '-median' argument that is equivalent to '-percentile 50' ### email_message - Bugfixes and minor API improvements. ### incremental - Add README.org to Incremental. - Added some type annotations based on comments by @def-lkb about lack of principality. - Switched to ppx. ### jenga - Restructure the code in a way that allows to build binaries that statically link jenga with the rules. This is useful because some debugging/profiling tools don't work in the presence of dynamically loaded code very well. - Switch to PPX. - Change the gc info output by jenga so it shows heap size and top heap size, instead of live and heap size. The live part is not super useful given how random it is. I have seen cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at the end so the top heap size avoids being tricked. - First half of the fixes no packing: sharing the structures of dependencies, so they take less space on disk (and in memory as well, when they are loaded from disk, but not really when building from scratch given the way we will use them). The sexp format also has sharing, because it would also blow up in size otherwise (this is different from the interning of paths, where the interning saves a constant factor). And of course, it makes it possible to see the actual on-disk representation which is nice. Also fix unhelpful error (contains no information) when the db can't be loaded. Break the thing that avoids rerunning rules when the set of dependencies decreases. I think it was never useful anyway. - Better error on duplicate targets in the same rule. - To prevent running more than one jenga in a repository, use a local lock rather than an nfs one. We need a transition period though, so for now we use both kinds of locks. Building on nfs is slow, so I don't think there's any downside is not supporting nfs this way. And maybe inotify doesn't work. The upside is that we don't step into Lock.Nfs bugs where if a process is interrupted at the wrong time (when the two lock files are empty) the locks won't be cleaned up automatically, forcing someone to get rid of the lock files manually. Also rename `.jenga/.jenga.*` to `.jenga/*`, because all these prefixes are annoying. - Added a couple of options to turn off some part of jenga, which I used to check how they impacted performance, and could still be handy later. - Optionally display additional information about much allocation was done, at the end of builds. Used it to try to improve memory usage of full tree builds without actually doing full tree builds. - Make stat'ing faster. Hash cons some tenacious that build mtimes map to avoid a huge increase of memory use. - Got rid of noise when stopping jenga. - Some changes to the implementation of Tenacious to improve memory efficiency. Includes the following changes: - Remove `strong_refs` field of `Heart.fragile` type and instead insert links from the `clients` `Ring.t` to its parent using `Ring.keep_alive`. - Replace the `Ring.t` in the `triggers` field of `Hearts.fragile` with an `Ivar.t` since all uses of `triggers` were producing their own equivalent `IVar.t`s. Remove the functions broken by this because they were unused. - Make the `Tenacious.t` type a concrete datatype, and optimize pure computations by partially evaluating this datatype directly in the pure case. - Rather than building separate `Heart.t`s for cancellation and the result, split the cancellation heart into two hearts `cancel` and `dep` and then use `dep` as the result heart. This means that a tenacious is cancelled if either `cancel` or `dep` is broken, and it must return a heart representing the validity of its result combined with `dep`. Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution time when building the lib directory from scratch. - Adding some `sexp_of` functions, since they're always missing and it's a pain when debugging. - Adding direct support for `Dep.map`. Even now that `Tenacious` is smarter, this still creates less `bind`s. Doesn't seem to make much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if nothing else, it's much less surprising to think that `Dep.Map` becomes `Tenacious.Map`. - Added Dep.List.concat. - Make it possible to turn off the behavior where jenga rejects commands that output on stderr. It increases slightly the footprint of the in memory db, but the difference is tiny compared to the rest of the memory usage. - Added a few tests about `Jenga_lib.Api.Reflect` ### ocaml_plugin - Switch to ppx. - Allow ppx-style code to be loaded by plugin-applications build using ocaml\_plugin. - Follow Core & Async evolution. ### patdiff - `patdiff -location-style omake` should print the line number of the first difference in each hunk, skipping context lines. - Switched to PPX. - Added binding in patdiff to use the newly minted colors of Ansi\_terminal. This will be used notably by patdiff4 to produce better ddiff. Also, have the module `Color` and `Style` implement and export `Comparable.S`. This is useful for example to dedup styles from a list of styles without relying on the polymorphic equality. - Make it so that if you pass `-warn-if-no-trailing-newline-in-both false` then you get the warning only when one file has a trailing newline and the other file does not. If you pass `-warn-if-no-trailing-newline-in-both true` or omit this flag, then you get the current behavior of warning for each file independently. - Patdiff's unified-tests currently render colors codes in angle brackets. Change them to square brackets. Square brackets are word boundaries, so we'll get more legible diffs when tests fail. - Simple code change in patdiff to prepare more changes in patdiff4. This change is a pure refactoring and has zero runtime change. Just moving some functions around. - patdiff_core.ml is a very long module. start extracting module from it. start with format. in the process, expose in a private fashion the record `Rule.t`. - Continue on splitting the file patdiff_core.ml into smaller pieces. In this version, we extract each output mode into its own file. - Kill the generation of html diffs in patdiff. There are good third party tools that can convert efficiently ansi texts to html directly. We plan on simplifying a bit the patdiff source code to increase its maintainability, and dropping the requirement of producing html output seems a step in the right direction. Some pointers: http://www.pixelbeat.org/scripts/ansi2html.sh jane: app/ralloc/commander/ansi2html.ml ### patience_diff - Switch to PPX. ### ppx_assert - Update to follow evolution of `Ppx_core`. ### ppx_bench - Update to follow `Ppx_core` evolution. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by ppx\_inline\_test\_drop doesn't cause a failure. ### ppx_bin_prot - Minor changes, nothing worth mentionning. ### ppx_compare - Follow evolution of `Ppx_core` and `Type_conv`. ### ppx_core - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code. - Merlin uses `@merlin.* ...` attributes in different places. Which ppx\_driver reports as unused. Introduce the concept of reserved namespaces. When one declares the namespace "foo" as reserved then: - `foo.*` will never get reported as unused - it is impossible to `Attribute.declare "foo.*"` Mark the "merlin" namespace as reserved by default. - Don't print: Extension `foo' was not translated. Hint: Did you mean foo? - OCaml makes no distinctions between "foo" and `{whatever|foo|whatever}`. The delimiter choice is simply left to the user. Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept {whatever|foo|whatever}. - Avoid stupid hints like this one: Attribute `default' was not used. Hint: `default' is available for label declarations but is used here in the context of a label declaration. Did you put it at the wrong level? - Update the API for the common case of extension point expanders. Make it simpler to define ppx rewriters that locally expand extension points, which is the majority of our non-type-conv rewriters. Such expanders are run inside the same `Ast_traverse.map` in a top-down manner which: - probably improve speed - help with rewriters that capture a pretty-print of their payload - help with rewriter that interpret some extension points in a special way inside their payload - Fix the order in which errors are reported by ppx rewriters. Make them be reported in the same order as they appear. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by `ppx_inline_test_drop` doesn't cause a failure. ### ppx_csv_conv - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx/changes-for-public-release Changes required for the first public release of our ppx rewriters Improved the type_conv export to ppx_deriving --------------------------------------------- - improved how it works, to enable it we just need: Type_conv.Ppx_deriving_exporter.set (module Ppx_deriving) - Type_conv.add_alias now takes as argument registered type_conv deriviers. This ensure that we can resolve aliases right from the registration time. This simplify the export to ppx_deriving Split some ppx rewirters ------------------------ Split some functions out of their main ppx_XXX library that does the registration with ppx_driver or ppx_type_conv. For instance: - ppx_sexp_conv defines sexp_of_quote that is used by ppx_assert - ppx_here defines ast_of_pos that is used by ppx_fail and ppx_assert Making ppx_assert depends on ppx_sexp_conv and ppx_here is not good as it enables `@@deriving sexp` and `%sexp_of: ty` even if the user only writes ppx_assert in the jbuild. This is especially problematic for the public release as it makes ppx_assert incompatible with ppx_deriving. This feature moves these functions into libraries called ppx_XXX_expander (and ppx_inline_test_libname for ppx_inline_test). To help reviewing the changes, you can run: patdiff <(hg cat -r `fe show -base` ppx/ppx_compare/src/ppx_compare.ml) <(hg cat -r `fe show -tip` ppx/ppx_compare/expander/ppx_compare_expander.ml) patdiff <(hg cat -r `fe show -base` ppx/ppx_sexp_conv/src/ppx_sexp_conv.ml) <(hg cat -r `fe show -tip` ppx/ppx_sexp_conv/expander/ppx_sexp_conv_expander.ml) Move Ppx_type_conv.Std.Type_conv_path to Ppx_core ------------------------------------------------- It's not type-conv specific anymore and is needed by libraries that don't use type_conv otherwise. Rename some old runtime libraries --------------------------------- pa_test_lib --> ppx_assert_lib pa_bench_lib --> ppx_bench_lib * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx/delete-make-at-the-end Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply. For users ========= This feature moves a few values up in signatures. For instance in this interface `sexp_of_t` is now the first value instead of the second one as before this feature: type t `@@deriving sexp_of` val x : int In some cases this caused the OCaml compiler to complain about items in the signature being re-ordered. In these cases the signature was adapted to match the expected ordering. * Affected files ppx/ppx_csv_conv/src/ppx_csv_conv.ml - ppx An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures. * Affected files ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv - ppx/ppx_csv_conv-example Make ppx_csv_conv example do something. Testing ------- Added inline test. * Affected files ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv ### ppx_custom_printf - OCaml makes no distinctions between "foo" and `{whatever|foo|whatever}`. The delimiter choice is simply left to the user. Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept `{whatever|foo|whatever}`. - Fix missing location in errors for broken custom printf example like: printf !"%{sexp: int" 3;; - Update to follow `Ppx_core` evolution. ### ppx_driver - Disable safety check when code transformations are used as standard "-ppx" rewriters. - Introduce reserved namespaces, see `Ppx_core`'s changelog. Pass errors as attribute with -dparsetree to avoid "Error while running external preprocessor". - Update to follow `Ppx_core` evolution. ### ppx_enumerate - Update to follow type\_conv evolution. ### ppx_expect Initial release. ### ppx_fail - Added a README.md - Update to follow `Ppx_core` evolution. ### ppx_fields_conv - The `iter` function generated by ppx\_variants\_conv and ppx\_fields\_conv allowed one to give function which returned values of arbitrary types as iter function. This release constraint these functions to return unit. N.B. the signature generated by the use of `@@deriving variants` (resp. fields) in interface already constrained the type to unit. - Update to follow type\_conv's evolution. - Add `Fields.make_creator` to ppx\_fields\_conv's readme, since it appears to not be all that deprecated. ### ppx_here - Make ppx\_here translate `[%here]` instead of `_here_`. - Update to follow `Ppx_core` evolution. ### ppx_inline_test - Support literate-style .ml files that allow ocaml code interleaved with expected output annotations. Compiling with the `ppx_expect_test` generates a program that outputs the original source file, but with the actual output substituted for the expected-output annotations. Then we can pat-diff the original file against the output file. Testing ------- Examples in the test/ and example/ folders. - Expect-tests can now be written inline in libraries by using `let%expect_test`. The runtime library has been split into two components: the test runner, which collects the output of the test body, and registers enough information to construct the `*.ml.corrected` file from the input; and the test evaluator, which compares the test output against the expected output and generates the output files. - Update to follow `Ppx_core` evolution. - When an exception is raised inside a `let%test_module`, display the position and name of the TEST\_MODULE, same as for the `let%test`. - Mark attributes as handled inside explicitly dropped pieces of code. So that a `@@deriving` inside a let%test dropped by `ppx_inline_test_drop` doesn't cause a failure. ### ppx_optcomp - Change the way optcomp resolve filenames in #import directives Do the same as cpp, i.e. for relative filenames, consider they are relative to the directory of the file being parsed. This doesn't matter internally as build commands are always executed from the current directory, but it matters for the public release as everything is executed from the root. ### ppx_sexp_conv - Trying to improve the tests in ppx\_sexp\_conv because they are a mess. At least all tests are automatic now. And more things are tested like the sexpification of exceptions. - Update to follow `Type_conv` and `Ppx_core` evolution. - Make ppx\_sexp\_conv correctly handle aliases to polymorphic variants: type t = ` `A ` `@@deriving sexp` type u = t `@@deriving sexp` type v = ` u | `B ` `@@deriving sexp` Before, `v_of_sexp` would never manage to read `B. This problem is now fixed if you use `sexp_poly` on `u` instead of `sexp`, and if you don't, you get an "unbound value __u_of_sexp__". People should use `sexp_poly` when they have a polymorphic variant type that is not syntactically a polymorphic variant, but in practice it's simpler to replace `sexp` by `sexp_poly` when faced with the error above. The need for `sexp_poly` should happen only in one new case: an implementation says `type u = t `@@deriving sexp`` but the interface says `type u = ``A` `@@deriving sexp``. (the old case where it was already needed is when you have an interface that says `type u = t `@@deriving sexp`` and in some other implementation you try to say `type t = ` That_module.t | `A ` `@@deriving sexp``). ### ppx_type_conv - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code. - Cleanup in type\_conv: remove `Type_conv.Generator_result.make_at_the_end`, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply. - Update to follow `Ppx_core` evolution. ### ppx_typerep_conv - Update following `Ppx_core` and `Type_conv` evolution. - Add a README. ### ppx_variants_conv - The `iter` function generated by ppx\_variants\_conv and ppx\_fields\_conv allowed one to give function which returned values of arbitrary types as iter function. This feature constraint these functions to return unit. N.B. the signature generated by the use of `@@deriving variants` (resp. fields) in interface already constrained the type to unit. - Update to follow `Type_conv` evolution. ### re2 - Switched to PPX. - Add `Re2.Parser.any_string` combinator. There are no tests because `any_string` is constructed only from the tested API and there's almost no interesting properties of it that can be verified. ### rpc_parallel - Switched to PPX. - Expose the `connection_timeout` argument in rpc\_parallel. This argument exists in `Rpc_parallel_core.Parallel`, but it is not exposed in `Rpc_parallel.Parallel`. - Allow custom handling of missed async\_rpc heartbeats. - Give a better error message when redirecting output on a remote box to a file path that does not exist. - remove unncessary chmod 700 call on the remote executable - Give a clear error message for the common mistake of not making the `Parallel.Make_worker()` functor application top-level - Make errors/exceptions in `Rpc_parallel` more observable - Make stderr and stdout redirection mandatory in order to encourage logging stderr - Clean up the use of monitors across `Rpc_parallel` - Fix bug with exceptions that are sent directly to `Monitor.main` (e.g. `Async_log` does this) - Add the ability to explicitly initialize as a master and use some subcommand for the worker. This would allow writing programs with complex command structures that don't have to invoke a bunch of `Rpc_parallel` logic and start RPC servers for every command. - Add the ability to get log messages from a worker sent back to the master. In fact, any worker can register for the log messages of any other workers. ### sexplib - Switch code in `lib` subdir to ppx-style. ### textutils - Switched to PPX. - Fixed a bug where the computation of cell heights could cause division by zero in some cases. - Expose the constructors of `Ascii_table.Align.t` so that we can write Column.create ~align:Left ... instead of Column.create ~align:Align.left ### typerep - Add whether record fields are mutable. ### typerep_extended - Switched to ppx. - Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code. ## 113.00.00 ### async - Added `Async.Std.Printf` module so that one doesn't unintentionally use blocking `Core.Std.Printf` functions in an Async program. There was much pre-existing code that did this via: : open Core.Std : open Async.Std `Async.Std.Printf` defines blocking functions (e.g `printf`, `eprintf`) to cause a type error, but leaves Async-friendly functions (e.g. `sprintf`, `ksprintf`) untouched. Replaced uses of `Printf.*`, with `Core.Std.Printf.*` where needed. ### async_extended - Added a more raw interface to `Delimited.Csv`. ### async_extra - Added `Limiter` module. Implements an async aware throttling rate limiter on top of `Core.Limiter`. - Generalized `Persistent_rpc_client` to supports RPC connection types with additional information besides the `Rpc.Connection.t` itself. For instance: `Persistent_rpc_client.Versioned` has `Versioned_rpc.Connection_with_menu.t` as its connection type. - Changed the `Persistent_rpc_client.Make` functor to not erase the type `conn` from its output module's signature. This way, the output of `Make` can be fed to functors or functions that expect a module matching `Persistent_rpc_client.S`. - Moved `Log` from `Async_extra` to `Async_unix`, so that the scheduler can refer to it. - Fixed a bug where `Persistent_rpc_client.close` would hang waiting for a connection to close. ### async_inotify - Added `modify_event_selector` optional parameter to `Async_inotify.create`. ### async_kernel - Switched `Lazy_deferred` to use `Or_error.t` rather than `('a, exn) Result.t`. Note: There is difference in the `run` argument between `Monitor.try_with` and `Monitor.try_with_or_error`. In this module, the function is called already in a closure inside a bind, so that difference is acceptable. - Made `Deferred` match `Invariant.S1`. - Improved `Async_kernel.Scheduler.run_cycles_until_no_jobs_remain` to use `Timing_wheel.fire_past_alarms` rather than sleeping. - Added `Quickcheck` module. - Reworked the initialization of `Monitor.try_with_ignored_exn_handling` to log exceptions using `Async.Log` so that it doesn't rely on top-level effects, which may not happen without packed libraries. ### async_rpc_kernel - Fixed race in `Rpc` that caused double connection cleanup. Two errors, `Connection_closed` and a Writer error, `(Uncaught_exn(monitor.ml.Error_((exn(\"writer error\"....))))))`, occurring at the same time will cleanup the connection twice and call response_handler of open_queries twice with two different errors. (((pid 31291) (thread_id 0)) ((human_readable 2015-05-25T10:47:18+0100) (int63_ns_since_epoch 1432547238929886105)) "unhandled exception in Async scheduler" ("unhandled exception" ((monitor.ml.Error_ ((exn ("Ivar.fill of full ivar" (Full _) lib/async_kernel/src/ivar0.ml:329:14)) (backtrace ("Raised at file \"error.ml\", line 7, characters 21-29" "Called from file \"rpc.ml\", line 101, characters 8-31" "Called from file \"connection.ml\", line 251, characters 8-172" "Called from file \"core_hashtbl.ml\", line 244, characters 36-48" "Called from file \"connection.ml\", line 248, characters 2-278" "Called from file \"async_stream.ml\", line 49, characters 53-56" "Called from file \"async_stream.ml\", line 21, characters 34-39" "Called from file \"job_queue.ml\", line 124, characters 4-7" "")) (monitor (((name main) (here ()) (id 1) (has_seen_error true) (is_detached false) (kill_index 0)))))) ((pid 31291) (thread_id 0))))) - Fixed bugs in `Rpc` in which a TCP connection's reader was closed before its writer. - In `Versioned_rpc`, eliminated an unnecessary Async cycle when placing RPC messages. - Added `Rpc.Pipe_rpc.close_reason` and `Rpc.State_rpc.close_reason`, which give the reason why a pipe returned by an RPC was closed. These functions take the IDs that are returned along with the pipes by the dispatch functions, so the interface of `dispatch` did not need to change. - Made `Rpc.Expert.dispatch` expose that the connection was closed, just like `One_way.Expert.dispatch`. - Expose the name of the `Versioned_rpc.Menu` RPC. ### async_smtp - Improve the async_smtp client interface so that it is suitable as a replacement for Core_extended.Std.Sendmail. ### async_ssl - Added `Ssl.Connection.close`. ### async_unix - Made Async dump core when it reports a "bug in async scheduler". There is no change for toplevel unhandled user exceptions, for which Async does not dump core. - Added `Dump_core_on_job_delay.dump_core` function, which exposes the core-dumping functionality in the C stubs for `Dump_core_on_job_delay`. - Made `Dump_core_on_job_delay.How_to_dump` an ordinary variant and moved it into `Async_kernel.Config`. - Changed `Thread_safe_pipe` functions that write to the pipe to take an additional argument, an `If_closed.t`, that says how to behave if the pipe is closed. The previous behavior is achieved with `~if_closed:Raise`. One can also now use `~if_closed:Return` to they return a variant reporting whether the pipe was closed, rather than raising. Returning a variant allows callers to distinguish the pipe-closed case from other errors. This change also allows us to do a a single acquisition of the Async lock, with the pipe-closed check synchronously immediately preceding the operation, avoiding a race. - Added `Fd.with_file_descr_deferred_exn`. - Improved the performance of `Clock.every`, and in particular reduced its allocation. It now allocates much less, especially with `~continue_on_error:false`. Handled `Clock.every`'s `~stop` argument directly using timing-wheel alarms, rather than using `Deferred.choose`. Slightly changed the behavior of Clock.every' f ~continue_on_error:false in the corner case where `f` raises but its result also becomes determined. Prior to this feature, iteration would stop. After this feature, iteration will continue, because `~continue_on_error:false` just looks at the deferred resulting from `f`. This doesn't affect: Clock.every f ~continue_on_error:false because if `f` raises, then there is no resulting deferred. Benchmark: +----------------------------------------------------+----------+------------+----------+----------+------------+ | Name | Time/Run | mWd/Run | mjWd/Run | Prom/Run | Percentage | +----------------------------------------------------+----------+------------+----------+----------+------------+ | [clock_ns.ml:Clock.every] ~continue-on-error:false | 54.21us | 91.03w | 0.36w | 0.36w | 22.06% | | [clock_ns.ml:Clock.every] ~continue_on_error:true | 245.80us | 93_208.27w | 7.31w | 7.31w | 100.00% | +----------------------------------------------------+----------+------------+----------+----------+------------+ - Added to `Clock.Event.t` type parameters so that one can record a value in the event when it happens or is aborted, and read that value via `Event.status`. type ('a, 'h) t val status : ('a, 'h) t -> [ `Aborted of 'a | `Happened of 'h | `Scheduled_at of Time.t ] val run_at : Time.t -> ('z -> 'h) -> 'z -> (_, 'h) t val abort : ('a, 'h) t -> 'a -> [ `Ok | `Previously_aborted of 'a | `Previously_happened of 'h ] - Fixed a (never observed) race in the Async scheduler's closing of file descriptors. Previously, when the number of active system calls on an `Fd.t` reached zero, the scheduler would call a closure that would immediately schedule the `close()` system call in a thread. It was possible (albeit very unlikely) that that `close()` would run before the scheduler got a chance to update the epoll set, violating the invariant that `close` is only ever called on fds not in the epoll set. Now, the scheduler enqueues an ordinary Async job to do the `close`, and thus the `close` cannot happen until the next cycle, after the scheduler has updated the epoll set. - Changed `Reader` to treat `read` returning `EPIPE` as end-of-file rather than fail, to deal with OpenOnload. This fixes an issue where reading from a TCP connection can return an `EPIPE` if the tcp connection is immediately closed. This happens when the application is running with onload and when the tcp connection is closed immediately after creation. - Reduced allocation of the Async scheduler's `File_descr_watcher`, by using callbacks to handle ready file descriptors. - Fixed `In_thread.Helper_thread.create`'s error message if there are no available threads in the thread pool. The error message is now constructed eagerly. It had been constructed lazily, so by the time it was rendered, the state might have changed, possibly making threads available. This leads to a nonsensical-looking error message that claims that there are no available threads, immediately followed by a list of available threads. - Moved `Log` from `Async_extra` to `Async_unix`, so that the scheduler can refer to it. - When `Writer.with_file_atomic` is unable to clean up its temp file, raise synchronously rather than asynchronously. This eliminates complaints about an exception being thrown after a deferred has been computed. - Added `Log.rotate` to force log rotation. val rotate : t -> unit Deferred.t. - Fixed `Log` rotation to correctly reset the size and number of lines. ### bignum - Fixed a bug in the =Zarith= library's `to_float` function. These fixes first introduce tests from the base distribution, and then backport a bugfix to the handling of to_float. ### bin_prot - Switched build to use =config.h= rather than the command-line for preprocessor variables. - Switched from `ARCH_SIXTYFOUR` to `JSC_ARCH_SIXTYFOUR`. - Fixed to support 32-bit integers, which are used in `js_of_ocaml`. Do not make too many assumptions on integer size. Integers are 32bit in Javascript. Do not use the "get_float_offset" hack on 32bit as it cannot be implemented in javascript. ### core_extended - Added to `Interval_map` a more complete set of operations. - Removed `Core_extended.Sexp.filter_record`, which has been superseded by `Core.Std.Sexp.of_sexp_allow_extra_fields`. - Added to `Interval_map` an `Interval` module, with the type of intervals used in an interval map. - In `Color_print`, added `sprintf` functions, and changed formatting to compose properly. ### core_kernel - Added `Float.int63_round_nearest_exn`. val int63_round_nearest_exn : t -> Core_int63. - Changed `Hashtbl.sexp_of_t` so that keys are sorted in increasing order. This also applies to the `sexp_of_t` produced by `Hashtbl.Make` and `Make_binable`. Sorting by key is nice when looking at output, as well as in tests, so that the output is deterministic and so that diffs are minimized when output changes. - Added to `Info`, `Error`, and `Or_error` a `Stable.V2` module, whose `bin_io` is the same as the unstable `bin_io`. - Replaced `Map.prev_key` and `next_key` with `closest_key`. val closest_key : ('k, 'v, 'cmp) t -> [ `Greater_or_equal_to | `Greater_than | `Less_or_equal_to | `Less_than ] -> 'k -> ('k * 'v) option - Shared code between `Monad.Make{,2}` and `Applicative.Make{,2}`. - Added tests to make sure `round_nearest` and `int63_round_nearest_exn` don't allocate. - Added `Lazy.T_unforcing` module, with a custom `sexp_of_t` that doesn't force. This serializer does not support round tripping, i.e. `t_of_sexp`. It is intended to be used in debug code or `<:sexp_of< >>` statements. E.g: type t = { x : int Lazy.T_unforcing.t ; y : string } with sexp_of - Extended `Map.to_sequence` and `Set.to_sequence` to take any combination of upper bound, lower bound, and direction. - Added `Map.split`. - Added `Timing_wheel.fire_past_alarms`, which fires alarms in the current time interval's bucket whose time is in the past. - Added a `Total_map` module, for maps where every value of the key type is present in the map. - Added `Bigstring.compare` and `Bigstring.equal`. - Split `monad.ml` into three files: `monad.ml`, `monad.mli`, and `monad_intf.ml`. - Removed the last remaining dependence of `Core_kernel` on Unix, moving `Time_ns.pause` functions to `Core`. - Added optional arguments to `Hash_queue.create`, `?growth_allowed` and `size`, which then get passed to `Hashtbl.create`. - Added a `?strict:unit` argument to functions that ordinarily create lazy sexps, like `failwiths`. Info.create Error.create Error.failwiths Error.failwithp Or_error.error This makes it easy to force a use to be strict, which is sometimes useful to accurately capture the state of a mutable data structure at the time the error happens, lest it change by the time the error is rendered. - Removed `Interned_string` module. - In `Pooled_hashtbl`, avoid trying to create arrays bigger than `Sys.max_array_length`. The problem affected 32-bit platforms. - Added `Quickcheck` module. Supports automated testing with randomly-generated inputs in the style of Haskell's Quickcheck library. Our adaptation supports flexible probability distributions for values of a given type and uniqueness guarantees for generated values. - Made `Set.to_sequence` and `Set.split` have the same interface as `Map.to_sequence` and `Map.split`, respectively. - Fixed `Float` and `Timing_wheel` to compile on 32-bit platforms. - Added `Lazy.Stable.V1`. - Added `List.reduce_balanced`, which is like `reduce`, but relies on associativity of `f` to make nesting of calls to `f` logarithmic rather than linear in the input list length. - Added `String_id.Make_without_pretty_printer`. - Restricted `Time_ns.Span` values to be less than 135 years, which ensures the corresponding `float` `Time.Span` values have microsecond precision. Fixed a `Time_ns` test that recently started failing due to crossing the 135-year boundary. Reducing the range of `Time_ns.Span` required adjusting the implementation of `Core.Time_ns.Option.Stable.V1`, which (accidentally, incorrectly) incorporated the (unstabilized) `Core_kernel.Time_ns.Span.min_value` as the representation of `bid_none` and `.max_value` as `ask_none`. The prior representation is preserved, but some previously allowed values are no longer allowed and now raise exceptions! - Added `Rope` module, the standard data structure for efficient string manipulation. - Added `Sequence.unfold_with_and_finish`, a variant of `unfold_with` that can continue the sequence after the inner sequence finishes. - Replaced `Sequence.cycle` with `Sequence.cycle_list_exn`, to work around a bug in `Sequence.cycle` raising on the empty sequence. Sequence.cycle can cause an infinite loop if its input is empty. It is problematic to check whether the input sequence is empty. * If we check it eagerly, we have to turn `cycle` into `cycle_eagerly_exn`, and it will evaluate the first element twice. * If we check it lazily, we might raise an exception in a seemingly unrelated part of the code, and the usually-good habit of wrapping a function like `cycle_exn` in `try .. with ..` would not catch it. To get around these issues, [cycle] is changed to accept only lists as inputs, not sequences. It is now called [cycle_list_exn]. - Fixed assumptions about the size of integers, to support compiling to Javascript, where integers are 32-bit. - Fixed build on Mac OSX. Fix build when LINUX_EXT or TIMERFD are undefined. - Added `Caml.Bytes`. Add an alias for Bytes in Caml. Fixes janestreet/core_kernel#46. - In `Container`, exposed polymorphic functions individually building container functions using `fold` or `iter`. Exposed polymorphic functions in `Core_kernel.Container` for individually building each of the `Container` functions using `fold` or `iter`. E.g.: type ('t, 'elt, 'accum) fold = 't -> init:'accum -> f:('accum -> 'elt -> 'accum) -> 'accum type ('t, 'elt) iter = 't -> f:('elt -> unit) -> unit val length : fold:('t, _, int ) fold -> 't -> int val exists : iter:('t, 'a) iter -> 't -> f:('a -> bool) -> bool - Added container.mli, which was sorely missing. - Added `Doubly_linked.to_sequence`. - Added `Hash_queue.sexp_of_t`. ### core_profiler - Changed delta timers and probes so they record the total amount of time/value change between each start and pause. ### custom_printf - Upgraded from OCaml `4.02.1` to `4.02.2` ### email_message - Extended and improved Email_message API. ### fieldslib - Added `Fields.Direct.set_all_mutable_fields`, a function intended to guarantee when pooling records that one cannot forget to reinitialize some fields. Obviously one could achieve this through something like `Fields.Direct.iter`, but we want a more efficient version that doesn't force the call side to create closures. ### jenga - Treat output to stderr by an action as a failure. ### ocaml_plugin - Made `Ocaml_plugin.Plugin_cache.Config.t` stable. ### pa_bench - Add `-pa-bench-drop-with-deadcode` option. ### pa_ounit - Add `-pa-ounit-drop-with-deadcode` option ### pa_structural_sexp - Added support for records where some fields are optionally displayed. ### re2 - Improved `Re2.find_submatches` on big patterns with many submatches unmatched, e.g. =(ABC)|(DEF)|(GHI)|(KLM)|...=. Without the fix: +-------------------------------------------------------+--------------+------------+----------+----------+------------+ | Name | Time/Run | mWd/Run | mjWd/Run | Prom/Run | Percentage | +-------------------------------------------------------+--------------+------------+----------+----------+------------+ | [re2_internal.ml] find_submatches with many Nones:5 | 406.81ns | 30.00w | | | 0.08% | | [re2_internal.ml] find_submatches with many Nones:10 | 2_385.11ns | 207.00w | | | 0.47% | | [re2_internal.ml] find_submatches with many Nones:50 | 12_772.97ns | 2_072.00w | 0.33w | 0.33w | 2.53% | | [re2_internal.ml] find_submatches with many Nones:100 | 43_196.95ns | 7_191.00w | 2.03w | 2.03w | 8.56% | | [re2_internal.ml] find_submatches with many Nones:200 | 504_884.95ns | 29_316.00w | 16.05w | 16.05w | 100.00% | +-------------------------------------------------------+--------------+------------+----------+----------+------------+ With it: +-------------------------------------------------------+--------------+-----------+----------+----------+------------+ | Name | Time/Run | mWd/Run | mjWd/Run | Prom/Run | Percentage | +-------------------------------------------------------+--------------+-----------+----------+----------+------------+ | [re2_internal.ml] find_submatches with many Nones:5 | 408.24ns | 30.00w | | | 0.12% | | [re2_internal.ml] find_submatches with many Nones:10 | 1_607.67ns | 163.00w | | | 0.48% | | [re2_internal.ml] find_submatches with many Nones:50 | 3_223.89ns | 563.00w | | | 0.96% | | [re2_internal.ml] find_submatches with many Nones:100 | 5_288.09ns | 1_063.00w | 0.20w | 0.20w | 1.58% | | [re2_internal.ml] find_submatches with many Nones:200 | 334_107.81ns | 2_063.00w | 0.79w | 0.79w | 100.00% | +-------------------------------------------------------+--------------+-----------+----------+----------+------------+ - Fixed build on FreeBSD. Excise direct mention of g++ from re2 Makefile, preferring the inbuilt CXX macro. This fixes the build on FreeBSD (yes, really). - Added an applicative interface to building/using regular expressions. - Made Re2 depend only on `Core_kernel`, not `Core`. Fixes janestreet/re2#6 ### rpc_parallel - Fixed a file-descriptor leak There was a file descriptor leak when killing workers. Their stdin, stdout, and stderr remain open. We now close them after the worker process exits. ## 112.35.00 ### async - Include some previously-omitted benchmarks ### async_extended - Added `Ltl` module, an implementation of linear temporal logic, which can be used to run online queries on sequences of states. - Added `Interactive.Job`, for printing start/done messages for multiple simultaneous jobs. - Made `Any_error` be `Applicative`. - Added `Command_rpc` support for `Versioned_rpc.Both_convert.Plain`. ### async_extra - Added to `Log` a better mechanism for catching and handling background errors, via `set_on_error` and an `on_error` argument to `create`. - Added `Log.get_output : t -> Output.t list`. - Changed `Monitor.try_with` so that errors after the initial return are written to the global error log, rather than ignored. - Added `Monitor.try_with_or_error` and `try_with_join_or_error`. `try_with_or_error` is intended to someday be renamed as `try_with`. It also omits some of `try_with`'s optional arguments: `run` and `rest`. Different from `try_with`, `try_with_or_error` uses ``~run:`Now``, which we now believe is a more sensible behavior. - Fixed a bug in `Versioned_typed_tcp` that causes spurious and repeated reconnects when user-level code disconnects. - Added `Tcp.Server.create_sock`, to create TCP servers that don't use `Reader` and `Writer`. - Changed `Log.Level.arg` to accept lowercase, uppercase, and capitalized words. - Replaced `Unpack_sequence.unpack*` functions with `unpack_into_pipe` and `unpack_iter`, for reduced allocation. module Unpack_from : sig type t = | Pipe of string Pipe.Reader.t | Reader of Reader.t end val unpack_into_pipe : from : Unpack_from.t -> using : ('a, 'b) Unpack_buffer.t -> 'a Pipe.Reader.t * ('a, 'b) Unpack_result.t Deferred.t val unpack_iter : from : Unpack_from.t -> using : ('a, 'b) Unpack_buffer.t -> f : ('a -> unit) -> ('a, 'b) Unpack_iter_result.t Deferred.t - Added to `Log` support for user-defined rotation schemes. - Added `Log.is_closed`. - Moved `Async_extra.Rpc` to its own library, `Async_kernel_rpc`, and abstracted its transport layer. `Async_kernel_rpc` depends only on `Async_kernel`. This allows `Async_rpc` to be used in javascript or to try transports tuned for different use cases. `Versioned_rpc` was moved to `Async_rpc_kernel` as well. `Async_extra` still provides an `Rpc` module with the Unix-dependent part: - the `Rpc.Transport` module is augmented with `Async_unix.{Reader,Writer}` based transports - the `Rpc.Connection` module is augmented with helpers for TCP based connections - In sexp-formatted `Log` messages, output the sexp on a single line rather than in multi-line "hum"an format. This makes it possible to, among other things, easily grep such logs. - Fixed a (very small) space leak in `Persistent_rpc_client`. The fix was to use `Deferred.choose` and `Deferred.choice` instead of `Deferred.any` and `>>|`. The old implementation added a callback to the `t.close_started` ivar every time the connection transitioned from connected to disconnected. - Added `Persistent_rpc_client.create_generic`, which is like `create`, but generic in the function used to connect. - Fixed a race condition in the `Versioned_typed_tcp` interface that caused a worker to miss a `Connect` message if the box is under high load. `Query_client.create` is called from `Worker_impl.create` in a different async cycle than the following call to `Query_client.listen` (really, `Tail.collect` under the hood) which is made from `Worker_impl.run`. When the load on the box is heavy (many workers starting and connecting at the same time), the OS might take away the CPU from the worker process between the two async cycles. The TCP socket gets connected while the process is still waiting for its turn, and eventually, when it's the worker's turn to grab the CPU, Async scheduler might process the TCP event earlier than `Worker_impl.run`. - Improved `Udp.ready_iter` to avoid intermediate exceptions by using `Syscall_result`. UDP loops use that, so will benefit. Adjust the implementation slightly as well: made the inner loop always exit on `EAGAIN`/`EWOULDBLOCK` to wait until ready, and give other Async jobs a chance to run after `EAGAIN`/`EWOULDBLOCK` in the outer loop. ### async_kernel - Added `Clock.Event.run_at` and `run_after`. - Eliminated a space leak in `Clock.with_timeout`. Previously `Clock.with_timeout span d` created a deferred that waited for `span` even if `d` (and hence `with_timeout`) became determined sooner. Now, `with_timeout` aborts the clock alarm if `d` becomes determined, which saves space in Async's timing wheel. - Added `Clock.Event.fired`, and removed the `fired` value that was returned by `at` and `after`. val fired : t -> [ `Happened | `Aborted ] Deferred.t - Added `Clock.Event.reschedule_{at,after}`. - Fixed the space leak that caused nested `Monitor.try_with` to use linear space rather than constant space. Changed `Monitor.try_with_ignored_exn_handling` so that with ``Eprintf` or `` `Run f ``, the error processing runs in `Monitor.main` rather than in the monitor that called `Monitor.try_with`. This avoids space leaks due to chains of monitors, e.g.: open! Core.Std open! Async.Std let () = Monitor.try_with_ignored_exn_handling := `Run ignore; let num_monitors = 10_000_000 in let num_remaining = ref num_monitors in let rec loop n : unit = if n > 0 then upon (Monitor.try_with (fun () -> loop (n - 1); return ())) (function | Error _ -> assert false | Ok () -> decr num_remaining; if !num_remaining = 0 then shutdown 0) in loop num_monitors; never_returns (Scheduler.go ()); ;; Added a unit test to detect if nested monitors leak space. - Removed `Lazy_deferred.follow`, which is not used in the tree anymore. Removing this function allows `Lazy_deferred.t` to be implemented as: type 'a t = 'a Deferred.t Lazy.t although that's not done yet. - Added hooks to `Async_kernel.Scheduler` for `js_of_ocaml`. This hook aims to be called every time one add a job to the scheduler (enqueue + timing_wheel). - Made `Async_kernel` not depend on thread. - Added `Deferred.Memo`, which wraps functions in `Core.Memo` to correctly handle asynchronous exceptions. - Renamed `Pipe` and `Thread_safe_pipe` functions that clear their input queue so that their name starts with `transfer_in`, to make it clearer that they side effect their input. | old name | new name | | ------------------------- | ------------------------------ | | `write'` | `transfer_in` | | `write_without_pushback'` | `transfer_in_without_pushback` | - Added `Pipe.init_reader`, symmetric to `Pipe.init`. val init : ('a Writer.t -> unit Deferred.t) -> 'a Reader.t val init_reader : ('a Reader.t -> unit Deferred.t) -> 'a Writer.t - Changed `Async_kernel.Job_queue.run_jobs` to call `Exn.backtrace` immediately after catching an unhandled exception There should be no change in behavior. This change was to make it more clear that there is no intervening code that interferes with the global backtrace state. - Made `Deferred` functions that take an argument `?how:[ `Parallel | `Sequential ]` accept `` `Max_concurrent_jobs of int``, which operates in a sequence in parallel, limited via a throttle. - Made `Deferred.Or_error` match `Applicative.S`. - Fixed `Scheduler.run_cycles_until_no_jobs_remain` so that it continues running if one has done `Scheduler.yield`. - Split the implementation of `Deferred` into a number of files, to solve some problems with circularities. Split into: - deferred.ml - deferred_sequence.ml - deferred_list.ml - deferred_array.ml - deferred_queue.ml - deferred_map.ml - deferred_result.ml - deferred_option.ml For a sequence of multiple modules used to construct a module, switched from the `Raw_*` prefix convention to the numeric suffix convention. E.g. we now have `Deferred0`, `Deferred1`, `Deferred`. ### async_parallel - Renamed `Async_parallel` as `Async_parallel_deprecated`; one should use `Rpc_parallel` instead. ### async_rpc_kernel - Moved `Async_extra.Rpc` to its own library, `Async_kernel_rpc`, and abstracted its transport layer. `Async_kernel_rpc` depends only on `Async_kernel`. This allows `Async_rpc` to be used in javascript or to try transports tuned for different use cases. `Versioned_rpc` was moved to `Async_rpc_kernel` as well. - Added `Rpc.One_way` module, for RPCs that do not expect any response or acknowledgment. - Sped up `Rpc.Connection`. We have been able to send 6_000_000 (contentless) one-way messages per second under some idealized circumstances. - In `Rpc`, added an optional `heartbeat_config` argument to configure the heartbeat interval and timeout. - In `Rpc`, added some information to `Connection_closed` exceptions. - In `Rpc.Implementations.create`'s `on_unknown_rpc` argument, added a `connection_state` argument to the `` `Call`` variant. When using `Rpc.Connection.serve`, one can put client addresses in the `connection_state`, so this makes it possible to identify who sent the unknown RPC instead of just saying what the RPC's name and version are. - Added `Rpc.One_way.Expert`, which has an `implement` function that gives direct access to the internal buffer instead of using a bin-prot reader. - Made `Rpc` not raise an exception if a connection is closed due to heartbeating. - Reworked `Async_rpc_kernel`'s `Transport` implementation to not require a `transfer` function similar to `Async_unix.Writer.transfer`. Changed the RPC connection to flush the pipe when the writer is closed, which was the only behavior of `Async_unix.Writer.transfer` that had been relied on. With this change, if someone closes the underlying writer by hand the pipes won't be flushed, which should be expected anyway. - Fixed an issue where "normal" `Pipe_rpc` errors caused the connection to shutdown. Such errors include querying an unknown RPC or getting an exception raised by the RPC implementation shutdown. Now such errors behave like `Rpc` errors, i.e. they are completely ignored besides being put in the return value of `Pipe_rpc.dispatch`. Errors that occur later in `Pipe_rpc` still cause the connection to close, since these should only occur if there is a bug somewhere. - In `Rpc`, connection-closing errors no longer raise top-level exceptions. One can now call `close_reason : t -> Info.t Deferred.t` to get the reason the connection was closed. - Added `One_way` rpcs to `Versioned_rpc`. - Fixed `Rpc` to not write if the transport is closed. The fix doesn't check that the writer is closed, but instead ensure that we don't try to write when we shouldn't. This means that it will still fail if the user close the transport by hand, which they shouldn't do. This change requires transport implementations to fail after they have been closed. This is different from the semantics of `Async_unix.Writer`, but the latter is non-intuitive and makes it hard to check the correctness of the code. Moreover it is only required for `Async_unix.Writer.with_flushed_at_close` which we don't use. ### async_ssl - Fix github issue #4 (some comments swapped). ### async_unix - Made `Unix.File_kind.t` be `Comparable`, so it can be used in `<:test_result< >>`. - Reduced allocation in Async's scheduler in the common path. The allocation was in `Raw_scheduler.be_the_scheduler.compute_timeout`, which was (statistically, based on perf) the largest single allocator in one of our applications. Now, it way down the list. Note that the application is not a typical Async app in that it does not sit in epoll very much, due to the way we do low-latency I/O. This change will benefit everyone, but only a tiny bit. - Added `Writer.write_bin_prot_no_size_header`. This is needed for Async RPC as it writes a different size on its own. - Fixed a bug in `Writer.transfer`, which didn't close the pipe when the consumer leaves. Simplified the implementation of `Writer.transfer`: - replaced the big loop by a simple iteration function on the pipe that just stop without filling its ivar when it sees the iteration should stop for other reason that `` `Eof `` on the pipe: writer closed, consumer left or stop requested by the user. - replaced the various `choose` by a single one and deal with the closing reason only at this point. - Added `Writer.close_started`, symmetric to Writer.close_finished. ### bignum - Upgraded from Zarith 1.2 to 1.3. - Removed dependence on `Big_int`. ### bin_prot - Sped up `bin_io` of `float array`. `Bin_prot` already had special fast handling for `float array`'s but `with bin_io` did not use it except for the special type `float_array`. Now, there is fast handling for `float array` and its aliases, for example `price array` when `type price = float`. - Changed `Size.bin_size_array`, `Write.bin_write_array` and `Read.bin_read_array` short circuit to the fast path when it detects that `float array` is handled. Each of these functions receives a function for handling array elements and short circuits when the function for handling elements is equal to the function for handling floats, using physical equality of closures. - To cause short circuiting for aliases of `float`, changed `bin_io` so that aliased `bin_io` functions are equal the the `bin_io` functions of the original type. That is an optimization for itself regardless whether it's used for `float`. Before this change, every function generated for aliases were eta-expanded leading to different closures at runtime for each type. Short circuiting needs to apply to the handling function rather than to the value at hand because: * the value is available only in `size` and `write`, and we need a way to make `read` work as well. * even when the value is a float at runtime, the handling of a specific float alias may have been overridden by a custom one. Made a slight improvement to `bin_read_float_array`: since the array is going to be filled with read values, there is no need to fill it with `0.` after allocation: let next = pos + size in check_next buf next; -| let arr = Array.create len 0. in +| let arr = Array.make_float len in unsafe_blit_buf_float_array buf arr ~src_pos:pos ~dst_pos:0 ~len; pos_ref := next; The difference in speed when optimal and non optimal way of handling floats is used: | Name | Time/Run | mWd/Run | mjWd/Run | |--------------------------------------------|-------------|-----------|----------| | [bench.ml:float array] size non optimal | 3_403.80ns | 2_000.00w | | | [bench.ml:float array] size float_array | 5.55ns | | | | [bench.ml:float array] size Price.t array | 6.18ns | | | | [bench.ml:float array] write non optimal | 7_839.89ns | 2_000.00w | | | [bench.ml:float array] write float_array | 292.42ns | | | | [bench.ml:float array] write Price.t array | 293.16ns | | | | [bench.ml:float array] read non optimal | 9_665.06ns | 2_002.00w | 1.00kw | | [bench.ml:float array] read float_array | 461.01ns | 2.00w | 1.00kw | | [bench.ml:float array] read Price.t array | 449.43ns | 2.00w | 1.00kw | There is no observed speed penalty for runtime check for short circuiting. The following benchmark shows the speed of handling `int array` without and with the check: | Name | Time/Run | mWd/Run | mjWd/Run | |--------------------------------------------|-------------|-----------|----------| | [bench.ml:float array] int array size | 3_910.64ns | | | | [bench.ml:float array] int array write | 6_548.40ns | | | | [bench.ml:float array] int array read | 14_928.11ns | 2.00w | 1.00kw | | Name | Time/Run | mWd/Run | mjWd/Run | |--------------------------------------------|-------------|-----------|----------| | [bench.ml:float array] int array size | 3_906.86ns | | | | [bench.ml:float array] int array write | 5_874.63ns | | | | [bench.ml:float array] int array read | 14_225.06ns | 2.00w | 1.00kw | ### core - Tweaked `Unix.stat`'s C code to reduce float rounding error, by using double-precision rather than single-precision floats. Following Xavier's comment: http://caml.inria.fr/mantis/view.php?id=6285 - Added `Date.O` module. - In `Interval`, exposed `compare` in stable types by having the appropriate modules match the `Stable` signature. - Added `Iobuf.Fill.decimal` and `Poke.decimal`, for efficiently writing integers in decimal format. - Removed `Filename.O` (and `/^`), since it's not used that much, and is inconsistent with the older operator for this that's exposed in `Core.Std`: `^/`. - Improved `Command` autocompletion to work even if some arguments can't be parsed. This is useful because the completion mode does not get fed precisely the same arguments that it would get if you hit RETURN. As a simple example, if completion is set up for `my-exe` which takes a sexp as its first argument, then: my-exe '(a b)' will run `my-exe` in completion mode with: '(a b)' as its first argument, rather than: (a b) as would be passed if RETURN had been pressed instead. Since `Sexp.of_string "'(a b)'"` fails, in this example tab completion won't work. Changed the internals a bit to make this possible. Most notably, the `Parser` module is now applicative rather than monadic, which required only a few simple changes to support. - Made `Time_ns.to_time` and `Time_ns.Span.to_span` round to the nearest microsecond in all cases. Previously, `Time_ns.Span.to_span` sometimes rounded incorrectly for negative spans. - Added `Time.Zone.prev_clock_shift`, the analog of `next_clock_shift`. val prev_clock_shift : t -> before:Time_internal.T.t -> (Time_internal.T.t * Span.t) option Implemented `next_clock_shift` and `prev_clock_shift` using `Array.binary_search_segmented`. - Added `Lock_file.get_pid : string -> Pid.t option`. - Added `val random: unit -> int` to `Time_ns` and `Time_ns.Span`. - Renamed `Iobuf.sub` as `Iobuf.sub_shared`. This more closely matches `Bigstring`, and clarifies the semantics of `Iobuf.sub` vs `sub` in `Blit_intf.S`. - Added `Iobuf` blit modules: `Blit`, `Blit_consume`, `Blit_fill`, `Blit_consume_and_fill`. - Added `Piecewise_linear.first_knot` and `last_knot`. val first_knot : t -> (key * value) option val last_knot : t -> (key * value) option - Made `Unix.Cidr` match `Comparable.S_binable`, and added `Cidr.create` and `Cidr.netmask_of_bits`. - Moved `Unix.tm` and `Unix.strftime` from `Core_kernel` to `Core`. - Made `Crc.crc32` return `Int63.t` rather than `int64`, and added `Crc.bigstring_crc32` and `Iobuf.crc32`. Cleaned up old cruft in the C stubs for CRC checking. - Added `Iobuf.narrow_lo` and `narrow_hi`, which comprise `Iobuf.narrow`. - Changed `Linux_ext.Timerfd`, `Epoll.wait`, and `Unix.select` to use (`int`) `Time_ns` rather than (`float`) `Time`. This avoids spurious float conversions and rounding problems. Made all timeouts consistently treat negative timeouts as "timeout immediately". This fixes an incorrect behavior of `Linux_ext.Timerfd.set_after` and `set`, which had been rounding to the nearest microsecond, which was particularly bad for time spans smaller than 500ns, which would be rounded to zero, and then would cause the timerfd to never fire. Now, the small span is directly fed to `timerfd_settime`. We also changed a span of zero to be treated as `1ns`, to avoid the behavior of `timerfd_settime 0`, which causes the timerfd to be cleared and never fire. - Made `Or_error` match `Applicative.S`. - Added `Command.Param`, with the intention of one day replacing `Command.Spec` and providing an applicative interface for command-line parsing. This change required lots of rearrangement of `command.mli` so that `Command.Param` and `Command.Spec` could share large portions of their interface. As a side effect, the interface is more sweeksy than before. - Added `Command.shape`, for exposing the shape of a command, including what subcommands its has. - Changed `Syscall_result.to_result` to return a preallocated object, and changed many uses to take advantage of this property. Pattern matching on results is much clearer than if-analysis and avoids double-checking errors in many cases. `Syscall_result.to_result` can only return preallocated results for a few `Ok` values from `Syscall_result.Int`, of course, and likewise for other large `ok_value` types. We have initially, and arbitrarily, limited preallocation to 64 `errno`'s and 2048 `ok_value`'s. - Added `Sys.big_endian : bool`, from `Caml.Sys`. - Disabled unit tests in `Time_ns` that started failing around 10:40pm NYC 2015-05-15. The tests indicate an off-by-one-microsecond error in round tripping between `Time.Span.t` and `Time_ns.Span.t`. ### core_bench - Exposed the equality of `Core_bench.Std.Bench.Test.t` with `Core_bench.Test.t`, so that one can get the name of a test. This is useful for filtering based on test name. ### core_extended - Removed the `Stats_reporting` module. - Renamed `Quickcheck` module to `Quickcheck_deprecated`. It's replaced by Janecheck, which for now is a separate library in the core_extended package, but will soon be merged into core. - Moved the `Selector` module to its own library. This is for internal reasons related for code review; it is included as a library within the core_extended package for now, but may move to another home in the future. - Added `Extended_unix.terminal_width : int Lazy.t`. - Added `Interval_map` module. - Added to `Sendmail.send` additional optional arguments: `?message_id:string`, `?in_reply_to:string`. ### core_kernel - Added an Applicative interface to Core (a.k.a. idioms or applicative functors) - Generalized the signature of `Hashtbl.merge_into` to allow the types of `src` and `dst` to be different. - Made `Day_of_week.of_string` accept additional formats (integers 0-6, full day names). - Added `Day_of_week.to_string_long`, which produces the full day name. - Changed `Hashtbl.add_exn` to not create a new exception constructor when it raises due to a duplicate key. - Added `Map.nth`, which returns the nth element of a map, ordered by key rank. - Added `Binable.Of_binable` functors, similar to `Sexpable.Of_sexpable` One should use `Binable.Of_binable` rather than the functionally equivalent `Bin_prot.Utils.Make_binable`. - Added `Either` module, with `type ('a, 'b) t = First of 'a | Second of 'b`. - Added to `Univ_map` a functor that creates a new `Univ_map` type in which the type of data is a function of the key's type, with the type function specified by the functor's argument. Normally, a `Univ_map.t` stores `('a Key.t * 'a)` pairs. This feature lets it store `('a Key.t * 'a Data.t)` pairs for a given `('a Data.t)`. - Made `Day_of_week.Stable` be `Comparable` and `Hashable`. - Fixed a couple `Exn` unit tests that mistakenly relied on the global setting of `Printexc.get_backtrace`. Now the tests locally set it to what they need. This avoids unit-test failures when running with no `OCAMLRUNPARAM` set: File "exn.ml", line 130, characters 2-258: clear_backtrace threw "Assert_failure exn.ml:133:4". in TEST_MODULE at file "exn.ml", line 127, characters 0-1057 - Renamed `Monad.ignore` as `Monad.ignore_m`, while preserving `ignore = ignore_m` in existing modules (e.g. `Deferred`) that used it. We can later consider those modules on a case-by-case basis to see whether we want to remove `ignore`. - Added `Set.symmetric_diff`. - Added `Timing_wheel.reschedule`, which reschedules an existing alarm. - Added `Applicative.S2`, analogous to `Monad.S2`. - Added combinators to `Either`. - Added `Hashtbl.add_or_error` and `create_with_key_or_error`, which use `Or_error` and are more idiomatic ways of signalling duplicates. - Added `Sexpable.Of_sexpable1` functor, for one-parameter type constructors. - Made `Timing_wheel_ns` keys be `Int63.t` rather than `int`, so that behavior is consistent on 32-bit and 64-bit machines. Also, made `Timing_wheel.Interval_num` an abstract type. - Hid the `bytes` type in `Core.Std`, so that type errors refer to `string` rather than `bytes`. Added `Bytes` module so that people can say `Bytes.t` if they need to. Now we get reasonable error messages: String.length 13 --> Error: This expression has type int but an expression was expected of type string "" + 13 --> Error: This expression has type string but an expression was expected of type int - Modernized the coding style in `Timing_wheel`. - Replaced `Unpack_buffer.unpack` with `unpack_into` and `unpack_iter`, to avoid allocation. `Unpack_buffer.unpack` created a (vector-backed) `Core.Std.Queue` for each call. When unpacking a buffer containing many values, resizing of the buffer can be costly and in some cases leads to promotions of short-lived data to the major heap. The new functions avoid allocating the queue: val unpack_into : ('value, _) t -> 'value Queue.t -> unit Or_error.t val unpack_iter : ('value, _) t -> f:('value -> unit) -> unit Or_error.t - Cleaned up the implementation of `Gc.tune`. - Change `Unit` implementation to use `Identifiable.Make` instead of applying functors separately. - Added `val random: unit -> int` to `Int63`. - Reworked `Float.iround_*_exn` functions to not allocate in the common case. - Added `Fqueue.singleton` and `Fdeque.singleton`. - Moved `Unix.tm` and `Unix.strftime` from `Core_kernel` to `Core`. Added external time formatting: float (* seconds *)-> string (* format *) -> string = "..." - Made `String_id.Make` call `Pretty_printer.Register`. - Changed `String_id` to allow the pipe character in identifiers. - Made `List.compare` have the usual type from `with compare`, `val compare : ('a -> 'a -> int) -> 'a t -> 'a t -> int`. Previously, `List.compare`'s type was: val compare : 'a t -> 'a t -> cmp:('a -> 'a -> int) -> int - Made stable `Map`'s and `Set`'s conform to the `Stable1` interface. - Reworked `Hashtbl.find_exn` to not allocate. Previously, `Hashtbl.find_exn` allocated because it called `Hashtbl.find`, which allocates an option (partially because `Avltree` allocates options in its `find` function). ### jenga - Make `.jenga.db` be format version aware, and store `.jenga` files in a subdirectory. - Switch md5 computation to use a C binding, fixes #10. - Move `jem.exe` to `jenga.exe monitor`, `jenga_offline.exe` to `jenga.exe offline`, `jenga.exe -cat-api` to `jenga.exe cat-api`. - Remove `Path.dotdot`. - Adjust behaviour of `Path.Repo.is_descendant` to be more consistent. - Repo-root invariance: remove `Path.Abs.the_root`, `Path.Rel.to_absolute_string`, `Path.Rel.create_from_absolute`, and make `Path.relative Path.the_root ".."` fail. - Add a flag `-sandbox-action` to `jenga build` to run the action in an environment that (attempts to) detect missing dependencies or overly large sets of targets. - Don't interpret paths of shape `./foo.ext` as aliases. - Always interpret `Dep.path path` as the file at `path` rather than the default alias for the directory at `path` - Make jenga look for the variable JENGA_OPTIONS, and use it to add a debug setting. - Fix handling of directories created during the build. - Basic support for symlink resolution. - Add an option to the monitor command to only display a single snapshot of the progress, rather than a continually updated pipe. - Fix a deadlock where File_access throttle and directory lock are obtained in the wrong order. ### ocaml_plugin - In `copy_source_files_to_working_dir`, exclude files that start with a dot. emacs creates temporary files that cannot be read with names like `.#foo.ml`, and attempting to copy those causes this function to fail. ### pa_ounit - Made `Pa_ounit_runtime` not depend on `OUnit` anymore. The dependency hasn't been needed in more than 2 years. ### re2 - Fixed a bug in `Re2.find_all_exn`, extant since 2014-01-23, in which it returns spurious extra matches. Using pattern `b` and input `aaaaaaaaaaaab` is expected to return a single match at the end of the input but instead returned the match multiple times, approximately as many times as `input length / min(match length, 1)`. Added tests for this function and also `get_matches` which uses the same code. - Updated to new version of upstream library. ### sexplib - Inline some calls that js_of_ocaml was unable to recognise as tail-recursive (cf. issue #14) ## 112.24.00 ### async Keep up to date with interface changes in `Async_kernel`, `Async_extra` and `Async_unix`. ### async_extended - Fixed misspelling in `Command_rpc.Connection`, renaming `propogate_stderr` as `propagate_stderr`. ### async_extra - Changed `Log` to not eagerly run the rotation loop when an `Output.Rotating_file` is created. - Changed `Log.Output.combine` to write log outputs in sequence rather than parallel, to give the correct semantics when the same output is included multiple times in `Log.create`. This fixed a test that was failing in `lib_test/log_test.ml`. - Remove `Log.Rotation.t_of_sexp`. - Made `Command.async*` functions flush `stdout` and `stderr` before calling shutdown, to avoid timeouts causing data to be dropped. For now, we're making this change in `Command` rather than `Writer`. `Writer` already has `at_shutdown` handlers. We've observed that they don't behave well for command-line programs w.r.t. stderr. So, the thinking of this feature is to try out a different `at_shutdown` behavior, just for `Command` executables and just for `stdout` and `stderr`. If it works out, maybe we move it into `Writer` proper. Putting the change in `Command` for now reduces the scope of what is affected by the experiment, and hopefully correlates well with where the change is likely to help. - In `Rpc`, catch exceptions raised by blocking-RPC implementations. - Added functionality to `Versioned_typed_tcp.Repeater`. Added to `create` an argument `is_client_allowed : Client_name.t -> bool`. Added to `start` an argument: on_connecting_error : (client_name : Client_name.t -> server_name : Server_name.t -> Error.t -> unit) - Fixed a race in `Versioned_typed_tcp` in which a message can be dropped between `Server.create` and `Server.listen` - Simplified the implementation of `Rpc`. Rpc has an internal Response_handler module, which is just a record containing a response-handling function and an `already_removed` boolean field. It turns out that this is unnecessary: `already_removed` is set to true when the function returns ``remove`, but if it returns ``remove` then it will also be removed from a hash table, and we only call the function immediately after looking it up in that hash table. This wasn't always pointless: this function used to return deferred values and run inside a throttle. The simplification is only possible because we made it synchronous a while ago. - Added `Tcp.Server.num_connections` function. - Added creation functions for `Versioned_rpc` menus, for use in clients of an RPC proxy. In `Menu`: val create : Implementation.Description.t list -> t In `Connection_with_menu`: val create_directly : Connection.t -> Menu.t -> t These are for use in clients of an RPC proxy, which can't use the regular menu mechanism since they each need to have many menus (one for each potential target) but should only need to have one connection (to the proxy). - Added to `Rpc` expert submodules, `Implementations.Expert` and `Rpc.Expert`, with low-level access for implementing a proxy that can handle queries without knowing their names, types, etc. in advance. - Renamed `Rpc.Implementation.Description` as `Rpc.Description`. - Added `Rpc.{Rpc,Pipe_rpc,State_rpc}.description` accessor functions. - Added `Rpc.Implementation.descriptions`, which returns all RPCs in an `Rpc.Implementations.t`. This was needed for the `rpc_discovery` library: given an `Implementations.t` we want to advertise at prefix///host_and_port = - Added combinators to `Rpc.Implementations`: `lift`, `add`, `add_exn`. ### async_kernel - Now depends on `Core_kernel` instead of `Core`. `Async_kernel.Clock` uses `Core_kernel.Time_ns` and `Core_kernel.Timing_wheel_ns` rather than `Core.Time` and `Core.Timing_wheel_float`. - Added `Async_kernel.Types` module to deal with the mutual recrsion of `Async_kernel`'s types. This should help eliminate the complexity and make it easier to make changes without running into as many constraints due to module/type ordering. Merged `Types.Jobs` into `Types.Scheduler`. - Improved the performance of `Deferred.bind`, eliminating an allocation in `Ivar.connect`. - Optimized `Deferred.bind`, removing a closure allocation by inlining `Deferred.create`. - Added `Pipe.interleave_pipe`, which is like `interleave`, but takes a pipe rather than a list. ### async_ssl - By default OpenSSL ignores the result of certificate validation, so we need to tell it not to. - Expose session details such as checked certificates and negotiated version. Add session resumption. ### async_unix - Made `Process.env` type equal `Core.Std.Unix.env` type, effectively adding the ``Replace_raw` variant. - Renamed `Process.wait` as `collect_output_and_wait`, and added a `wait` function that is a thin wrapper around `waitpid`. Also renamed: wait_stdout --> collect_stdout_and_wait wait_stdout_lines --> collect_stdout_lines_and_wait - Added `Unix.getgrouplist`, a wrapper around the eponymous function in core - Change the Async scheduler to run external actions immediately upon dequeueing them, rather than first enqueueing them in the normal job queue. Also, made external actions be jobs rather than closures. - Changed `Unix.Inet_addr.of_string_or_gethostbyname` to not use a sequencer. We had used a sequencer to workaround bugs in winbind, which we don't use anymore. Reported on github: https://github.com/janestreet/async_unix/issues/4 ### bignum - Fixed exception raised by `Bignum.sexp_of_t` when the denominator is zero. ### bin_prot Minor commit: comments. ### core - Renamed `Dequeue` as `Deque`. - Added `Fdeque`, a functional deque (a double-ended `Fqueue`, or a functional `Deque`). - Changed `Fqueue`'s bin-io format, and added a stable type. Deprecated deque-like functions in favor of `Fdeque`. - Added `Fheap`, a functional heap implementation based on pairing heaps. - Reverted the change to `Or_error`'s bin-io format made in 112.17, going back to the format in 112.16 and before. - Added to `Unix.env` type a ``Replace_raw` variant, as used in `exec` and `fork_exec`. - Added `Array.Permissioned` module, which has a permissioned array type and permissioned versions of all the regular array functions. - Added `Date.week_number : t -> int`. - Added values to `Day_of_week`: `of_int_exn`, `iso_8601_weekday_number`, `weekdays`. val of_int_exn : int -> t val iso_8601_weekday_number : t -> int val weekdays : t list (- [ Mon; Tue; Wed; Thu; Fri ] *) - Switched `Float` IEEE functions to use `Int63.t` for the mantissa rather than `int`, so they work on 32-bit platforms. - Added a `length` field to the `Map.t` record, making `Map.length` `O(1)` rather than `O(n)`. - Moved a fragment of `Time_ns` from `Core` to `Core_kernel`, enough so that `Async_kernel` can use `Core_kernel.Time_ns` and ultimately only depend on `Core_kernel`. - Fixed compilation of `Time_ns` 32-bit Linux. - Added `Bounded_int_table.clear`. - Fixed the `module_name` passed to `Identifiable.Make` for a number of modules. The module name must be an absolute module path. Reported here: https://github.com/janestreet/core/issues/52 - Added `Tuple.Binable` functor, for making binable tuples. - Sped up a `Time_stamp_counter` unit test. `Time_stamp_counter` unit test has an 18s unit test, which seems excessive. Take a couple of orders of magnitude off the number of iterations. - Added `Time_ns.pause`, whose implementation is the same as `Time.pause`. This involved moving the `nanosleep` C code from `Core` to `Core_kernel`. This was necessary so that `Async_kernel` can pause without introducing a dependence of Async on Core. - Made `Core_kernel.Time_ns.Alternate_sexp` use a similar format to `Core.Time_ns`. This was needed so that `Async_kernel` can use a nice sexp format for time spans. - Changed `Timing_wheel` implementation to use `Time_ns`, and moved to `Core_kernel.Timing_wheel_ns`; made `Core.Timing_wheel` a wrapper around `Timing_wheel_ns`. Generalized the timing-wheel interface to be parametric in `Time`, so that one interface applies to both `Timing_wheel` and `Timing_wheel_ns`. Generalized the timing-wheel unit tests to a functor, `Timing_wheel_unit_tests.Make`, that is used to test both `Timing_wheel_ns` and `Timing_wheel_float`. Moved a few tests that depend on `Time` and `Date` from the functor into `timing_wheel_float_unit_tests.ml`. Split out `Timing_wheel.Debug` into a separate functor, `Timing_wheel_debug.Make`. This was done in so that `Async_kernel` can depend only on `Core_kernel` and not `Core`. - Added optional arguments to `Command.group`: `?body` and `?preserve_subcommand_order`. `preserve_subcommand_order : unit` causes subcommands to be in the order they are specified, rather than sorted. `body : (path:string list -> unit)` is called when no additional arguments are passed. - Added accessor function `Command.summary : t -> string`. - Fixed a bug in `Time.Span` robust comparison. - Changed `Command`'s tab-completion bash code so that it is possible for programs to return completions containing spaces. Actually knowing when and how to do so is difficult, because of course there's escaping to worry about. Adding helper functions to make that sort of thing manageable is left for future work. - In `Command`, made the `-version` and `-build-info` flags work at the top level when there are subcommands. - Added `Sequence.interleaved_cartesian_product`, which implements cartesian product of potentially infinite sequences. ### core_extended - Added to `Shell.set_defaults` a `?preserve_euid:bool` argument, which causes `Shell` to use `bash -p`. - Removed `Array.Access_control`, now that there is `Core.Std.Array.Permissioned`. - Removed `Fast_int_div`. ### core_kernel - Added `Time_ns` module. A fragment of `Core.Std.Time_ns` is now in `Core_kernel.Std.Time_ns` such that `Async_kernel` can use `Time_ns` and only depend on `Core_kernel`. - Renamed `Dequeue` as `Deque`. `Dequeue` remains for backward compatibility, but should not be used anymore. Use `Deque` instead. - Added `Fdeque` module, a functional version `Deque`. Deprecate deque-like functions in `Fqueue`. ### custom_printf - Added syntax to use `Sexp.to_string_mach` rather than `Sexp.to_string_hum`. Added syntax `%{sexp#mach:}`, which is like `%{sexp:}`, except it calls `Sexplib.Sexp.to_string_mach` instead of `Sexplib.Sexp.to_string_hum`. In fact, you can put any identifier after the `sexp#` and it will call `Sexplib.Sexp.to_string_`; however, there are no other such functions right now. ### jenga - Interns strings, significantly reducing memory use. ### ocaml_plugin Minor update: follow Async evolution. ### pa_ounit - Added `-verbose` switch to time unit tests, so we can easily see which ones are taking a long time. ### pa_test - Reduce code generated by `pa_test` in favor of more code in `pa_test_lib`. + Less generated code means less time spent compiling it. + Reducing code in `pa_test.ml` is good, because: a) Staging semantics make this code hard to understand. b) This code is written using the less familiar revised OCaml syntax. b) We get less code to migrate to syntax extensions. - Re-used location code in `pa_here`; side benefit that we get full path names in (Loc...) instead of just the file's basename. ### patdiff Minor update: doc. ### patience_diff Update references to `Core.Std.Dequeue` to refer to `Core.Std.Deque` ### rpc_parallel - Added `Parallel.State.get` function, to check whether `Rpc_parallel` has been initialized correctly. - Added `Map_reduce` module, which is an easy-to-use parallel map/reduce library. It can be used to map/fold over a list while utilizing multiple cores on multiple machines. Also added support for workers to keep their own inner state. - Fixed bug in which zombie process was created per spawned worker. Also fixed shutdown on remote workers - Made it possible for workers to spawn other workers, i.e. act as masters. - Made the connection timeout configurable and bumped the default to 10s. ### sexplib Minor update: documentation. ### typerep - Remove unused "bin_proj" rewriter. ## 112.19.00 ### core_profiler Initial release ## 112.17.00 ### async Added tests and updated examples ### async_extended - Added `Interactive` module for terminal interaction with users by command-line executables. `Interactive` was previously in `Iron_common`. - In `Process`, added an `?env` argument to some functions. - Allowed `Command_rpc` implementations to access the `Rpc.Connection.t` used to invoke them. There is an option to invoke `Command_rpc` implementations via sexp communication instead of `Rpc`, so implementations are given a value of a variant type `Sexp | Bin_io of Rpc.Connection.t`. - Added `Resource` module, which abstracts the idea of acquiring and releasing a handle to a resource. ### async_extra - Modernized code style in `Async_extra`. This was mostly whitespace changes, plus deletions of unneeded module paths. - Added `with sexp_of` to `Tcp.Where_to_listen` for debugging. - In `Versioned_typed_tcp`, check that the writer on the other side is not closed in the `Pass_on` case. - Added a new way to implement an RPC, where the implementation doesn't return a deferred. This "blocking" rpc implementation guarantees that the rpc will in fact be fully dispatched by the time the implementation returns. This can be used to skip the deserialization of the query, and instead operate directly in the message contents as received. Also, fixed a bug in which the query handler (and therefore the connection state) was being called before the internal async rpc handshake was finished. - Added an optional `job_tag` argument to `Sequencer_table.enqueue`, to display for debugging. - Added an optional argument to TCP-connection functions to control the local interface used to connect. To implement this this, extended `Tcp.connect` to work on a bound socket. - Added `with compare` to `Process.Output.t`. - Added `Process.Output.Stable` module. - Exposed concrete rpc in `Versioned_rpc.Both_convert`. - Changed `Cpu_usage` to take its first sample after waiting, rather than immediately. This fixes a problem where the first sample could be `NAN` or `Inf`. - Made `Log` buffer-age be unlimited, to avoid exceptions when log writes are blocked for long periods. - Improved `Log.t_of_sexp`'s error message. - Changed `Rpc.Connection.client` and `with_client` to raise some errors which had been dropped during RPC dispatch. Previously, errors dispatching `Rpc.Rpc.t`'s were handled correctly and returned or raised by the relevant dispatch functions. However, errors that occurred in the middle of handling a `Rpc.Pipe_rpc.t` or `Rpc.State_rpc.t` were swallowed. This is because they happen after the dispatch functions have returned, and the dispatch interface doesn't allow for errors to occur in the middle of the pipe -- they must be raised to the monitor in effect when the `Rpc.Connection.t` is created. Errors could be raised to the effective monitor at the dispatch call, but the failure causes the entire connection to go into error, so the connection's monitor seems more appropriate. These errors weren't propagated to the caller because `client` and `with_client` both used `Monitor.try_with` without `rest` handling, causing /any/ errors caused while handling the connection (after the `Connection.t` has been returned to the user) to be dropped. - In `Rpc`, exposed some optional parameters from the `Tcp` module: `?max_pending_connections` and `?buffer_age_limit`. ### async_kernel - Fixed a space leak in `Clock.Event.abort`, making it free the job that was created and stored in the timing wheel. - Moved `Scheduler.yield` from `Async_unix`. - Fixed a bug in `Scheduler.yield_every`, so that it doesn't initialize the scheduler until the staged function is called. - Added `concat_map` function to `Monad_sequence` interface. - Added `Shutdown.shutdown_on_unhandled_exn`. - Added some functions to `Deferred.Or_error` to parallel `Core.Or_error`: `errorf`, `tag`, `tag_arg`. ```ocaml val errorf : ('a, unit, string, _ t) format4 -> 'a val tag : 'a t -> string -> 'a t val tag_arg : 'a t -> string -> 'b -> ('b -> Sexp.t) -> 'a t ``` - Added `Gc.Alarm`, an Async-friendly wrapper around `Core.Gc.Expert.Alarm`. - Removed `Gc.Expert`, whose functions are superseded by Async-friendly functions in `Gc` proper. - Added `Pipe.read_now_at_most`. - Changed `Pipe.merge` to check whether its output is closed, and if so, stop rather than write to it (which raised). Also, made `Pipe.merge` close its inputs whenever its output is closed. - Changed `Clock.at` to return `Deferred.unit` if it is supplied a time in the past. Previously, it would create an empty ivar and a job to fill it that would run in the next cycle. - Changed `Clock.Event.status` to return ````Will_happen_at of Time.t``` rather than ````Waiting```, if applicable. - Added `Ivar.create_full`. - Moved the use of `Linux_ext` to `Async_unix`. This is one of the necessary steps in making `Async_kernel` depend on `Core_kernel` rather than `Core`. ### async_parallel - Modernize the code ### async_smtp Moved from janestreet-alpha ### async_ssl - moved ffi_bindings and ffi_stubgen in separate libraries ### async_unix - Moved `Scheduler.yield` to `Async_kernel`. - Added `Reader.load_annotated_sexp*` functions. These are like the existing `Reader.load_sexp*` functions, except they return annotated sexps rather than sexps. Having annotated sexps is useful so that one can report error positions to the user when processing values built by the `t_of_sexp_` functions /after/ they return. I.e. when there aren't syntax errors in the sexps, but rather semantic errors detected later. - Removed noise and redundancy from `Reader.load_sexp` error messages. - Added `Writer.save_sexps`, analogous to `Reader.load_sexps`. - Made `Writer` errors raised by the background flush job include the entire `Writer.t`, rather than just the `Fd.t`. - Added to `Writer.transfer` an optional argument to limit the number of values read at once from the pipe. The old behavior is to have no limit and remains the default. - Added to `Writer` some missing checks for functions that should ensure the input writer isn't closed. - Changed `Scheduler.run_cycles_until_no_jobs_remain` to pause so that alarms scheduled to fire in the past actually fire. This is necessary because of the timing-wheel, which doesn't guarantee to fire an event until alarm-precision after it was scheduled. Without this change, some tests unexpectedly fail, due to jobs not running that should have. ### bignum - Added `Bigint.random` function, which produces a uniformly distributed value. ### bin_prot - Added `Bin_prot.Blob`, formerly known as `Core_extended.Wrapped`, which has efficient handling of size-prefixed bin-io values in cases where serialization can be bypassed. ### core - Deprecated the single-line files that simply `include` the corresponding Core_kernel module. Those are unnecessary, because people should use `Core.Std`. We will keep these aliases around for a version before deleted them entirely. - Changed finalizers and signal handlers to, upon an unhandled exception, exit nonzero rather than asynchronously raise. - Removed `Time.Zone.find_office`. Replaced uses with the still-blocking `Time.Zone.find_exn` - Made many changes to `Time` to make the time zone explicit instead of implicitly using the local timezone. Added `zone:Time.Zone.t` parameters to many functions. In almost all cases, used `~zone:Time.Zone.local` where previously it was implicit. Removed `of_local_ofday` and `to_local_ofday` in favor of the explicit versions (with `Time.Zone.local`). Removed `Time.Zone.machine_zone ()` in favor of `local`. - Exported `Core.Std.With_return`. - Exposed `Core.Std.with_return_option`. - Fixed `Time_ns.Ofday.of_span_since_start_of_day` to check its input. - Changed `Time_ns.to_span` and `of_span` to round to microseconds, for round trippability. - Added `Unix.Error` module, for the `Unix.error` type. - Added `Unix.Syscall_result`, a new abstract type representing the result of a Unix system call as an `int`, to avoid allocation. A lot of Unix system calls return an integer on success, so for ones that are called a lot, we can encode errors as `-errno`. This module abstracts this concept. - Changed `Iobuf.recvmmsg` functions to return the new `Unix.Syscall_result`. - Changed `Unix.exec`'s `?env` argument to support extending the environment in addition to replacing it. - Added `with compare` to `Unix.Exit.t` and `Unix.Exit_or_signal.t`. - Moved `Backtrace` to `Core_kernel`. Deleted `backtrace_stubs.c`, now that we have `Printexc.get_callstack`. - Changed `Bigstring.read_assume_fd_is_nonblocking` and `send_nonblocking_no_sigpipe` to return `Unix.Syscall_result.t`, to reduce allocation. - Changed `Iobuf.send_nonblocking_no_sigpipe` to handle `EINTR` like `EAGAIN`, instead of raising. - Added `Command.Spec.char`. - Changed `Process_env.parse_ssh_client` to accept an `SSH_CLIENT` that is just IP address without ports. ### core_bench - Updated code to follow some core changes ### core_extended - Added functions to `Low_level_debug` to get a sexp or string representation of any type. This could be handy when debugging polymorphic code. - Renamed `String.is_substring` to `is_substring_deprecated`. Use `Core.String.is_substring` instead. - Fixed a bug in `Bin_io_utils.save`. - Made `Unix.Mac_address` match `Hashable.S`. ### core_kernel - Added `List.is_prefix`. ```ocaml val List.is_prefix : 'a t -> prefix:'a t -> equal:('a -> 'a -> bool) -> bool ``` - Made `String_id.Make` functor generative, which exposes that the result has `type t = private string`. Previously the result of `String_id.Make` didn't expose `type t = private string` due to a type-checker bug: * http://caml.inria.fr/mantis/view.php?id=6485 * http://caml.inria.fr/mantis/view.php?id=6011 - Used generative functors, e.g. for `Unique_id`. Used generative functors (new feature in 4.02) where previously we used dummy `M : sig end` arguments in the signature and `(struct end)` when applying the functor. Just to note the difference between applicative and generative functors. Suppose we have: ```ocaml module F (M : sig end) : sig type t end ``` and we apply it several times ```ocaml module A = F (struct end) module B = F (struct end) module C = F (String) module D = F (String) ``` Then we have that `A.t <> B.t` but `C.t = D.t`. This can lead to subtle bugs, e.g. `Unique_id.Int (Unit)`. Note that it is perfectly valid to apply any module to `F`, even though that is certainly not what we want. In 4.02, we can explicitly say that functor generates new types, i.e. it is generative. For this we use argument `()`. So `F` becomes ```ocaml module F () : sig type t end ``` You can only apply `F` to `()` or `(struct end)` but each application yields a new type `t`. ```ocaml module A = F () module B = F () module C = F (struct end) module D = F (String) (* illegal *) ``` and now `A.t`, `B.t` and `C.t` are all different. Note that `F (struct end)` is still allowed but was converted to to `F ()` for consistency with signatures. Propagated generativity where necessary. If inside a functor we use generative functor that creates new types, then we also need to make the enclosing functor generative. For functors that don't create types (like `Async.Log.Make_global`), generative or applicative functors are the same, but the syntax of generative functors is lighter. - Exported `Core_kernel.Std.With_return`. - Exposed the record type of `Source_code_position.t`. - In `Weak_hashtbl.create`, exposed the `?growth_allowed` and `?size` arguments of the underlying `Hashtbl.create`. - Added `with compare` to `Array`. - Sped up `Int.pow`. Benchmarks before: | Name | Time/Run | mWd/Run | Percentage | |-----------------------------------------------|--------------|---------|------------| | [int_math.ml:int_math_pow] random[ 5] x 10000 | 140_546.89ns | | 53.98% | | [int_math.ml:int_math_pow] random[10] x 10000 | 173_853.08ns | | 66.77% | | [int_math.ml:int_math_pow] random[30] x 10000 | 219_948.85ns | | 84.47% | | [int_math.ml:int_math_pow] random[60] x 10000 | 260_387.26ns | | 100.00% | | [int_math.ml:int_math_pow] 2 ^ 30 | 11.34ns | | | | [int_math.ml:int_math_pow] 2L ^ 30L | 21.69ns | 3.00w | | | [int_math.ml:int_math_pow] 2L ^ 60L | 22.95ns | 3.00w | | and after: | Name | Time/Run | mWd/Run | Percentage | |-----------------------------------------------|--------------|---------|------------| | [int_math.ml:int_math_pow] random[ 5] x 10000 | 105_200.94ns | | 80.78% | | [int_math.ml:int_math_pow] random[10] x 10000 | 117_365.82ns | | 90.12% | | [int_math.ml:int_math_pow] random[30] x 10000 | 130_234.51ns | | 100.00% | | [int_math.ml:int_math_pow] random[60] x 10000 | 123_621.45ns | | 94.92% | | [int_math.ml:int_math_pow] 2 ^ 30 | 8.55ns | | | | [int_math.ml:int_math_pow] 2L ^ 30L | 22.17ns | 3.00w | 0.02% | | [int_math.ml:int_math_pow] 2L ^ 60L | 22.49ns | 3.00w | 0.02% | - Removed the old, deprecated permission phantom types (`read_only`, etc.) and replaced them with the new =Perms= types. The old types had subtyping based on covariance and `private` types. The new types have subtyping based on contravariance and dropping capabilities. Renamed `read_only` as `read`, since `Perms` doesn't distinguish between them. The idiom for the type of a function that only needs read access changed from: ```ocaml val f : _ t -> ... ``` to ```ocaml val f : [> read ] t -> ... ``` This mostly hit `Iobuf` and its users. - Added `String.is_substring`. - Added `With_return.prepend`, and exposed `With_return.t` as contravariant. ```ocaml (** [prepend a ~f] returns a value [x] such that each call to [x.return] first applies [f] before applying [a.return]. The call to [f] is "prepended" to the call to the original [a.return]. A possible use case is to hand [x] over to an other function which returns ['b] a subtype of ['a], or to capture a common transformation [f] applied to returned values at several call sites. *) val prepend : 'a return -> f:('b -> 'a) -> 'b return ``` - Moved the `Gc` module's alarm functionality into a new `Gc.Expert.Alarm` module. The was done because the Gc alarms introduce threading semantics. - Exposed modules in `Core_kernel.Std`: `Int_conversions`, `Ordered_collection_common` - Removed `Pooled_hashtbl` from `Hashable.S`, to eliminate a dependency cycle between `Int63` and `Pool`. This was needed to use `Int63` in `Pool`. Previously, `Int63 <- Int <- Hashable <- Pool`, which made it impossible to use `Int63` in `Pool`. So, we are removing the dependency `Hashable <- Pool`, simplifying `Hashable` to not include `Pooled_hashtbl`, and letting users call the `Pooled_hashtbl` functor directly when necessary. - Added to `Pool.Pointer.Id` conversions to and from `Int63`. - Made `Pooled_hashtbl.resize` allocate less. - Removed `Pool.pointer_of_id_exn_is_supported`, which was always `true`. - Added `with compare` to `Info`, `Error`, `Or_error`. - Moved `Backtrace` from `Core` - In C stubs, replaced `intxx` types by `intxx_t`. Following this: http://caml.inria.fr/mantis/view.php?id=6517 Fixes #23 - Removed `Backtrace.get_opt`, which is no longer necessary now that `Backtrace.get` is available on all platforms. - Added module types: `Stable`, `Stable1`, `Stable2`. - Exposed `Core_kernel.Std.Avltree`. - Removed from `Binary_packing` a duplicated exception, `Pack_signed_32_argument_out_of_range`. Closes #26 - Made `Info`, `Error`, and `Or_error` stable. The new stable serialization format is distinct from the existing unstable serialization format in the respective modules, which wasn't changed. - Add `Sequence.Step.sexp_of_t`. ### core_profiler Initial release ### custom_printf - Changed `%{M.x}` to call `M.x` instead of `M.Format.x`. This is more natural and makes it easy to call arbitrary other functions. - Added syntax `%{M#x}`, which calls `M.to_string_x`. This makes it easy to use existing modules, since calling functions `to_string` or `to_string_x` is already the convention. ### email_message Moved from janestreet-alpha ### jenga - Fixed byte-compile targets to avoid stale artifact deletion of `.for-byte-compile.cmt` file when compilation fails. This avoids polling jenga trigger loop. - Show `Removed stale build artifact` messages only when `-act` flag is given. - Extended Jenga API with `val file_existence : Path.t -> unit t`. `file-existence` has same relationship to `file-exists` as `glob_change` has to `glob_listing`. - Fixed memory leak in tenacious hearts by using weak references and finalizers. * Strip code for obsolete versions of hearts. * Strip code for `OLD_TENACIOUS=true`. * Ensure `Ring` support preemptive calls to `detach`. Add ring tests. * Keep message showing `Live(Kb-delta)`. - Fixed curly braces in globs. - Throttled calls to `Writer.save`, to fix the `too many open files` bug. Throttled calls to `Writer.save` from `save_description.run` sharing the same throttle used for all FD access, with `~max_concurrent_jobs:500` - Added to `jenga -progress` `~save=XXX`, the number of calls to `Writer.save` in `Progress.saves_run`. ### ocaml_plugin - Fixed spurious `interface mismatch` error when a plugin cache is shared by incompatible compilers. When a plugin cache directory is used by several executables with incompatible cmis/compilers, and the cache config option `try_old_cache_with_new_exec` is set to true, this could lead to the following error: ```ocaml Plugin failed: (ocaml_dynloader.ml.Dynlink_error "interface mismatch") ``` This feature fixes this. Since it modifies some record, for later changes it seems easier and more conservative to allow field additions without breaking older version. Thus we allow extra fields in persisted records. ```ocaml let t_of_sexp = Sexp.of_sexp_allow_extra_fields t_of_sexp ``` New executables can read both old and new caches, but old executables will either blow away new caches, or if the config says the cache is read-only, fail. Take the chance to modernize part of the code. - Switched tests to unified tests. - Fixed bugs dealing with paths with spaces in them. - Check that plugins have the expected type before running them rather than after, which is what one would expect. Also check that runtime and compile types match in `check_ocaml_src_files` and `compile_ocaml_src_files_into_cmxs_file`. ### pa_ounit - only spend time to format test description when tests are run ### patdiff - The call to Pcre.full_split in patdiff_core.ml rely on a bug of pcre-ocaml <= 7.1.2. To get the same behavior with pcre-ocaml >= 7.1.3 we need to pass ~max:(-1). See this bug for more details: https://github.com/mmottl/pcre-ocaml/issues/1 ### rpc_parallel - Follow changes in Async RPC ### sexplib - Added `sexp_of_` support for GADTs, and remove the not-quite-working support for `of_sexp`. ### textutils - Added a `~narrow` argument to `Text_graph.render` ### typerep - Split out typerep_extended which is now using core_kernel ### typerep_extended - typerep_extended now use core_kernel ## 112.06.00 ### async_extended - Unwound a recent change to `Mailbox` where one invocation of `receive` would put aside items, preventing other invocations from noticing them. - Added `Delimited.Row.nth_conv_exn`, as a counterpart to `get_conv_exn`. - Fixed `File_updates` handling of identical mtimes. ### async_extra - In `Log`, exposed the raw message. - Changed `Rpc` creators' `connection_state` to be a function that takes the connection and returns the state. This makes it possible for the connection state to actually get a handle on the connection itself, which simplifies a number of idioms for using RPC. In particular, it makes it easier to respond with an RPC back to a client over client's own connection. - Fixed some nondeterministically failing tests. - In `Log`, made logs discard messages when their output list is empty. Also, removed redundant tracking of current level. - Moved `Udp.bind_to_interface_exn` to `Unix` module in `async_unix`. - Added `Versioned_typed_tcp.Repeater`. Repeater is used in the cases where we want to inspect and possibly alter the flow between a client and a server without having to change either the client or the server or the protocol between them. ### async_kernel - Added `Deferred.Sequence` module, analogous to `Deferred.List` but for `Core_kernel.Std.Sequence`. - Modernized code style. ### async_unix - Added `Writer.behave_nicely_in_pipeline`, which makes a program behave nicely when used in a shell pipeline where the consumer goes away. - Modernized code style. - Removed spurious overrides in `Signal`: `set` and `signal`. These overrides are no longer necessary because the functions moved from `Core.Signal` to `Core.Signal.Expert`. - Moved `async_extra`'s `Udp.bind_to_interface_exn` to `Unix`. ### bignum - Added functions to round from `Bignum.t` to `Bigint.t`, and to convert `Bigint.t` into `Bignum.t`. ### bin_prot - Sped up `float` and `float array` operations. - Removed a use of `Obj.magic` in code generated by `pa_bin_prot` for polymorphic variants that led to memory unsafety. Previously, `pa_bin_prot` generated this kind of code for polymorphic variants: match Obj.magic (read_int buf pos) with | `A as x -> x | `B as x -> x | `C -> `C (read_float buf pos) | _ -> fail and this caused the compiler to assume the result is an immediate value. To fix this we removed the `as x -> x` and used the computed integer hash. ### core - Renamed `Linux_ext.gettid` as `Unix.gettid`, and added OpenBSD support. `SYS_gettid` is not available on OpenBSD, but is used in `Core_extended`. See the mailing list discussion about this here: https://groups.google.com/forum/#!topic/ocaml-core/51knlnuJ8MM Seems like the OpenBSD alternative is: pid_t getthrid(void); although it's not defined in any header file, which is a bit unfortunate. - Added `Piecewise_linear.precache`, which computes a lookup table that speeds up subsequent calls to `Piecewise_linear.get`. - Added `Time_ns` module, representing times as 63-bit integers of nanoseconds since the epoch. - Fixed build of `unix_stubs.c` on OpenBSD. - In `Daemon`, fixed an error message regarding `WSTOPPED` (fixes #47). - Added `Time.Span.Stable.V2`, with sexps that use new suffixes for microseconds (`us`) and nanoseconds (`ns`). `Time.Span.of_string` supports the new format, but `Time.Span.to_string` doesn't yet produce it -- we plan to change that later, after the new `of_string` has made it out more widely. - Added `Time.Span.to_string_hum`, which gives more options for rendering time spans. - Merged the `recvmmsg` stubs in `Bigstring` and `Iobuf`. Factored out a shared underlying `recvmmsg` call that both stubs use. Restored `-pedantic` by avoiding a C99 feature (variable-length stack arrays). - Made `Date.t` abstract, and changed its representation from a 4-word record to an immediate int (packing year, month, day). - In `Daemon`, changed the permissions of the `std{err,out}` files generated during daemonization from `0o777` to `0o644`. - Moved `Thread_safe_queue` from `core` to `core_kernel`. This was done so that `Async_kernel` can use it, eliminating one of `Async_kernel`'s dependencies on `Core`. `Thread_safe_queue_unit_tests` remains `Core`, at least for now, because it has some dependencies on other stuff in `Core`. ### core_bench - Solved a problem in which OCaml 4.02 was optimizing away benchmarks, making them meaningless. ### core_extended - Sped up `String.is_substring` by replacing the OCaml implementation with a call to libc `memmem`. `memmem` runs in 20% of the time, incurs minimal GC pressure, is portable among UNIXen that we target, AND it's clearer than the ML version. - Made `Float_ref` support `bin_io` and `sexp`. - Removed `gettid`, which is now available in `Core.Unix`. - Added `Fast_int_div` module, which speeds up integer division by a fixed divisor. - Moved `Sexp.of_sexp_allow_extra_fields` to core_kernel. ### core_kernel - Made `String_id` have `Stable_containers.Comparable`. - Changed `Gc.disable_compaction` to require an `allocation_policy`. - Made `Option` match `Invariant.S1`. - Added `Sequence.filter`, `compare`, and `sexp_of_t`. - Added `With_return.with_return_option`, abstracting a common pattern of `with_return`. val with_return : ('a return -> 'a ) -> 'a val with_return_option : ('a return -> unit) -> 'a option - Install a handler for uncaught exceptions, using `Printexc.set_uncaught_exception_handler`, new in OCaml 4.02. - Changed `Day_of_week` representation to a normal variant. - Changed `Exn.handle_uncaught` so that if it is unable to print, it still does `exit 1`. - Added `Sexp.of_sexp_allow_extra_fields`, previously in `Core_extended.Sexp`. - Changed the implementation of `Exn.raise_without_backtrace` to use `raise_notrace`, new in OCaml 4.02. - Added `Float` functions for converting to and from IEEE sign/exponent/mantissa. - Added `String.Caseless` module, which compares and hashes strings ignoring case. - Reimplemented `Type_equal.Id` using extensible types (new in OCaml 4.02), removing a use of `Obj.magic`. Changed `Type_equal.Id.same_witness` to return `option` rather than `Or_error`, which allows it to be implemented without allocation. - Removed a reference to the `Unix` module. Applications using `core_kernel` should be able to link without `unix.cma` again. - Made `Char.is_whitespace` accept `\f` and `\v` as whitespace, matching C. ### core_profiler - Solved a problem in which OCaml 4.02 was optimizing away benchmarks, making them meaningless. ### jenga - Support for user control of stale-artifact deletion, by allowing specification of an artifact-determination policy. - Expose jenga's internal (and better - only quotes when necessary) definition of `Shell.escape` in `Api` - Removed `Action.shell` from the API, superseded by `Action.process`. - Changed RPC interface as needed for build manager to switch from scraping error messages to RPCs. - Fixed jenga's per-rule memo table, which mistakenly kept stale values. - Show what target is being demanded, useful for debugging rules. - Run user action when persistent format changes. - When filtering buildable targets by globs, pay attention to the kinds allowed by the glob. Specifically, if the kinds don't include `` `File `` (i.e. only include `` `Directory ``) then we should not see any `buildable_targets` in the filtered list. ### ocaml_plugin - Stopped using the `~exclusive` with `Reader`, because it doesn't work on read-only file systems. It's not even needed because these files are written atomically. - Used a generative functor in the generated code, so the user code can apply generative functors at toplevel, or unpack first class modules that contain type components. - Fixed bug when mli file references something defined only in another ml. - Made it possible to compile a plugin in one process, and dynload the compiled `cmxs` file without starting async in another process. This was done with two new APIs in `Ocaml_dynloader.S`: val compile_ocaml_src_files_into_cmxs_file : dynloader -> string list -> output_file:string -> unit Deferred.Or_error.t val blocking_load_cmxs_file : string -> t Or_error.t - Allowed plugins to optionally have a shebang line. - Made `Ocaml_dynloader.find_dependencies` also support files with shebang lines. ### pa_bench - Made the code generated by `pa_bench` for `BENCH` not use `ignore`, because OCaml 4.02 will remove dead code in some cases, meaning the benchmarks are no longer measuring what they should. Instead the ignore is deep inside `Core_bench`, which is likely out of reach of the compiler. The result of the user functions given to `BENCH_FUN` and `BENCH_INDEXED` are changed so they don't have to return unit and people are encouraged not to use `ignore` when these functions don't return `unit` (you will get the same warning though, i.e. a warning if the result of your function is a function too, thus preventing unintended partial applications). For example, here are a few benchmarks and their output before the fix: let x = if Random.bool () then 100 else 1001 let r = ref 0 BENCH "ig-1" = 10 / x BENCH "ig-2" = () BENCH "ig-3" = phys_equal (10 / x) (Obj.magic 0) BENCH "ig-4" = r := (10 / x) BENCH "ig-5" = r := x +----------------+----------+------------+ | Name | Time/Run | Percentage | +----------------+----------+------------+ | [misc.ml] ig-1 | 3.92ns | 29.30% | | [misc.ml] ig-2 | 3.34ns | 24.95% | | [misc.ml] ig-3 | 3.91ns | 29.23% | | [misc.ml] ig-4 | 13.37ns | 100.00% | | [misc.ml] ig-5 | 3.24ns | 24.20% | +----------------+----------+------------+ Many of the the numbers above are much lower than they should be because of the implicit ignores inserted by the benchmark caused the division to to eliminated by the compiler. After the fix, the same benchmarks produced more meaningful numbers: +----------------+----------+------------+ | Name | Time/Run | Percentage | +----------------+----------+------------+ | [misc.ml] ig-1 | 12.78ns | 94.55% | | [misc.ml] ig-2 | 3.23ns | 23.90% | | [misc.ml] ig-3 | 13.51ns | 99.94% | | [misc.ml] ig-4 | 13.52ns | 100.00% | | [misc.ml] ig-5 | 3.30ns | 24.40% | +----------------+----------+------------+ ### sexplib - Improved the implementation of `Exn.sexp_of_t`, using the unique id in exceptions in OCaml 4.02. We use the identifier to map exception constructors to converters. ## 112.01.00 ### async - update tests ### async_extended - Clarified an error in `Rpc_proxy`. ### async_extra - Changed `Persistent_rpc_client.connected` to avoid returning a connection that is closed at the time it was called. - Optimized `Rpc.implement` so that if a server's implementation returns a determined deferred, then the output is immediately serialized and written out for the client. This reduces memory consumption, improves throughput and latency. Measurements with the `pipe_rpc_test program` showed that a server went from processing 600\_000 msg/sec, to 2\_200\_000 msg/sec before pegging the CPU. - Changed `Log`'s output processor's batch size from `1_000` to `100`. - Added `Persistent_rpc_client.close` and `close_finished`. - In `Rpc.Connection.client` and `with_client`, used the `handshake_timeout` as the `timeout` passed to `Tcp.connect`. `handshake_timeout` was previously used only for the `Rpc` module's handshake timeout. - Changed `Rpc.create`'s `on_unknown_rpc` argument, renaming `\`Ignore` as `\`Close_connection`, and requiring `\`Call` to return `\`Close_connection` or `\`Continue`. `\`Ignore` was renamed because it was a poor name, since in fact it closed the connection. Added a `\`Continue` option, whic allows one to keep the connection open. Changed `\`Call` to return `\`Continue` or `\`Close_connection`, where the old `unit` return value meant `\`Close_connection`. - In `Versioned_typed_tcp`, enabled the use of "credentials" in the "Hello" message. Propagate credentials to the user code when it arrives on the wire. ### async_kernel - Optimized `Monitor.try_with ~run:\`Now f` to return a determined deferred if `f ()` returns a determined deferred. Previously, `Monitor.try_with ~run:\`Now f` always introduced a `Deferred.map`, which made it impossible to do some optimizations that bypass the scheduler overhead. - Added an `ASYNC_CONFIG` field that causes the program to dump core if Async jobs are delayed too much. The new field is `dump_core_on_job_delay`. - Switched `Async_kernel` from using `Core.Sys` to `Pervasives.Sys` eliminating one of the dependencies on `Core`. ### async_unix - Changed `Writer.transfer write pipe` to close `pipe` when the `writer`, is closed. Previously, `Writer.transfer` did not close the pipe when the underlying writer is closed. This was strange because: 1. Callers would have to consistently check for the writer being closed and close the `Pipe.Reader`t= themselves 2. The analogous function `Pipe.transfer` closes the reader on similar circumstances. The absence of the close was noticed as a bug in `Rpc`, which assumed that `Writer.transfer` did the close. - Fixed a bug in `Scheduler.yield` that caused it to pause for 50ms if there is no other pending work and no I/O. - Exposed type equivalence between `Unix.Passwd.t` and `Core.Std.Unix.Passwd.t`. - Changed `Writer.write_bin_prot` to use the new `Bigstring.write_bin_prot`. ### bignum - Added `Bignum.Bigint` module, with arbitrary-precision integers based on `Zarith`, which is significantly faster than the `Num.Big_int` library. ### bin_prot - In `Write`, improved some OCaml macros to name values and avoid calling C functions multiple times. ### core - Removed vestigial code supporting OCaml 4.00. - Added `Command` support for flags that are passed one or more times. Added `Command.Spec.one_or_more` and `Command.Spec.non_empty_sequence` to deal with the cases where you expect a flag or anonymous argument (respectively) to be passed one or (optionally) more times. This is common enough and distinct from the case where you want the argument passed zero or more times that it seems like we should canonize it in the library. - In `Lock_file`, made stale lock detection more robust. Made `Lock_file.create foo` succeed if `foo` is absent and `foo.nfs_lock` file is present and stale. Previously, it would fail. - Removed `Syslog.syslog`'s `add_stderr` argument; use the `PERROR` option instead. - Fixed `unix_stubs.c` compilation on NetBSD Closes #45 - Added `Filename` operators `/^` and `/@`, and `of_parts`, like the same functions for Catalog paths. - Changed `Iobuf` functions that advance the iobuf to not also return a redundant number of bytes processed. This avoids a small allocation (in the case of the `int option` functions) and normalizes the result (so the same information isn't returned two ways). Actually, it doesn't yet avoid the allocation in the implementation, as the corresponding `Bigstring` functions must still return the number of bytes processed, and currently do so as an option. We hope to eventually change that. In the future I expect we will change `unit` to some `error` variant to also avoid the exception construction for `EWOULDBLOCK/EAGAIN`. We can even make Unix syscalls `noalloc` if we're careful. - In `Unix` module, added unit tests for `Cidr.does_match`. ### core_bench - fixed legacy format string ### core_extended - Added `Float_ref` module, which is like `float ref` but faster for sets due to bypassing the write barrier. Benchmark results on Sandy Bridge: | [float\_ref.ml:] float ref set | 2\_886.94ns | 8.00w | | | [float\_ref.ml:] Float\_ref.set | 355.76ns | 6.00w | | | [float\_ref.ml:] float ref get | 415.52ns | 6.00w | | | [float\_ref.ml:] Float_ref.get | 416.19ns | 6.00w | | - Added `Bin_io_utils.Wrapped.t`, which defines an `'a t with bin_io` that supports size-prefixed serialization and deserialization. `Wrapped` has two useful submodules, `Opaque` and `Ignored`, for efficient handling of size-prefixed bin-io values in cases where serialization can be bypassed. See the comments in the module for more details. ### core_kernel - Removed vestigial code supporting OCaml 4.00. - Used `{Hashable,Comparable}.S_binable` in `Day_of_week` and `Month`. - Improved the performance of `Set_once.set`. - Added `Type_equal.Lift3` functor. - Replaced occurrences of `Obj.magic 0` with `Obj.magic None`. With the former the compiler might think the destination type is always an integer and instruct the GC to ignore references to such values. The latter doesn't have this problem as options are not always integers. - Made `String_id.of_string` faster. - Added `Bigstring` functions for reading and writing the size-prefixed bin-io format. - `bin_prot_size_header_length` - `write_bin_prot` - `read_bin_prot` - `read_bin_prot_verbose_errors` - Added `{Info,Error}.to_string_mach` which produces a single-line sexp from an `Error.t`. - Added `{Info,Error}.createf`, for creation from a format string. - Added new `Perms` module with phantom types for managing access control. This module supersedes the `read_only`, `read_write`, and `immutable` phantom types, which are now deprecated, and will be removed in the future. This module uses a different approach using sets of polymorphic variants as capabilities, and contravariant subtyping to express dropping capabilities. This approach fixes a bug with the current phantom types used for `Ref.Permissioned` in which `immutable` types aren't guaranteed to be immutable: ```ocaml let r = Ref.Permissioned.create 0 let r_immutable = (r : (int, immutable) Ref.Permissioned.t) let () = assert (Ref.Permissioned.get r_immutable = 0) let () = Ref.Permissioned.set r 1 let () = assert (Ref.Permissioned.get r_immutable = 1) ``` The bug stems from the fact that the phantom-type parameter is covariant, which allows OCaml's relaxed value restriction to kick in, which allows one to create a polymorphic value, which can then be viewed as both immutable and read write. Here's a small standalone example to demonstrate: ```ocaml module F (M : sig type +'z t val create : int -> _ t val get : _ t -> int val set : read_write t -> int -> unit end) : sig val t : _ M.t end = struct let t = M.create 0 let t_immutable = (t : immutable M.t) let () = assert (M.get t_immutable = 0); M.set t 1; assert (M.get t_immutable = 1); ;; end ``` The new approach fixes the problem by making the phantom-type parameter contravariant, and using polymorphic variants as capabilities to represent what operations are allowed. Contravariance allows one to drop capabilities, but not add them. - Added `Int.Hex` module, which has hexadecimal sexp/string conversions. - Added `Gc.major_plus_minor_words`, for performance reasons. ### core_profiler - fixed legacy format string ### custom_printf - Fixed uses of `printf=-style`format strings that have unspecified behavior in OCaml 4.02 and will become errors. - Support substitution in format string (lost with 4.02 compatibility) ### jenga - Don't show noisy `glob..changed` messages except with `-show-glob-changed` flag. - Support shared build rules via `${jenga}/share`. - Detect cycle in dep scheme instead of hanging. - Made standalone actions atomic, just like actions associated with target files. Running actions and recording the result in the persistent `.jenga.db` should be performed atomically for standalone actions, as it is for actions which are associated with target files ### ocaml_plugin - Changed to not use `rm -r` when it is expected to remove one file. ### rpc_parallel Initial import. ### sexplib - Replaced occurrences of `Obj.magic 0` with `Obj.magic None`. With the former the compiler might think the destination type is always an integer and instruct the GC to ignore references to such values. The latter doesn't have this problem as options are not always integers. ### textutils - Added `Ascii_table.Table_char`, to expose the special table border characters. ### type_conv - Updated ast matching for 4.02 ## 111.31.00 ### jenga - Switched API to composable generator schemes. - Support `-api` flag to show the embedded API. - New examples. ## 111.28.00 ### async_extra - Added to `Versioned_rpc` a non-functor interface. - Added `Log.level`, which returns the last level passed to `set_level`. - Enabled Async-RPC pushback in the `Tcp_file` protocol. ### async_unix - Added `Shutdown.set_default_force`, which allows one to change the default `force` value used by `shutdown`. This is useful for applications that call `shutdown` indirectly. val set_default_force : (unit -> unit Deferred.t) -> unit ### core - Added `Piecewise_linear.create_from_linear_combination`. val create_from_linear_combination : (t * float) list -> t Or_error.t - Added `Time.is_{earlier,later} : Time.t -> than:Time.t -> bool`, which are easier to read than `Time.(<)` and friends. - Added `Command.exec`, which allows one to include the `Command` hierarchy from one executable in another. `Command.exec` takes the file path to an executable that uses the `Command` module and returns a `Command.t` that integrates the executable (by exec'ing it), including providing recursive help and autocompletion as if it were a standard `Command.t`. - Replaced most uses of `Hashtbl.replace` with `Hashtbl.set`. - Renamed `Float.epsilon` to `robust_comparison_tolerance`, to avoid confusion with `epsilon_float`. ### core_extended - Implemented `Int.gcd` using binary GCD in C, for improved performance. - Added `Bin_io_utils.Serialized`, which stores a value in memory as its bin-io representation. Writing such a value just blits the value. - Moved `Text_block` from `Core_extended` into `Textutils`. - Added modules `Hashtbl2` and `Hashtbl2_pair`. ### core_kernel - Added `Pooled_hashtbl.resize` function, to allow preallocating a table of the desired size, to avoid growth at an undesirable time. - Added `Pooled_hashtbl.on_grow` callback, to get information about hashtbl growth. - Changed `Hashable.Make` to not export a `Hashable` module. The `Hashable` module previously exported was useless, and shadowed `Core.Std.Hashable`. - Moved `Common.does_raise` to `Exn.does_raise`, to make it easier to find. - Added `Float.one`, `minus_one`, and `~-`. (fixes #12). - Removed `Core.Std.unimplemented` and renamed it as `Or_error.unimplemented`. It is not used enough to live in the global namespace. ### jenga - Fixed problem that caused `rule failed to generate targets`. ### ocaml_plugin - Fixed a bug in tests that could leave the repository in a state where running the tests would fail. The bug happened if the tests were interrupted after creating read-only directories but before cleaning then up. ### pa_ounit - Added a flag to disable embedding of unit tests/inline benchmarks. (`janestreet/core_kernel#13`) ### textutils - Moved `Text_block` from `Core_extended` into `Textutils`. ## 111.25.00 ### async - add a dns example ### async_extra - Removed `lazy` from the core of `Log`. - Made `Log.Message.t` have a stable `bin_io`. The `Stable.V1` is the current serialization scheme, and `Stable.V0` is the serialization scheme in 111.18.00 and before, which is needed to talk to older systems. - Changed `Rpc` to return `Connection_closed` if a connection ends before a response makes it to the caller. Previously, the dispatch output was never determined. Also, removed an unused field in one of the internal data structures of Async RPC. - In `Versioned_rpc`, added `version:int` argument to `implement_multi` functions. - In `Versioned_rpc`, the `Pipe_rpc.Make` functors now return an additional output functor. `Register'` is like `Register` but has in its input module: ```ocaml val response_of_model : Model.response Queue.t -> response Queue.t Deferred.t ``` rather than ```ocaml val response_of_model : Model.response -> response ``` This is analogous to `Pipe.map'` and `Pipe.map`. - Added to `Log` a `V2` stable format and better readers for time-varying formats. - In `Log`, added an optional `?time:Time.t` argument to allow callers to pass in the logged time of an event rather than relying on `Time.now ()`. ### async_kernel - Fixed `Clock.run_at_intervals` to make the initial callback at an interval multiple. Previously, if `start` was in the past, `f` would run immediately rather than waiting for an interval of the form `start + i * span`. Now it always waits for an interval, even the first time it runs. ### async_parallel - improve error handling ### async_unix - Added `Unix.Addr_info` and `Name_info`, which wrap `getaddrinfo` and `getnameinfo`. - Improved the scheduler's error message when the thread pool is stuck. ### core - Added `Gc.disable_compaction` function. - Added `Time.to_string_abs_trimmed`, which prints a trimmed time and takes a `zone` argument. - Fixed `unix_stubs.c` to suppress a warning when building with some versions of gcc. - Changed `Time.Zone` to allow the zoneinfo location to be specified by an environment variable. Closes #40 - Fix compatibility with 4.02 ### core_extended - Moved `Quickcheck` from `core`. - Added [Int.gcd]. ### core_kernel - Fix build on FreeBSD Closes #10 - Added functions to `Container` interface: `sum`, `min_elt`, `max_elt`. ```ocaml (** Returns the sum of [f i] for i in the container *) val sum : (module Commutative_group.S with type t = 'sum) -> t -> f:(elt -> 'sum) -> 'sum (** Returns a min (resp max) element from the collection using the provided [cmp] function. In case of a tie, the first element encountered while traversing the collection is returned. The implementation uses [fold] so it has the same complexity as [fold]. Returns [None] iff the collection is empty. *) val min_elt : t -> cmp:(elt -> elt -> int) -> elt option val max_elt : t -> cmp:(elt -> elt -> int) -> elt option ``` - Made `Core_hashtbl_intf` more flexible. For instance supports modules that require typereps to be passed when creating a table. Address the following issues: The type `('a, 'b, 'z) create_options` needs to be consistently used so that `b` corresponds with the type of data values in the returned hash table. The type argument was wrong in several cases. Added the type `('a, 'z) map_options` to `Accessors` so that map-like functions -- those that output hash tables of a different type than they input -- can allow additional arguments. - Fixed a bug in `Dequeue`'s `bin_prot` implementation that caused it to raise when deserializing an empty dequeue. - Made `Container.Make`'s interface match `Monad.Make`. - Deprecated infix `or` in favor of `||`. - Simplified the interface of `Arg` (which was already deprecated in favor of `Command`). - Replaced `Bag.fold_elt` with `Bag.filter`. - `Memo.general` now raises on non-positive `cache_size_bound`. - Removed `Option.apply`. - Removed `Result.call`, `Result.apply`. - Moved `Quichcheck` to `core_extended`. It should not be used in new code. ### custom_printf - Fix 4.02 compatibility. ### jenga - Switched to un-version-numbered API. - Renamed `Tenacious_sample_lib.Tenacious` to `Tenacious_sample_lib.Tenacious_sample` to avoid conflicts in the public release. - Write `buildable_targets.list` (on alias `.info`). ### ocaml_plugin - ignore more warnings by default ### patdiff - add a `?file_names` argument to `Compare_core.diff_strings` ### patience_diff - refactoring and more unit tests ### sexplib - Fix compatibility with OCaml 4.02 ## 111.21.00 ### async_extra - Added `Sexp_hum` `Log.Output.format`, which is useful for making logs more human readable. - Added `with compare` to `Rpc.Implementation.Description`. ### async_ssl - Upgraded to use new ctypes and its new stub generation methods. ### async_unix - Added `Process.wait_stdout` and `wait_stdout_lines`, which are like `run` and `run_lines`, but take a `Process.t`. ### core - Fixed an issue where `Time.Zone.init` would not properly traverse the directory containing timezone information. - Added `Time.With_utc_sexp`, which has stable serialization of `Time.t` that is byte-for-byte equal across timezones. - Made `Uuid` stable. - Made `Piecewise_linear` stable. ### core_kernel - Removed our custom C stub for closing channels, reverting to the one in the OCaml runtime. A long time ago we found that the OCaml runtime did not release the lock before calling `close` on the fd underlying a channel. On some filesystems (e.g. smb, nfs) this could cause a runtime hang. We filed a bug with INRIA and wrote our own `close` function which `In_channel` calls to this day. The bug has long been fixed, and our function is probably buggy, so this reverts us to the runtime's `close`. - Added `Float.{of,to}_int64_preserve_order`, which implement the order-preserving zero-preserving bijection between non-NaN floats and 99.95% of `Int64`'s. Used the new function to improve `one_ulp`, which is now exposed: (** The next or previous representable float. ULP stands for "unit of least precision", and is the spacing between floating point numbers. Both [one_ulp `Up infinity] and [one_ulp `Down neg_infinity] return a nan. *) val one_ulp : [`Up | `Down] -> t -> t - Changed `Map.symmetric_diff` to return a `Sequence.t` instead of a `list`. - Added `Sequence.filter_map`. - Improved `Stable_unit_test.Make_sexp_deserialization_test`'s error message so that it includes the expected sexp. ### custom_printf - Fixed a bug in which custom-printf syntax was incompatible with labeled arguments. For example, the preprocessor used to raise an exception on this code: let f ~labeled_arg:() fmt = ksprintf (fun _ -> ()) fmt in f !"hello" ~labeled_arg:() ### jenga - Introduced jenga API v3, a small cleanup of v2 which has been planned for a while. ### ocaml_plugin - Fixed a bug in `ocaml_embed_compiler` on 32-bit machines. `ocaml_embed_compiler` tries to read the full contents of the file as a string, but the string might be too big on 32bits: https://github.com/ocaml/opam-repository/pull/2062#issuecomment-43045491 ### patdiff - Added `Patdiff_core.iter_ansi`. (** Iter along the lines of the diff and the breaks between hunks. Offers more flexibility regarding what the caller wants to do with the lines *) val iter_ansi : f_hunk_break:((int*int) -> (int*int) -> unit) -> f_line:(string -> unit) -> string Patience_diff.Hunk.t list -> unit ### patience_diff - Added plain differ `Plain_diff` and use it in some cases for improved results. - Move modules under `Patience_diff_lib.Std`. ## 111.17.00 ### async_extra - Added module `Persistent_rpc_client`, an RPC client that attempts to reconnect when the connection is lost, until a new connection is established. - Significantly sped up the `Rpc` module by removing `Bigstring` serialization. Performance of the two implementations was tested by building a simple client/server executable that would count major cycles. Sending 100 byte messages at a rate of 50k/second shows (on both sides of the RPC): original: * ~160 major cycles in 30s * CPU usage around 60% new: * ~10 major cycles in 30s * CPU usage <= 2% - Enabled a version of `Pipe_rpc` and `State_rpc` where the consumer can pushback on the producer if it can't consume the contents of the pipe fast enough. - Added `Log.Level.arg : Log.Level.t Command.Spec.Arg_type.t` for defining command lines that accept (and autocomplete) log levels. - Added `Command.async_or_error` and renamed `Command.async_basic` to `Command.async`, leaving `async_basic` a deprecated alias for the new name. `Command.async_or_error` is similar to `Command.basic` and `Command.async`, but accepts a `unit Or_error.t Deferred.t` type. - Added `Persistent_rpc_connection.current_connection`, so that one can detect whether one is currently connected. ```ocaml val current_connection : t -> Rpc.Connection.t option ``` ### async_inotify - Upgraded library to use inotify 2.0 ### async_kernel - Renamed `Monitor.errors` to `Monitor.detach_and_get_error_stream` and `Monitor.error` as `Monitor.get_next_error`. The use of `detach` in the name is intended to make clear that errors do not propagate to the parent. Added several other non-stream =Monitor= functions to capture common use cases of `Monitor.detach_and_get_error_stream`: ```ocaml detach_and_get_next_error detach_and_iter_errors detach ``` ### bignum - Improved the performance of binprot deserialization by removing the allocation of an intermediate type. ### core - Fixed a bug in `Bigstring.really_recv` if `recv` doesn't receive all the data it wants. This bug has been around forever; it may not have caused trouble because `Bigstring.really_recv` (1) is barely used (the only use is in `Bigstring.unmarshal_from_sock`) and (2) passes `recv` the `MSG_WAITALL` flag, so it will read the full amount unless it gets interrupted by a signal. - Fixed `Bigstring.read`'s handling of `EINTR` so that it retries rather than returning zero. This fixes a bug introduced in 111.09 in the interaction between `Bigstring.read` and `Async.Reader`. Prior to 111.09, `Bigstring.read` would raise on `EINTR`, and `Async.Reader` would propagate the exception. From 111.09 to 111.16, `Bigstring.read` would return zero, which would confuse `Async.Reader` into thinking it reached EOF when it hadn't. From 111.17, `Bigstring.read` will retry and not return zero when not at EOF. We believe the bug was rare, because otherwise we would have frequently seen `EINTR` exceptions prior to 111.09. - Added `Command.Spec.apply` and `pair`, which allow one to program more with `Spec.param` rather than `Spec.t`. ```ocaml val apply : ('a -> 'b) param -> 'a param -> 'b param val pair : 'a param -> 'b param -> ('a * 'b) param ``` - Added `Command.Spec.file`, which builds an `Arg_type` value with the same autocompletion as `Spec.file`. ```ocaml (** [file] defines an [Arg_type.t] that completes in the same way as [Command.Spec.file], but perhaps with a different type than [string] or with an autocompletion key. *) val file : ?key:'a Univ_map.Multi.Key.t -> (string -> 'a) -> 'a t ``` ### core_extended - Added some functions to `Splay_tree`: * `length` * `keys` * `data` * `to_alist` * `delete_{after,before}` * `map` * `map_range` * `split`. ### core_kernel - In `Bigstring`, made many operations use compiler primitives new in OCaml 4.01. Exposed `Bigstring.get` and `set` as compiler primitives in the interface. Added `Bigstring.unsafe_get_int64_{le,be}_trunc`. - Made `Error` round trip `exn`, i.e. `Error.to_exn (Error.of_exn exn) = exn`. - Added to `failwiths` an optional `?here:Lexing.position` argument. - Added `with typerep` to `Flags.S`. - Optimized `List.dedup []` to return immediately. - Added `data` argument to polymorphic type `Hashtbl_intf.Creators.create_options`. This allows implementations of `Hashtbl_intf.Creators` to have constructor arguments that depend on the type of both key and data values. For example: ```ocaml module type Hashtbl_creators_with_typerep = Hashtbl_intf.Creators with type ('key, 'data, 'z) create_options = typerep_of_key:'key Typerep.t -> typerep_of_data:'data Typerep.t -> 'z ``` - Improved the interface for getting `Monad.Make` to define `map` in terms of `bind`. Instead of passing a `map` function and requiring everyone who wants to define `map` using `bind` to call a special function, we use a variant type to allow the user to say what they want: ```ocaml val map : [ `Define_using_bind | `Custom of ('a t -> f:('a -> 'b) -> 'b t) ] ``` - Improved the performance of many `Dequeue` functions. Previously, many `Dequeue.dequeue`-type functions worked by raising and then catching an exception when the dequeue is empty. This is much slower than just testing for emptiness, which is what the code now does. This improves the performance of `Async.Writer`, which uses `Dequeue.dequeue_front`. ### patdiff - Removed latex output. ### patience_diff - Exposed `Patience_diff.matches`. ### sexplib - Make the camlp4 dependency optional ## 111.13.00 ### async_extra - For `Typed_tcp.create`, added a `Client_id.t` argument to the `auth` callback. ### async_unix - Added `Scheduler.yield_every`, which returns a function that calls `Scheduler.yield` every n-th call. This is useful in circumstances where you don't have strict control over where a deferred is examined, as in a `Deferred.List.iter`. ### bignum - Eliminated the dependence of `Bignum` on `Re2`, and reduced dependence from `Core` to `Core_kernel`. - Extended the rounding interface to bring it in line with int and float rounding. - Improved the performance of `Bignum`'s binprot. `Bignum`'s binprot had been to just binprot the decimal string representation. This is both slow to do and unnecessarily big in the majority of cases. Did something better in the majority of cases and fell back to this representation in the exceptional case. $ ./inline_benchmarks_runner Estimated testing time 20s (2 benchmarks x 10s). Change using -quota SECS. | Name | Time/Run | mWd/Run | Percentage | |------------------------------------------------------|----------|-----------|------------| | bignum0.ml:Stable:Bignum binprot roundtrip compact | 7.87us | 490.00w | 32.88% | | bignum0.ml:Stable:Bignum binprot roundtrip classic | 23.94us | 1_079.00w | 100.00% | ### core_extended - Moved `Patience_diff` out of `Core_extended` into its own library. ### core_kernel - Added a `Sequence` module that implements polymorphic, on-demand sequences. Also implemented conversion to `Sequence.t` from various containers. - Improved the explicitness and expressiveness of `Binary_searchable.binary_search`. `binary_search` now takes an additional (polymorphic variant) argument describing the relationship of the returned position to the element being searched for. val binary_search : ?pos:int -> ?len:int -> t -> compare:(elt -> elt -> int) -> [ `Last_strictly_less_than (** {v | < elt X | v} *) | `Last_less_than_or_equal_to (** {v | <= elt X | v} *) | `Last_equal_to (** {v | = elt X | v} *) | `First_equal_to (** {v | X = elt | v} *) | `First_greater_than_or_equal_to (** {v | X >= elt | v} *) | `First_strictly_greater_than (** {v | X > elt | v} *) ] -> elt -> int option - Added a new function, `Binary_searchable.binary_search_segmented`, that can search an array consisting of two segments, rather than ordered by `compare`. (** [binary_search_segmented ?pos ?len t ~segment_of which] takes an [segment_of] function that divides [t] into two (possibly empty) segments: {v | segment_of elt = `Left | segment_of elt = `Right | v} [binary_search_segmented] returns the index of the element on the boundary of the segments as specified by [which]: [`Last_on_left] yields the index of the last element of the left segment, while [`First_on_right] yields the index of the first element of the right segment. It returns [None] if the segment is empty. By default, [binary_search] searches the entire [t]. One can supply [?pos] or [?len] to search a slice of [t]. [binary_search_segmented] does not check that [segment_of] segments [t] as in the diagram, and behavior is unspecified if [segment_of] doesn't segment [t]. Behavior is also unspecified if [segment_of] mutates [t]. *) val binary_search_segmented : ?pos:int -> ?len:int -> t -> segment_of:(elt -> [ `Left | `Right ]) -> [ `Last_on_left | `First_on_right ] -> int option - Made `Queue` match `Binary_searchable.S1`. - Made `Gc.Stat` and `Gc.Control` match `Comparable`. - Fixed some unit tests in `Type_immediacy` that were fragile due to GC. ### patience_diff - Moved `Patience_diff` out of `Core_extended` into its own library depending only on `Core_kernel`. ### sexplib - In `Sexplib.Std`, renamed `Macro` as `Sexp_macro`. ### type_conv - Removed some unused-value warnings when `with` is used in signatures. Removed warnings in cases like: include (module_expr : sig type t with bin_io end) ## 111.11.00 ### async - Updated the sound.ml example ### async_extra - Made `Log` more fair with respect to other Async jobs, by working on fixed-length groups of incoming log messages. Previously, `Log` had processed everything available. The change gives other Async jobs more of a chance to run. ### async_kernel - Added `Clock.run_at_intervals`, which runs a job at regular intervals. ### async_unix - Added `val Scheduler.yield : unit -> unit Deferred.t`, which becomes determined after the current cycle finishes. - Improved the behavior of the scheduler's thread pool when `Thread.create` raises. With this improvement, when the thread pool is unable to create a thread, it presses on with the threads it has rather than raise. Subsequent attempts to add work to the thread pool will cause the pool to attempt to create the thread, as long as enough time has passed since the most recent thread-creation failure. Before this change, the thread pool wouldn't handle a `Thread.create` exception, and the exception would get raised to whatever code happened to be calling the `Thread_pool` function that tried to create a thread, e.g. `Thread_pool.add_work`. This caused `In_thread.run` to unexpectedly raise, and in turn `In_thread.syscall` to unexpectedly raise, leading to: ``` "Fd.syscall_in_thread bug -- should be impossible" ``` Also, changed `should be impossible` text to `please report`, since there may be other lurking rare exceptions that `In_thread.syscall` can raise, and we'd like to hear about them. We rely on thread-pool-stuck detection to report problems where the inability to create threads causes the inability of the thread pool to make progress. A tweak was needed to make that work -- now the thread-pool-stuck warning is based on whether the thread pool has unstarted work, rather than on whether the thread pool has an "available thread". The latter would no longer work, since it is now possible for the thread pool to have unstarted work and to appear to have an available thread, i.e. `num_threads < max_num_threads`. ### core - Change some `Bigstring` functions to retry on `EINTR` rather than raise. The following functions (and their unsafe versions) were affected: * `read` * `really_read` * `really_recv` * `really_write` * `really_send_no_sigpipe` Some other `Bigstring` functions, like `input` and `output`, already retried on `EINTR`, so this change has precedent. All of the affected stubs raise `Bigstring.IOError` on failure, rather than `Unix_error`, which means the normal method for retrying on `EINTR` doesn't work. In particular `Async.Reader` didn't retry them, even though it was supposed to. Additionally, the documentation for the following functions was corrected to say that they raise =Unix_error= rather than =IOError=: * `pread_assume_fd_is_nonblocking` * `pwrite_assume_fd_is_nonblocking` - Eliminated global binary-to-decimal tables computed at startup for converting `Date` and `Of_day` to string. Used an efficient implementation of division by 10, rather than the `sprintf` tables in `time_internal.ml`. This result in much less allocation at startup and it is a bit faster: * before: | Name | Time/Run | mWd/Run | Percentage | |----------------|----------|---------|------------| | Date.to_string | 69.39ns | 3.00w | 100.00% | - after: | Name | Time/Run | mWd/Run | Percentage | |----------------|----------|---------|------------| | Date.to_string | 53.38ns | 3.00w | 100.00% | - Fixed `Time.Zone` tests so that they are deterministic. - Added `Iobuf.to_string_hum`, which produces a readable, multi-line representation of an iobuf, intended for debugging. - Fixed brittle unit tests of `Command`. ### core_extended - For `Flang`, added ordering to fields, and added `abs`, `min`, and `max` to the language. - Removed `Loggers` module. ### core_kernel - Added to `String` functions for substring search and replace, based on the KMP algorithm. Here are some benchmarks, comparing `Re2` for a fixed pattern, Mark's kmp from extended_string, and this implementation ("needle"). The pattern is the usual `abacabadabacabae...`. The text looks similar, with the pattern occurring at the very end. For =Re2= and =Needle= search benchmarks, the pattern is preprocessed in advance, outside of the benchmark. FWIW: I've also tried searches with pattern size = 32767, but =Re2= blows up, saying: ``` re2/dfa.cc:447: DFA out of memory: prog size 32771 mem 2664898 ``` | Name | Time/Run | mWd/Run | mjWd/Run | Prom/Run | Percentage | |-------------------------------|-----------------|---------------|-------------|----------|------------| | create_needle_15 | 102.56ns | 21.00w | | | | | re2_compile_15 | 6_261.48ns | | 3.00w | | 0.01% | | create_needle_1023 | 13_870.48ns | 5.00w | 1_024.01w | | 0.03% | | re2_compile_1023 | 107_533.32ns | | 3.03w | | 0.24% | | create_needle_8191 | 90_107.02ns | 5.00w | 8_192.01w | | 0.20% | | re2_compile_8191 | 1_059_873.47ns | | 3.28w | 0.28w | 2.37% | | create_needle_524287 | 6_430_623.96ns | 5.00w | 524_288.09w | | 14.35% | | re2_compile_524287 | 44_799_605.83ns | | 3.77w | 0.77w | 100.00% | | needle_search_15_95 | 349.65ns | 4.00w | | | | | re2_search_15_95 | 483.11ns | | | | | | mshinwell_search_15_95 | 1_151.38ns | 781.01w | | | | | needle_search_15_815 | 2_838.85ns | 4.00w | | | | | re2_search_15_815 | 3_293.06ns | | | | | | mshinwell_search_15_815 | 8_360.57ns | 5_821.07w | 0.55w | 0.55w | 0.02% | | needle_search_15_2415 | 8_395.84ns | 4.00w | | | 0.02% | | re2_search_15_2415 | 9_594.14ns | | | | 0.02% | | mshinwell_search_15_2415 | 24_602.09ns | 17_021.16w | 1.62w | 1.62w | 0.05% | | needle_search_1023_6143 | 14_825.50ns | 4.00w | | | 0.03% | | re2_search_1023_6143 | 40_926.59ns | | | | 0.09% | | mshinwell_search_1023_6143 | 81_930.46ns | 49_149.66w | 1_025.65w | 1.65w | 0.18% | | needle_search_1023_52223 | 126_465.96ns | 4.00w | | | 0.28% | | re2_search_1023_52223 | 365_359.98ns | | | | 0.82% | | mshinwell_search_1023_52223 | 527_323.73ns | 371_715.39w | 1_033.17w | 9.17w | 1.18% | | needle_search_1023_154623 | 377_539.53ns | 4.00w | | | 0.84% | | re2_search_1023_154623 | 1_001_251.93ns | | | | 2.23% | | mshinwell_search_1023_154623 | 1_499_835.01ns | 1_088_518.15w | 1_033.19w | 9.19w | 3.35% | | needle_search_8191_49151 | 115_223.31ns | 4.00w | | | 0.26% | | re2_search_8191_49151 | 559_487.38ns | | | | 1.25% | | mshinwell_search_8191_49151 | 653_981.19ns | 393_219.50w | 8_201.01w | 9.01w | 1.46% | | needle_search_8191_417791 | 976_725.24ns | 4.00w | | | 2.18% | | re2_search_8191_417791 | 4_713_965.69ns | | | | 10.52% | | mshinwell_search_8191_417791 | 4_224_417.93ns | 2_973_709.32w | 8_202.37w | 10.37w | 9.43% | | needle_search_8191_1236991 | 2_912_863.78ns | 4.00w | | | 6.50% | | re2_search_8191_1236991 | 14_039_230.59ns | | | | 31.34% | | mshinwell_search_8191_1236991 | 11_997_713.73ns | 8_708_130.87w | 8_202.47w | 10.47w | 26.78% | - Added to `Set` functions for converting to and from a `Map.t`. ```ocaml val to_map : ('key, 'cmp) t -> f:('key -> 'data) -> ('key, 'data, 'cmp) Map.t val of_map_keys : ('key, _, 'cmp) Map.t -> ('key, 'cmp) t ``` This required adding some additional type trickery to `Core_set_intf` to indicate that the comparator for a given module may or may not be fixed. - Added an optional `iter` parameter to `Container.Make`. A direct implementation of `iter` is often more efficient than defining `iter` in terms of `fold`, and in these cases, the results of `Container.Make` that are defined in terms of `iter` will be more efficient also. - Added `Int.pow` (and for other integer types), for bounds-checked integer exponentiation. ### ocaml_plugin - Added a tag to exceptions coming from the toplevel execution of plugins so that we do not confuse them with exceptions coming from the library. Also, added a function to check a plugin without executing it. And captured the common pattern of checking the compilation of a plugin in a `Command.t` offered in the library. ### sexplib - Added error locations to `Macro`-expansion errors. ## 111.08.00 ### async_extra - Added `Log.Message.add_tags`, which extends a message with a list of key-value pairs. val add_tags : t -> (string * string) list -> t ### async_kernel - Changed low-level error messages to use `Sexp.to_string_hum` rather than `to_string_mach`. ### async_ssl - Improved the propagation of SSL errors to the caller. ### async_unix - Added `Sys.when_file_changes : string -> Time.t Pipe.Reader.t`. - Added `Time.now ()` to some error messages. ### core - Improved `Command` to print a good error message if command-line parsing raises. `Command`'s `Exn.handle_uncaught` now protects the phase of parsing command-line arguments in addition to protecting the phase of running the `main` function as it did already. ### core_kernel - Added `Hashtbl.for_all` and `for_alli`. - Added `Float.to_padded_compact_string` for converting a floating point number to a lossy, compact, human-readable representation. E.g., `1_000_001.00` becomes `"1m "`. - Tweaked the form of the definition of `Blang.Stable.V1`. Removed a `type t_` that is not necessary now that we can use `nonrec` without triggering spurious warnings. ### enumerate - Restricted the signature of the `all` value generated by `with enumerate`. Now, in: type t = [ `Foo | ... ] with enumerate the type of `all` will be `t list`, instead of `[> t] list`. When taking unions of polymorphic variant types, inserted the appropriate coercions. Fixes #1. ### jenga - Fix a hang. Jenga could reach a state with a non-zero todo-count, but have no jobs actually running, and then hang in this state forever. The hang would be evident from a progress line with not all targets built and with `j=0+0` such as: todo: 17 (100406 / 100423) j=0+0 con=149956 act=3303, finish at: 16:20 ### ocaml_plugin - Use `ocamldep` to generate the dependencies of an `.ml` file, if requested. Added a function to find the dependencies of a module, but did not change the existing behavior and interface of the library if one does not choose to use this functionality. ### pa_test - Made it possible to use `<:test_eq< >>` and friends in `Core_kernel`. Removed the dependency of the code generated by `pa_test` on `Core_kernel`, so `pa_test` can be used there. ### re2 - Upgraded to upstream library version 20140304. Added options `Dot_nl` and `Never_capture`. ## 111.06.00 ### async_extra - Added `?on_wouldblock:(unit -> unit)` callback to `Udp.recvmmsg_loop` and `recvmmsg_no_sources_loop`. - For functions that create `Rpc` connections, added optional arguments: `?max_message_size:int` and `?handshake_timeout:Time.Span.t`. These arguments were already available to `Connection.create`, but are now uniformly available to all functions that create connections. ### async_kernel - Improved the performance of `Pipe.filter_map` by using batching. ### async_ssl Initial release ### async_unix - In the `Busy_pollers.t` record, made the `kernel_scheduler` field be `sexp_opaque`. Did this so that one doesn't get two copies of the kernel scheduler in sexps of the scheduler, which already has its own `kernel_scheduler` field. ### core - Added inline benchmarks for =Iobuf= and =Time=. Hera are some of the results from the new benchmarks, with some indexed tests dropped. | Name | Time/Run | mWd/Run | Percentage | |--------------------------------------|----------|---------|------------| | [time.ml:Time] Time.to_string | 848.74ns | 249.98w | 100.00% | | [time.ml:Time] Time.to_ofday | 59.66ns | 38.00w | 7.03% | | [time.ml:Time] Time.now | 39.78ns | 2.00w | 4.69% | | [time.ml:Time] Time.Zone.find_office | 83.64ns | 4.00w | 9.85% | | [time.ml:Time] Time.Span.of_hr | 3.71ns | 2.00w | 0.44% | | [time.ml:Time] Time.Span.of_min | 3.69ns | 2.00w | 0.44% | | [time.ml:Time] Time.Span.of_sec | 2.72ns | | 0.32% | | [time.ml:Time] Time.Span.of_ms | 6.02ns | 2.00w | 0.71% | | [time.ml:Time] Time.Span.of_ns | 5.98ns | 2.00w | 0.71% | | Name | Time/Run | Percentage | |------------------------------------------|----------|------------| | [iobuf.ml:Blit tests] functor blit:5 | 15.53ns | 7.66% | | [iobuf.ml:Poke tests] char:0 | 4.11ns | 2.03% | | [iobuf.ml:Poke tests] uint8:0 | 5.35ns | 2.64% | | [iobuf.ml:Poke tests] int8:0 | 4.59ns | 2.26% | | [iobuf.ml:Poke tests] int16_be:0 | 5.19ns | 2.56% | | [iobuf.ml:Poke tests] int16_le:0 | 5.14ns | 2.53% | | [iobuf.ml:Poke tests] uint16_be:0 | 5.11ns | 2.52% | | [iobuf.ml:Poke tests] uint16_le:0 | 5.12ns | 2.53% | | [iobuf.ml:Poke tests] int32_be:0 | 5.17ns | 2.55% | | [iobuf.ml:Poke tests] int32_le:0 | 4.91ns | 2.42% | | [iobuf.ml:Poke tests] uint32_be:0 | 5.73ns | 2.83% | | [iobuf.ml:Poke tests] uint32_le:0 | 5.74ns | 2.83% | | [iobuf.ml:Poke tests] int64_be:0 | 5.33ns | 2.63% | | [iobuf.ml:Poke tests] int64_le:0 | 4.93ns | 2.43% | | [iobuf.ml:Peek tests] char:0 | 5.50ns | 2.71% | | [iobuf.ml:Peek tests] uint8:0 | 4.68ns | 2.31% | | [iobuf.ml:Peek tests] int8:0 | 4.91ns | 2.42% | | [iobuf.ml:Peek tests] int16_be:0 | 5.19ns | 2.56% | | [iobuf.ml:Peek tests] int16_le:0 | 4.90ns | 2.42% | | [iobuf.ml:Peek tests] uint16_be:0 | 5.17ns | 2.55% | | [iobuf.ml:Peek tests] uint16_le:0 | 5.10ns | 2.51% | | [iobuf.ml:Peek tests] int32_be:0 | 5.17ns | 2.55% | | [iobuf.ml:Peek tests] int32_le:0 | 4.92ns | 2.42% | | [iobuf.ml:Peek tests] uint32_be:0 | 5.45ns | 2.69% | | [iobuf.ml:Peek tests] uint32_le:0 | 5.46ns | 2.69% | | [iobuf.ml:Peek tests] int64_be:0 | 6.61ns | 3.26% | | [iobuf.ml:Peek tests] int64_le:0 | 6.31ns | 3.11% | - Re-implemented `Thread_safe_queue` to improve performance and reduce allocation. The new implementation requires 3 words per element, down from the 7 words required by the old implementation. The new implementation pools elements so that they can be reused, so there is no allocation in steady-state use. The new implementation has `dequeue_exn` rather than `dequeue`, so that one can dequeue without allocating 2 words. Eliminated `create'`. One should just use `create` and explicit calls to `enqueue` and `dequeue_exn`. Eliminated `dequeue_until_empty`. One should use an explicit while loop guarded by `length` and using `dequeue_exn`. Moved `Thread_safe_queue` from `Core_kernel` to `Core`, since it's thread related. All in, there is now no allocation in a steady-state usage of enqueueing and dequeueing elements, as opposed to 9 words per `enqueue+dequeue` in the old implementation. This reduces the cost from `enqueue+dequeue` taking 166-216ns to `enqueue+dequeue_exn` taking 48-82ns (plus eliminating gc impacts). Here are some `BENCH` results, the first table being the old implementation, and the second table the new. | Name | Time/Run | mWd/Run | mjWd/Run | |------------------------------------------------------------|----------|---------|----------| | [thread_safe_queue.ml] enqueue + dequeue of immediate | 183.89ns | 9.00w | 7.02w | | [thread_safe_queue.ml] enqueue + dequeue of young object | 216.69ns | 11.00w | 9.01w | | [thread_safe_queue.ml] enqueue + dequeue_exn of old object | 166.75ns | 9.00w | 7.02w | | Name | Time/Run | mWd/Run | |--------------------------------------------------------------|----------|---------| | [thread_safe_queue.ml] enqueue + dequeue_exn of immediate | 48.20ns | | | [thread_safe_queue.ml] enqueue + dequeue_exn of young object | 81.96ns | 2.00w | | [thread_safe_queue.ml] enqueue + dequeue_exn of old object | 48.30ns | | - Changed `{Bigstring,Iobuf}.recvmmsg_assume_fd_is_nonblocking`, when no message is available, to return a negative number rather than raise. This was done for performance reasons, because raising an exception is expensive, due to the stashing of the backtrace and the string creation. - Added `Iobuf.unsafe_resize`. - Changed `Bigstring.blit` so that it doesn't release the OCaml lock on `map_file` bigstrings. The old behavior of releasing the lock for blits of (small) bigstrings involving mmapped files was problematic and inconsistent. Its cost is high, and fundamentally any access to a mapped bigstring could cause some level of blocking. - Added time-related `Arg_type.t` values to `Command.Spec`. - Added module `Type_immediacy`, which has witnesses that express whether a type's values are always, sometimes, or never immediate. This code used to be in the `Typerep_immediate` library in typerep. ### core_kernel - Added inline benchmarks for `Array` Here are some of the results from the new benchmarks, with some indexed tests dropped. | Name | Time/Run | mWd/Run | mjWd/Run | |-----------------------------------------------------|-------------|---------|-----------| | [core_array.ml:Alloc] create:0 | 13.65ns | | | | [core_array.ml:Alloc] create:100 | 99.83ns | 101.00w | | | [core_array.ml:Alloc] create:255 | 201.32ns | 256.00w | | | [core_array.ml:Alloc] create:256 | 1_432.43ns | | 257.00w | | [core_array.ml:Alloc] create:1000 | 5_605.58ns | | 1_001.01w | | [core_array.ml:Blit.Poly] blit (tuple):10 | 87.10ns | | | | [core_array.ml:Blit.Poly] blito (tuple):10 | 112.14ns | 2.00w | | | [core_array.ml:Blit.Poly] blit (int):10 | 85.25ns | | | | [core_array.ml:Blit.Poly] blito (int):10 | 107.23ns | 2.00w | | | [core_array.ml:Blit.Poly] blit (float):10 | 84.71ns | | | | [core_array.ml:Blit.Poly] blito (float):10 | 86.71ns | 2.00w | | | [core_array.ml:Blit.Int] blit:10 | 19.77ns | | | | [core_array.ml:Blit.Int] blito:10 | 23.54ns | 2.00w | | | [core_array.ml:Blit.Float] blit:10 | 19.87ns | | | | [core_array.ml:Blit.Float] blito:10 | 24.12ns | 2.00w | | | [core_array.ml:Is empty] Polymorphic '=' | 18.21ns | | | | [core_array.ml:Is empty] Array.equal | 8.08ns | 6.00w | | | [core_array.ml:Is empty] phys_equal | 2.98ns | | | | [core_array.ml:Is empty] Array.is_empty (empty) | 2.98ns | | | | [core_array.ml:Is empty] Array.is_empty (non-empty) | 3.00ns | | | - Moved `Thread_safe_queue` to core - Generalized the type of `Exn.handle_uncaught_and_exit` to `(unit -> 'a) -> 'a`. In the case where `handle_uncaught_and_exit` succeeds, it can return the value of the supplied function. It's type had been: ```ocaml val handle_uncaught_and_exit : (unit -> never_returns) -> never_returns ``` - Added `Int.round*` functions for rounding to a multiple of another int. ```ocaml val round : ?dir:[ `Zero | `Nearest | `Up | `Down ] -> t -> to_multiple_of:t -> t val round_towards_zero : t -> to_multiple_of:t -> t val round_down : t -> to_multiple_of:t -> t val round_up : t -> to_multiple_of:t -> t val round_nearest : t -> to_multiple_of:t -> t ``` These functions were added to `Int_intf.S`, implemented by `Int`, `Nativeint`, `Int32`, and `Int64`. Various int modules were also lightly refactored to make it easier in the future to implement common operators available for all modules implementing the int interface via a functor to share the code. ### jenga - Improved the error message when the same library is defined multiple times. - Fixed an issue where jenga sometimes would sometimes complain about a self cycle when `foo.ml` uses a module `Foo`. - With `-no-notifiers`, jenga doesn't use `inotify` to watch for file changes. This is useful for linting `jengaroot.ml`. - Allowed writing jenga rules which restrict dependencies from an initial conservative approximation to a more accurate set discovered after an action is run ### re2 - Added `Re2.Std`, so that one should now use `Re2` via `module Re2 = Re2.Std.Re2`. At some future date, we will rename the `Regex` module to `Re2_internal` to force the stragglers to update to the new convention. ### typerep - Renamed `Typerep` libraries for more consistency with the rest of the framework. ```ocaml Typerep_kernel --> Typerep_lib Typerep_core --> Typerep_extended Typereplib --> Typerep_experimental ``` ## 111.03.00 ### async_extra - Add `?max_connections:int` argument to `Rpc.Connection.serve`. `max_connections` is passed to `Tcp.Server.create`, and limits the number of connections that an Rpc server will accept. - Improved `Log.Rotation`: - Made `Log.Rotation.t` abstract; use `create` rather than an explicit record. - Added a `` `Dated`` `naming_scheme`. - Add `Log.Rotation.default`, for getting a sensible default rotation scheme. - Added an optional (but discouraged) option to symlink the latest log file. - Every log rotation scheme has an associated `Time.Zone.t`. - Changed the internal representation of `Log.Rotation.t`, but `t_of_sexp` is backwards compatible, so existing config files will continue to work. - Changed `Udp.bind_any` to use `Socket.bind ~reuseaddr:false`, to ensure a unique port. - Added `Tcp.Server.listening_socket`, which returns the socket the server is listening on. Changed `Tcp.Server` so that if the listening socket is closed, the server is closed. - Added to `Udp.Config.t` a `max_ready : int` field to prevent UDP receive loops from starving other async jobs. - Improved `File_tail` to cut the number of `fstat` calls in half. `File_tail` uses a stat loop to monitor a file and continue reading it as it grows. We had made two `fstat` invocations per loop iteration, using `Async.Std.Unix.with_file` which constructs an `Fd.t` and therefore does it own `fstat`. Switching to `Core.Std.Unix.with_file` with `In_thread.run` eliminated the extra `fstat`. ### async_unix - Improved `Socket.accept` to abort and return `` `Socket_closed`` when the file descriptor underlying the socket is closed. - Added to `Socket.bind` a `?reuseaddr:bool` argument, preserving the default as `true`. - Added `Fd.close_started`, which becomes determined when `close` is called on an `Fd.t`. ### bin_prot - Fixed build on ARM. ### core - Added `Unix.Syslog` module. - Changed `Command.run` to no longer ignore the first element of its `~argv` parameter. - Made `Time.Span.to_short_string` show microsecond precision. ### core_extended - Added `Set_lang`, a DSL for sets with constants, union, intersection, and difference. - In `Process`, use `epoll` rather than `select` when possible, This prevents errors when selecting on file descriptors numbered greater than `FD_SETSIZE` (1024). - Removed `Syslog` module. There is now `Unix.Syslog` in core instead; the APIs are not compatible, but they are similar. ### core_kernel - Added `Error.to_string_hum_deprecated` that is the same as `Error.to_string_hum` pre 109.61. - Changed `Error.to_string_hum` so that `Error.to_string_hum (Error.of_string s) = s`. This fixed undesirable sexp escaping introduced in 109.61 and restores the pre-109.61 behavior for the special case of `Error.of_string`. A consequence of the removal of the custom `to_string_hum` converter in 109.61 was that: ```ocaml Error.to_string_hum (Error.of_string s) = Sexp.to_string_hum (Sexp.Atom s) ``` That introduced sexp escaping of `s`. - Added to `Doubly_linked` functions for moving an element within a list. ```ocaml val move_to_front : 'a t -> 'a Elt.t -> unit val move_to_back : 'a t -> 'a Elt.t -> unit val move_after : 'a t -> 'a Elt.t -> anchor:'a Elt.t -> unit val move_before : 'a t -> 'a Elt.t -> anchor:'a Elt.t -> unit ``` - Improved `Core_map_unit_tests.Unit_tests` to allow arbitrary data in the map, not just `ints`. This was done by eta expansion. ### custom_printf - Simplified the code generated by `pa_custom_printf` to make it more readable. ### re2 - Fixed a bug with `replace_exn` and anchoring. Fixed this bug: $ R.replace_exn ~f:(fun _ -> "a") (R.create_exn "^") "XYZ";; - : string = "aXaYaZa" $ R.replace_exn ~f:(fun _ -> "a") (R.create_exn "^X") "XXXYXXZ";; - : string = "aaaYXXZ" ### textutils - Changed `Textutils.Console` to not reference `Async.Log`, so that building inline benchmark runners no longer requires building `Async`. The change makes `Textutils`, and by extension `Core_extended` and `Core_bench`, not depend on `Async`. ## 110.01.00 ### async_extra - Added `Cpu_usage.Sampler` for directly sampling CPU usage. - Fixed `Log.rotate` to never raise. - Fixed two bugs in `Log` rotation. * Log rotation had used the wrong date when checking whether it should rotate. * Made `Rotation.keep = \`At_least` delete the oldest, rather than the newest, logs. ### async_kernel - Added `Deferred.Result.map_error`. ### core - Fixed `Time` unit tests that failed in London because of timezone dependence. - Added `Iobuf.protect_window_and_bounds`, which calls a user function and restores the iobuf's bounds afterwards. - Fixed compilation on OpenBSD, which doesn't support `Unix.mcast_join`'s `?source : Inet_addr.t` argument. ### core_kernel - Changed `Queue` from a linked to an array-backed implementation. Renamed the previous implementation to `Linked_queue`. Renamed `transfer`, which was constant time, as `blit_transfer`, which is linear time. Removed `partial_iter`. One can use `with_return`. Added `singleton`, `filter`, `get`, `set`. - For `Error` and `Info`, changed `to_string_hum` to use `sexp_of_t` and `Sexp.to_string_hum`, rather than a custom string format. - Changed the output format of `Validate.errors` to be a sexp. - Added `Hashtbl.of_alist_or_error` and `Map.of_alist_or_error`. - Added `String_id.Make` functor, which includes a module name for better error messages. - Exposed `Bucket.size`. - Changed the default for `Debug.should_print_backtrace` to be `false` rather than `true`. Usually the backtraces are noise. - Removed the tuning of gc parameters built in to Core, so that the default is now the stock OCaml settings. Such tuning doesn't belong in Core, but rather done per application. Also, the Core settings had fallen way out of date, and not kept up with changes in the OCaml runtime settings. We have one example (lwt on async) where the Core settings significantly slowed down a program. - Added `Exn.raise_without_backtrace`, to raise without building a backtrace. `raise_without_backtrace` never builds a backtrace, even when `Backtrace.am_recording ()`. - Made `with_return` faster by using `Exn.raise_without_backtrace`. - Improved `with_return` to detect usage of a `return` after its creating `with_return` has returned. ### ocaml_plugin - Added `cmi`'s so that plugins can use `lazy`, recursive modules, and objects. ### pa_test - For `<:test_result< >>`, renamed `~expected` to `~expect` ### sexplib - Added `with sexp` support for mutually recursive types with common fields. For instance: ```ocaml type a = { fl : float } and b = { fl : int } with sexp ``` Closes #3 - Fixed the behavior of sexplib on `private` types. sexplib used to ignore the `private` modifier, which means generated functions had the wrong type. Now, it generates a function with the right type for `sexp_of` and refuses to generate anything for `of_sexp`. - Added `Macro.expand_local_macros`, which macro expands sexps, failing on `:include` macros. - Fixed `Macro`'s handling of nested `:include`'s which was broken with respect to paths. Prior to this fix, `:include`'s were broken with respect to the path used to resolve the filename. Including a file outside the current directory which included another file relative to that one would break. ## 109.60.00 ### async_extra - Replaced `Tcp_file.serve`'s `~port:int` argument with `Tcp.Where_to_listen.inet`. ### async_kernel - Changed the scheduler to clear a job from its queue when it runs the job, eliminating a performance regression from 109.57. Clearing avoids spurious promotion of what would otherwise be dead data associated with already-executed jobs. ### async_unix - Fixed a bug in detection of the thread pool being stuck that could overstate the amount of time the pool was stuck. It had been incorrectly reporting the duration of the thread pool being stuck if the pool had no work in it and then got enough jobs to be stuck. In that situation, the duration included the time span before the pool got stuck. If the pool had been idle long enough, this could even spuriously abort the program. ### comparelib - Fixed a type error in `with compare` of polymorphic variant inclusions. ### core - Added `Iobuf.unsafe_advance`. This can be used to benchmark inner loops that have redundant bounds checking, to see if the checks matter. For example, see the following two `advance` calls: let rec process_buffer buf ~f = let len = Iobuf.length buf in if header_len <= len then let msg_len = header_len + Iobuf.Unsafe.Peek.uint16_be buf ~pos:0 in if msg_len <= len then begin let len = msg_len - header_len in Iobuf.advance buf header_len; f (Protocol.packed_of_iobuf buf); Iobuf.advance buf len; process_buffer buf ~f end - Added `Weak_hashtbl.add_exn` and `sexp_of_t`. - Fixed `Lock_file.create` to behave correctly if the target mountpoint is out of space. Previously in this situation, `Lock_file.create` would create an empty lock and exit with exception trying to write pid/message there. Subsequent runs would not able to read pid out of empty pid file and `blocking_create` would block instead of removing defective lock. ### core_kernel - Added `Gc.keep_alive`, which ensures its argument is live at the point of the call. - Added `Sexp.With_text` module, which keeps a value and the a sexp it was generated from, preserving the original formatting. ### custom_printf - Compatibility with warning 7 (method override) ### faillib - Compatibility with warning 7 (method override) ### pipebang - Compatibility with warning 7 (method override) ### sexplib - Moved unix-specific code to a new object section, sexplib_unix ### type_conv - Compatibility with warning 7 (method override) ## 109.58.00 ### async_extra - Changed `Cpu_usage` to use `Core.Percent` instead of `float` where appropriate. - Made `Bus.unsubscribe` check that the subscriber is subscribed to the given bus. - Made `Log.t` support `with sexp_of`. - Fixed `Tcp.on_port 0` to return the port actually being listened on, like `Tcp.on_port_chosen_by_os`. Previously, a serverlistening on `Tcp.on_port 0` would have its `Tcp.Server.listening_on` as `0`, which of course is not the port the server is listening on. ### async_kernel - Renamed the `Async_core` library as `Async_kernel`, to parallel `Core_kernel`. Someday `Async_core` will depend only on `Core_kernel`, but not yet. - Added a thread-safe queue of "external actions" that is checked after each job. - Fixed a race condition in `Clock.Event.abort`. Here is the race condition: * `Clock.Event.at` adds an alarm, its value is a job (let's call it job1) with this run function: ```ocaml let fire () = t := Happened; Ivar.fill ready `Happened; ``` * later a job (let's call it job2) aborting the clock event is queued in the async scheduler * in the same cycle, the `Timing_wheel.advance_clock` fires the alarm and job1 scheduled * at this point: + job1 and job2 are still pending + the alarm was removed so it is invalid + the clock event is still in the state `Waiting` * job2 is executed before job1: the clock event is still in the `Waiting` state, so the abort tries to remove the alarm from the timing wheel: CRASH The bugfix is for `Clock.Event.abort` to check if the alarm has already been removed from the timing wheel and if so, don't remove it again. - Changed `Monitor.try_with` when run with ``~rest:`Ignore``, the default, so that the created monitor is detached from the monitor tree. The detached monitor has no parent, rather than being a child of the current monitor. This will eliminate recently observed space leaks in `Sequencer_table` and `Throttle`, like: ```ocaml let leak () = let seq = Throttle.Sequencer.create () in let rec loop n = Throttle.enqueue seq (fun () -> loop (n + 1); Deferred.unit ) |> don't_wait_for in loop 0 ``` - Changed Async's scheduler to pool jobs rather than heap allocate them, decreasing the cost of a job by 30-40%. Changed the main scheduler queue of jobs to be an `Obj_array.t` that is essentially a specialized `Flat_queue` (the specialization was necessary for speed). Also, cleaned up the scheduler run-job loop. With these changes, the cost of a simple job decreases significantly (30-40%), across a range of live data sizes. Here are the nanoseconds-per-job numbers for a microbenchmark with the old and new approaches. | num live jobs | old ns/job | new ns/job | |---------------|------------|------------| | 1 | 74 | 53 | | 2 | 75 | 47 | | 4 | 76 | 41 | | 8 | 63 | 39 | | 16 | 62 | 38 | | 32 | 61 | 37 | | 64 | 61 | 37 | | 128 | 60 | 37 | | 256 | 60 | 38 | | 512 | 60 | 38 | | 1024 | 60 | 39 | | 2048 | 61 | 40 | | 4096 | 67 | 41 | | 8192 | 65 | 45 | | 16384 | 75 | 56 | | 32768 | 115 | 67 | | 65536 | 171 | 108 | | 131072 | 255 | 158 | | 262144 | 191 | 130 | | 524288 | 216 | 139 | | 1048576 | 238 | 152 | See async/bench/nanos\_per\_job.ml for the benchmark. - Removed `debug_space_leaks` from Async's internals. It hadn't been used in years. ### async_unix - Improved fairness of the async scheduler with respect to external threads, including I/O done in external threads. The change is to add a thread-safe queue of "external actions" that is checked after each job. Previously, when a job given to `In_thread.run` finished, `In_thread.run` would take the async lock, fill the result ivar and run a cycle. The problem is that in some situations, due to poor OS scheduling, the helper thread never had a chance to grab the lock. Now, `In_thread.run` tries to take the lock: - if it can it does as before - if it can't it enqueues a thunk in the external actions queue and wakes up the scheduler With this change, the helper thread doing an `In_thread.run` will always quickly finish once the work is done, and the async scheduler will fill in the result ivar as soon as the current job finishes. - Fixed `Epoll_file_descr_watcher.invariant` to deal with the timerfd, which has the edge-triggered flag set. - Added `Writer.write_gen`, a generic functor for blitting directly to a writer's buffer. ### core - Added `Debug.should_print_backtrace : bool ref`, to control whether `Debug.am*` functions print backtraces. - Added to `Float` inline benchmarks. - Moved all of the `Gc` module into `Core_kernel`. Part of the `Gc` module used to be in `Core` because it used threads. But it doesn't use threads anymore, so can be all in `Core_kernel`. - Improved `Iobuf` support for copying to/from strings and bigstrings. The new modules are `Iobuf.{Consume,Peek}.To_{bigstring,string}`. They match a `Blit`-like interface. We don't yet implement the `Blit` interface in all applicable contexts, but do use `Blit.Make` and expose some of the operations we want in the forms we expect them to take under a `Blit` interface. - Added `Linux_ext.Timerfd.to_file_descr`. - Added to `Time.next_multiple` an optional argument to control whether the inequality with `after` is strict. - Added `Time.Zone.local`, a lazily cached `Time.Zone.machine_zone ()`. This is the first stage in a plan to make explicit timezones more pervasive. First, they are made more convenient, by replacing the relatively wordy `Time.Zone.machine_zone ()` with `Time.Zone.local`. This involves making the underlying timezone type lazy. The next stage will be to remove `machine_zone` and use `Time.Zone.local` everywhere instead. Then (it is hoped) instead of `of_local_whatever`, we just say e.g. `of_date_ofday Time.Zone.local` and currently-implicitly-local functions will be able to switch over to explicit timezones without becoming too verbose. - Added `Timing_wheel.Alarm.null`. - Made `Unix.File_descr.t` have `with sexp`. Closes janestreet/async_unix#3 - Fixed OpenBSD compilation failures in C stubs. - Fixed `Lock_file.is_locked` to require read permission, not write permission, on the lock file. - Added to `Unix.mcast_join` an optional `?source:Inet_addr.t` argument. From pull-request on bitbucket: https://bitbucket.org/janestreet/core/pull-request/1/receive-source-specific-multicast/diff ### core_bench - Added support for saving inline benchmark measurements to tabular files for easy loading into Octave. ### core_extended - Cleaned up the `Stats_reporting` module ### core_kernel - Moved all of the `Gc` module into `Core_kernel`. Part of the `Gc` module used to be in `Core` because it used threads. But it doesn't use threads anymore, so can be all in `Core_kernel`. - Made `Stable.Map` and `Set` have `with compare`. - Added `String.rev`. Closes janestreet/core#16 We will not add `String.rev_inplace`, as we do not want to encourage mutation of strings. - Made `Univ_map.Key` equivalent to `Type_equal.Id`. - Added `Univ.view`, which exposes `Univ.t` as an existential, `type t = T : 'a Id.t * 'a -> t`. Exposing the existential makes it possible to, for example, use `Univ_map.set` to construct a `Univ_map.t`from a list of `Univ.t`s. This representation is currently the same as the underlying representation, but to make changes to the underlying representation easier, it has been put in a module `Univ.View`. ### core_profiler - Added support for saving inline benchmark measurements to tabular files for easy loading into Octave. ## 109.55.00 ### async_extra - Added `Udp.recvmmsg_no_sources_loop`, a specialization of `recvmmsg_loop` for improved performance. This improvement was driven by profiling at high message rates. ### async_unix - Fixed `Scheduler.is_running` to not initialize the scheduler. - Added `Writer.make_write`, which is a general function for blitting directly to a writer's buffer. - Added `Writer.transfer'`, which is like `Writer.transfer` but allows async actions in the callback. This was requested in pull request #1. - Added `Writer.write_iobuf`, which blits the contents of an iobuf directly to a writer's buffer. ### core - Fixed building on FreeBSD and OpenBSD. - Added `with typerep` to many `Core` types. - Made `open Core.Std` support `with typerep`. - Added `Iobuf.recvmmsg_assume_fd_is_nonblocking_no_options`, a specialization of `recvmmsg_assume_fd_is_nonblocking` for improved performance. This improvement was driven by profiling at high message rates. - Changed `Unix.Rlimit.virtual_memory` be an `Or_error.t`, for platforms where it is undefined. ### core_bench - Improved `bench.mli`'s generated docs and added some usage examples. This also partly satisfies issue #3. - Added the ability to create groups of benchmarks with a common prefix. For example, the prefix "Perf" below is created in created using `create_group`: ```ocaml let command = Bench.make_command [ Bench.Test.create ~name:"Time.now" (fun () -> ignore (Time.now ())); ... Bench.Test.create_group ~name:"Perf" [ Bench.Test.create ~name:"TSC.now" ... ``` and the output shows: ``` Estimated testing time 7s (7 benchmarks x 1s). Change using -quota SECS. ┌───────────────────────────────────────────┬──────────┬─────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ Percentage │ ├───────────────────────────────────────────┼──────────┼─────────┼────────────┤ │ Time.now │ 41.38ns │ 2.00w │ 16.72% │ │ Calibrator.calibrate │ 247.42ns │ 32.00w │ 100.00% │ │ Perf/TSC.now │ 7.84ns │ │ 3.17% │ │ Perf/TSC.to_time │ 9.35ns │ 2.00w │ 3.78% │ │ Perf/TSC.to_time (TSC.now ()) │ 13.22ns │ 2.00w │ 5.34% │ │ Perf/TSC.to_nanos_since_epoch │ 10.83ns │ │ 4.38% │ │ Perf/TSC.to_nanos_since_epoch(TSC.now ()) │ 14.86ns │ │ 6.00% │ └───────────────────────────────────────────┴──────────┴─────────┴────────────┘ ``` ### core_extended - Added `Service_command.acquire_lock_exn`, for acquiring a service's lock. ### core_kernel - Added `with typerep` to many `Core` types. - Changed `Flat_queue` to raise if the queue is mutated during iteration. - Improved `Map.merge` to run in linear time. ### core_profiler - Improved `bench.mli`'s generated docs and added some usage examples. This also partly satisfies issue #3. - Added the ability to create groups of benchmarks with a common prefix. For example, the prefix "Perf" below is created in created using `create_group`: ```ocaml let command = Bench.make_command [ Bench.Test.create ~name:"Time.now" (fun () -> ignore (Time.now ())); ... Bench.Test.create_group ~name:"Perf" [ Bench.Test.create ~name:"TSC.now" ... ``` and the output shows: ``` Estimated testing time 7s (7 benchmarks x 1s). Change using -quota SECS. ┌───────────────────────────────────────────┬──────────┬─────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ Percentage │ ├───────────────────────────────────────────┼──────────┼─────────┼────────────┤ │ Time.now │ 41.38ns │ 2.00w │ 16.72% │ │ Calibrator.calibrate │ 247.42ns │ 32.00w │ 100.00% │ │ Perf/TSC.now │ 7.84ns │ │ 3.17% │ │ Perf/TSC.to_time │ 9.35ns │ 2.00w │ 3.78% │ │ Perf/TSC.to_time (TSC.now ()) │ 13.22ns │ 2.00w │ 5.34% │ │ Perf/TSC.to_nanos_since_epoch │ 10.83ns │ │ 4.38% │ │ Perf/TSC.to_nanos_since_epoch(TSC.now ()) │ 14.86ns │ │ 6.00% │ └───────────────────────────────────────────┴──────────┴─────────┴────────────┘ ``` ### pa_bench - Added support for inline benchmarks using the `BENCH` syntax, similar to `TEST`. This feature allows users to specify inline benchmarks in any library. One can specify a benchmark using the following syntax: ```ocaml BENCH "name" = expr ``` In the above, the value of `expr` is ignored. This creates a benchmark for `expr`, that is run using the `inline_benchmarks_runner` script from the command-line. This workflow is similar to that of inline unit tests. One can specify benchmarks that require some initialization using `BENCH_FUN`. For example: ```ocaml BENCH_FUN "name" = let t = create () in (fun () -> test_something t) ``` The function returned on the RHS of `BENCH_FUN` should have type `unit -> unit`. One can specify benchmarks that have a variable parameter using `BENCH_INDEXED`. For example: ```ocaml BENCH_INDEXED "Array.create" len [1;10;100;1000] = (fun () -> Array.create ~len 0) ``` The above snippet measures the time taken to create arrays of different length. Indexed tests are useful in highlighting non-linearities in the execution time of functions. We can group benchmarks together into modules and the output of `inline_benchmarks_runner` will reflect this grouping. ```ocaml BENCH_MODULE "Blit tests" = struct ..some benchmarks.. end ``` For examples of all of the above see `core_gc.ml` and `core_array.ml`. Only the generated `inline_benchmarks_runner.exe` depends on `Core_bench` and other libraries. The library that includes the the benchmarks itself does not have a dependency on `Core_bench`. Doing this is important so that we can add benchmarks to `Core` and still avoid cyclic dependencies. Finally, it is important to note that adding inline benchmarks should have little effect on the execution or module initialization time. ## 109.53.00 ### async - Bump version number ### async_extra - Added module `Bus`, which is an intraprocess "broadcast" communication mechanism. - Added `Tcp.to_inet_address` and `to_unix_address`. - Added `Tcp.to_socket` which creates a `Tcp.where_to_connect` from a `Socket.Address.Inet.t`. - Module `Weak_hashtbl` is now implemented as a wrapper around `Core.Weak_hashtbl`. No intended change in behavior. ### async_unix - Changed the scheduler to calibrate `Time_stamp_counter` every second. - Improved error messages in the scheduler when `epoll` functions raise. - Made `Scheduler.reset_in_forked_process` close the `epoll` file descriptor. ### bin_prot - Bump version number ### core - Added `Linux_ext.Epoll.close`. - Added `Weak_hashtbl` module, moved from `Async`. It had only been in `Async` to use `Async`'s finalizers. The move to `Core` exposes a bit more with respect to finalization so that one can still implement `Async.Weak_hashtbl`, as well as do other things (e.g. use `Weak_hashtbl` in `Incremental`, which does not use `Async`). Simplified the implementation of `Weak_hashtbl` to eliminate "entry ids". They were only used to avoid removing a table entry that was in use. But there is a more direct way to test for that -- just check if the weak is `None` or `Some`. - Added an autoload file for utop - Disabled warning 40 in corebuild ### core_bench - Fixed a bug in `Core_bench` where the linear regression was sometimes supplied with spurious data. This showed up when doing custom regressions that allow for a non-zero y-intercept. ### core_extended - Fixed `Flang` and `Olang` to round-trip via sexps, i.e. `(t_of_sexp (sexp_of_t t)) = t`. ### core_kernel - Added `Float.to_string_round_trippable`, which produces a string that loses no precision but (usually) uses as few digits as possible. This can eliminate noise at the end (e.g. `3.14` not `3.1400000000000001243`). Benchmarks: New sexp: | Name | Time/Run | mWd/Run | Percentage | |------------------------|----------|---------|------------| | new Float.sexp_of 3.14 | 463.28ns | 6.00w | 48.88% | | new Float.sexp_of e | 947.71ns | 12.00w | 100.00% | Old sexp: | Name | Time/Run | mWd/Run | Percentage | |------------------------|----------|---------|------------| | old Float.sexp_of 3.14 | 841.99ns | 178.00w | 98.03% | | old Float.sexp_of e | 858.94ns | 178.00w | 100.00% | Much of the speedup in the 3.14 case comes from the fact that `format_float "%.15g"` is much faster than `sprintf "%.15g"`. And of course the above does not capture any of the benefits of dealing with shorter strings down the road. Here are some detailed benchmarks of the various bits and pieces of what's going on here: | Name | Time/Run | mWd/Run | Percentage | |-------------------------------------|------------|---------|------------| | format_float '%.15g' 3.14 | 335.96ns | 2.00w | 32.71% | | format_float '%.17g' 3.14 | 394.18ns | 4.00w | 38.38% | | format_float '%.20g' 3.14 | 459.79ns | 4.00w | 44.77% | | format_float '%.40g' 3.14 | 638.06ns | 7.00w | 62.13% | | sprintf '%.15g' 3.14 | 723.71ns | 165.00w | 70.47% | | sprintf '%.17g' 3.14 | 803.44ns | 173.00w | 78.23% | | sprintf '%.20g' 3.14 | 920.78ns | 176.00w | 89.66% | | sprintf '%.40g' 3.14 | 990.09ns | 187.00w | 96.41% | | format_float '%.15g' e | 357.59ns | 4.00w | 34.82% | | format_float '%.17g' e | 372.16ns | 4.00w | 36.24% | | format_float '%.20g' e | 434.59ns | 4.00w | 42.32% | | format_float '%.40g' e | 592.78ns | 7.00w | 57.72% | | sprintf '%.15g' e | 742.12ns | 173.00w | 72.26% | | sprintf '%.17g' e | 747.92ns | 173.00w | 72.83% | | sprintf '%.20g' e | 836.30ns | 176.00w | 81.43% | | sprintf '%.40g' e | 1_026.96ns | 187.00w | 100.00% | | valid_float_lexem 12345678901234567 | 76.29ns | 9.00w | 7.43% | | valid_float_lexem 3.14 | 9.28ns | 5.00w | 0.90% | | float_of_string 3.14 | 130.19ns | 2.00w | 12.68% | | float_of_string 1234567890123456.7 | 184.33ns | 2.00w | 17.95% | | to_string 3.14 | 316.47ns | 7.00w | 30.82% | | to_string_round_trippable 3.14 | 466.02ns | 9.00w | 45.38% | | to_string e | 315.41ns | 7.00w | 30.71% | | to_string_round_trippable e | 949.12ns | 15.00w | 92.42% | - Replaced `Float.min_positive_value` with `min_positive_normal_value` and `min_positive_subnormal_value`. - Added some functions to `Float.O`: `abs`, `of_float`, and `Robustly_comparable.S`. - Small improvements to the `Heap` module. Implemented `Heap.iter` directly rather than in terms of `fold`. In `heap.ml`, fixed the idiom for using `Container.Make`. - Added an `Int.O` and other `Int*.O` modules, with arithmetic operators, infix comparators, and a few useful arithmetic values. - Added `Int.( ~- )`, for unary negation. - Added `Pool.unsafe_free`. - Added `Percent` module. ### core_profiler - Fixed a bug in `Core_bench` where the linear regression was sometimes supplied with spurious data. This showed up when doing custom regressions that allow for a non-zero y-intercept. ### ocaml_plugin Bump version number ### pa_ounit - Bump version number ### pa_test - Bump version number ### patdiff - Bump version number ### re2 - Bump version number ### sexplib - Changed `sexp_of_float` to (usually) use as few digits as possible, without losing precision. - Split the part of `Sexplib` that depends on `Num` into a separate library, `Sexplib_num`. This was done to eliminate the dependence of `Core_kernel` on `Num`, which is not usable on javascript. ### textutils - Bump version number ### type_conv Bump version number ## 109.52.00 ### async_extra - Added module `Cpu_usage`, which publishes CPU-usage statistics for the running process. - Fixed `Sequencer_table.enqueue` so that there is no deferred between finding the state and calling the user function. ### async_kernel - Changed `Pipe.iter_without_pushback` to never call `f` after `Pipe.close_read` has been called. The new behavior is like `Pipe.iter`. - Changed the implementation of `Pipe.fold_gen` and `Pipe.transfer_gen` to be analogous to `Pipe.iter_without_pushback`, and in particular to process as many elements as possible before calling `values_available`. ### async_unix - Fixed a bug in `Unix.mkstemp`, which had a race because it used `Fd.create` in a thread. This bug affected `Writer.with_file_atomic`, `save`, `save_lines`, and `save_sexp`, and could cause corruption of the async scheduler data structures. - Changed async to never do `set_nonblock` on `std{in,out,err}`, which allows `Core` I/O to use the standard file descriptors simultaneously with async. Before this change, the Core I/O libraries could (and sometimes did) fail due to `Sys_blocked_io`. - Changed `Pipe.iter_without_pushback` to never call `f` after `Pipe.close_read` has been called. The new behavior is like `Pipe.iter`. Changed the implementation of `Pipe.fold_gen` and `Pipe.transfer_gen` to be analogous to `Pipe.iter_without_pushback`, and in particular to process as many elements as possible before calling `values_available`. - Added `?expand_macros:bool` argument to `Reader.load_sexp*` functions, to support the new `Sexplib` macros. - Added an optional argument to `Process.run` to accept nonzero exits as successful runs. ### core - Added `Unix.File_descr.equal`. - Added `Lock_file.Nfs.unlock`, the `Or_error` version of `unlock_exn`. - Improved the detail of the error messages exposed by `Lock_file.Nfs.create{,_exn}`. - Added `Unix.set_mcast_ifname`, to control the interface used for UDP multicast traffic. Added bindings for setsockopt `IP_MULTICAST_IF`. See 6.3 in: http://www.tldp.org/HOWTO/Multicast-HOWTO-6.html - Changed `Command` argument processing to treat a single dash (`-`) as an anonymous argument rather than a flag. This change follows the unix convention of passing `-` as an anonymous argument meaning `stdin`. - Added more bin-prot support to `Iobuf`: `Consume.bin_prot`, `Fill.bin_prot`, `Peek.bin_prot`, `Poke.bin_prot`. Framing doesn't do much for `Iobuf`, so these are to be more standard, unframed accessors, as opposed to `fill_bin_prot`. - Added `Core.Debug.am`, `amf`, and `ams`, for outputting debugging messages showing the current source-code position. Unfortunately, these aren't available in `Core.Std.Debug`, but only in `Core.Debug`. That will be fixed in 109.49. - Made `Time_stamp_counter` compile on non x86-64 platforms. - Made `Core.Std.Debug` be `Core.Debug` rather than `Core_kernel.Debug`. This exposes the `Debug.am*` functions added in 109.48. ### core_bench - Exposed an extensible form of `make_command` so that inline-benchmarking and the other tools can add more commandline flags. - A significant rewrite of `Core_bench`. The rewrite provides largely the same functionality as the older version. The most visible external change is that the API makes it clear that `Core_bench` performs linear regressions to come up with its numbers. Further, it allows running user-specified multivariate regressions in addition to the built in ones. The underlying code has been cleaned up in many ways, some of which are aimed at improving the implementation of inline benchmarking (the `BENCH` syntax, which has not yet been released). ### core_extended - Removed `Sexp.load_sexp_with_includes`; one should use the new `Sexplib.Macro` functions. - Added `Blang`-like languages `Flang` and `Olang`. * `Flang` -- terms over a field. * `Olang` -- predicates over an ordered set. ### core_kernel - Added to `Binary_packing` module functions for packing and unpacking signed 64-bit ints in little- and big-endian. - Changed the `Comparator` interfaces to no longer have `with bin_io` or `with sexp`. The `Comparator` interfaces are now just about having a comparator. Also, renamed `type comparator` as `type comparator_witness`. And, removed `Comparator.S_binable`, since one can use: ```ocaml type t with bin_io include Comparator.S with type t :` t ``` - Changed `Comparator.Make` to return a module without a type `t`, like other `*able` functors, This made it possible to remove the signature constraint when `Comparator.Make` is applied. - Made `Comparable.S_binable` be like `Comparable.S` and not have `type t with sexp`. The following two functors now fail to type check: ```ocaml module F1 (M : Comparable.S ) : sig type t with sexp end ` M module F2 (M : Comparable.S_binable) : sig type t with sexp end ` M ``` whereas previously `F1` was rejected and `F2` was accepted. - Changed the `Monad.Make` functor to require a `val map` argument. This was done since we almost always want a specialized `map`, and we kept making the mistake of not overriding the generic one in the three places needed. Added `Monad.map_via_bind`, which one can use to create a standard `map` function using `bind` and `return`. - Removed unnecessary signature constraints on the result of applying `Monad.Make`. Some time ago, `Monad.Make` changed from returning: ```ocaml S with type 'a t ` 'a M.t ``` to returning: ```ocaml S with type 'a t :` 'a M.t ``` so we no longer need to constrain the result of `Monad.Make` at its uses to remove `t`. - Changed `String.exists` and `String.for_all` to iterate by increasing index rather than decreasing. - Added `with compare` to module `Ref`. - Made `Flags` be `Comparable`, with the order consistent with bitwise subset. - Cleaned up the implementation of `Union_find`. Improvemed the code in `union_find.ml`: * Removed an assert false. * do not reallocate a parent node during compress. This should result in more stability for sets memory wise. * Added implementation notes. * Renamed internal variant constructors. * Added unit tests. - Added `Float.O`, a sub-module intended to be used with local opens. The idea is to be able to write expressions like: ```ocaml Float.O.((3. + 4.) > 6. / 2.) ``` This idiom is expected to be extended to other modules as well. - Added a `sexp_of_t` converter to `Type_equal.Id`. - Replaced `Univ.Constr` with `Type_equal.Id`. - Added `Debug.eprintf`, analogous to `eprint` and `eprints`. ### core_profiler - Exposed an extensible form of `make_command` so that inline-benchmarking and the other tools can add more commandline flags. - A significant rewrite of `Core_bench`. The rewrite provides largely the same functionality as the older version. The most visible external change is that the API makes it clear that `Core_bench` performs linear regressions to come up with its numbers. Further, it allows running user-specified multivariate regressions in addition to the built in ones. The underlying code has been cleaned up in many ways, some of which are aimed at improving the implementation of inline benchmarking (the `BENCH` syntax, which has not yet been released). ### pa_ounit - Added `-stop-on-error` flag to `inline_test_runner`, to stop running tests at the first failure. This is useful if the remaining tests are likely to fail too or just long to run. ### pa_test - Added an optional `?message` argument to `<:test_eq<>>`, `<:test_result<>>` and `<:test_pred<>>`. ### re2 - Fixed bugs in `Re2.Regexp.find_all` and `Re2.Regexp.find_first`. * memory leaks on errors * unlikely garbage in their return values, or even segfaults (especially unlikely for `find_first`) ### sexplib - Added a `Macro` module, with `load_sexp*` functions that support file includes and templates. The following new syntaxes are supported: ```ocaml (:include filename) (:let f (arg1 ... argn) sexp1 ... sexpn) (:use f (arg1 valn) ... (argn valn)) (:concat a1 ... an) ``` - Added support to `with sexp` for a subset of GADTs. The new support is for types that use existentially quantified variables or plain variants written with GADT syntax. Existentially quantified variables still have to be wrapped with `sexp_opaque` generate compiling code. - Fixed a type error in the code generated by `with sexp` in some cases of variant inclusions. ### textutils - Added `Console.Log` module for writing colorized `Async.Log` messages. ### type_conv - Removed comments from pretty-printed types in `type_conv` error messages. ## 109.47.00 ### async_extra - Added `with sexp` to `Log.Output.machine_readable_format` and `format`. ### async_kernel - Fix a bug introduced in `Monitor.error` in 109.28, in which the error wasn't seen unless someone is listening to the monitor. ### async_unix - Added `Socket.Address.Inet.to_host_and_port`. - Changed `Fd_by_descr` so that it actually calls `Bounded_int_table.invariant`. ### bin_prot - Compilation fix for 32-bit systems ### core - Added `Time_stamp_counter` module, which has fast (< 10 nanos) access to the hardware time-stamp counter. This module provides the fast function `Time_stamp_counter.now ()` which is our best effort high-performance cycle counter for a given platform. For x86 systems this retrieves the CPU's internal time stamp counter using the `RDTSC` instruction. For systems that do not have a RDTSC instruction, we fallback to using `clock_gettime(CLOCK_MONOTONIC)`. Here is a benchmark of execution time in nanos and allocations in words: ``` Name Time/Run Minor ------------------------------- ---------- ------- Time.now 39.02 2.00 TSC.now 7.54 TSC.to_time 4.88 2.00 TSC.to_time (TSC.now ()) 8.54 2.00 TSC.to_time_nanos 4.49 TSC.to_time_nanos(TSC.now ()) 8.95 Calibrator.calibrate 279 34.00 ``` Historically, the rate of increment of the TSC (sometimes referred to as the TSC frequency) varied based of CPU overclocking, temperature, load etc. On modern Intel CPU's the TSC is expected to be stable. On Linux systems, the "constant_tsc" in `/proc/cpuinfo` indicates that the machine has a stable TSC rate. While this module assumes that the TSC is relatively stable, it can adapt to small variations in the TSC frequency. - Changed `Daemon.daemonize` and `daemonize_wait` to leave the `umask` alone by default. Previously, these had alwasy set the umask to `0`, which means that all app-harness programs and all binaries run from grass were creating world-writeable (`0666`) files by default. ### core_kernel - Added `Error.to_info` and `of_info`. - Significantly sped up `Float.iround_*` functions. For `iround_down_exn`, the new version appears to use about 25% of the CPU time of the old version on non-negative floats. For negative floats it uses around 60% of the CPU time. | Name | Time (ns) | % of max | |-------------------------|-----------|----------| | old iround_down_exn pos | 15.02 | 95.23 | | new iround_down_exn pos | 3.75 | 23.75 | | old iround_down_exn neg | 15.78 | 100.00 | | new iround_down_exn neg | 9.80 | 62.10 | - Added `Binary_searchable.Make` functor to core, and used it in `Array` and `Dequeue`. - Fixed `Bounded_int_table` to match `Invariant.S2`. - Added to `Pool` support for `10-`, `11-`, and `12-` tuples. - Added functions to the `Gc` module to get usage information without allocating. Added these functions, all of type `unit -> int`: ``` minor_collections major_collections heap_words heap_chunks compactions top_heap_words ``` They all satisfy: ```ocaml Gc.f () = (Gc.quick_stat ()).Gc.Stat.f ``` They all avoid the allocation of the stat record, so one can monitor the garbage collector without perturbing it. ### pa_test - Added new syntax `<:test_result<>>`, which is similar to `<:test_eq<>>`. `test_result` is meant for the common case of comparing some hardcoded expected value against one returned by some code you're testing. In this case you can now write, for example: ```ocaml <:test_result> 3 ~expected:4 ``` which behaves exactly the same way as: ```ocaml <:test_eq> 3 4 ``` except that the error message is something like: ``` ("got unexpected result" ((got 3) (expected 4) (Loc "File \"z.ml\", line 44, characters 25-25"))) ``` rather than: ``` ("comparison failed" (3 vs 4 (Loc "File \"z.ml\", line 44, characters 25-25"))) ``` The additional labels on the values help identify the problem earlier. ### type_conv - Made `type nonrec` work when a type has both an equation and a representation. For example: ```ocaml type t = A of t module T = struct type nonrec t = t = A of t end ``` ## 109.45.00 ### async_extra - Added `?abort:unit Deferred.t` argument to `Lock_file.waiting_create`, `Lock_file.Nfs.waiting_create` and `critical_section`. ### async_kernel - Removed internal `Backpatched` module. Async used to use this module, but it doesn't anymore. ### async_unix - Added `Fd.every_ready_to` `Fd.interruptible_every_ready_to` which register a callback to be called every time the fd becomes ready. These can significantly reduce allocation. - Renamed `Fd.ready_to_interruptible` as `Fd.interruptible_ready_to`. - Changed `Fd.ready_fold` to use `Fd.interruptible_ready_to`, to improve its performance. ### core - Added `Core.Std.phys_same`, which is like `phys_equal`, except has a more general type. ```ocaml val phys_equal : 'a -> 'a -> bool val phys_same : _ -> _ -> bool ``` `phys_same` is useful when dealing with existential types, and one has a packed value and an unpacked value that one wants to check are physically equal. One can't use `phys_equal` in such a situation because the types are different. - Added `Iobuf.set_bounds_and_buffer` and `set_bounds_and_buffer_sub`, which make it easier to use with zero allocation. ```ocaml (** [set_bounds_and_buffer ~src ~dst] copies bounds (ie limits + window) and shallowly copies the buffer from [src] to [dst]. [read_write] access is required on [src] because the caller might have [read_write] access to [dst], and would after the call then effectively have [read_write] access to [src]. *) val set_bounds_and_buffer : src:(read_write, _) t -> dst:(_, seek) t -> unit (** [set_bounds_and_buffer_sub ?pos ?len ~src ~dst ()] is a more efficient version of: [set_bounds_and_buffer ~src:(Iobuf.sub ?pos ?len src) ~dst]. [set_bounds_and_buffer ~src ~dst] is not the same as [set_bounds_and_buffer_sub ~dst ~src ()], because the limits are narrowed in the latter case. *) val set_bounds_and_buffer_sub : ?pos:int -> ?len:int -> src:(read_write, _) t -> dst:(_, seek) t -> unit -> unit ``` - Added `?timeout:Time.Span.t` argument to `Lock_file.blocking_create`, `Lock_file.Nfs.blocking_create` and `critical_section`. ### core_extended - Fixed `Core_extended.Sys.groups` to use `Unix.Group.getbygid` rather than `Unix.Group.getbygid_exn`. This handles when a group is deleted and its gid remains in the cache, which causes `Unix.Group.getbygid_exn` to fail because the gid no longer resolves to a group. ### core_kernel - Changed `Blang.bind` to short-circuit `And`, `Or`, and `If` expressions. For example if `bind t1 f ` false`, then `bind (and_ t1 t2) ` false`, and will not evaluate `bind t2 f`. - Renamed `Dequeue.get` as `get_opt`, and `get_exn` as `get`, to be consistent with other containers which don't use the `_exn` suffix for subscripting exceptions. - Removed `Source_code_position.to_sexp_hum`, in favor of `sexp_of_t_hum`, which works smoothly with `with sexp`. - Changed `Flat_queue_unit_tests` to run `Flat_queue.invariant`, which was mistakenly not being used. ### ocaml_plugin - Made executables link without error even if no archive is embedded in them. This is often the desired behavior (for inline tests of libraries using transitively ocaml-plugin for instance). ## 109.44.00 ### async_extra - Fixed a time-based race condition in `Log` rotation. ### async_kernel - Documented that `Throttle.enqueue t f` never runs `f` immediately, and added unit tests. ### bin_prot - Remove "unwrapped" pointers used by `Bin_prot`, with the bug from 109.41 fixed. Unwrapped pointers cannot coexist with the remove-page-table optimization. Removed all the C stubs for reading/writing and used instead either the new primitives of the next OCaml or standard OCaml code reading/writing integers byte by byte. Since we don't have unsafe/safe functions anymore but only safe ones, removed all the `bin_{read,write}_t_` functions. Also renamed `bin_read_t__` to `__bin_read_t__` for the same reason as sexplib: to avoid confusion with the function generated for `t_` and hide it in the toplevel. ### core - Added `val Day_of_week.num_days : from:t -> to_:t -> int`. - Added `Time.of_date_ofday_precise` and `Time.Zone.next_clock_shift`, to deal with clocks going forward and backward. Due to clocks going forward/backward, some local times occur twice, and some never occur at all. `Time.of_date_ofday_precise` identifies these cases and returns all of the relevant information. - Added accessors for `Unix.Cidr`: `base_address` and `bits`. ```ocaml (** Accessors. - [base_address 192.168.0.0/24 ` 192.168.0.0] - [bits 192.168.0.0/24 ` 24]. *) val base_address : t -> Inet_addr.t val bits : t -> int ``` ### core_kernel - Implemented `Dequeue.iter` directly, instead of as a specialization of `fold`. Extended random tests to cover `iter`. ## 109.42.00 ### async_extra - Fixed `Log.Blocking` so that when async is running it writes the message in syslog before failing with an exception. ### async_kernel - In `ASYNC_CONFIG`, replaced `alarm_precision` and `timing_wheel_level_bits` with `timing_wheel_config`. This parallels the new `Timing_wheel.Config` module. ### async_unix - Added `Reader.drain`. ```ocaml val drain : t -> unit Deferred.t ``` - Added `Writer.with_close`. ```ocaml val with_close : t -> f:(unit -> 'a Deferred.t) -> 'a Deferred.t ``` ### bin_prot - Backed out the changes introduced in 109.41 ### core - Removed `Zone` from `Core.Std`; use `Time.Zone` instead. - Removed `Time.Date`; use `Date` instead. - Improved the performance of `Piecewise_linear` by using arrays with binary search on indices. The previous implementation `Piecewise_linear` used `(float * float) list` (a list of (x,y) pairs) to represent piecewise linear functions, with a linear search through the knots to evaluate the function at a point. This type is now: ```ocaml { x : float array ; y : float array } ``` and the implementation uses binary search to identify the correct array index. Here are the costs of the `get` function under the old (list) and new (array) implementations for various numbers of knots: ``` knots | old | new ------+------+----- 1 | 11ns| 12ns 2 | 18ns| 14ns 5 | 27ns| 19ns 100 | 182ns| 38ns 1000 |1974ns| 52ns ``` - Added module `Time.Ofday.Zoned`, which is a pair of an `Time.Ofday.t` and a `Time.Zone.t`. - Added `with compare` to `Time.Zone.Stable.t`. - Added `Timing_wheel` functionality. * Added `Config` module, which combines `alarm_precision` and `timing_wheel_level_bits`. * Removed the need to supply a dummy value to `create`. * Added `mem` and `clear` functions. * Added functions for dealing with the interval number: `interval_num`, `now_interval_num`, `interval_num_start`, `add_at_interval_num`. This makes it easier to use a timing wheel with `int` interval numbers, which are more efficient than than `float` times. ### core_kernel - Added `Array.is_sorted_strictly` and `List.is_sorted_strictly`. ```ocaml val is_sorted_strictly : 'a t -> cmp:('a -> 'a -> int) -> bool ``` - Added `Array.find_consecutive_duplicate` and `List.find_consecutive_duplicate`. ```ocaml val find_consecutive_duplicate : 'a t -> equal:('a -> 'a -> bool) -> ('a * 'a) option ``` - Added `Array.truncate`, which changes (shortens) the length of an array. ```ocaml val truncate : _ t -> len:int -> unit ``` - Improved the debugging message in `Bounded_int_table.remove` to show the data structure's details. - Added `Float.iround_lbound` and `iround_ubound`, the bounds for rounding to `int`. - Added `Hashtbl.similar`, which is like `equal`, but allows the types of the values in the two tables to differ. - Added `Pool.Pointer.phys_compare`, which is analagous to `phys_equal`, and does not require an argument comparison function. ```ocaml val phys_compare : 'a t -> 'a t -> int ``` - Exposed that `Pool.Debug`'s output types are the same as its input types. ## 109.41.00 ### async_parallel Rename library from Parallel to Async_parallel ### async_unix - Changed `Reader` to treat more `errno` responses to a `read()` system call as `EOF` rather than raise. The following are now treated as `EOF`. ``` ENETDOWN ENETRESET ENETUNREACH ETIMEDOUT ``` - Improved the error message that async prints when the thread pool is stuck, including the time of the error. ### bin_prot - Remove all uses of "unwrapped" pointers Unwrapped pointers cannot coexist with the remove-page-table optimization. Removed all the C stubs for reading/writing and used instead either the new primitives of the next OCaml or standard OCaml code reading/writing integers byte by byte. Since we don't have unsafe/safe functions anymore but only safe ones, removed all the `bin_{read,write}_t_` functions. Also renamed `bin_read_t__` to `__bin_read_t__` for the same reason as sexplib: to avoid confusion with the function generated for `t_` and hide it in the toplevel. ### core - Added `Command.Spec.map_anon` and `map_flag`. ```ocaml (** [map_flag flag ~f] transforms the parsed result of [flag] by applying [f] *) val map_flag : 'a flag -> f:('a -> 'b) -> 'b flag (** [map_anons anons ~f] transforms the parsed result of [anons] by applying [f] *) val map_anons : 'a anons -> f:('a -> 'b) -> 'b anons ``` - Fixed `Unix.open_flag` to compile with OCaml 4.01. It needed the additional constructor `O_CLOEXEC`. ### core_bench - Columns that have a `+` prefix are now always displayed, whereas columns that don't are displayed only if they have meaningful data. - Added the ability to reload saved metrics (benchmark test data) so that bench can re-analyze them. ### core_kernel - Added `Map.of_alist_reduce`. This function is a natural addition alongside `of_alist_fold`. Its advantage is that it does not require an `init` argument like `of_alist_fold`. Moreover, it does not involve `option` types, like `List.reduce` does in order to handle the empty list case. ### core_profiler - Columns that have a `+` prefix are now always displayed, whereas columns that don't are displayed only if they have meaningful data. - Added the ability to reload saved metrics (benchmark test data) so that bench can re-analyze them. ### ocaml_plugin - Added option `-strict-sequence`, which is set to `true` by default. ### type_conv - Fixed the generated code of `typerep` and `sexplib` on sum types containing `True` or `False`. Without this fix, `typerep` would wrong constructor names for `Blang.t`, for instance. `Variantslib` has the same problem but applying the same fix there would still not make the generated code compile because the generated code would contain labels and variable named `true` or `false`. Other syntax extensions should not be affected because they don't build strings from constructor names. ## 109.40.00 ### async_extra - Added to `Udp.Config` the ability to stop early, via `stop : unit Deferred.t`. ### async_unix - Added value `Socket.Type.unix_dgram`, which represents a UNIX domain datagram socket. - Added UDP socket functionality: `Socket.Opt.mcast_{loop,ttl}` and `Socket.mcast_{join,leave}`. - Improved `Fd.ready_fold` to accept `?stop:(unit Deferred.t)` rather than `?stop:('a -> bool)`. ### core_extended - Added `Stats_reporting.Delta`, for recording deltas of values. ## 109.39.00 ### async_unix - Added "thread-local" storage, `Scheduler.{find,with}_local`, for `LWT` emulation. ```ocaml (** [with_local key value ~f] runs [f] right now with the binding [key ` value]. All calls to [find_local key] in [f] and computations started from [f] will return [value]. *) val with_local : 'a Univ_map.Key.t -> 'a option -> f:(unit -> 'b) -> 'b (** [find_local key] returns the value associated to [key] in the current execution context. *) val find_local : 'a Univ_map.Key.t -> 'a option ``` ### core_bench - Added support for additional predictors like minor/major GCs and compactions, using multi-variable linear regression. Replaced linear regression with multi-variable linear regression. The original algorithm estimated the cost of a function `f` by using a linear regression of the time taken to run `f` vs the number of runs. The new version adds the ability to include additional predictors such as minor GCs, compactions etc. This allows a more fine-grained split-up of the running costs of a function, distinguishing between the time spent actually running `f` and the time spent doing minor GCs, major GCs or compactions. - Added a forking option that allows benchmarks to be run in separate processes. This avoids any influence (e.g. polluting the cache, size of live heap words) they might otherwise have on each other. ### core_kernel - Implemented `Heap.iter` directly instead of in terms of `fold`. ### core_profiler - Added support for additional predictors like minor/major GCs and compactions, using multi-variable linear regression. Replaced linear regression with multi-variable linear regression. The original algorithm estimated the cost of a function `f` by using a linear regression of the time taken to run `f` vs the number of runs. The new version adds the ability to include additional predictors such as minor GCs, compactions etc. This allows a more fine-grained split-up of the running costs of a function, distinguishing between the time spent actually running `f` and the time spent doing minor GCs, major GCs or compactions. - Added a forking option that allows benchmarks to be run in separate processes. This avoids any influence (e.g. polluting the cache, size of live heap words) they might otherwise have on each other. ## 109.38.00 ### async_extra - In `Rpc`, exposed accessors for binary protocol values. For example, this allows one to write a wrapper for `Pipe_rpc` that allows for the easy re cording and replaying of values the come over the pipe. ### async_unix - Added `Reader.of_pipe` and `Writer.of_pipe`, for converting from `string Pipe.t`'s. These can be used to add arbitrary transformations (e.g. encryption, compression) to code that expects plain file- or socket-based readers and writers. ## 109.37.00 ### core - Command.run now calls Exn.handle_uncaught so you don't have to. - Fixes for building on FreeBSD. - Fixed Blang to build with OCaml 4.01. In blang.mli: Blang.t is a private variant type, Blang.Stable.V1.t is a private variant type, and client code knows Blang.t = Blang.Stable.V1.t. Previously, this was done in a strange way, using with type 'a t = private 'a t on the signature of Blang.Stable.V1. In addition to being strange, this line no longer builds in OCaml 4.01, which caused problems for building Real World Ocaml. This patch changed the code to something much more straightforward, although not quite so straightforward as we expect to be able to achieve once a nonrec bug is fixed. ### core_kernel - Added Core.Std.Poly as a short name for Core.Std.Polymorphic_compare. - Exposed module Core.Std.Decimal. ## 109.36.00 ### async_unix - Added `Process.run_lines`, which runs a process and returns stdout as a list of strings. ### core_extended - In `Sexp` module, added ability to expand and compress bash-like brace wildcards. ### core_kernel - Made `Hashtbl.Poly.hash` equal `Caml.Hashtbl.hash`, and changed changed `String.hash` and `Float.hash` to match OCaml's hash function. Previously, `Core.Poly.hash` had been defined as: ```ocaml let hash x = hash_param 10 100 x ``` This fell out of sync with OCaml's hash function, and was providing worse hash values. - Fixed `Obj_array.singleton` to never create a float array. Also made it clearer that `Obj_array.copy` could never create a float array. - Changed `Pool.create` to allow zero-length pools. Previously, `Pool.create ~capacity:0` had raised, which made it easy to write code that blows up on edge cases for no apparent reason. For example, `Heap.copy` was written in a way that copying an empty heap would blow up (regardless of its capacity), and `Heap.of_array` would also blow up on an empty array. - Added `String.split_lines`. ```ocaml (** [split_lines t] returns the list of lines that comprise [t]. The lines do not include the trailing ["\n"] or ["\r\n"]. *) val split_lines : t -> t list ``` ### pa_ounit - Simplified so that it does not generate unnecessary top-level bindings. It had been hiding quite a few `unused import` warnings. ### textutils - In `Ascii_table` module, added support for displaying table using ASCII characters instead of Unicode. This is motivated by the need to use `Core_bench` in contexts where the extended ASCII character set is not suitable for displaying tables. The default style is the following: ``` ┌─────────────────┬───────────┬──────────┐ │ Name │ Time (ns) │ % of max │ ├─────────────────┼───────────┼──────────┤ │ quick_stat │ 93.11 │ 100.00 │ │ counters │ 33.24 │ 35.70 │ │ allocated_bytes │ 37.03 │ 39.77 │ └─────────────────┴───────────┴──────────┘ ``` The new style is as follows: ``` $ ./test_bench.exe gc -q 0.5 -ascii Estimated testing time 1.5s (change using -quota SECS). Name Time (ns) % of max ----------------- ----------- ---------- quick_stat 93.17 100.00 counters 34.56 37.09 allocated_bytes 37.06 39.78 ``` ## 109.35.00 ### async_extra - Added module `Async.Udp`, aimed at high-performance UDP applications. - Added module `Lock_file.Nfs`, which wraps the functions in `Core.Std.Lock_file.Nfs`. ### async_kernel - Added new configuration options for Async, `max_inter_cycle_timeout` and `max_num_jobs_per_priority_per_cycle`. ```ocaml val max_inter_cycle_timeout : Time.Span.t val max_num_jobs_per_priority_per_cycle : int ``` These are configurable as usual via `ASYNC_CONFIG`. - Added an `ASYNC_CONFIG` option to debug the `Shutdown` module. - Added `find` and `find_map` to `Deferred.Monad_sequence`. ### async_unix - Made some configuration possible via additional optional arguments to `go_main`. ```ocaml ?max_num_open_file_descrs:int ?max_num_threads:int ``` - Made some aspects of the async scheduler configurable via functions in `Scheduler`. ```ocaml val set_check_invariants : bool -> unit val set_detect_invalid_access_from_thread : bool -> unit val set_max_inter_cycle_timeout : Time.Span.t -> unit val set_record_backtraces : bool -> unit ``` - Added a dynamic check in `Pipe` that a consumer is used with the correct pipe. Specifically, check that a consumer supplied to a read operation on a `Pipe.Reader.t` was previously created by `add_consumer` with that same reader. - Renamed `Pipe.fold` as `fold_without_pushback` and added `Pipe.fold` with an analogous type to `Pipe.iter`. - Fixed a bug in `Pipe.merge`, which did not always close the resulting pipe when the merge was finished. This had prevented medusa regtests from working correctly. - In `Writer`, changed the default `buffer_age_limit` for files to ``Unlimited`. THis was done for the same reason that we treat files specially in flush on close -- slowness will likely be resolved eventually with a file, unlike with a socket. ### core - Added `or_error` functions in `Unix.Exit_*` types to `unit Or_error.t`. This makes it easier to deal with combining with infix operators `>>=?` and `>>|?` ### core_extended - Added stable versions of types contained in the `Selector` module. ### core_kernel - Added `with compare` to `List.Assoc.t`. - Made `Pooled_hashtbl.create` handle non-positive and very large `size`s in the same way as `Core.Hashtbl`. - Added `is_error`, `is_ok`, and `does_raise` to `Core.Std`. ```ocaml let is_error ` Result.is_error let is_ok ` Result.is_ok val does_raise : (unit -> _) -> bool ``` - Reimplemented `Heap` and reworked the interface to be more standard. The new implementation uses pairing heaps and `Pool`. - Added a module `Pool.Unsafe`, which is like `Pool`, except that `create` doesn't require an initial value. This makes it unsafe to access pool pointers after they have been freed. But it is useful for situations when one isn't able to create an initial value, e.g. `Core.Heap`. - Removed `Time.to_localized_string` and `Time.to_string_deprecated`. These did not include the time-zone offset. Instead, use `Time.to_string` and `Time.to_string_abs`, which do include the time-zone offset. - Exposed that `Int63.t = private int` on 64-bit machines. This lets the OCaml compiler avoid `caml_modify` when dealing with it. - Added `Gc` stat functions that don't allocate: `Gc.minor_words`, `Gc.major_words`, `Gc.promoted_words`. Added the following `Gc` functions: ```ocaml Gc.minor_words : unit -> int Gc.major_words : unit -> int Gc.promoted_words : unit -> int ``` such that these functions cause no allocations by themselves. The assumption being that 63-bit ints should be large enough to express total allocations for most programs. On 32-bit machines the numbers may overflow and these functions are not as generally useful. These functions were added because doing memory allocation debugging with `Gc.quick_stat` as the primary means of understanding allocations is difficult: tracking down allocations of the order of a few hundred words in a hot loop by putting in lots of `quick_stat` statements becomes too intrusive because of the memory allocations they cause. Here are some benchmarks of existing `Gc` functions and the newly added functions: ``` $ ./test_bench.exe -q 2 -clear name time +alloc +time-err Estimated testing time 12s (change using -quota SECS). ``` | Name | Time (ns) | 95% ci | Time R^2 | Minor | |-----------------|-----------|-------------|----------|-------| | quick_stat | 92.16 | +0.72 -0.64 | 1.00 | 23.00 | | counters | 33.63 | +0.26 -0.23 | 1.00 | 10.00 | | allocated_bytes | 37.89 | +0.34 -0.32 | 1.00 | 12.00 | | minor_words | 4.63 | +0.03 -0.02 | 1.00 | | | major_words | 4.36 | +0.02 -0.02 | 1.00 | | | promoted_words | 4.10 | +0.03 -0.02 | 1.00 | | ### faillib - New syntax extension to get improved location information on calls to `failwiths`. Added `failwithp`, which is like `failwiths` but takes a `_here_` argument: ```ocaml val failwithp : Lexing.position -> string -> 'a -> ('a -> Sexp.t) -> _ ``` `pa_fail` automatically converts `failwiths` into `failwithp _here_` If you don't want to compile with `pa_fail`, then you can manually write `failwithp _here_`. Otherwise, running the same source through `pa_fail` gets you location information for free. ### ocaml_plugin - Changed the execution of plugin's toplevel to run in async instead of `In_thread.run`, unless a config parameter says otherwise. ### textutils - Added new module `Textutils.Text_graph` for plotting text graphs on a terminal. Here is an example density plot for `minor_words` from `mcquote-stats.data`. ``` (354 (3%) values outside the range, bucket size is 38.58) 74.00 3786 |----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----| 112.58 0 | 151.17 0 | 189.75 0 | 228.33 0 | 266.92 0 | 305.50 0 | 344.08 0 | 382.67 0 | 421.25 0 | 459.83 3653 |----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+- 498.42 4 | 537.00 1055 |----+----1----+----2----+-- 575.58 308 |----+--- 614.17 70 |- 652.75 404 |----+----1 691.33 71 |- 729.92 32 | 768.50 18 | 807.08 151 |--- 845.67 17 | 884.25 4 | 922.83 8 | 961.42 66 |- 1000.00 0 | ``` Each `-` is approximately 37.860 units. ## 109.34.00 ### async_kernel - Added a function to `Pipe` that merges a list of sorted pipes `val Pipe.merge : 'a Reader.t list -> cmp:('a -> 'a -> int) -> 'a Reader.t` - Improved the performance of `Ivar.squash` by removing the allocation of the list of ivars. Instead, `squash` does one loop to find the end of the chain and a second loop to set all the indirections. - Changed `Scheduler.run_cycles_until_no_jobs_remain` to raise an exception if there is an unhandled exception when it finishes. ### async_unix - Changed the scheduler to detect when the thread pool is stuck, i.e. when all threads are blocked and haven't made progress. Added default handlers for when the thread pool is stuck, and the ability for the user to configure their own handlers. - Changed low-level I/O to not use nonblocking I/O for char devices. This fixes a problem due to `epoll` not working with `/dev/null`. For example: ```ocaml let () = Reader.read_line (Lazy.force Reader.stdin) >>> fun _ -> shutdown 0 ;; let () = never_returns (Scheduler.go ()) ``` had failed when run like: ``` + ./z.exe t option`. This is more convenient to use than `Backtrace.get`, which is an `Or_error.t`. - Moved functions for dealing with finalizers into the `Gc.Expert` module. This was done to make people be very explicit and sure that they want finalizers, which are very hard to use because they essentially introduce multithreading semantics. One should typically use async finalizers. - Eliminated the thread that had been used to sequentialize all finalizers. ### core_extended - Improved `Sexp.Diff`. Labeled arguments, put them in the right order (old before new), and rework the code to follow the same convention, and produce the output where deletions precede insertions. ### core_kernel - Added a new module, `Flat_queue`, which is a queue of flat tuples. This is essentially: ```ocaml ('a1 * .. * 'aN) Queue.t ``` However the queue is implemented as a `Flat_array`, so the tuples are layed out flat in the array and not allocated. - Improved `Bounded_int_table.remove`'s error message when it detects an internal inconsistency. - Added new `Debug` module. - Changed `Invariant.invariant` to take `_here_` rather than a string. - Made `Float` satisfy the `Identifiable` interface. ### pa_test - Added new quotation expanders `<:test_eq< >>` and `<:test_pred< >>`. These can be used to reduce boilerplate in testing code. - `<:test_eq< type >>` expands to a function of type ```ocaml ?(f = (fun x y -> <:compare< type >> x y = 0)) -> ?here:Lexing.position list -> type -> type -> unit ``` that throws a nice exception if the values are not equal according to `f`. - `<:test_pred< type >>` expands to a function of type ```ocaml ?here:Lexing.position list -> (type -> bool) -> type -> unit ``` that throws a nice exception if the predicate fails on the value. ## 109.33.00 ### async_extra - Change `Log.Global` to by default send all output, including `` `Info ``, to `stderr`. Replaced `Log.Output.screen` with `Log.Output.stderr`. There is now also and `Log.Output.stdout`. ## 109.32.00 ### async_extra - Added `Dynamic_port_writer`. `Dynamic_port_writer` solves the problem of communicating a dynamically selected tcp port from a child process to its parent. ### async_kernel - Improved the batching of `Pipe.fold` and other `Pipe` functions that handle batches. Previously, such functions used a loop with `Pipe.read`. This didn't batch as well as it might. If values were put in the pipe after the `read` becomes determined but before the values are handled, then they wouldn't be handled until the next batch. Now, batching functions use `values_available` and then pull elements out of the pipe synchronously after waking up. This makes the batch as large as possible. ### core - Normalized `Command`'s help messages. Made anonymous argument names uppercase and subcommand names lowercase. - In `Iobuf`, added duals to `flip` and `snapshot` to work on the high end of the window. `flip` has been renamed to `flip_lo`. The dual of `flip_lo` is the newly added `flip_hi`, and shifts the window forward to continue reading, rather than back to switch from writing to reading, as `flip_lo` does. `flip_hi`, in practice, needs snapshots of the upper bound of the window, we split `Snapshot` into `Lo_bound` and `Hi_bound` and introduced bounded versions of `flip_lo`, `compact`, and `flip_hi` to support buffers which are only partially filled, but have substructure, like packet buffers. Here's the new API. ```ocaml module type Bound = sig type ('d, 'w) iobuf (** Expose =t = private int= only if a =t= is stored in a mutable data structure somewhere and leads to a measurable =caml_modify= performance problem. *) type t with sexp_of val window : (_, _) iobuf -> t val limit : (_, _) iobuf -> t val restore : t -> (_, seek) iobuf -> unit end module Lo_bound : Bound module Hi_bound : Bound val flip_lo : (_, seek) t -> unit val bounded_flip_lo : (_, seek) t -> Lo_bound.t -> unit val flip_hi : (_, seek) t -> unit val bounded_flip_hi : (_, seek) t -> Hi_bound.t -> unit ``` ### core_bench - Changed `-save` to output compaction information. - Added indexed tests. These are benchmarks of the form `int -> unit -> unit`, which can be profiled for a list of user specified `int`s. ### core_kernel - Added `val Option.merge: 'a t -> 'a t -> f:('a -> 'a -> 'a) -> 'a t`. - Added `val Validate.failf : ('a, unit, string, t) format4 -> 'a`. - In `Validated.Make_binable`, made it possible to apply the validation function when un-bin-io-ing a value. - Added `module Pooled_hashtbl` to `module type Hashable`. This is an alternative implementation to `Core.Hashtbl`. It uses a standard linked list to resolve hash collisions, and `Pool` to manage the linked-list nodes. ### core_profiler - Changed `-save` to output compaction information. - Added indexed tests. These are benchmarks of the form `int -> unit -> unit`, which can be profiled for a list of user specified `int`s. ### ocaml_plugin - Fixed the slow and memory-consuming compilation of > 100MB `.c` files generated by `ocaml_embed_compiler`. This was done by having them contain one big string instead of one big array. - Added more unused-value warnings in plugins. If { `Ui` , `M` } are the modules that constitute a given plugin of expected module type `S`, then previously we generated a file like: ```ocaml module Ui : sig ... end = struct ... end module M : sig ... end = struct ... end let () = ##register (M : S) ``` Doing that, we did not get unused variables: 1. for the toplevel of `Ui` if `Ui` does not have a `mli`. 2. for unused values of `Ui` and `M` if they have an `mli` exporting them. OCaml plugin now allows one to get these warnings. Since (2) is rather annoying for utils kind of file, this comes only if a config flag is enabled. ### re2 - Fixed a bug in the C bindings that could cause a segfault. Fixed a bug where `mlre2__create_re` in C can give OCaml a freed C string. The bug was in: ```c if (!compiled->ok()) { compile_error = compiled->error().c_str(); delete compiled; compiled = NULL; caml_raise_with_string(*caml_named_value("mlre2__Regex_compile_failed"), compile_error); } ``` This is in `mlre2__create_re` if we fail to compile the regular expression. Notice how we delete the re object before we use the pts to its' error string. (Notice that in C++ `c_str()` returns a pointer to the internal data of the string object it does NOT create a copy and `error()` just returns a reference to the regular expression objects error string member `*error_`). So if `caml_raise_with_string` has to allocate on the heap to create the exception or the copy of the string that might invalidate the ptr before we will copy it. ## 109.31.00 ### async_unix - Renamed `Reader.read_one_chuck_at_a_time_until_eof` as `Reader.read_one_chunk_at_a_time`, and added a new case to the `handle_chunk` callback. The name change is to reflect that one can stop early, before EOF. The new `handle_chunk` case allows one to specify the number of bytes that were read in the `Stop` case. Also, changed `read_one_chunk_at_a_time` to use `do_read` instead of just locking the file without unlocking it. This allows subsequent read operations to read from the file. ### core_kernel - Renamed some functions in module `Lazy`: dropped the `lazy_` prefix from `is_val`, `from_val`, and `from_fun`. ### ocaml_plugin - Fixed OCaml Plugin on CentOS 5 -- it had problems because the generated c files did not end with a newline. - Finished the transition from `Command_deprecated` to `Command`. ## 109.30.00 ### async_kernel - Added function `Throttle.kill`. `Throttle.kill` aborts all jobs that have been enqueued but not started, and immediately aborts all jobs subsequently enqueued. Split out `Throttle` debugging and unit-testing code into their own modules. - Changed the semantics of `Throttle.enqueue` on a dead throttle so that the exception is sent to the monitor rather than raised synchronously. This gives more uniform treatment to the race between enqueueing a job and an already running job raising. Now the enqueued job is always aborted, whether enqueued before or after the raise. - Added an ASYNC_CONFIG option to print debug messages when `Monitor.send_exn` is called. This is useful when one is debugging an application in which an exception is being unexpectedly swallowed. - Allow one to dynamically configure the behavior of =Monitor.try_with=. This is to allow experimentation with different handling of asynchronous exceptions after `Monitor.try_with` has become determined. ### async_unix - Changed aync's scheduler to use `epoll` rather than `select` by default. This is based on a dynamic test to see whether `timerfd_create` works. - Added support for "busy polling". This runs a thread that busy loops running user-supplied polling functions. The busy-loop thread is distinct from async's scheduler thread, but it acquires the async lock so that the user-supplied function can do ordinary async operations, e.g. fill an ivar. Busy polling is useful for a situation like a shared-memory ringbuffer being used for IPC. One can poll the ringbuffer with a busy poller, and then when data is detected, fill some ivar that causes async code to handle the data. - Added `Async.Fd.clear_nonblocking`. This clears the nonblocking bit on the file descriptor underlying the fd, and causes async to treat the fd as though it doesn't support nonblocking I/O. This is useful for applications that want to share a file descriptor between async and non-async code and want to avoid `Sys_blocked_io` being seen by the non-async code. ### core - Created submodule `Core.Signal.Expert` module. This is for functions previously in `Core.Signal` that introduce multithreading semantics, and are hence hard to reason about and should only be used by experts. ### core_bench - Report compaction stats ### core_kernel - Added module, `Core.Blit`, which codifies the type, implementation, and unit-testing of blit functions. - Added `remove_zero_flags` option to `Flags.Make`, to support flags that are zero. This fixes a problem with `Flags.Make` on CentOS 5 because `O_CLOEXEC` is `0` there. - Removed `Pool.None`, and folded `Pool.Obj_array` into `Pool` proper. `Pool.None` had its day, but `Pool.Obj_array` dominates it, so we don't need it any more. ### core_profiler - Report compaction stats ### ocaml_plugin - Support for Mac OSX Removed the dependency of `ocaml-plugin` on `objcopy` and `/proc`. ## 109.28.00 ### async_extra - Fixed an error message in `Versioned_rpc` that was swapping which versions were supported by the caller and the callee. ### async_kernel - Eliminated a messy dependency cycle in `async_core`, so that `Monitor.t` no longer contains a `Tail.t`. `async_core` was messy because of cycle between the following types: ``` Execution_context --> Scheduler --> Ivar --> Deferred --> Tail --> Monitor ``` This messiness caused the need for the `Raw` signature, for the various `Scheduler_dependent` functors, for making various types polymorphic in `execution_context`, and then instantiating the polymorphism later. The cause of the problem was that `Monitor` contained a `Tail`. We eliminated that, so that there is no longer a cycle, and defined things in order: ``` Monitor Execution_context Job Scheduler Ivar Deferred Tail Stream ``` Replaced the `errors` tail from the monitor type: ```ocaml errors : (exn, 'execution_context) Raw_tail.t; ``` with a list of handlers: ```ocaml mutable error_handlers : (exn -> unit) list; ``` Cleaned up all the messiness caused by the cycle -- eliminated the `Raw` signature, the `Scheduler_dependent` functors, and the unnecessary polymorphism. Cleaned up the long standing annoyance with `Async.Stream`, in which we couldn't expose the `next` sum type and people had to use annoying `Stream.of_raw` calls. `Stream.of_raw` is now gone. ### core - Moved `Timing_wheel` from `Zero`. ### core_extended - In `Shell` functions, made the amount of captured stderr/stdout configurable. ### core_kernel - Moved all the contents of the `Zero` library into `Core`, mostly into `Core_kernel`. We want to start using `Zero` stuff more in `Core`, which couldn't be done with `Zero` as a separate library. Everything moved into `Core_kernel`, except for `Timing_wheel`, which moved into `Core` proper, due to its dependence on `Time`. - Renamed `Flat_tuple_array` as `Flat_array`. - Added `Dequeue.{front,back}_index_exn` These are more efficient than using `{front,back}_index` and then `Option.value_exn`. - Exposed `Core.String.unsafe_{get,set}`. ### pipebang - Fixed `pa_pipebang` in the toplevel. `pa_pipebang` had registered an AST filter for implementations but not for the toplevel. ### type_conv - Fixed an issue with `type_conv` in the toplevel. Used AST filters for the `_no_unused_value_warning_` machinery. `type_conv` modifies the parser but it didn't work well in the toplevel. Removed the `parsing_mli` reference, an instead always add the special `_no_unused_value_warning_` type and just strip it for signature items. ## 109.27.00 ### async_extra - Added function `Versioned_typed_tcp.Client.shutdown`. - Added new module `Sequencer_table`, which is a table of `Throttle.Sequencer`'s indexed by keys. ### async_kernel - Fixed `Monitor.catch_stream` to prevent missing a synchronous exception. ### async_unix - Fixed a performance problem in the scheduler due to repeated calls of `Timing_wheel.min_elt`. `Timing_wheel.min_elt` is an important part of async, since the scheduler calls it once per cycle to know when to timeout for `epoll` or `select`. This causes a problem if `min_elt` is slow and called repeatedly, which happens in an application where the next clock event is a second out, and yet there are lots of cycles per second. `Timing_wheel.min_elt` now caches the minimum element, which eliminates the problem. - Fixed async's clock to work on 32-bit machines. With the change to `Timing_wheel` in 109.22, async no longer worked on 32-bit machines, due to the clock overflowing. This is because it is initialized to `Time.epoch`, and can only handle 6 days. The fix now in place is to start the clock at `Time.now ()` rather than `Time.epoch`. - Added many functions to `Async.Sys` so that it looks more like `Core.Sys`. - Changed `Reader.read_one_chunk_at_a_time_until_eof` to not destroy the reader buffer. Destroying the buffer failed if user code held on to the buffer. ### comparelib - Changed how `with compare` treats `option`'s so that `None < Some`, like `Pervasives.compare`. Previously, `with compare` had treated `Some < None`. ### core - Disabled use of `recvmmssg`, which isn't available on our CentOS 5 machines. - Defined `Option.compare` using `with compare` so that their comparisons are consistent. - Cleaned up the `Dequeue` module's interface and implementation. The interface now matches the conventions used elsewhere in `Core`. The new implementation is also cleaner and more efficient. - Reimplemented the `Stack` module to improve performance, and renamed the old implementation as `Linked_stack`. The new `Stack` is implemented with this type: ```ocaml type 'a t ` { dummy : 'a; mutable length : int; mutable elts : 'a array; } ``` `Linked_stack` is implemented with this type: ```ocaml type 'a t ` { mutable length : int; mutable elts : 'a list; } ``` Using an array rather than a linked list is a more efficient and traditional implementation. Pushing to the stack now does not require allocation, except in the rare case when the stack grows. One downside is that `Stack.clear` is now O(n) rather than O(1). This change also eliminates the `Stack.Empty` exception, so any code matching on that exception should fail to compile, and should be changed to depend on option-returning `top` and `pop` operations. - Improved `Lock_file.Nfs`. * Allowed an arbitrary message to be stored and retreived. * Fixed a case where `create` might throw an exception. * Delete both files used for locking when we unlock. - Split `Core` into `Core_kernel` and `Core`. - `Core_kernel` is composed of all modules of `Core` that do not depend on unix or threads, and `Core` contains the rest and depends on `Core_kernel`. The goal is to have a very portable standard library that can especially run on exotic environment such as Javascript. So that code that directly refers to `Core` (rather than `Core.Std`) for modules that have moved to `Core_kernel`, we included "proxy" modules in `Core` that simply include the corresponding module from `Core_kernel`. - Fixed `Core.Flags` to build on 32-bit machines. It had contained a unit test with an integer literal too large to be accepted by the compiler when building on a 32-bit machine. ### core_bench - Added R^2 error estimation. Adding this metric should give us a sense of how closely the given values fit a line. Even dots that are fairly scattered can give tight confidence intervals. We would like to have to number to have a sense of how much noise we have. ### core_extended - In module `Sexp`, changed and renamed `load_includes_in_sexp`. From: ```ocaml val load_includes_in_sexp : ?max_depth:int -> Sexp.t -> Sexp.t ``` to: ```ocaml val load_sexp_with_includes: ?max_depth:int -> ?buf:string -> string -> Sexp.t ``` - Added function `Sexp.Diff.to_string`. - Previously the only option was to print to `Out_channel`. ### core_profiler - Added R^2 error estimation. Adding this metric should give us a sense of how closely the given values fit a line. Even dots that are fairly scattered can give tight confidence intervals. We would like to have to number to have a sense of how much noise we have. ### custom_printf - Added missing registration of `custom_printf`'s Camlp4 filter so that it works in the toplevel. ### pa_ounit - Removed comments from test names displayed by `pa_ounit`. Before: ``` File "iobuf.ml", line 141, characters 0-34: <<(** WHEN YOU CHANGE THIS, CHANGE iobuf_fields `...`>> threw ("Iobuf.create got nonpositive len" 0). ``` After: ``` File "iobuf.ml", line 141, characters 0-34: <> threw ("Iobuf.create got nonpositive len" 0). ``` ## 109.24.00 ### async_extra - Made the `Caller_converts` interface in `Versioned_rpc` use the `Connection_with_menu` idea introduced in `Both_convert`. ### async_kernel - Reworked the `Throttle` module. Made both `Throttle.t` and `Throttle.Sequencer.t` instances of the same type, using a phantom type to distinguish them. Removed all `Throttle.Sequencer` functions -- one can now use the `Throttle` functions directly. Added new functions: ```ocaml (*** [max_concurrent_jobs t] returns the maximum number of jobs that [t] will run concurrently. *) val max_concurrent_jobs : (_, _) t_ -> int (*** [num_jobs_running t] returns the number of jobs that [t] is currently running. It is guaranteed that if [num_jobs_running t < max_concurrent_jobs t] then [num_jobs_waiting_to_start t = 0]. That is, the throttle always uses its maximum concurrency if possible. *) val num_jobs_running : (_, _) t_ -> int (*** [num_jobs_waiting_to_start t] returns the number of jobs that have been [enqueue]d but have not yet started. *) val num_jobs_waiting_to_start : (_ , _) t_ -> int (*** [capacity_available t] becomes determined the next time that [t] has fewer than [max_concurrent_jobs t] running, and hence an [enqueue]d job would start immediately. *) val capacity_available : (_, _) t_ -> unit Deferred.t ``` Replaced the `Pipe` used inside a `Throttle` with a `Queue`, and simplified the implementation. This fixed a bug in `num_jobs_waiting_to_start`, which could have missed a job that was not in the pipe but had not started. ### async_unix - Changed `Reader.close` so that it frees the reader buffer using `Bigstring.unsafe_destroy`. This is an improvement over the previous situation, in which the buffer wasn't freed until its finalizer fired. - Fixed a bug in `Reader.read_bin_prot`. It was missing a try-with that could cause it to raise without cleaning up the reader. ### core - Added module `Core.Iobuf`, a module aimed at zero-copy I/O. An iobuf is a bigstring combined with a position and length, that defines a contiguous region of bytes in the bigstring. Operations on an iobuf operate relative to start of the region and cannot look outside the region. - Added module `Core.Validated` for creating an abstract type that ensures a validation function has been run. - Added function `Bigstring.recvmmsg_assume_fd_is_nonblocking`, which allows one to read a number of UDP packets with a single system call. - Fixed `Unix.create_process` on OSX. ## 109.23.00 ### core - Exposed `Core.Std.Flags` module. - Made the `Heap` module implement `Container.S1`. - Added module `Ref.Permissioned`, which is a ref with `read_only` / `read_write` access control. - Exposed the unique id in `Type_equal.Id`. This allows, e.g. hash tables indexed by unique ids. - Removed the use of `Obj.magic` from the implementation of `Type_equal.Id.same`. It is not needed because the `Id.t` contains a `Uid.t` and we can just use `Uid.equal`. ## 109.21.00 ### async_unix - Added `Unix.remove`. ### core - Massively improved the signatures of `Map` and `Set`, both for readability and ocamldoc, as well as improved type error messages. For instance the type of `Int.Set.singleton` was: ```ocaml ('a, 'comparator, 'a Core.Std.Int.Set.elt_ -> ('a, 'comparator) Core.Std.Int.Set.t_) Core.Core_set_intf.without_comparator ``` Now it is simply: ```ocaml int -> Int.Set.t ``` - Added an optional argument to `Command.run` that can be used to specify default flags from a user config file. The optional argument can extend the command line based on the path to the command. - Rename module `Weekday` as `Day_of_week`. The name `Weekday` conflicted with ordinary usage of "weekday" to mean Monday through Friday. - Changed `sexp_of_t` for `{Month,Ofday,Time,Time.Span}.{Set,Map}` to use the nice sexp format of the underlying atomic type. Previously, the converter had used thes raw type (`float`, `int`, etc.). `t_of_sexp` still accepts both formats; we will remove the ability to accept the raw format in the distant future. This output-format change was planned when we originally in 108.06b improved those `t_of_sexp` functions to accept both formats. - Added `Unix.remove`. - Removed some `IFDEF`'s connected to OCaml <4 support. ## 109.20.00 ### async_kernel - Added the ability for a `Throttle` to have resources that are exclusively available to running jobs. ### async_unix - Set `close-on-exec` for both ends of the pipe used to wake up the scheduler. ### core - Wrapped `Unix.wordexp` in an `Or_error.t` since it is not available on all systems. - Added function `Process_env.parse_ssh_client`. This gets the address from which you're currently ssh'd in. - Added to `Unix` module the ability to get and set `IP_MULTICAST_LOOP` and `IP_MULTICAST_TTL`. - Exposed module `Core.Std.Ref`, which was previously only available via `Core.Ref`. - Remove `Mutex.am_holding_mutex` and changed the type of `Mutex.try_lock`. With NPTL it is impossible to determine which thread is holding the lock. So, `Mutex.am_holding_mutex` is unimplementable. Also, `Mutex.try_lock` was incorrect because it claimed to raise if one was attempting to recursively lock. Since it's not possible to distinguish between recursive locking and the lock being held by another thread, we changed the type to make this clear: ```ocaml val try_lock : t -> [ `Already_held_by_me_or_other | `Acquired ] ``` - Removed our custom version of the OCaml runtime's `core_sys_open` function. There used to be a bug in the OCaml runtime, PR#5069, in which `open_{in,out}_gen` could block while holding the OCaml lock, because they made a call to `fcntl` outside the blocking section. We had our own C code with the bug fix and re-exposed the fixed versions of the functions in `Core`. The bug in OCaml has been fixed, so we have removed our patched function from `Core`. - In `unix_stubs.c`, switched from using `FNM_FILE_NAME` to `FNM_PATHNAME`. The GNU project introduced FNM_FILE_NAME as a non-portable synonym for FNM_PATHNAME. We were using pre-processor macros to define FNM_FILE_NAME as FNM_PATHNAME if unavailable, but it is simpler to just use the more portable FNM_PATHNAME everywhere. ### ocaml_plugin - Removed a test that (rarely) failed nondeterministically. ### sexplib - Renamed converter generated by `with sexp` for polymorphic variants so it is hidden from the toplevel. `of_sexp` created a value named `_of_sexp__` to handle polymorphic variants. To hide it from the toplevel, we renamed it as `___of_sexp__`. We kept the `__` suffix to avoid any confusion with a type named `__`. ### type_conv - Removed some warnings caused by generated signatures. 1. In signatures on local modules. 2. When there are duplicate signature items like in this example: ```ocaml module Warnings : sig type t = private { foo : int } with fields (** used to say unused value foo *) val foo : string end = struct type t = { foo : int } with fields let foo = "a" end ``` 3. In the signatures of all the parameters of functors that take multiple parameters; this used to work only for the last parameter. ## 109.19.00 ### async_extra - Added function `Versioned_typed_tcp.Client.flushed : t -> [ `Flushed | `Pending of Time.t Deferred.t ]`. This exposes whether the underlying `Writer.t` has been flushed. ### async_unix - Reworked a number of `Reader` functions to improve performance by avoiding deferreds. This is a followup to the `Reader` improvements in 109.14, and eliminates some last vestiges of performance degradation that had been introduced in 109.04. - Added function `Reader.lseek : t -> int64 -> mode:[< `Set | `End] -> int64 Deferred.t`. `lseek t offset ~mode` clears `t`'s buffer and calls `Unix.lseek` on `t`'s file descriptor. - Added function `Writer.bytes_received : t -> int`. - Added function `Unix.mkfifo : ?perm:file_perm -> string -> unit Deferred.t`, which was mistakenly missing. This is a simple wrapper around `Core.Unix.mkfifo`. ### core - Changed `Time.to_string` and `Time.sexp_of_t` to include the timezone. This is an incompatible change with very old programs in which `Time.of_string` and `Time.t_of_sexp` did not support the timezone. If you have programs that are: * very old and do Time string/sexp handling * rely on reading in time values without using `Time.of_string` and `Time.t_of_sexp`. * rely on chains of writing/reading/writing times across machines and timezones where the time is always intended to be taken as the local time on the currently reading machine you should recompile/review your code to make sure you won't have issues. - Added function `List.remove_consecutive_duplicates : 'a t -> equal:('a -> 'a -> bool) -> 'a t`. This returns the input list with consecutive duplicates removed, and doesn't change the order of the remaining elements. - Added module `User_and_group`, which is a pair of a unix username and primary unix group. The string/sexp converters follow the usual unix convention of `:`. - Added function `Date.first_strictly_after : t -> on:Weekday.t -> t`. `first_strictly_after t ~on:day_of_week` returns the first occurrence of `day_of_week` strictly after `t`. - Added functor `Type_equal.Lift`. It is always safe to conclude that if type `a` equals `b`, then type `a X.t` equals `b X.t`, for any type `X.t`. The OCaml type checker uses this fact when it can. However, sometimes, e.g. when using `Type_equal.conv`, one needs to explicitly use this fact to construct an appropriate `Type_equal.t`. The `Type_equal.Lift*` functors do this. ```ocaml module Type_equal : sig type ('a, 'b) t ... module Lift (X : T1) : sig val lift : ('a, 'b) t -> ('a X.t, 'b X.t) t end end ``` ### fieldslib - Made `with fields` generate the same functions in the `Fields` and `Fields_of_*` modules whether the type is called `t` or not. ## 109.18.00 ### async_unix - added `Async.Unix.fcntl_{get,set}fl`. Made `Reader` and `Writer` detect if they are passed a file descriptor with incorrect permissions (`O_WRONLY` for `Reader`, `O_RDONLY` for `Writer`). ### core - changed implementation of `Array.sort` to use introsort. See http://en.wikipedia.org/wiki/Introsort. - tweaked a unit test in `Core.Flags` to not print a message to stderr. ### pa_ounit - a number of improvements to `inline_tests_runner`, including a `-verbose` flag. 1. Made pa_ounit errors more readable. 2. Added `-verbose` flag. 3. Made the `-only-test` locations compatible with those displayed by the `-verbose` flag. 4. Renamed `-display` as `-show-counts` to avoid confusion with `-verbose`. 5. Improved errors when parsing the command line. 6. Updated the readme. 7. Added a `-list-test-names` which shows what tests would be run, if this option was not given. ## 109.17.00 ### async_extra - Added an option to `Async.Log.Rotation` to include the date in logfile names. This is mostly for archiving purposes. - Made `Versioned_rpc.Callee_converts.Pipe_rpc.implement_multi` agree with `Rpc.Pipe_rpc.implement` on the type of pipe rpc implementations. - Improved the performance of `Versioned_typed_tcp`. Avoided creating deferreds while reading the incoming messages. ### core - Fixed `Random.self_init`, which was broken since 109.00.00 with the upgrade to OCaml 4.0 The fix changed the type signature expressed in `core_random.ml` of the standard OCaml `caml_sys_random_seed` C function from `unit -> int` from `unit -> int array`. That C function changed between OCaml 3.12 and 4.0. - Moved module `Core_extended.Unix.Cidr` into `Core.Unix`. - Wrapped `Unix.wordexp` into an `Or_error.t` to handle systems that does not implement it in the libc. - Fixed two other printer names - Added `Array.int_blit` and `Array.float_blit`, which are specialized fast blits for `int array` and `float array`. For motivation underlying this change and other design alternatives please see Section 3 "Fast, Slow and Incorrect Array blits" of http://janestreet.github.com/ocaml-perf-notes.html - Added `Unpack_buffer.Unpack_one.sexp` for parsing sexps using the `Unpack_buffer` interface. ## 109.15.00 ### async_extra - In `Rpc.client` and `Rpc.with_client`, allowed the client to implement the rpcs. Added a new optional argument: `?implementations:_ Client_implementations.t`. - Added new module `Versioned_rpc.Both_convert` to allow the caller and callee to independently upgrade to a new rpc. This is a new flavor of `Versioned_rpc` in which both sides do some type coercions. ### async_unix - The `epoll`-based scheduler now supports sub-millisecond timeouts, using `Linux_ext.Timerfd`. Async still uses the `select`-based scheduler by default. We plan to switch the default to `epoll` in a few weeks, once we have done more testing. - Eliminated module `Work_group`, which was for limiting the number of threads used by jobs. This was a little-used module that significantly complicated the implementation of the Async thread pool. One should consider using a `Throttle` instead. Along the way, fixed a bug in Async helper threads in which the finalizer could fire too early, causing an unhandled exception. The fix involves relaxing the requirements on when `Thread_pool.finished_with_helper_thread` functions can be called, allowing it to be called while the helper thread still has work, but so long as no future work will be added. ### core - Changed the tolerance of `Time.Robustly_compare` functions from `1E-7` to `1E-6`. - Fixed the names of some toplevel pretty-printers, which referred to nonexistent modules. Fix some of the `pp`'s for Core which are used to install printers in the top-level. Some of the toplevel printers refer to non-existent modules like `Core.Nativeint.pp`; this feature changed to the correct name, like `Core.Std.Nativeint.pp`. - Added to module `Unix` functionality for getting and setting flags in the open-file-descriptor table. ```ocaml module Open_flags : sig type t include Flags.S with type t :` t ... end val fcntl_getfl : File_descr.t -> Open_flags.t val fcntl_setfl : File_descr.t -> Open_flags.t -> unit ``` - Added module `Linux_ext.Timerfd`. This allows one to create a file descriptor that can be monitored by `epoll` or `select` and notify them at a certain time. It makes it possible to use `epoll` with sub-millisecond timeouts. - Added `Version_util.application_specific_fields`, which allows custom build-time information to be included in an executable. ## 109.14.00 ### async - Added function `Monitor.kill`, which kills a monitor and all its descendants. This prevents any jobs from ever running in the monitor again. ### async_unix - Fixed major performance degradation (since 109.04) in `Reader.read*` functions. - Added function `Rpc.Implementation.map_inv`. ```ocaml val map_inv : 'a t -> f:('b -> 'a) -> 'b t ``` - Add functions `Reader.file_lines` and `Writer.save_lines`. These deal with files as lists of their lines. ```ocaml val Reader.file_lines : string -> string list Deferred.t val Writer.save_lines : string -> string list -> unit Deferred.t ``` - Added a `?wakeup_scheduler:bool` optional argument to functions in the `Thread_safe` module. The default is `true`, which continues the behavior that has been in place since 109.09. However, once can use `~wakeup_scheduler:false` to reduce CPU use, in return for increased latency (because the scheduler won't run a cycle immediately). ### core - Fixed major performance problem with hashing in `Int.Table`. Our `Int.Table.replace` was 3 times slower than polymorphic hash table and `find` was _8_ times slower. This was caused by using: ```ocaml external seeded_hash_param : int -> int -> int -> 'a -> int = "caml_hash" "noalloc" ``` in `Int.Table` but: ```ocaml external old_hash_param : int -> int -> 'a -> int = "caml_hash_univ_param" "noalloc" ``` everywhere else. The `seeded_hash_param` was introduced in Caml 4. We fixed this problem by changing `Int.hash` from: ```ocaml let hash (x : t) = Hashtbl.hash x ``` to: ```ocaml let hash (x : t) = if x >= 0 then x else ~-x ``` - Added `Bigstring.{pread,pwrite}`, which allow reading and writing at a specific file offset. - Added module `Nothing`, which is a type with no values. This is useful when an interface requires you to specify a type that you know will never be used in your implementation. - Changed `Identifiable.Make` so that it registers a pretty printer. `Identifiable.Make` now uses `Pretty_printer.Register`. This requires all calls to `Identifiable.Make` to supply a `val module_name : string`. - Made `Core.Zone` match the `Identifiable` signature. - Made polymorphic equality always fail on `Core.Map.t` and `Core.Set.t`. Before this change, polymorphic equality on a `Core.Map` or a `Core.Set` could either raise or return `false`. It returnd `false` if the data structures were unequal, and raised if the data structures were equal. This is because their type definitions looked like: ```ocaml type ('k, 'v, 'comparator) t = { tree : ('k, 'v) Tree0.t; comparator : ('k, 'comparator) Comparator.t; } ``` and polymorphic equality visits a block's fields in order. So, it will detect unequal trees and return false, but if the trees are equal, it will compare the comparators and raise because of the functional value. This change reversed the order of the fields so polymorphic equality always fails. ### custom_printf - initial import Added support for `%{}` in `printf`-style format strings. If you put `!` before a format string, it allows the use of a spec like `%{}` in the format string. For example, using `%{Time}` wraps `Time.to_string` around the appropriate argument. It also allows different formats for a given type: `%{.}` wraps `.Format.` around the appropriate argument. For example, `%{Float.pretty}` would wrap `Float.Format.pretty` around the appropriate argument. ### fieldslib - Made `with fields` expose first-class fields for private types while preserving privacy. There is now an additional phantom type in a first-class field that prevents building or modifying elements of a private type. One consequence of this change is that the `Field.t` type is now an abstract type -- it used to be exposed as a record type. So, one must, e.g., change `field.Field.name` to `Field.name field`. ## 109.13.00 ### async_kernel - Fixed `Pipe.iter`'s handling of a closed pipe. Fixed the handling by `Pipe.iter` and related foldy functions that handle one element at a time, which behaved surprisingly with a pipe whose read end has been closed. These functions had worked by reading a queue as a batch and then applying the user function to each queue element. But if the pipe's read end is closed during the processing of one queue element, no subsequent element should be processed. Prior to this fix, the `iter` didn't notice the pipe was closed for read until it went to read the next batch. - Renamed `Pipe.read_one` as `Pipe.read_one`', and added `Pipe.read_one` that reads a single element. ### async_unix - Added `Writer.write_line`, which is `Writer.write` plus a newline at the end. - Added `?close_on_exec:bool` argument to `{Reader,Writer}.open_file` and `Async.Unix.open_file`. Made the default `close_on_exec:true` for `Reader` and `Writer`. - Added a `compare` function to `Socket.Address.Inet`. ### core - Added `Command.Spec.flags_of_args_exn`, for compatibility with OCaml's standard library. This function converts a `Core.Std.Arg.t` into a `Command.Spec.t`. - Made various modules `Identifiable`: `Char`, `String`, and the various `Int` modules. In particular, `Int` being identifiable is useful, because one can now write: ```ocaml module My_numeric_identifier : Identifiable ` Int ``` You might think that we could now delete `String_id`, and just write: ```ocaml module My_string_identifier : Identifiable ` String ``` But this is not quite equivalent to using `String_id`, because `String_id.of_string` enforces that its argument is nonempty. - Removed module `Space_safe_tuple`, which became unnecessary in OCaml 4.00.0. OCaml 4.00.0 included Fabrice's patch to fix the space leak that `Space_safe_tuple` was circumventing (PR#5288, commit SVN 11085). - Added `Exn.to_string_mach`, for single-line output. - Added `Linux_ext.bind_to_interface`, to improve security of UDP applications. ```ocaml val bind_to_interface : (File_descr.t -> string -> unit) Or_error.t ``` This uses the linux-specifc socket option `BINDTODEVICE` to prevent packets being received from any interface other than one named. - Fixed `Unix.mkdir_p` on Mac OS X. ## 109.12.00 ### async_extra - Made explicit the equivalence between type `Async.Command.t` and type `Core.Command.t`. ### async_unix - Fixed a bug in `Fd.syscall_in_thread`. The bug could cause: ```ocaml Fd.syscall_in_thread bug -- should be impossible ``` The bug was that `syscall_in_thread` raised rather than returning `Error`. - Changed `Tcp.connect` and `Tcp.with_connect` to also supply the connected socket. Supplying the connected socket makes it easy to call `Socket` functions, e.g. to find out information about the connection with `Socket.get{peer,sock}name`. This also gives information about the IP address *after* DNS, which wouldn't otherwise be available. One could reconstruct the socket by extracting the fd from the writer, and then calling `Socket.of_fd` with the correct `Socket.Type`. But that is both error prone and not discoverable. - Added `Writer.schedule_bigsubstring`, which parallels `Writer.schedule_bigstring`. ### core - Add some functions to `Byte_units`. - Added functions: `to_string_hum`, `scale`, `Infix.//`. - Eliminated the notion of "preferred measure", so a `Byte_units.t` is just a `float`. - Improved the performance of `Array.of_list_rev`. The new implementation puts the list elements directly in the right place in the resulting array, rather that putting them in order and then reversing the array in place. Benchmarking shows that the new implementation runs in 2/3 the time of the old one. - Fixed `Fqueue.t_of_sexp`, which didn't work with `sexp_of_t`. There was a custom `sexp_of_t` to abstract away the internal record structure and make the sexp look like a list, but there wasn't a custom `t_of_sexp` defined, so it didn't work. - Added `Stable.V1` types for `Host_and_port`. - Removed `Identifiable.Of_sexpable` and `Identifiable.Of_stringable`, in favor of `Identifiable.Make` `Identifiable.Of_sexpable` encouraged a terrible implementation of `Identifiable.S`. In particular, `hash`, `compare`, and bin_io were all built by converting the type to a sexp, and then to a string. `Identifiable.Of_stringable` wasn't as obviously bad as `Of_sexpable`. But it still used the string as an intermediate, which is often the wrong choice -- especially for `compare` and `bin_io`, which can be generated by preprocessors. Added `Identifiable.Make` as the replacement. It avoids using sexp conversion for any of the other operations. - Added `List.intersperse` and `List.split_while`. These came from `Core_extended.List`. ```ocaml val intersperse : 'a list -> sep:'a -> 'a list val split_while : 'a list -> f:('a -> bool) -> 'a list ** 'a list ``` - Added a functor, `Pretty_printer.Register`, for registering pretty printers. The codifies the idiom that was duplicated in lots of places: ```ocaml let pp formatter t = Format.pp_print_string formatter (to_string t) let () = Pretty_printer.register "Some_module.pp") ``` ### fieldslib - Added back `Fields.fold` to `with fields` for `private` types. We had removed `Fields.fold` for `private` types, but this caused some pain. So we're putting it back. At some point, we'll patch `with fields` to prevent setting mutable fields on private types via the fields provided by `fold`. ### sexplib - A tiny lexer improvement in `lexer.mll`. Used `lexbuf.lex_{start|curr}_pos` instead of `lexbuf.lex_{start|curr}_p.pos_cnum` for computing the length of a lexeme since the difference is the same. ## 109.11.00 ### async_extra - Exposed a `version` function in `Pipe_rpc` and `State_rpc`. ### async_kernel - Extended `Deferred.Or_error` to parallel almost all of the `Core.Or_error` interface. - Improved the performance of `Clock.at`, and added a more efficient version, `Clock.run_at`. Reworked the async heap of clock alarms to use async jobs as alarms. Reworked `Clock.at` to use this and to not use abortable events, which is a performance improvement. Added a more efficient version of `Clock.at`, for the common situation when one doesn't need a deferred. ```ocaml (*** [run_at time ~f] is a more efficient version of [at time >>> f]. *) val run_at : Time.t -> f:(unit -> unit) -> unit ``` ### async_unix - Added a check to fail if `Scheduler.go` is called more than once. ### core - Added module `Interned_string` This has a functor for creating modules of interned strings. It uses the very simple mechanism of mapping distinct strings to consecutive ints. - Added value `Hashtbl.find_and_remove`. ### fieldslib - `with fields`, for a type `u` that isn't named `t`, creates module `Fields_of_u` rather than module `Fields`. This allows one to us `with fields` on several types in the same structure. ## 109.10.00 ### async_extra - Fixed a race condition in `Pipe_rpc` and `State_rpc`. This race could cause an exception to be raised on connection closing. ### async_unix - Added `Shutdown.do_not_finish_shutdown_before`. This allows one to add `unit Deferred.t`'s that will delay the `shutdown` from finishing. The implementation is more efficient than using `at_shutdown`. ### bin_prot - Improved error messages in presence of GADTs. ### comparelib - Improved error messages in presence of GADTs. ### core - Added `|>`, which means the same as `|!`, but is likely to replace it someday. This is mostly because `|>` is an accepted notation elsewhere, particularly in F#. In the future, we will consider eliminating `|!` in favor of `|>`, so as to avoid the duplication. - Made `module Lazy` into a monad. - Renamed `List.stable_dedup_involving_an_application_of_the_set_functor` as `List.stable_dedup_staged`. Made it use `Staged.t` to make explicit the expectation of partial application. - Added pretty printers for the toplevel to `Error` and `Info`. ### fieldslib - Changed `with fields` on `private` types to not expose mutators or creators. ### pa_ounit - Rewrote `pa_ounit` to simplify execution order and work better with functors. Rewrote `pa_ounit` to solve its shortcomings with functors, namely that functors need to be applied with `TEST_MODULE` for their tests to be registered. The order of execution is also much simpler: tests are executed inline, at the toplevel (or functor application time). There is still a limitation: when a library doesn't have any occurrence of `TEST`, `TEST_UNIT`, or `TEST_MODULE` inside of it, the test runners are not set up, so tests inside of functors (from other libraries) will not be executed. Running `inline_test_runner.exe` is not going to run tests anymore; people should run the `inline_test_runner` script instead. Backtraces are now properly shown when exceptions are thrown. ### sexplib - Improved error messages in presence of GADTs. - Made `with sexp` work with types containing `as` in signatures. ### variantslib - Improved error messages in presence of GADTs. ## 109.09.00 ### async - Switched `Async.Std`'s toplevel bindings for `Deferred.Or_error`'s `bind` and `map` to use `Deferred.Result`. This allows them to be used with any `'error` type, rather than just `Error.t`. ### async_kernel - Fixed bug in `Async.Throttle`, in which jobs weren't started in order. ### async_unix - Added module `Thread_safe_pipe`, for streaming data outside async into async. This a more efficient and feature-ful way to send a sequence of values from outside async into async than `Thread_safe.pipe`, which has been eliminated. - Changed functions in `Thread_safe` to always wake up the scheduler. Changed `Thread_safe.run_in_async{,_exn}` to not run a cycle, and instead rely on the scheduler to run the cycle. ### core - In `Core.Std`, exposed `Or_error.ok_exn` and `Or_error.error` - Removed some values exported by `Core.Std`. Removed some values from `Core.Std` that weren't widely used, or we didn't think should be exposed, including `ascending`, `descending`, and `equal`, which use polymorphic comparison, and we want to discourage. Here's a guide to some of what was removed, and what one should now use instead. | removed | replace with | |-----------------------------------+---------------------------------------| | `Int_replace_polymorphic_compare` | `Int.Replace_polymorphic_compare` | | `ascending` | `Polymorphic_compare.ascending` | | `descending` | `Polymorphic_compare.descending` | | `equal` | `Polymorphic_compare.equal` | | `ifprintf` | `Printf.ifprintf` | | `sscanf` | `Scanf.sscanf` | | `Scan_failure` | `Scanf.Scan_failure` | | `string_of__of__sexp_of` | `Sexplib.Conv.string_of__of__sexp_of` | | `of_string__of__of_sexp` | `Sexplib.Conv.of_string__of__of_sexp` | | `type vec` | `type float64_vec` | - Disallowed `<:sexp_of<` with two underscores; using a single underscore instead. - Added `Command.Spec.Arg_type.of_alist_exn` as an alternative for `of_map`. This captures the common pattern to create the map from an alist. - Improved the performance of `Hashtbl`. Constrained hashtbl size to a power of two and used a bitmask rather than mod operation for finding hash buckets. - Improved the performance of `Univ`, using the `Type_equal` GADT. The new implementation improves the run-time and space usage over the old one. In the old implementation, a `Univ.t` was represented as record with three fields: an exception, a string, and a closure. Creating a univ required allocating three heap blocks, the exception (3 words), the closure (3 words), and the three-field record (4 words). In the new implementation, a `Univ.t` is represented as a 2-field heap block containing the `Constr.t` and the value. Creating a univ allocates that single 3-word block, improving on the 10 words needed previously. Matching on univs is also faster. In the old implementation, matching on a univ required making a function call, testing exception equality, and allocating a `Some` block. Now, it does just the test and allocation. Furthermore, it is possible to use `does_match` and `match_exn` to avoid the allocation. - Added `Version_util.build_info_as_sexp`. - Added `_squelch_unused_module_warning_` to `Comparable.S.Replace_polymorphic_compare`. ### sexplib - Fixed an `unused rec` warning in the code generated by `pa_sexp` in rare cases. ## 109.08.00 ### async_extra - Added module `Async.Command` This is `Core.Command` with additional async functions. In particular it contains a function `async_basic` that is exactly the same as `Core.Command.basic`, except that the function it wraps returns `unit Deferred.t`, instead of `unit`. `async_basic` will also start the async scheduler before the wrapped function is run, and will stop the scheduler when the wrapped function returns. ### async_unix - Added module `Async.Process` This is a new module for creating and dealing with child processes. - For `Writer.save`, replaced the `temp_prefix` argument with `temp_file`. - Added `Ivar.invariant` function. - Added value `Scheduler.fold_fields` This lets one fold over the fields in the scheduler, eliminates an annoying place in catalog browser that reached into the internals of async to compute the sizes of the scheduler fields ### core - Cleaned up and updated the `README`. - Changed executables to enable backtraces if `OCAMLRUNPARAM` is not set. - Changed `Command` so that executables show build info and version info This happens when an executatble is called as: foo.exe version Before this change, rather than display build info, executables would display the not-so-helpful: (no option given - printing version) - Added back `Float` rounding functions with a hardcoded direction. - Exposed `with bin_io` and `with compare` for the =sexp_bool= type. - Added value `Core.Never_returns.sexp_of_t`. - Added values `Or_error.tag{,_arg}` These are analogous to `Error` functions of the same name. - Added functor `Sexpable.Of_sexpable` This is for serializing values of one type as though it were some other isomorphic type. - Added module `Backtrace.Exn` This exposes OCaml stdlib's `Printexc` functions for backtraces. - Added module `Flags` This implements Unix-style sets of flags that are represented as an `int` with various bits set, one bit for each flag, e.g., `Linux_ext.Epoll.Flag`. - Added module `Uuid` This module implements universally unique identifiers based on version 3 of the UUID specification. It used to be in `Core_extended=` - Added module `Type_equal`, which defines the "equality" GADT. ### type_conv - Fixed type_conv to stop dropping parens in arguments such as: type t = { a : int with default(1), sexp_drop_if(fun x -> (x + 1) * 2 = 4) } with sexp ## 109.07.00 ### async_unix - Changed the async scheduler so that if there are no upcoming events, it times out in 50ms rather than waiting forever. - Improved `Reader.read_one_chunk_at_a_time_until_eof`: - the callback need not consume everything - add `\`Eof_with_unconsumed_data` as a possible result - grow internal buffer of the reader when needed - Added `Shutdown.exit`, removed `Shutdown.shutdown_and_raise`. - Added `Scheduler.force_current_cycle_to_end`. ### core - Added a number of functions to =Bounded_int_table=: =equal=, =exists{,i}=, =for_all{,i}=, =filter_map{,i}=, =map{,i}=. Also added a functor, =Bounded_int_table.With_key=, that makes a bounded-int table binable and sexpable, and adds =of_alist= and =of_alist_exn=. - Added =Doubly_linked.iter_elt= and =Bag.iter_elt=. - Added =module Invariant=, which defines signatures that are to be included in other signatures to ensure a consistent interface to invariant-style functions. - Added =module Ordering=, which defines: =type t = Less | Equal | Greater= ## 109.06.00 ### core - Added [Map.symmetric_diff], for returning a list of differences between two maps. It has a fast-path implementation for maps that share a large amount of their internal structure. ## 109.05.00 ### async - Added `val _squelch_unused_module_warning_` to `Async.Std`. ### core - Updated [Core.Unix.stat] so that access, modify, and change times have nanosecond precision. - Fixed a bug in [Nano_mutex.invariant]. - Simplified the implementation of [with_return] using a local explicit polymorphic type variable. ## 109.04.00 ### core - Fix [Backtrace.get], which was broken in 109.00, with the switch to OCaml 4.0. - Added [Heap.iter_el]. ## 109.02.00 ### core - Add Char.of_string ## 2012-07-15 ### bin_prot - Rewrote README in Markdown and improved documentation. - Eliminated new warnings available in OCaml 4.00. ### sexplib - Added support for S-expression default record fields. - Added syntax for S-expression comments and for nested block comments. - Fixed a few minor bugs and inconsistencies in the parsers and updated their whitespace handling to conform with the upcoming OCaml 4.00 compiler. The parser specification now also supports Menhir. - Rewrote README in Markdown and improved documentation. - Minor bug fix in the preprocessing module. - Eliminated new warnings available in OCaml 4.00. ### type_conv - Added support for record field annotations and defaults. - Fixes for upcoming OCaml 4.00 release. ## 2012-02-28 ### bin_prot - Improved portability by better supporting the C99-standard and non-GNU compilers. ## 2011-11-10 ### bin_prot - Improved portability to older glibc distributions. ## 2011-09-18 ### sexplib - Improved documentation. ## 2011-09-15 ### bin_prot - Fixes to improve package dependency resolution. ### sexplib - Fixes to improve package dependency resolution. ### type_conv - Fixes to improve package dependency resolution. ## 2011-08-02 ### type_conv - Added missing module type case for "module type of". ## 2011-07-05 ### sexplib - Fixed a parser position bug. Parser positions passed by the user were not updated correctly. - Internal code beautification. ## 2011-07-04 ### bin_prot - Internal updates to sync with Jane Street. ### sexplib - Internal updates to sync with Jane Street. ### type_conv - Merged with Jane Street version. API changes: Removed functions: * Gen.ide * Gen.idp Removed location arguments from: * type_is_recursive * drop_variance_annotations ## 2011-06-29 ### bin_prot - Fixed bigstring layout bug, which should only affect value comparisons with OCaml 3.12.1 or later. - Made 64-bit detection more reliable on Mac OS X. ## 2011-01-30 ### sexplib - Fixed a code generation bug with toplevel entries. Thanks to Yong Lu for the report! ## 2010-12-27 ### sexplib - Added support for MoreLabels.Hashtbl and improved reporting of error locations with preprocessor. ## 2010-12-26 ### sexplib - Worked around a compiler bug that is expected to be fixed in OCaml 3.12.1. This workaround temporarily removes the interface for module Conv (conv.mli), thus exposing the internals. This should not cause any problems for end users as long as they do not depend on the exported internal representations. The interface will become constrained again as soon as the fixed compiler is out. ## 2010-12-22 ### sexplib Major release. - Merged with Jane Street version. This has caused an API-change that requires "open Sexplib.Conv" at the top of files that use the syntax extension. - Renamed functions: * sexp_of_lazy -> sexp_of_lazy_t * lazy_of_sexp -> lazy_t_of_sexp - Some standard library modules are now re-exported with predefined S-expression converters in module Conv. ### type_conv - Merged with Jane Street version. No code changes. ## 2010-09-25 ### sexplib - Fixed inferred types of generated functions when dealing with arrow types. ### type_conv - Added a missing type case to type_is_recursive. Thanks to Michael Wawrzoniak for this patch! ## 2010-08-26 ### sexplib - Fixed a lexer bug when parsing comments. ## 2010-07-07 ### type_conv - Major changes for compatibility with OCaml 3.12. ## 2010-06-03 ### type_conv - Improved determination of type_conv paths. Thanks to Jacques Le Normand for this patch! ## 2010-05-21 ### sexplib - Added support for sexp_bool record field annotations. ## 2010-05-18 ### sexplib - Improved performance of converting S-expressions to strings. ## 2010-04-12 ### sexplib - Changed API of Of_sexp_error exception. ## 2010-04-07 ### sexplib - Added of_(big)string_conv_exn functions. ## 2010-04-01 ### sexplib - Merged with Jane Street version. Major new features (various functions): * Type-annotated parsing for better error messages * Greatly improved performance of exception converters ## 2010-03-20 ### bin_prot - Fixed linking of toplevels to require bigarrays. - Improved compilation on Mac OS X. ## 2010-03-17 ### bin_prot - Fixed small name capture bug. ## 2009-12-21 ### bin_prot - Updated contact information. ### sexplib - Improved saving of files. ## 2009-10-12 ### sexplib - Added sexp_array record field extension. ## 2009-09-19 ### bin_prot - Added missing type cases for supporting variant types. - Fixed handling of variance annotations. ### sexplib - Added missing variant type cases. - Fixed handling of variance annotations. ### type_conv - Added missing type cases for supporting variant types. ## 2009-09-15 ### sexplib - Internal cleanups. ## 2009-07-28 ### sexplib - Added better support for conversion of exception types. ## 2009-07-27 ### bin_prot - Fixed build problem with gcc 4.4 due to stricter checking for empty macro arguments. Thanks to Nobuyuki Tomiza for the patch! ## 2009-07-20 ### bin_prot - Merged tiny Jane Street improvements. ## 2009-07-03 ### bin_prot - Made byte swapping more portable. ## 2009-07-02 ### bin_prot - Added support for network byte order integers. ## 2009-06-23 ### sexplib - Fixed build problem. Thanks to Sylvain Le Gall for the patch! ## 2009-05-08 ### sexplib - Fixed build problems on Windows and OCamlMakefile issues. Thanks to Sylvain Le Gall for the patch! ## 2009-04-22 ### bin_prot - Added macro support for all kinds of vectors (vec, float32_vec, float64_vec) and matrices (mat, float32_mat, float64_mat), and for bigstrings (bigstring). ### sexplib - Added macro support for all types of vectors, matrices, and for bigstrings. ## 2009-04-21 ### sexplib - Merged with Jane Street version, no user-relevant changes. ## 2009-04-16 ### bin_prot - Fixed a bug leading to an exception when writing extremely large values (>4 GB buffer size). Does not cause data corruption. ## 2009-03-09 ### sexplib - Merged with Jane Street version, no user-relevant changes. ## 2009-03-01 ### sexplib - Fixed build problem on Mac OS X by updating OCamlMakefile. ## 2009-01-20 ### sexplib - Automatically add S-expression pretty-printers to toplevels. ## 2009-01-14 ### type_conv - Added support for type converters that take arguments. Thanks to Jérémie Dimino for this patch! Added support for deprecated OCaml syntax, since the compiler still supports it, too. ## 2008-10-22 ### type_conv - Fixed bug preprocessing labeled arguments in function types. ## 2008-10-18 ### type_conv - Fix for upcoming OCaml release 3.11. ## 2008-10-07 ### type_conv - Added a patch to improve handling of type conversion paths. Thanks to David Rajchenbach-Teller for the patch! ## 2008-09-30 ### sexplib - Added a new feature: sexp_opaque. It prevents the need for / use of type converters for a given type in a particular type context. Removed abstract types to unify them with this new concept. ## 2008-09-29 ### sexplib - Added a new feature: sexp_list. This is similar to the handling of sexp_option. By default an empty list is assumed for unspecified records using sexp_list as qualifier. Such record fields bound to empty lists will also not be printed anymore for better readability. ## 2008-09-23 ### sexplib - Added missing Not_found-exception to standard exception converters. ## 2008-08-20 ### sexplib - Removed dependency on threads. Fixed build problems. ### type_conv - Added support for exception converters. ## 2008-08-08 ### sexplib - Nifty new feature: exceptions can now be converted to S-expressions, too! The "with sexp" syntax extension can be used with exceptions, thus registering a conversion function. A global exception conversion function can then be called to convert an arbitrary exception into an S-expression, which can then be printed out. This should greatly improve readability of uncaught exceptions while making life extremely easy for the developer. - Renamed the ParseError exception to Parse_error to be more compliant with Jane Street naming conventions. ## 2008-07-25 ### sexplib - Added utilities for conversion error handling. Minor fixes. ### type_conv - Fixed bug concerning variance annotations in type definitions within structures. ## 2008-04-24 ### sexplib - Made Sexp-interface manifest. ## 2008-03-20 ### sexplib - Fixed META-file (missing num dependency). ## 2008-03-17 ### sexplib - Improved META-file. ### type_conv - Improved META file and support of toplevel interpreter. ## 2008-03-13 ### sexplib - Fully allow function types in converters. Raise runtime exceptions on converting from S-expressions instead when function type encountered. ## 2008-02-11 ### sexplib - Fixed code generation problems with variance annotations in signatures, and empty types. ### type_conv - Added support for handling variance annotations in signatures, and for empty types. ## 2007-12-17 ### sexplib - Added support for generating signature entries for S-expression converters. Thanks to Till Varoquaux for the patch! ## 2007-11-29 ### sexplib - Added support for converting big_int, nat, num, and ratio. ## 2007-11-26 ### sexplib - Added support for parsing from bigstrings (char bigarrays). ## 2007-11-02 ### sexplib - Added syntax support for option types to use the ordinary sum type syntax. This should improve readability. The old syntax will be accepted, too, if Conv.read_old_option_format is set to true (this is currently the default). The old format will be used for writing if Conv.write_old_option_format is true (currently the default). The old syntax is deprecated and will probably not be supported by default in the near future. Reading new-style option values will always succeed. ## 2007-10-14 ### type_conv - Initial release. ## 2007-09-14 ### sexplib - Fixed bug in S-expression preprocessor concerning record field names. ## 2007-08-06 ### sexplib - Added support for converting functions to S-expressions. ## 2007-07-20 ### sexplib - Fixed position information and improved speed of S-expression parser. Fixed S-expression macro bug concerning contained polymorphic variants. ## 2007-06-28 ### sexplib - Improved Sexplib code generation. ## 2007-06-22 ### sexplib - Fixed escaping bug in S-expression parser. ## 2007-06-01 ### sexplib - Added correct handling of recursive types + test case. ## 2007-04-18 ### sexplib - Added missing conversion functions from S-expressions to pairs and triples. ## 2007-03-21 ### sexplib - Updated OCamlMakefile. ## 2007-03-02 ### sexplib - Improved error messages when parsing illegal type definitions. ## 2007-01-30 ### sexplib - Added triple conversions. ## 2006-11-22 ### sexplib - Updated OCamlMakefile. ## 2006-10-13 ### sexplib - Improved checking of records for extra or duplicate fields. ## 2006-09-06 ### sexplib - Added support for polymorphic record fields. ## 2006-09-05 ### sexplib - Added support for manifest types. ## 2006-08-16 ### sexplib - Improved error messages. ## 2006-07-28 ### sexplib - Added a new, hand-written S-expression parser that supports partial parsing and should be approx. 10x faster than the previous one. ## 2006-06-20 ### sexplib - Fixed a code generation problem leading to compilation errors concerning the use of type aliases within polymorphic variant type definitions. This fix also solves potential erroneous appearances of backtracking exceptions in user code. ## 2006-03-21 ### sexplib - Added -for-pack option to Makefile and cleaned up distribution for a new public release. ## 2006-03-13 ### sexplib - Sexplib now accepts capitalized booleans. ## 2006-03-03 ### sexplib - Added customizable indentation levels. - Improved documentation. - Fixed API-problem concerning backward compatibility. ## 2006-03-01 ### sexplib - Added a missing flush for string conversions with a buffer. ## 2006-02-08 ### sexplib - Eliminated unused variable warnings in Sexplib-generated code. ## 2006-01-11 ### sexplib - Added functions for pretty-printing to buffers. - Improved performance of outputting S-expressions to channels. ## 2006-01-09 ### sexplib - Added functions load_sexp and load_sexps. ## 2006-01-04 ### sexplib - Changed float conversion from %E to %G (more readable). ## 2005-12-28 ### sexplib - Made machine representation for S-expressions more compact ## 2005-12-15 ### sexplib - Fixed a problem appearing with OCaml-release 3.08.4: CamlP4 obviously performs more strict checking on some constructs now and crashed with an exception when generating S-expression code for records containing only one field ("singleton tuple problem"). This problem is fixed now. ## 2005-11-25 ### sexplib - Fixed problem with type variables that could not be generalized. ## 2005-11-23 ### sexplib - Added a missing case in type definitions (path alias) ## 2005-11-17 ### sexplib - Major release: 2.0 Fixed a major design problem. The user now has to pass lex buffers instead of channels to input-functions. Reason: trailing characters in channels were lost due to ocamllex buffering them in the non-exposed lex buffer. This lex buffer is now exposed. The functions have been renamed ("input_X" -> "scan_X") to reflect this change. ## 2005-11-16 ### sexplib - Added label to conversion function "input_cnv_sexps". ## 2005-11-11 ### sexplib - Fixed a bug in the pretty-printer: strings in atoms were not escaped in the function "to_string_mach" (and therefore also "to_string"). ## 2005-11-07 ### sexplib - Initial release.