113.24.00
async
Keep up to date with interface changes in Async_kernel
, Async_extra
and
Async_unix
.
async_extended
Make LTL predicates comparable by tagging and id to each one. Fixes a functional comparison bug.
Switched to PPX.
Add an mli for
async_extended/src/reader_ext.ml
and remove a couple of unused functions (notablyReader_ext
had its own version ofReader.read_char
).Add async-friendly color print
Ltl.eval
should close the pipe after it is done with it.Deleted
Async_extended.Cml
.Remove
Async_extended.Std.Gzip
and redirect references toAsync_gzip.Std.Gzip
.Update
Command_rpc.Connection
to check the program before exec'ing it. The filename must now be absolute, exist, and be executable. Previously errors with nonexistent or nonexecutable files would only be found out after forking.-
Change
Command_rpc.Command
to useVersioned_rpc.Callee_converts
instead ofVersioned_rpc.Both_convert
so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate.Removed
val rpcs
fromCallee_converts
interfaces because nothing appears to use it, andBoth_convert
does not provide it. NowBoth_convert.S
can be supplied to satisfyCallee_converts.S
. Add simple example of Command_rpc
Add
Deferred_cache
.-
Fixing a couple of issues noticed in
Command_rpc
:If
propagate_stderr
is false, the child's stderr is now drained instead of ignored.When connections are closed, stderr is now closed as well, which prevents a file descriptor leak if the child process is unresponsive.
async_extra
N.B. some changes happening for this release are not listed in this changelog since they appear only as a consequence of changes in core or async_kernel.
When
Transfer.Writer.send*
raises, send an error to the client.Add a new rpc that enables a "push" rather than a "poll" model.
Switched to PPX.
For connected UDP sockets, expose
send
in the same fashion assendto
.-
Tcp.Server
is documented to refuse excess connections beyondmax_connections + max_pending_connections
, but it treats them as pending connections in our standard OS configuration. In fact, research indicates that the documented behavior is nearly impossible to obtain directly and consistently fromlisten
.Clarify the name and role of the
backlog
argument tolisten
and rename and update documentation formax_pending_connections
to clarify what it actually does, in light of some research:`listen` does not generally respect the backlog argument as an upper limit, but as a lower limit (mod `tcp_max_syn_backlog`) and, with `tcp_abort_on_overflow=0`, `listen` will ignore excess connections rather than actively refusing them. (With `syncookies=1`, this can look like an indefinite backlog.)
Existing, working code can substitute
max_pending_connections -> backlog
and move on. The behavior is not changed.When possible, consider architecting applications so the server can simply accept and close excess connections, rather than relying on the
listen
backlog to return an active indication to the client that they won't be serviced. To make sure the client receives an RST rather than an orderly shutdown, you can set the linger time to 0 before closing the socket. (Added to unit tests.)Direct
Tcp.Server
support for this paradigm is left for future work. Make
Rpc_low_latency_transport
treat disconnections as eof, likeAsync_unix.Reader
does.Add an implementation of Mvars to Async
Allow custom handling of missed async_rpc heartbeats.
adds a configuration limit on the number of tokens that can be in-flight
-
Replace an
#include <sys/errno.h>
by#include <errno.h>
.Fixes janestreet/async_extra#4
Added
Tcp.Server.sexp_of_t
-
Adds
Rpc.Pipe_rpc.dispatch_iter
, plus a bunch of additional types to support it. The main reason for this is to reduce space usage:Pipe_rpc.dispatch
followed byPipe.iter_without_pushback
consumes ~105 words in the steady state (i.e., when no messages are coming in) whiledispatch_iter
consumes ~15. I'm suredispatch
can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small asdispatch_iter
.Both cases can be made smaller by making
Connection.response_handler
a GADT instead of a closure. I plan to do this later.One annoying property of the interface is that the only way to cancel a subscription is to use
Pipe_rpc.abort
, which has a terrible interface. The logical way to improve the interface is to return a record of aPipe_rpc.t
, aConnection.t
, and aQuery_id.t
, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. Added
Tcp.Server.listening_on_address
, so that one can get the address a server is listening on, as compared withlistening_on
, which just returns the port.-
Marked Command.async_basic as deprecated using the appropriate ocaml attribute.
@@ocaml.deprecated
(http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec241)
Extend the interface of
Persistent_rpc_client
to make the "address" type - previously fixed asHost_and_port.t
- abstract. This is helpful for integrating with libraries that have a different notion of an address, e.g.rpc_discovery_lib
.Typed_tcp
mutated a Hashtbl while iterating over it when closing.-
Added
Async.Bus.first_exn
, which takes a bus and a function, and returns a deferred that becomes determined when the first event is published to the bus for which the function returnsSome
.This function is useful to reduce boilerplate for dealing with unsubscription.
-
Reduced the number of threads required by tests in:
async_extra/src/tcp.ml
Added to the error message
Bus.subscribe_exn called after first write
the source-code position of the caller, in case there isn't a backtrace, to make the source of the problem clearer, and to avoid confusion with other source-code positions of subscribers already in the bus.Added to
Bus.first_exn
aSource_code_position.t
argument, so that in the event of subscription failure, we can see who caused the subscription to the bus.-
Added to
Tcp.Server.close
an optional argument:?close_existing_connections : bool
This closes the sockets of all existing connections.
Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket).
Improved
Async.Udp.bind
's error message when it fails tomcast_join
a multicast group.Change
~callback
to~f
throughout theBus
interface
async_find
Initial release.
async_kernel
N.B. some interface change in Core (notably to Hashtbl
and Map
) implied some
interface change in this package as well, although they are not mentionned in
this changelog.
Switched to ppx.
Improved the Async scheduler's to allocate a
handle_fired
function once, rather than every time it callsadvance_clock
.-
Removed configurability of Monitor's
try_with
-ignored-exception handling, i.e. removedMonitor.try_with_rest_handling
andMonitor.try_with_ignored_exn_handling
.The behavior of exceptions raised to a monitor after it returns is unchanged -- i.e. they are logged, as they have been since 112.28.
Changed
Monitor.try_with
's?rest
argument from:?rest :
Ignore |
Raise
?rest :Log |
Raise
This naming reflects the fact that subsequent exceptions are logged, not ignored.
-
In
Async_kernel
, movedScheduler.set_execution_context
from theTypes.Scheduler
module to its own file. BecauseTypes
is amodule rec
,set_execution_context
hadn't been inlined and was called viacaml_apply2
. In its own file, it will be inlined.This release creates a new scheduler0.ml, and moves the old scheduler0.ml to scheduler1.ml.
-
Fixed a space leak in
Pipe
due to a pipe holding pointers to itsupstream_flusheds
after they are closed. The leak shows up inPipe.transfer
and related functions, e.g. with:Pipe.transfer temporary_pipe long_lived_pipe
called repeatedly, in which
long_lived_pipe
would accumulate a large number ofupstream_flusheds
.The fix is to maintain
upstream_flusheds
as aBag.t
, and to remove an upstream pipe when it is closed. Implement
Pipe.of_sequence
Improved the error message when an exception is raised to a
Monitor.try_with
that has returned before Async has initializedMonitor.try_with_log_exn
.-
Improved the implementation of
Monitor.get_next_error
, by replacing the monitor's list of handlers:; mutable handlers_for_next_error : (exn -> unit) list
with a single ivar:
; mutable next_error : exn Ivar.t
I think this wasn't done originally because of a dependency cycle. But now that we have types.ml, we can do the clear thing.
Improved the implementation of Monitor exception handling, i.e.
detach_and_iter_errors
to make it clear thatMonitor.send_exn
does not run user code -- it only schedules jobs.Fix an error message in
Pipe
to match the condition that led to it.-
Add a new pipe constructor:
val unfold : 'b -> f:('b -> ('a * 'b) option Deferred.t) -> 'a Reader.t
unfold
is more powerful than the combination ofUseful for, e.g., creating a pipe of natural numbers:
Pipe.unfold 0 ~f:(fun n -> return (Some (n, n+1)))
-
Add
Deferred.Map.all
similar toDeferred.List.all
.This does what you would expect:
val all : ('a, 'b Deferred.t, 'cmp) Map.t -> ('a, 'b, 'cmp) Map.t Deferred.t
-
Added some simple functions that seem missing from
Deferred
andIvar
.val Ivar.peek : 'a t -> 'a option val Ivar.value_exn : 'a t -> 'a val Deferred.value_exn : 'a t -> 'a
Add
Monitor.catch_error
, which provides error handling for processes/subsystems intended to run forever.-
Added to the Async scheduler a configurable:
min_inter_cycle_timeout : Time_ns.Span.t
When scheduler calls epoll(), it uses a timeout of at least
min_inter_cycle_timeout
.min_inter_cycle_timeout
can be configured viaASYNC_CONFIG
or viaval Scheduler.set_min_inter_cycle_timeout : Time_ns.Span.t -> unit
This allows one to tweak the scheduler to be more fair w.r.t. threads, e.g. with:
Scheduler.set_min_inter_cycle_timeout <- Time_ns.Span.of_us 1.;
Optimized
Scheduler.schedule'
to avoid a closure allocation.Removed
Monitor.kill
, which was unused internally. This removes thekill_index
field fromExecution_context.t
, which saves us a word everytime we allocate or store an execution context.Assert that
Deferred.forever
never returns statically, rather than dynamically.Changed
Async.Std
to not includeDeferred.Monad_syntax
, so that one must explicitly opt in (viaopen Deferred.Monad_syntax
) to uselet%bind
syntax with Async.Add
Pipe.to_sequence
-
Make
Stream.closed s
return immediately whens
is already closed.Currently the following property holds:
for any s, Deferred.peek (Stream.closed s) = None
-
For
Pipe
functions that deal with batches of elements in a queue, added an optional argument:?max_queue_length : int (** default is
Int.max_value
*)This limits the size of the queue that is used, which can improve Async fairness.
Affected functions are:
filter_map filter_map' fold' iter' map' read' read_now' transfer' transfer_id
This also obviates
read_at_most
andread_now_at_most
, which we will deprecate in a later release.Removed a couple helper types,
iter
andfold
, that had been used to express commonality among functions, but were becoming unwieldy due to differences. Changed
Pipe
's defaultmax_queue_length
fromInt.max_value
to 100.-
Added to
Pipe.iter_without_pushback
an optional argument:?max_iterations_per_job : int (** default is
Int.max_value
*)iter_without_pushback
will not make more thanmax_iterations_per_job
calls tof
in a single Async_job; this can be used to increase Async-scheduling fairness. -
Added
Pipe.write_if_open
which does exactly what it says. This is a common pattern. Also added a pushback-oblivious variantwrite_without_pushback_if_open
.Call sites for these two new functions were introduced wherever I found that doing so would not introduce any side effects (even counting allocation) in the case of a closed pipe.
async_parallel
- Switched to ppx.
async_rpc_kernel
When
Transfer.Writer.send*
raises, send an error to the client.Added
Pipe_rpc
inVersioned_rpc.Both_convert
.Switched to ppx.
Expose the lower-level registration hook in
Versioned_rpc
.Allow custom handling of missed async_rpc heartbeats.
-
Client-side State_rpc
dispatch
function does not behave well when the reader side of the pipe is closed.It should gracefully abort the rpc instead of raising exceptions, or whatever it currently does.
-
Add
Rpc.Expert.implement
andRpc.Expert.implement'
with a similar interface asOne_way.Expert.implement
but for 2-way rpcs.Exceptions raised by an expert implementations are handled as follow:
- if the query has already been answered, stop the server (as for one-way expert)
- if not, send a
Rpc_error.Uncaught_exn
(as for 2-way rpc)
-
Adds
Rpc.Pipe_rpc.dispatch_iter
, plus a bunch of additional types to support it. The main reason for this is to reduce space usage:Pipe_rpc.dispatch
followed byPipe.iter_without_pushback
consumes ~105 words in the steady state (i.e., when no messages are coming in) whiledispatch_iter
consumes ~15. I'm suredispatch
can be improved a lot, but a pipe by itself is 46 words, so it can't possibly become as small asdispatch_iter
.Both cases can be made smaller by making
Connection.response_handler
a GADT instead of a closure. I plan to do this later.One annoying property of the interface is that the only way to cancel a subscription is to use
Pipe_rpc.abort
, which has a terrible interface. The logical way to improve the interface is to return a record of aPipe_rpc.t
, aConnection.t
, and aQuery_id.t
, which allocates an additional few words. I'd kind of like to do this but it seems counter to the goal of reducing space usage. -
Adds
Rpc.Pipe_rpc.implement_direct
, which uses a "direct stream writer" to write results to the other side, rather than using aPipe.Writer.t
. This consumes much less memory, ~15 words per open query as opposed to ~225 for a regularPipe_rpc
implementation.A large part of the diff in this feature is the addition of a module
Implementation_types
, which just contains copies of type definitions fromImplementation
andImplementations
. This is necessary to handle some cyclic type definitions (both of these modules now depend on types defined in the other module).This is the implementation-side dual of
Pipe_rpc.dispatch_iter
. -
Change
Command_rpc.Command
to useVersioned_rpc.Callee_converts
instead ofVersioned_rpc.Both_convert
so that commands can be constructed without client-side conversions. Clients remain free to use conversions or not, as appropriate.Removed
val rpcs
fromCallee_converts
interfaces because nothing appears to use it, andBoth_convert
does not provide it. NowBoth_convert.S
can be supplied to satisfyCallee_converts.S
. Annotate errors returned by the async-rpc library with the name of the RPC for which the error was returned (if it's an rpc-level error) and a description of the remote side of the connection (the ip:host if connected via a network socket).
Bring back
val rpcs
in versioned_rpc modules.
async_shell
Initial release.
async_smtp
Switched to PPX.
Follow Core & Async evolution.
async_ssl
- Switched to ppx.
async_unix
-
In the periodic check for a Writer buffer have too old data, eliminated allocation and generally improved performance.
This eliminated a large source of allocation in a simple TCP pingpong benchmark
bench/pingpong
. -
Removed allocation in the Async scheduler's code that detects the thread-pool being stuck. This involved switching it to use
Time_ns
rather thanTime
.This eliminates a relatively large source of allocation in a simple TCP-pingpong benchmark
bench/pingpong
. Switched to ppx.
Improved the Async scheduler's to allocate a
handle_fired
function once, rather than every time it callsadvance_clock
.Added
Fd_by_descr.find_exn
, to avoid the option allocated byFd_by_descr.find
. Used it to reduce allocation in the Async scheduler.Improved
Reader.load_sexp*
functions to behave better when loading from a non files, e.g. a pipe. Previously, it produced an empty error message because it mistakenly attempted to read the sexp a second time in order to determine the error position. Now it produces a good error message, but without the (impossible to obtain) error position.In
Async_unix.Syscall
, added a zero-allocation syscall interface, removing sources of allocation as observed when running a simple TCP pingpong benchmark (found inbench/pingpong
).-
Added
val time_spent_waiting_for_io : unit -> Time_ns.Span.t
to
Scheduler
which returns the amount of time that the Async scheduler has spent in calls toepoll_wait
(orselect
) since the start of the program. -
Changed
In_thread.Helper_thread.create
from:val create : ?priority:Priority.t -> ?name:string -> unit -> t Or_error.t
to:
val create : ?priority:Priority.t -> ?name:string -> unit -> t Deferred.t
Kept around the prior function, renamed as
create_now
.Switching
create
to return a deferred allows it to, when there are no available threads, wait until one becomes available. This, in turn, avoids rare nondeterminstic failures in programs that make heavy use of the thread pool and create a helper thread, when the creation happens at just the wrong time, when no thread is available.Split out
Thread_safe_ivar
from the internals ofThread_pool
, so that it can be used in other tests, and in particular in a new unified test added by this feature. Make
Unix_syscalls.Stats
bin-io-able.Fixed a bug in
Thread_safe.block_on_async*
, in which the execution context wasn't properly restored before returning.Add a function in
Process
that expects empty output, mirroringShell.run
.-
Added
Reader.read_one_iobuf_at_a_time
, which is likeread_one_chunk_at_a_time
, except that the user-suppliedhandle_chunk
function receives its data in anIobuf.t
, and uses theIobuf
position to communicate how much data was consumed.This facilitates using reader in scenarios (such as with protogen) where
Iobuf
s are expected (and presently allocated around the bigstring at each call) and the calculation of consumed bytes from theIobuf
is duplicated in few places. Log.message
used to always logs the message, even if its log level was too low. This has been fixed.Add writer functions to schedule an iobuf to be written out.
Add
Unix.Inet_addr.Stable
.-
Alter
Async.Std.Socket.Address.Inet.Blocking_sexp
to expose the polymorphic variant functions, so that you can include it in a separate polymorphic variant type.Also, expose
Async.Std.Socket.Address.Inet.__t_of_sexp__
to give a deprecation message, instead of a message about the function not existing. -
Fixed a bug in Async's
In_thread.Helper_thread
, which wasn't finalizing helper threads, due to a bug inThread_pool
, which wasn't finalizing helper threads. The fix was to move the finalization out ofThread_pool
, where we don't care about it, toIn_thread.Helper_thread
, where we do.Added
Scheduler.max_num_threads : unit -> int
. -
Make
Epoll_file_descr_watcher
trigger callbacks for error conditions such as closed pipes.Three new unit tests, all validating appropriate behavior in the case that a Unix pipe is opened, then the read end is closed after reading only part of the data sent by the write end.
A test in writer.ml verifying that
Writer.consumer_left
is triggered. Before the fix toEpoll_file_descr_watcher
,Writer.consumer_left
would never become determined in this case.A test in fd.ml verifying that
Fd.ready_to
is triggered for the writing fd. Before the fix toEpoll_file_descr_watcher
,Fd.ready_to
would never become determined in this case.A test in linux_ext.ml verifying that
Epoll.wait_timeout_after
is triggered. This test shows that epoll reports the ERR flag for the file descriptor in this case, and therefore thatEpoll_file_descr_watcher
needs to pay attention to the ERR flag.
Added to
Writer.write_sexp
an optional?terminate_with
argument, that specifies how to terminate the string representation of the sexp. This also makes it clear that the default behavior,~terminate_with:Space_if_needed
, might append a space to the sexp you are outputting if its representation is not enclosed in either () or "" .Sexp.output_hum
anSexp.output_mach
do not have this behavior, so porting non-async code to async could introduce unexpected differences in the output.Add an Async wrapper for
Core.Std.Unix.getifaddrs
.
bignum
Switched to PPX.
The library used polymorphic compare, rather than
Zarith.Q
's compare, in a few locations. Fixed this.Previously stable types in Bignum were defined with unstable types in the scope. Fixd this.
Update to zarith-1.4
-
Bignum.of_string
needs to handle different formats for its input. The previous version of the code was trying to parse the common format (floats), and in case of failure, was attempting to use a different format (based on the error). This resulted in the string being parsed twice in some cases.This version is a complete rewriting of
of_string
to do the parsing in one step. The new code forto_string
encode an automaton and remembers the positions of the various elements of the string (depending on the format).This feature uses a function which has been upstreamed in the new version of ZArith (1.4) which is a variant of the
Zarith.of_string
function to work with substrings. This variant alone is responsible for a big part of the performance improvement.The new version of the code performs better than the original one in all cases. The performance improvement are variable depending on the micro benchmark. See below.
We also tried to implement the lexing engine using OCamllex. This makes for a much more concise description, but the performance are significantly lower. OCamllex produces code which allocates some table and some state, which is avoided in the hand written code. Also, it will allocate the sub strings matched.
New version (patch for ZArith + of_substring, reimplementation of of_string)
┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │
bigint\_bench.ml
random │ 48_381.13ns │ 7_166.00w │ 1.24w │ 1.24w │ 45.12% │ │bigint\_bench.ml:vs. Big\_int
plus_self │ 293.96ns │ 72.00w │ │ │ 0.27% │ │bigint\_bench.ml:vs. Big\_int
plus_other │ 807.62ns │ 124.00w │ │ │ 0.75% │ │bigint\_bench.ml:vs. Big\_int
mult_self │ 353.98ns │ 91.00w │ │ │ 0.33% │ │bigint\_bench.ml:vs. Big\_int
mult_other │ 783.78ns │ 128.00w │ │ │ 0.73% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (decimal) │ 14_415.44ns │ 475.00w │ │ │ 13.44% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (scientific) │ 61_363.80ns │ 3_929.00w │ │ │ 57.23% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (fraction) │ 24_957.02ns │ 303.00w │ │ │ 23.28% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (decimal) │ 15_867.52ns │ 1_523.00w │ │ │ 14.80% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (scientific) │ 33_345.31ns │ 4_206.00w │ │ │ 31.10% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (fraction) │ 31_770.26ns │ 3_779.00w │ │ │ 29.63% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (decimal) │ 9_726.82ns │ 380.00w │ │ │ 9.07% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (scientific) │ 28_141.40ns │ 2_059.00w │ │ │ 26.25% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (fraction) │ 70_436.16ns │ 5_541.00w │ │ │ 65.69% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (decimal) │ 27_000.73ns │ 1_994.00w │ │ │ 25.18% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (scientific) │ 66_057.63ns │ 6_217.00w │ │ │ 61.61% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (fraction) │ 107_219.89ns │ 8_097.00w │ │ │ 100.00% │ │bignum\_bench.ml:Bignum binprot
roundtrip compact │ 5_997.81ns │ 581.00w │ │ │ 5.59% │ │bignum\_bench.ml:Bignum binprot
roundtrip classic │ 18_522.20ns │ 779.00w │ │ │ 17.27% │ │bignum\_bench.ml:round
round_decimal:0 │ 8_479.49ns │ 463.00w │ │ │ 7.91% │ │bignum\_bench.ml:round
round_decimal:3 │ 24_621.71ns │ 2_115.00w │ │ │ 22.96% │ │bignum\_bench.ml:round
round_decimal:6 │ 26_896.35ns │ 2_437.00w │ │ │ 25.09% │ │bignum\_bench.ml:round
round_decimal:9 │ 29_428.19ns │ 2_730.00w │ │ │ 27.45% │ │bignum\_bench.ml:round
round │ 8_452.31ns │ 459.00w │ │ │ 7.88% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘Original version
┌─────────────────────────────────────────────────────────────────────┬──────────────┬───────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼───────────┼──────────┼──────────┼────────────┤ │
bigint\_bench.ml
random │ 51_218.04ns │ 7_166.00w │ 1.25w │ 1.25w │ 43.26% │ │bigint\_bench.ml:vs. Big\_int
plus_self │ 336.84ns │ 72.00w │ │ │ 0.28% │ │bigint\_bench.ml:vs. Big\_int
plus_other │ 837.73ns │ 124.00w │ │ │ 0.71% │ │bigint\_bench.ml:vs. Big\_int
mult_self │ 411.03ns │ 91.00w │ │ │ 0.35% │ │bigint\_bench.ml:vs. Big\_int
mult_other │ 808.03ns │ 128.00w │ │ │ 0.68% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (decimal) │ 29_650.60ns │ 2_415.00w │ │ │ 25.04% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (scientific) │ 92_495.93ns │ 6_465.00w │ │ │ 78.12% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (fraction) │ 39_482.77ns │ 2_060.00w │ │ │ 33.35% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (decimal) │ 16_195.93ns │ 1_523.00w │ │ │ 13.68% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (scientific) │ 34_227.78ns │ 4_059.00w │ │ │ 28.91% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (fraction) │ 32_856.17ns │ 3_779.00w │ │ │ 27.75% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (decimal) │ 19_745.71ns │ 2_149.00w │ │ │ 16.68% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (scientific) │ 51_024.99ns │ 3_853.00w │ │ │ 43.09% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (fraction) │ 88_884.15ns │ 7_819.00w │ │ │ 75.07% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (decimal) │ 32_812.27ns │ 2_498.00w │ │ │ 27.71% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (scientific) │ 77_518.77ns │ 6_369.00w │ │ │ 65.47% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (fraction) │ 118_402.78ns │ 8_907.00w │ │ │ 100.00% │ │bignum\_bench.ml:Bignum binprot
roundtrip compact │ 8_947.02ns │ 371.00w │ │ │ 7.56% │ │bignum\_bench.ml:Bignum binprot
roundtrip classic │ 22_799.74ns │ 1_039.00w │ │ │ 19.26% │ │bignum\_bench.ml:round
round_decimal:0 │ 8_176.74ns │ 463.00w │ │ │ 6.91% │ │bignum\_bench.ml:round
round_decimal:3 │ 25_798.77ns │ 2_115.00w │ │ │ 21.79% │ │bignum\_bench.ml:round
round_decimal:6 │ 28_561.23ns │ 2_437.00w │ │ │ 24.12% │ │bignum\_bench.ml:round
round_decimal:9 │ 30_861.38ns │ 2_730.00w │ │ │ 26.06% │ │bignum\_bench.ml:round
round │ 8_237.26ns │ 459.00w │ │ │ 6.96% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴───────────┴──────────┴──────────┴────────────┘Tentative version using OCamllex
┌─────────────────────────────────────────────────────────────────────┬──────────────┬────────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├─────────────────────────────────────────────────────────────────────┼──────────────┼────────────┼──────────┼──────────┼────────────┤ │
bigint\_bench.ml
random │ 48_164.21ns │ 7_166.00w │ 1.25w │ 1.25w │ 39.99% │ │bigint\_bench.ml:vs. Big\_int
plus_self │ 285.84ns │ 72.00w │ │ │ 0.24% │ │bigint\_bench.ml:vs. Big\_int
plus_other │ 768.12ns │ 124.00w │ │ │ 0.64% │ │bigint\_bench.ml:vs. Big\_int
mult_self │ 343.14ns │ 91.00w │ │ │ 0.28% │ │bigint\_bench.ml:vs. Big\_int
mult_other │ 780.00ns │ 128.00w │ │ │ 0.65% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (decimal) │ 26_931.12ns │ 3_108.00w │ │ │ 22.36% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (scientific) │ 79_750.28ns │ 6_599.00w │ 0.11w │ 0.11w │ 66.21% │ │bignum\_bench.ml:Bignum of\_string/to\_string
of_string (fraction) │ 34_988.94ns │ 4_300.00w │ │ │ 29.05% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (decimal) │ 15_958.17ns │ 1_523.00w │ │ │ 13.25% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (scientific) │ 32_495.25ns │ 4_059.00w │ │ │ 26.98% │ │bignum\_bench.ml:Bignum of\_string/to\_string
to_string (fraction) │ 31_802.75ns │ 3_779.00w │ │ │ 26.40% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (decimal) │ 18_742.81ns │ 2_924.00w │ │ │ 15.56% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (scientific) │ 45_282.09ns │ 4_622.00w │ │ │ 37.60% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
of_sexp (fraction) │ 86_907.83ns │ 8_777.00w │ 0.15w │ 0.15w │ 72.16% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (decimal) │ 35_727.73ns │ 4_493.00w │ │ │ 29.66% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (scientific) │ 82_247.61ns │ 8_273.00w │ 0.13w │ 0.13w │ 68.29% │ │bignum\_bench.ml:Bignum of\_sexp/to\_sexp
to_sexp (fraction) │ 120_445.25ns │ 10_688.00w │ 0.12w │ 0.12w │ 100.00% │ │bignum\_bench.ml:Bignum binprot
roundtrip compact │ 6_734.49ns │ 371.00w │ │ │ 5.59% │ │bignum\_bench.ml:Bignum binprot
roundtrip classic │ 21_773.79ns │ 1_890.00w │ │ │ 18.08% │ │bignum\_bench.ml:round
round_decimal:0 │ 8_306.45ns │ 463.00w │ │ │ 6.90% │ │bignum\_bench.ml:round
round_decimal:3 │ 24_714.96ns │ 2_115.00w │ │ │ 20.52% │ │bignum\_bench.ml:round
round_decimal:6 │ 26_894.27ns │ 2_437.00w │ │ │ 22.33% │ │bignum\_bench.ml:round
round_decimal:9 │ 29_343.81ns │ 2_730.00w │ │ │ 24.36% │ │bignum\_bench.ml:round
round │ 8_296.05ns │ 459.00w │ │ │ 6.89% │ └─────────────────────────────────────────────────────────────────────┴──────────────┴────────────┴──────────┴──────────┴────────────┘
bin_prot
-
Bin_prot can be configured to use the primitives to read/write integers from bigarrays. This was never enabled due to missing tests that selecting this code path doesn't change the format.
This version add these tests and enable the use of the fast primitives.
Add benchmarks for all exposed bin_prot read/write functions. These are intended to check performance regressions.
-
Remove most use of cpp in bin_prot.
Replace the pre-processor conditionals by runtime one. This make the code a little less obfuscated.
Remove big literals so that the compiler does not complain in 32bit
core
N.B. Some interface change were made which are not listed here, as they are only
cascading from core_kernel
. Look at core_kernel's CHANGES.md file to get a
complete history of the changes made for this release.
Add
Core.Command.Arg_type.Export.time_zone
-
Fix
Command.shape
to run external programs only once to get their sexp.Introduces a new variant of Command.t called Proxy. The Exec variant represents the top-level command of an external executable that has not yet been run; the Proxy variant represents an arbitrary subcommand that has been extracted by running an external executable. Command.exec constructs an Exec variant; Command.shape of an Exec variant runs the executable and generates a tree of Proxy variants representing all the information from the generated sexp, so the executable will not need to be re-run.
-
A version of recvmmsg that pre-allocates and reuses the iovec record. Profiling indicates this is a non-trivial amount of our I/O loop (under very heavy load).
Since nobody is using the heavyweight features of the existing recvmmsg, replace it with the lightweight one. This leads to minor but important changes to the interfaces of
Iobuf.recvmmsg_assume_fd_nonblocking
andUdp.recvmmsg_loop
. Switch to ppx.
-
Time.set_sexp_zone
affectst_of_sexp
If we're willing to read sexps without zones, we should be willing to let you control what timezone they're read with.
Sped up
Llimiter
.Make
Interval.Int
implementContainer
andBinary_searchable
.Add
Identifiable.S
toUnix.Cidr
so that it supports hash tables. UpdateUnix.Cidr.t
to normalize values, e.g. "192.168.1.101/24" ==> "192.168.1.0/24".Added "noalloc" attribute to
Linux_ext.unsafe_timerfd_settime
.-
In Iobuf, made some functions take:
(
> write
, _) Iobuf.trather than:
(read_write, _) Iobuf.t
if they only need to write to the iobuf and don't need to read it.
Time.of_string_abs
didn't support ISO 8601 time zone strings without colons, or those specified as locations. This version adds support for time zones without colonsUnix.Passwd.getpwents
takes the lock, partially appliesExn.protect
, then releases the lock, then completes the application and actually runs stuff.-
Command.file-completion
Files are now completed correctly when paths contain a directory.
Previously, completion when pointed at a directory would put a space at the end. This would cause the user to hit backspace every time a directory was in the path.
Add
diff_weekdays
anddiff_weekend_days
functions to date module.Time.to_date_ofday_precise
implements a complete inverse forof_date_ofday
. This is needed to give the DWIM-est semantics to Schedule.t that we can think of.Reduce allocation in Linux_ext.Epoll
Add
Time_ns.Of_day.(add_exn, sub_exn, diff)
.Adding
head_padded_fixed_string
to Iobuf.Move
Core_extended.Std.Sys.home
toCore.Std.Sys.home_directory
.Add
Iobuf.{read,write,input,output}
akin to the bigstring versions.Add expert iobuf functions for extracting bigstrings and iovecs that share the iobuf's underlying storage.
Add stable
Int63.t
conversions for types inTime_ns
.Rename DNS-based
Inet_addr
sexp conversions to expose their blocking nature.Add
Unix.Inet_addr.Stable
.Add [Core.Std.Schedule].
-
Fixed
Iobuf_packet.iter
, which behaved incorrectly if the packet didn't start at the lo_min of the iobuf. It usedIobuf.rewind
when it should have usedIobuf.Lo_bound.restore
.Added
@@deriving compare
toIobuf.Bound
. Add ability to
Time.format
for a specific time zone-
Make more information available via Command.Shape.t
Expose machine-readable info on anonymous arguments.
Remove unnecessary rebinding of
(^/)
incore_filename.ml
. It had one call site and wasn't exposed. The(^/)
everyone uses comes from std.ml-
Add
getifaddrs
toCore.Std.Unix
.Handles Packet (for interfaces that do not have an address on Linux systems only), IPv4 and IPv6 address families.
Implement
Time_ns.Span.to_string_hum
by analogy toTime.Span.to_string_hum
. The code and tests are essentially copied from "lib/core/src/span.ml".-
Remove our stubs for
Unix.stat
.They were upstreamed in 4.02.2.
core_bench
- Switched to ppx.
core_extended
N.B. Some interface changes occured in Core which are repercuted in this package, they are not all list in this file though.
Switched to PPX.
Upgrade
Interval_map.t
with monad operations.-
Update the
interval_map_intf.ml
file to try to make the documentation clearer.This mostly constitutes splitting out the core operations into a separate module type earlier in the file, so that their documentation occurs before the various more specific module types in reading order.
Various bits of documentation have been tweaked with examples or laws.
Add underscores to color print names Improve and uniformize the behavior of colorprintf functions at the cost of changing the type slightly
-
Fix core_extended stubs on openbsd
Closes #7 Closes #2
Move
Core_extended.Std.Sys.home
toCore.Std.Sys.home_directory
.-
Add a module whose type
'a t
acts as a container of ordered items of type 'a (morally, a'a list
) but which supports efficient append operations.Sometimes called a Rope, or Concatenable_list.
-
Expose the constructors of
Ascii_table.Align.t
so that we can writeColumn.create ~align:Left ...
instead of
Column.create ~align:Align.left
Fix sexp diffing on records The wrong comparaison was leading to huge diff as soon as one field was missing on one side
core_kernel
Add
Container.Make0
for monomorphic container types.Improved the performance of the implementation of
Bounded_int_table.find
.Switched to ppx
Remove references to
Core_list
fromSequence
.Added functions to
Bigstring
andIobuf
for reading unsigned 64-bit integers.Move
Comparable.bound
toMaybe_bound.t
. The purpose is to break up dependencies between the two.Doubly_linked
allocated during iteration. This became a large source of allocation for simple benchmarks like TCP pingpong (async/bench/pingpong
). Some unnecessary allocations have been removed.Added
Timing_wheel.next_alarm_fires_at_exn
, which is useful to avoid allocation when you know the timing wheel isn't empty.Make versions of
Binary_searchable.Make*
that don't require aFor_test
argument. This allowsBinary_searchable.Make
to be used for types that don't easily convert from arrays.-
Add
Quickcheckable
interface to Core and move generators/observers into type modules.Renames core_list.ml to core_list0.ml, then adds a new core_list.ml with quickcheck generators and observers. This allows quickcheck.ml to use core_list0.ml without a dependency cycle.
The feature also moves the contents of quickcheck.mli into quickcheck_intf.ml.
-
Made
Core.Unpack_buffer.Unpack_one.t
be a unary type rather than a binary one, by hiding itspartial_unpack
type under an existential.This makes it possible to make
Unpack_one
into a monad because we can combine twoUnpack_one.t
's with differentpartial_unpack
types into a newUnpack_one.t
with a differentpartial_unpack
type. https://github.com/janestreet/core\_kernel/pull/20 Core.Std module is not found when compiling lib_test/pool_caml_modify_check.ml.
-
Added an optional argument
?key_order
for specifying the order of Map.to_alist output: eitherIncreasing or
Decreasing.The default key order is no longer left unspecified: we're now committed to the `Increasing, which was the old behavior.
Add Sexpable.Of_sexpable2 functor, for symmetry with Binable.Of_binable2. Add sexpable.mli
Added a function for sequencing computations stored in a total map:
Total_map.sequence
.Added
Core.Bus
, a publisher/subscriber system within the memory space of the program. This is a synchronous version ofAsync.Bus
.Added
Core_map.fold2
(fold based on the contents of two maps side-by-side).Core.Interfaces
defines theUnit
module type to besig end
. Increase uniformity with its other definitions by defining it to beUnit.S
instead.Adapt
Core_random.int
to accept larger values than1 lsl 30
.Mark the
Sexpable.Of_*
andBinable.Of_*
functors as stable.In
Core_char.int_is_ok
, used byof_int
andof_int_exn
, use int compare instead of polymorphic compare.Fix a few files where toplevel side effects might not be running when we don't pack libraries anymore and use -no-alias-deps.
In
Char.For_quickcheck
, memoize construction of the filtered chars generators, since if they are used once, they are likely to be used many times, and the construction is costly compared to generating a single char.Extend
Core_map
to implement quickcheckable ExtendCore_set
to implement quickcheckableIn
Avltree.add
, replace?(replace = true)
with~replace
. This both makes the behavior more explicit, and saves some allocation occasionally.Reimplement
Avltree.iter
directly, rather than usingfold
, which requires allocating a closure. This winds up being expensive duringHashtbl.iter
.-
Add a function in Blang to deal with boolean expressions E representing the membership of elements in a set, given a universe U and a function projecting each atoms of E to a subset of U.
{
Blang.eval_set ~universe:js_tech resolve_named_set ("(or (and has-blue-eyes has-brown-hair) (and has-brown-eyes has-blue-hair))" |> Sexp.of_string |> t_of_sexp)
} Expose more functions in univ_map interface
Made
Random.self_init
by default raise if used in inline tests. One can opt out by passing~allow_in_tests:true
.In core_hashtbl.ml,
maybe_resize_table
allocates the same closure in each iteration of a for loop. Allocate it just once.Hashtbl.remove_one
andHashtbl.remove_multi
are the same function, written twice. Removeremove_one
and replace uses withremove_multi
.-
Bigstring.unsafe_{get,set}-{,u}int8
used the generic bigarray access function without a type annotation. As a result the compiler generated a call to the generic C function.Fixed this by adding type annotations.
Add new functions to map that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules.
-
Made
Unpack_buffer.Unpack_one
monadic so that users can easily compose file parsersAdded a couple simple parsers as examples and for testing.
-
Avoid use of polymorphic compare in Quickcheck. Make
Quickcheck.Generator.bind_choice
lazy: do not eagerly descend into all branches.Reduces memory overhead by setting a threshold on the probability of choices that are remembered and discarded by
Quickcheck.iter
and friends.Motivation: Currently,
Quickcheck.iter
and related functions guarantee never to repeat a choice from a generator. This winds up recording every choice ever made, which for a lot of generators is a prohibitive cost in space, and most of the recorded values are very unlikely to be repeated anyway.Implementation: This feature sets a probability threshold below which choices will not be remembered. Choosing a fairly low, but still non-zero, threshold means values are still very unlikely to be repeated, but memory usage stays low.
As of this version, the benefits of "forgetting" unlikely-to-repeat values:
┌──────────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │
quickcheck.ml:Quickcheck.iter
remember │ 20.26ms │ 16.33Mw │ 100.85kw │ 100.85kw │ 100.00% │ │quickcheck.ml:Quickcheck.iter
forget │ 17.65ms │ 16.21Mw │ 34.83kw │ 34.83kw │ 87.10% │ └──────────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ -
Optimizations to:
- various
Float.t
validation functions - various functions in
Validate
List.fold_right
- various
Made the type of
Option.compare
compatible with@@deriving compare
.Fixed an example code fragment in a comment in applicative_intf.ml
-
In
Core_hashtbl
, theadd_worker
function used abool ref
both internally and to pass toAvltree
to track whether a new key is added. This was allocated on every call toadd
orset
, andset
didn't even use its contents.This version pre-allocates the
bool ref
inside eachCore_hashtbl.t
and reuses it. It still can't be amutable
field because it does need to be passed toAvltree
.After change:
┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set
no collisions
│ 84.73ns │ 3.00w │ 0.83w │ 0.83w │ 0.01% │ │ Hashtbl.setw/ collisions
│ 112.46ns │ │ │ │ 0.02% │ │ Hashtbl.changeno collisions
│ 82.74ns │ 3.50w │ 0.53w │ 0.53w │ 0.01% │ │ Hashtbl.changew/ collisions
│ 191.50ns │ 4.56w │ 1.15w │ 1.15w │ 0.03% │ │ Hashtbl.mergeno collisions
│ 292_976.43ns │ 26_669.00w │ 15_381.62w │ 12_305.62w │ 48.52% │ │ Hashtbl.mergew/ collisions
│ 603_822.86ns │ 33_001.00w │ 20_037.22w │ 16_961.22w │ 100.00% │ │ Hashtbl.add_exnno resize, no collisions
│ 80_992.57ns │ 3_088.00w │ 4_102.63w │ 3_077.63w │ 13.41% │ │ Hashtbl.add_exnno resize, w/ collisions
│ 178_080.05ns │ 4_621.00w │ 5_668.61w │ 4_643.61w │ 29.49% │ │ Hashtbl.add_exnw/ resize, no collisions
│ 176_442.98ns │ 16_403.00w │ 9_222.64w │ 6_148.64w │ 29.22% │ │ Hashtbl.add_exnw/ resize, w/ collisions
│ 297_577.29ns │ 19_472.00w │ 12_292.13w │ 9_218.13w │ 49.28% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘Before change:
┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set
no collisions
│ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.setw/ collisions
│ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.changeno collisions
│ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.changew/ collisions
│ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.mergeno collisions
│ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.mergew/ collisions
│ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exnno resize, no collisions
│ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exnno resize, w/ collisions
│ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exnw/ resize, no collisions
│ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exnw/ resize, w/ collisions
│ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ -
In
Core_hashtbl.add_worker
, removed amatch
that avoided callingAvltree.add
, but actually did hurt performance overall.Perhaps at some point before cross-module inlining, this was a helpful optimization. Right now it bypasses the mutation inside
Avltree
, so replacing a value in a non-colliding bucket (aLeaf
) causes unnecessary re-allocation of the leaf.After changes:
┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set
no collisions
│ 52.19ns │ 2.00w │ │ │ │ │ Hashtbl.setw/ collisions
│ 112.04ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.changeno collisions
│ 87.25ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.changew/ collisions
│ 195.85ns │ 5.56w │ 1.29w │ 1.29w │ 0.04% │ │ Hashtbl.mergeno collisions
│ 308_164.10ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.48% │ │ Hashtbl.mergew/ collisions
│ 526_914.80ns │ 38_119.00w │ 20_013.81w │ 16_937.81w │ 100.00% │ │ Hashtbl.add_exnno resize, no collisions
│ 76_983.60ns │ 5_135.00w │ 4_100.44w │ 3_075.44w │ 14.61% │ │ Hashtbl.add_exnno resize, w/ collisions
│ 174_712.92ns │ 6_668.00w │ 5_667.47w │ 4_642.47w │ 33.16% │ │ Hashtbl.add_exnw/ resize, no collisions
│ 176_681.57ns │ 19_476.00w │ 9_231.75w │ 6_157.75w │ 33.53% │ │ Hashtbl.add_exnw/ resize, w/ collisions
│ 280_448.62ns │ 22_545.00w │ 12_293.32w │ 9_219.32w │ 53.22% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘Before changes:
┌───────────────────────────────────────────────────┬──────────────┬────────────┬────────────┬────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├───────────────────────────────────────────────────┼──────────────┼────────────┼────────────┼────────────┼────────────┤ │ Hashtbl.set
no collisions
│ 104.88ns │ 5.00w │ 1.26w │ 1.26w │ 0.02% │ │ Hashtbl.setw/ collisions
│ 114.33ns │ 2.00w │ │ │ 0.02% │ │ Hashtbl.changeno collisions
│ 85.79ns │ 4.50w │ 0.58w │ 0.58w │ 0.02% │ │ Hashtbl.changew/ collisions
│ 198.75ns │ 5.56w │ 1.28w │ 1.28w │ 0.04% │ │ Hashtbl.mergeno collisions
│ 307_857.59ns │ 31_787.00w │ 15_380.91w │ 12_304.91w │ 58.19% │ │ Hashtbl.mergew/ collisions
│ 529_054.02ns │ 38_119.00w │ 20_015.32w │ 16_939.32w │ 100.00% │ │ Hashtbl.add_exnno resize, no collisions
│ 77_708.20ns │ 5_135.00w │ 4_101.83w │ 3_076.83w │ 14.69% │ │ Hashtbl.add_exnno resize, w/ collisions
│ 180_950.23ns │ 6_668.00w │ 5_638.77w │ 4_613.77w │ 34.20% │ │ Hashtbl.add_exnw/ resize, no collisions
│ 177_492.82ns │ 19_476.00w │ 9_237.07w │ 6_163.07w │ 33.55% │ │ Hashtbl.add_exnw/ resize, w/ collisions
│ 285_298.72ns │ 22_545.00w │ 12_330.90w │ 9_256.90w │ 53.93% │ └───────────────────────────────────────────────────┴──────────────┴────────────┴────────────┴────────────┴────────────┘ Add new functions to hashtbl that add common missing functionality and/or that makes the interface more uniform and consistent with other container modules.
Add a bunch of functions to list and array that add common missing functionality and/or that make their interfaces more uniform and consistent with other container modules.
-
Rewrite
Hashtbl.merge
to be simpler and faster.After changes:
┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge
no collisions
│ 172.57us │ 17.44kw │ 9.22kw │ 7.69kw │ 48.76% │ │ Hashtbl.mergew/ collisions
│ 284.55us │ 20.61kw │ 11.53kw │ 9.99kw │ 80.41% │ │ Pooled_hashtbl.mergeno collisions
│ 260.57us │ 5.20kw │ 19.18kw │ 3.09kw │ 73.63% │ │ Pooled_hashtbl.mergew/ collisions
│ 353.88us │ 5.20kw │ 19.18kw │ 3.09kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘Before changes:
┌──────────────────────────────────────┬──────────┬─────────┬──────────┬──────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├──────────────────────────────────────┼──────────┼─────────┼──────────┼──────────┼────────────┤ │ Hashtbl.merge
no collisions
│ 309.59us │ 31.79kw │ 15.38kw │ 12.30kw │ 48.91% │ │ Hashtbl.mergew/ collisions
│ 526.67us │ 38.12kw │ 19.97kw │ 16.90kw │ 83.21% │ │ Pooled_hashtbl.mergeno collisions
│ 469.41us │ 7.32kw │ 35.29kw │ 3.12kw │ 74.16% │ │ Pooled_hashtbl.mergew/ collisions
│ 632.96us │ 7.32kw │ 35.29kw │ 3.12kw │ 100.00% │ └──────────────────────────────────────┴──────────┴─────────┴──────────┴──────────┴────────────┘ -
Make
Hashtbl
functions raise an exception if a callback passed in as an argument mutates one of the hash tables being worked on.Usually, though not always, this comes up for iteration functions. Once a hash table has been mutated, it is unsafe to continue operating on it, as its structure may have changed. Buckets and their contents may have been moved or resized; continuing may result in skipping key/value pairs, repeating key/value pairs, or executing unsafe operations.
This feature adds a
mutation_allowed
flag to hash tables. Each mutating operation first checks the flag, and raises if it is not set. Each operation with callbacks that must not mutate unsets the flag before calling the callbacks, and restores the flag's original value when it finishes.We compared the timing of this implementation to an alternate implementation using a mutation counter, and the time and space used for this implementation was much better for iteration and within epsilon of the other for single-key operations like
set
. -
Array function names related to zipping are all over the place. Make them match List, which has a nice uniform naming scheme.
- Rename
combine
->zip_exn
- Rename
split
->unzip
- (
zip
remains named aszip
)
- Rename
Add
~key
and~data
labels toHashtbl.filteri_inplace
Added
Hash_set.to_hashtbl
, by analogy toSet.to_map
.-
Since we are mutating avltrees in place, make sure the compiler sees the type parameters as invariant.
Tested that a segfaulting example doesn't compile anymore.
-
Add label
f
to Hashtbl.change, Map.change, & family.Introduce the new function
update
in those modules, which enforces statically the presence of a resulting valueExample:
-|val Hashtbl.change : 'a t -> key -> ('a option -> 'a option) -> unit
+|val Hashtbl.change : 'a t -> key -> f:('a option -> 'a option) -> unit +|val Hashtbl.update : 'a t -> key -> f:('a option -> 'a) -> unit
The motivation for the introduction of
update
is that in an overwhelming majority of the places whereHashtbl.change
is used in our codebase, it is statically known that a new value shall be computed and stored. The use of the dynamism offered bychange
, which can return an option, is error prone.The addition of the label is considered acceptable in consideration to external libraries depending on core, because a missing label is just a warning, and we do not guarantee stability in the presence of -warn-error = true.
-
Changed
Source_code_position.t
from:@@deriving bin_io, sexp
to:
@@deriving sexp_of
and made
sexp_of
use the human-readable format,"FILE:LINE:COL"
, rather than the unreadable format. RemovedSource_code_position.t_hum
, which is now obsolete.If one wants a serialized source-code position, one can use
Source_code_position.Stable
. -
Added
Ref.set_temporarily
, for temporarily setting a ref to a value for the duration of a thunk.val set_temporarily : 'a t -> 'a -> f:(unit -> 'b) -> 'b
Add the function
singleton : 'a -> 'a t
in the stack containers. It cannot be added toContainer.S
directly because some container never have exactly 1 element.Made
Core.Array
matchInvariant.S1
.-
Change the interface of
Make_iterable_binable*
to give the control back to the user when deserializing Bin_protted data.Improve the bin_prot deserialization of
Map
s andSet
s. We construct a balanced tree directly instead of relying onMap.add
/Set.add
. This is possibile because the size of the map is known and elements are sorted.The complexity goes down from n.log(n) to n.
In case the comparison function changes (and the invariant is not respected), there is a fallback to reconstruct the whole map from scratch.
Add a function to blit a
Rope.t
into aBuffer.t
.-
Hashtbl differs from some other core containers with idiosyncratic naming of iteration functions. Change to be consistent and to more closely match the conventions for List and Array.
Hashtbl:
- Copy
iter
->iteri
. - Add a deprecation tag to
iter
.
- Copy
Made
Bag.invariant
andDoubly_linked.invariant
matchInvariant.S1
.-
Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current
Map.iter
vsMap.map
vsMap.mapi
). Change to be consistent and to more closely match the conventions for List and Array.Map:
- Copy
filter
->filteri
. - Add a deprecation tag to
filter
.
- Copy
-
Map differs from some other core containers with idiosyncratic naming of iteration functions. The current Map name conventions are also internally inconsistent as well (ex: current
Map.iter
vsMap.map
vsMap.mapi
). Change to be consistent and to more closely match the conventions for List and Array.Map:
- Copy
iter
->iteri
. - Add a deprecation tag to
iter
.
- Copy
Made
Core.Set_once
matchInvariant.S1
.Add
Bigstring.concat
.For
Core.Unique_id
, exposed@@deriving typerep
.Expose Hashtbl.hashable, analogous to Map.comparator.
Adds a constant-time
val mem_elt : 'a t -> 'a Elt.t -> bool
to Doubly_linked and BagAdd
Ordering.to_int
which can be useful when one is writing a comparison function. Instead of dealing with the int directly, one can return Ordering.t values and transform them later into ints.Float.int_pow
: Fast computation ofx ** n
when n is an integer.Make
Core_kernel.Std.Nothing.t
enumerable. There's no particular reason not to.Minor improvements to queue interface
-
Call
Caml.Pervasives.do_at_exit
before printing an exception and exitingThe default ocaml uncaught exception handler does this. It is especially useful for curses applications as the
at_exit
handler has a chance to put back the terminal in a good state before printing the exception and backtrace.Do the same in Core and Async.
Removed big literals so that the compiler does not complain in 32bit
Add
List.range'
, a generalization ofList.range
.-
Add some functions to
Map
that are present inHashtbl
:remove_multi
partition_tf
partitioni_tf
partition_map
partition_mapi
Add a
Map.nth_exn
as a missing complementary function to nthRenamed
Validate.fail_sexp
asfail_s
, to follow our new*_s
convention forSexp.t
-taking functions.-
Sequence.split_n_eagerly
returns a pair of sequences, but every element of the first sequence has already been evaluated by the time it returns. This feature just makes the first component of the tuple a list instead of a sequence, and renamessplit_n_eagerly
tosplit_n
.Additionally, this feature adds a new
chunks_exn
function, which just appliessplit_n
until the input sequence is empty. Removed
Timing_wheel
's defaultalarm_precision
, to force people to think about the precision they want when they create a timing wheel.In
Timing_wheel.Config.sexp_of_t
, used@sexp_drop_default
withlevel_bits
.-
Write a better-performing
Array.filter_mapi
function, and implementArray.filter_map
,Array.filter_opt
,Array.partitioni_tf
, andArray.partition_tf
in terms of it.Slightly worse for zero-length input arrays, about unch'd if we're filtering out almost everything (
eq_zero
), better on most everything else.┌────────────────────────────────────────────────────┬─────────────────┬─────────────┬─────────────┬─────────────┬────────────┐ │ Name │ Time/Run │ mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │ ├────────────────────────────────────────────────────┼─────────────────┼─────────────┼─────────────┼─────────────┼────────────┤ │
core\_array.ml:filter
old-filter-even:0 │ 12.37ns │ 9.00w │ │ │ │ │core\_array.ml:filter
old-filter-even:1 │ 77.44ns │ 15.00w │ │ │ │ │core\_array.ml:filter
old-filter-even:10 │ 207.10ns │ 36.00w │ │ │ │ │core\_array.ml:filter
old-filter-even:100 │ 1_699.41ns │ 261.00w │ │ │ │ │core\_array.ml:filter
old-filter-even:1000 │ 56_320.50ns │ 1_009.00w │ 2_506.01w │ 1_004.01w │ 0.30% │ │core\_array.ml:filter
old-filter-even:10000 │ 469_134.89ns │ 10_009.00w │ 25_007.38w │ 10_005.38w │ 2.46% │ │core\_array.ml:filter
old-filter-even:100000 │ 4_421_742.22ns │ 100_009.00w │ 250_130.09w │ 100_128.09w │ 23.17% │ │core\_array.ml:filter
new-filter-even:0 │ 13.87ns │ 14.00w │ │ │ │ │core\_array.ml:filter
new-filter-even:1 │ 57.64ns │ 18.00w │ │ │ │ │core\_array.ml:filter
new-filter-even:10 │ 196.28ns │ 35.00w │ │ │ │ │core\_array.ml:filter
new-filter-even:100 │ 1_361.04ns │ 215.00w │ │ │ │ │core\_array.ml:filter
new-filter-even:1000 │ 21_473.76ns │ 1_014.00w │ 1_001.02w │ │ 0.11% │ │core\_array.ml:filter
new-filter-even:10000 │ 204_033.12ns │ 10_014.00w │ 10_001.14w │ 0.14w │ 1.07% │ │core\_array.ml:filter
new-filter-even:100000 │ 2_058_144.47ns │ 100_014.00w │ 100_002.00w │ 1.00w │ 10.78% │ │core\_array.ml:filter
old-filter-eq_zero:0 │ 12.21ns │ 9.00w │ │ │ │ │core\_array.ml:filter
old-filter-eq_zero:1 │ 71.23ns │ 15.00w │ │ │ │ │core\_array.ml:filter
old-filter-eq_zero:10 │ 174.80ns │ 24.00w │ │ │ │ │core\_array.ml:filter
old-filter-eq_zero:100 │ 1_212.70ns │ 114.00w │ │ │ │ │core\_array.ml:filter
old-filter-eq_zero:1000 │ 23_347.51ns │ 13.00w │ 1_007.00w │ 6.00w │ 0.12% │ │core\_array.ml:filter
old-filter-eq_zero:10000 │ 210_509.83ns │ 13.00w │ 10_007.00w │ 6.00w │ 1.10% │ │core\_array.ml:filter
old-filter-eq_zero:100000 │ 1_912_253.91ns │ 13.00w │ 100_007.01w │ 6.01w │ 10.02% │ │core\_array.ml:filter
new-filter-eq_zero:0 │ 13.70ns │ 14.00w │ │ │ │ │core\_array.ml:filter
new-filter-eq_zero:1 │ 56.56ns │ 18.00w │ │ │ │ │core\_array.ml:filter
new-filter-eq_zero:10 │ 179.42ns │ 27.00w │ │ │ │ │core\_array.ml:filter
new-filter-eq_zero:100 │ 1_254.49ns │ 117.00w │ │ │ │ │core\_array.ml:filter
new-filter-eq_zero:1000 │ 20_968.06ns │ 16.00w │ 1_001.02w │ │ 0.11% │ │core\_array.ml:filter
new-filter-eq_zero:10000 │ 204_299.82ns │ 16.00w │ 10_001.13w │ 0.13w │ 1.07% │ │core\_array.ml:filter
new-filter-eq_zero:100000 │ 2_019_283.81ns │ 16.00w │ 100_001.91w │ 0.91w │ 10.58% │ │core\_array.ml:filter
old-filter-neq_zero:0 │ 12.14ns │ 9.00w │ │ │ │ │core\_array.ml:filter
old-filter-neq_zero:1 │ 32.72ns │ 11.00w │ │ │ │ │core\_array.ml:filter
old-filter-neq_zero:10 │ 219.18ns │ 48.00w │ │ │ │ │core\_array.ml:filter
old-filter-neq_zero:100 │ 1_902.76ns │ 408.00w │ 0.12w │ 0.12w │ │ │core\_array.ml:filter
old-filter-neq_zero:1000 │ 82_032.44ns │ 2_007.00w │ 3_998.20w │ 1_997.20w │ 0.43% │ │core\_array.ml:filter
old-filter-neq_zero:10000 │ 850_234.44ns │ 20_007.00w │ 40_014.86w │ 20_013.86w │ 4.46% │ │core\_array.ml:filter
old-filter-neq_zero:100000 │ 7_345_941.05ns │ 200_007.00w │ 400_407.82w │ 200_406.82w │ 38.49% │ │core\_array.ml:filter
new-filter-neq_zero:0 │ 13.66ns │ 14.00w │ │ │ │ │core\_array.ml:filter
new-filter-neq_zero:1 │ 18.26ns │ 14.00w │ │ │ │ │core\_array.ml:filter
new-filter-neq_zero:10 │ 201.04ns │ 43.00w │ │ │ │ │core\_array.ml:filter
new-filter-neq_zero:100 │ 1_404.33ns │ 313.00w │ │ │ │ │core\_array.ml:filter
new-filter-neq_zero:1000 │ 22_829.70ns │ 2_012.00w │ 1_001.02w │ │ 0.12% │ │core\_array.ml:filter
new-filter-neq_zero:10000 │ 218_872.52ns │ 20_012.00w │ 10_001.21w │ 0.21w │ 1.15% │ │core\_array.ml:filter
new-filter-neq_zero:100000 │ 2_121_340.68ns │ 200_012.00w │ 100_002.77w │ 1.77w │ 11.12% │ │core\_array.ml:filter
old-filter_map-int:0 │ 9.58ns │ 5.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-int:1 │ 68.46ns │ 11.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-int:10 │ 191.66ns │ 32.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-int:100 │ 1_492.60ns │ 257.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-int:1000 │ 57_155.42ns │ 1_005.00w │ 2_507.01w │ 1_005.01w │ 0.30% │ │core\_array.ml:filter
old-filter_map-int:10000 │ 522_177.50ns │ 10_005.00w │ 25_008.54w │ 10_006.54w │ 2.74% │ │core\_array.ml:filter
old-filter_map-int:100000 │ 5_945_405.67ns │ 100_005.00w │ 250_170.69w │ 100_168.69w │ 31.15% │ │core\_array.ml:filter
new-filter_map-int:0 │ 12.03ns │ 10.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-int:1 │ 53.63ns │ 14.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-int:10 │ 164.16ns │ 31.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-int:100 │ 1_263.42ns │ 211.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-int:1000 │ 23_113.12ns │ 1_010.00w │ 1_001.02w │ │ 0.12% │ │core\_array.ml:filter
new-filter_map-int:10000 │ 218_152.23ns │ 10_010.00w │ 10_001.15w │ 0.15w │ 1.14% │ │core\_array.ml:filter
new-filter_map-int:100000 │ 2_217_307.86ns │ 100_010.00w │ 100_002.11w │ 1.11w │ 11.62% │ │core\_array.ml:filter
old-filter_map-float:0 │ 9.32ns │ 5.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-float:1 │ 66.68ns │ 13.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-float:10 │ 182.86ns │ 42.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-float:100 │ 1_496.56ns │ 357.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-float:1000 │ 76_479.74ns │ 2_005.00w │ 3_507.02w │ 2_005.02w │ 0.40% │ │core\_array.ml:filter
old-filter_map-float:10000 │ 694_999.59ns │ 20_005.00w │ 35_011.08w │ 20_009.08w │ 3.64% │ │core\_array.ml:filter
old-filter_map-float:100000 │ 8_694_669.26ns │ 200_005.00w │ 350_476.44w │ 200_474.44w │ 45.56% │ │core\_array.ml:filter
new-filter_map-float:0 │ 12.29ns │ 10.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-float:1 │ 58.24ns │ 16.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-float:10 │ 142.67ns │ 41.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-float:100 │ 1_119.41ns │ 311.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-float:1000 │ 14_262.66ns │ 2_010.00w │ 1_001.02w │ │ 0.07% │ │core\_array.ml:filter
new-filter_map-float:10000 │ 136_448.05ns │ 20_010.00w │ 10_001.23w │ 0.23w │ 0.71% │ │core\_array.ml:filter
new-filter_map-float:100000 │ 1_282_005.01ns │ 200_010.00w │ 100_003.14w │ 2.14w │ 6.72% │ │core\_array.ml:filter
old-filter_map-boxed:0 │ 9.48ns │ 5.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-boxed:1 │ 71.16ns │ 13.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-boxed:10 │ 197.40ns │ 42.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-boxed:100 │ 1_762.40ns │ 357.00w │ │ │ │ │core\_array.ml:filter
old-filter_map-boxed:1000 │ 86_220.67ns │ 2_005.00w │ 3_507.02w │ 2_005.02w │ 0.45% │ │core\_array.ml:filter
old-filter_map-boxed:10000 │ 828_291.42ns │ 20_005.00w │ 35_011.84w │ 20_009.84w │ 4.34% │ │core\_array.ml:filter
old-filter_map-boxed:100000 │ 7_955_395.61ns │ 200_005.00w │ 350_441.44w │ 200_439.44w │ 41.68% │ │core\_array.ml:filter
new-filter_map-boxed:0 │ 14.43ns │ 10.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-boxed:1 │ 59.24ns │ 16.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-boxed:10 │ 198.19ns │ 41.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-boxed:100 │ 1_580.21ns │ 311.00w │ │ │ │ │core\_array.ml:filter
new-filter_map-boxed:1000 │ 52_045.31ns │ 2_010.00w │ 2_011.01w │ 1_010.01w │ 0.27% │ │core\_array.ml:filter
new-filter_map-boxed:10000 │ 479_239.44ns │ 20_010.00w │ 20_012.42w │ 10_011.42w │ 2.51% │ │core\_array.ml:filter
new-filter_map-boxed:100000 │ 4_389_392.06ns │ 200_010.00w │ 200_135.09w │ 100_134.09w │ 23.00% │ │core\_array.ml:filter
old-partition_tf:0 │ 16.55ns │ 16.00w │ │ │ │ │core\_array.ml:filter
old-partition_tf:1 │ 128.08ns │ 29.00w │ │ │ │ │core\_array.ml:filter
old-partition_tf:10 │ 554.15ns │ 111.00w │ │ │ │ │core\_array.ml:filter
old-partition_tf:100 │ 4_853.58ns │ 921.00w │ 0.46w │ 0.46w │ 0.03% │ │core\_array.ml:filter
old-partition_tf:1000 │ 201_289.06ns │ 5_016.00w │ 9_015.21w │ 5_010.21w │ 1.05% │ │core\_array.ml:filter
old-partition_tf:10000 │ 1_796_749.87ns │ 50_016.00w │ 90_040.96w │ 50_035.96w │ 9.41% │ │core\_array.ml:filter
old-partition_tf:100000 │ 19_084_871.85ns │ 500_016.00w │ 902_187.67w │ 502_182.67w │ 100.00% │ │core\_array.ml:filter
new-partition_tf:0 │ 28.29ns │ 23.00w │ │ │ │ │core\_array.ml:filter
new-partition_tf:1 │ 103.78ns │ 31.00w │ │ │ │ │core\_array.ml:filter
new-partition_tf:10 │ 504.10ns │ 96.00w │ │ │ │ │core\_array.ml:filter
new-partition_tf:100 │ 3_869.52ns │ 726.00w │ 0.23w │ 0.23w │ 0.02% │ │core\_array.ml:filter
new-partition_tf:1000 │ 122_807.29ns │ 4_023.00w │ 5_013.04w │ 2_010.04w │ 0.64% │ │core\_array.ml:filter
new-partition_tf:10000 │ 1_197_596.39ns │ 40_023.00w │ 50_020.05w │ 20_017.05w │ 6.28% │ │core\_array.ml:filter
new-partition_tf:100000 │ 10_458_344.09ns │ 400_023.00w │ 500_590.94w │ 200_587.94w │ 54.80% │ └────────────────────────────────────────────────────┴─────────────────┴─────────────┴─────────────┴─────────────┴────────────┘ Added
Binable.Of_sexpable
functor.Install the sexp exception printer sooner so that we can get proper
%test_result ...
errors in things that come beforecore_kernel
.In
Stable_unit_test.Make
functors, include all test failures rather than just the first. This is useful for updating batches of expectedbin_io
results when stabilizing a module.Remove an unnecessary cast in or_error.ml
core_profiler
Switched to ppx.
-
Minor adjustments to the command line of profiler_tool.exe:
- Make '-%' an alias for '-percentile'
- Make '-percentile' accept a comma-separated list of numbers
- Add '-median' argument that is equivalent to '-percentile 50'
email_message
- Bugfixes and minor API improvements.
incremental
Add README.org to Incremental.
Added some type annotations based on comments by @def-lkb about lack of principality.
Switched to ppx.
jenga
Restructure the code in a way that allows to build binaries that statically link jenga with the rules. This is useful because some debugging/profiling tools don't work in the presence of dynamically loaded code very well.
Switch to PPX.
Change the gc info output by jenga so it shows heap size and top heap size, instead of live and heap size. The live part is not super useful given how random it is. I have seen cases where jenga was using 20GB during building and jenga reported a heap size of 13GB at the end so the top heap size avoids being tricked.
-
First half of the fixes no packing: sharing the structures of dependencies, so they take less space on disk (and in memory as well, when they are loaded from disk, but not really when building from scratch given the way we will use them).
The sexp format also has sharing, because it would also blow up in size otherwise (this is different from the interning of paths, where the interning saves a constant factor). And of course, it makes it possible to see the actual on-disk representation which is nice.
Also fix unhelpful error (contains no information) when the db can't be loaded.
Break the thing that avoids rerunning rules when the set of dependencies decreases. I think it was never useful anyway.
Better error on duplicate targets in the same rule.
-
To prevent running more than one jenga in a repository, use a local lock rather than an nfs one. We need a transition period though, so for now we use both kinds of locks. Building on nfs is slow, so I don't think there's any downside is not supporting nfs this way. And maybe inotify doesn't work. The upside is that we don't step into Lock.Nfs bugs where if a process is interrupted at the wrong time (when the two lock files are empty) the locks won't be cleaned up automatically, forcing someone to get rid of the lock files manually.
Also rename
.jenga/.jenga.*
to.jenga/*
, because all these prefixes are annoying. Added a couple of options to turn off some part of jenga, which I used to check how they impacted performance, and could still be handy later.
Optionally display additional information about much allocation was done, at the end of builds. Used it to try to improve memory usage of full tree builds without actually doing full tree builds.
Make stat'ing faster. Hash cons some tenacious that build mtimes map to avoid a huge increase of memory use.
Got rid of noise when stopping jenga.
-
Some changes to the implementation of Tenacious to improve memory efficiency.
Includes the following changes:
Remove
strong_refs
field ofHeart.fragile
type and instead insert links from theclients
Ring.t
to its parent usingRing.keep_alive
.Replace the
Ring.t
in thetriggers
field ofHearts.fragile
with anIvar.t
since all uses oftriggers
were producing their own equivalentIVar.t
s. Remove the functions broken by this because they were unused.Make the
Tenacious.t
type a concrete datatype, and optimize pure computations by partially evaluating this datatype directly in the pure case.Rather than building separate
Heart.t
s for cancellation and the result, split the cancellation heart into two heartscancel
anddep
and then usedep
as the result heart. This means that a tenacious is cancelled if eithercancel
ordep
is broken, and it must return a heart representing the validity of its result combined withdep
.
Cursory benchmarking indicates a 23% improvement in maximum resident set size and a 10% improvment in (user) execution time when building the lib directory from scratch.
Adding some
sexp_of
functions, since they're always missing and it's a pain when debugging.Adding direct support for
Dep.map
. Even now thatTenacious
is smarter, this still creates lessbind
s. Doesn't seem to make much of a difference (perhaps 3-5% less allocation, on a null build of lib), but if nothing else, it's much less surprising to think thatDep.Map
becomesTenacious.Map
.Added Dep.List.concat.
-
Make it possible to turn off the behavior where jenga rejects commands that output on stderr.
It increases slightly the footprint of the in memory db, but the difference is tiny compared to the rest of the memory usage.
Added a few tests about
Jenga_lib.Api.Reflect
ocaml_plugin
Switch to ppx.
Allow ppx-style code to be loaded by plugin-applications build using ocaml_plugin.
Follow Core & Async evolution.
patdiff
patdiff -location-style omake
should print the line number of the first difference in each hunk, skipping context lines.Switched to PPX.
-
Added binding in patdiff to use the newly minted colors of Ansi_terminal. This will be used notably by patdiff4 to produce better ddiff.
Also, have the module
Color
andStyle
implement and exportComparable.S
. This is useful for example to dedup styles from a list of styles without relying on the polymorphic equality. -
Make it so that if you pass
-warn-if-no-trailing-newline-in-both false
then you get the warning only when one file has a trailing newline and the other file does not.If you pass
-warn-if-no-trailing-newline-in-both true
or omit this flag, then you get the current behavior of warning for each file independently. Patdiff's unified-tests currently render colors codes in angle brackets. Change them to square brackets. Square brackets are word boundaries, so we'll get more legible diffs when tests fail.
Simple code change in patdiff to prepare more changes in patdiff4. This change is a pure refactoring and has zero runtime change. Just moving some functions around.
patdiff_core.ml is a very long module. start extracting module from it. start with format. in the process, expose in a private fashion the record
Rule.t
.Continue on splitting the file patdiff_core.ml into smaller pieces. In this version, we extract each output mode into its own file.
-
Kill the generation of html diffs in patdiff. There are good third party tools that can convert efficiently ansi texts to html directly. We plan on simplifying a bit the patdiff source code to increase its maintainability, and dropping the requirement of producing html output seems a step in the right direction.
Some pointers:
http://www.pixelbeat.org/scripts/ansi2html.sh jane: app/ralloc/commander/ansi2html.ml
patience_diff
- Switch to PPX.
ppx_assert
- Update to follow evolution of
Ppx_core
.
ppx_bench
Update to follow
Ppx_core
evolution.-
Mark attributes as handled inside explicitly dropped pieces of code.
So that a
@@deriving
inside a let%test dropped by ppx_inline_test_drop doesn't cause a failure.
ppx_bin_prot
- Minor changes, nothing worth mentionning.
ppx_compare
- Follow evolution of
Ppx_core
andType_conv
.
ppx_core
Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.
-
Merlin uses
@merlin.* ...
attributes in different places. Which ppx_driver reports as unused.Introduce the concept of reserved namespaces. When one declares the namespace "foo" as reserved then:
-
foo.*
will never get reported as unused - it is impossible to
Attribute.declare "foo.*"
Mark the "merlin" namespace as reserved by default.
-
-
Don't print:
Extension `foo' was not translated. Hint: Did you mean foo?
-
OCaml makes no distinctions between "foo" and
{whatever|foo|whatever}
. The delimiter choice is simply left to the user.Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept {whatever|foo|whatever}.
-
Avoid stupid hints like this one:
Attribute
default' was not used. Hint:
default' is available for label declarations but is used here in the context of a label declaration. Did you put it at the wrong level? -
Update the API for the common case of extension point expanders.
Make it simpler to define ppx rewriters that locally expand extension points, which is the majority of our non-type-conv rewriters.
Such expanders are run inside the same
Ast_traverse.map
in a top-down manner which:- probably improve speed
- help with rewriters that capture a pretty-print of their payload
- help with rewriter that interpret some extension points in a special way inside their payload
Fix the order in which errors are reported by ppx rewriters. Make them be reported in the same order as they appear.
-
Mark attributes as handled inside explicitly dropped pieces of code.
So that a
@@deriving
inside a let%test dropped byppx_inline_test_drop
doesn't cause a failure.
ppx_csv_conv
-
ppx
An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
-
Affected files
ppx/ppx_csv_conv/src/ppx_csv_conv.ml
-
ppx/changes-for-public-release
Changes required for the first public release of our ppx rewriters
-
improved how it works, to enable it we just need:
Type_conv.Ppx_deriving_exporter.set (module Ppx_deriving)
Type_conv.add_alias now takes as argument registered type_conv deriviers. This ensure that we can resolve aliases right from the registration time. This simplify the export to ppx_deriving
Split some functions out of their main ppx_XXX library that does the registration with ppx_driver or ppx_type_conv.
For instance:
- ppx_sexp_conv defines sexp_of_quote that is used by ppx_assert
- ppx_here defines ast_of_pos that is used by ppx_fail and ppx_assert
Making ppx_assert depends on ppx_sexp_conv and ppx_here is not good as it enables
@@deriving sexp
and%sexp_of: ty
even if the user only writes ppx_assert in the jbuild. This is especially problematic for the public release as it makes ppx_assert incompatible with ppx_deriving.This feature moves these functions into libraries called ppx_XXX_expander (and ppx_inline_test_libname for ppx_inline_test).
To help reviewing the changes, you can run:
patdiff <(hg cat -r
fe show -base
ppx/ppx_compare/src/ppx_compare.ml) <(hg cat -rfe show -tip
ppx/ppx_compare/expander/ppx_compare_expander.ml) patdiff <(hg cat -rfe show -base
ppx/ppx_sexp_conv/src/ppx_sexp_conv.ml) <(hg cat -rfe show -tip
ppx/ppx_sexp_conv/expander/ppx_sexp_conv_expander.ml)It's not type-conv specific anymore and is needed by libraries that don't use type_conv otherwise.
pa_test_lib --> ppx_assert_lib pa_bench_lib --> ppx_bench_lib
-
-
Affected files
ppx/ppx_csv_conv/src/ppx_csv_conv.ml
-
ppx
An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
-
Affected files
ppx/ppx_csv_conv/src/ppx_csv_conv.ml
-
ppx/delete-make-at-the-end
Cleanup in type_conv: remove Type_conv.Generator_result.make_at_the_end, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply.
This feature moves a few values up in signatures. For instance in this interface
sexp_of_t
is now the first value instead of the second one as before this feature:type t `@@deriving sexp_of` val x : int
In some cases this caused the OCaml compiler to complain about items in the signature being re-ordered. In these cases the signature was adapted to match the expected ordering.
-
Affected files
ppx/ppx_csv_conv/src/ppx_csv_conv.ml
-
ppx
An umbrella feature for development on ppx syntax extensions. All work except for rebasing should be done in subfeatures.
-
Affected files
ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv
-
ppx/ppx_csv_conv-example
Make ppx_csv_conv example do something.
Added inline test.
-
Affected files
ppx/ppx_csv_conv/example/example.ml ppx/ppx_csv_conv/example/example.mli ppx/ppx_csv_conv/example/test.csv
ppx_custom_printf
-
OCaml makes no distinctions between "foo" and
{whatever|foo|whatever}
. The delimiter choice is simply left to the user.Do the same in our ppx rewriters: i.e. wherever we accept "foo", also accept
{whatever|foo|whatever}
. -
Fix missing location in errors for broken custom printf example like:
printf !"%{sexp: int" 3;;
Update to follow
Ppx_core
evolution.
ppx_driver
Disable safety check when code transformations are used as standard "-ppx" rewriters.
-
Introduce reserved namespaces, see
Ppx_core
's changelog.Pass errors as attribute with -dparsetree to avoid "Error while running external preprocessor".
Update to follow
Ppx_core
evolution.
ppx_enumerate
- Update to follow type_conv evolution.
ppx_expect
Initial release.
ppx_fail
Added a README.md
Update to follow
Ppx_core
evolution.
ppx_fields_conv
-
The
iter
function generated by ppx_variants_conv and ppx_fields_conv allowed one to give function which returned values of arbitrary types as iter function. This release constraint these functions to return unit.N.B. the signature generated by the use of
@@deriving variants
(resp. fields) in interface already constrained the type to unit. Update to follow type_conv's evolution.
Add
Fields.make_creator
to ppx_fields_conv's readme, since it appears to not be all that deprecated.
ppx_here
Make ppx_here translate
[%here]
instead of_here_
.Update to follow
Ppx_core
evolution.
ppx_inline_test
-
Support literate-style .ml files that allow ocaml code interleaved with expected output annotations. Compiling with the
ppx_expect_test
generates a program that outputs the original source file, but with the actual output substituted for the expected-output annotations. Then we can pat-diff the original file against the output file.Examples in the test/ and example/ folders.
-
Expect-tests can now be written inline in libraries by using
let%expect_test
.The runtime library has been split into two components: the test runner, which collects the output of the test body, and registers enough information to construct the
*.ml.corrected
file from the input; and the test evaluator, which compares the test output against the expected output and generates the output files. Update to follow
Ppx_core
evolution.When an exception is raised inside a
let%test_module
, display the position and name of the TEST_MODULE, same as for thelet%test
.-
Mark attributes as handled inside explicitly dropped pieces of code.
So that a
@@deriving
inside a let%test dropped byppx_inline_test_drop
doesn't cause a failure.
ppx_optcomp
-
Change the way optcomp resolve filenames in #import directives
Do the same as cpp, i.e. for relative filenames, consider they are relative to the directory of the file being parsed. This doesn't matter internally as build commands are always executed from the current directory, but it matters for the public release as everything is executed from the root.
ppx_sexp_conv
Trying to improve the tests in ppx_sexp_conv because they are a mess. At least all tests are automatic now. And more things are tested like the sexpification of exceptions.
Update to follow
Type_conv
andPpx_core
evolution.-
Make ppx_sexp_conv correctly handle aliases to polymorphic variants:
type t =
A
@@deriving sexp
type u = t
@@deriving sexptype v =
u |B
@@deriving sexp
Before,
v_of_sexp
would never manage to readB. This problem is now fixed if you use
sexp_polyon
uinstead of
sexp, and if you don't, you get an "unbound value __u_of_sexp__". People should use
sexp_polywhen they have a polymorphic variant type that is not syntactically a polymorphic variant, but in practice it's simpler to replace
sexpby
sexp_poly` when faced with the error above.The need for
sexp_poly
should happen only in one new case: an implementation saystype u = t
@@deriving sexpbut the interface says `type u =
A@@deriving sexp
. (the old case where it was already needed is when you have an interface that says `type u = t `@@deriving sexp
and in some other implementation you try to saytype t =
That_module.t |A
@@deriving sexp
`).
ppx_type_conv
Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.
Cleanup in type_conv: remove
Type_conv.Generator_result.make_at_the_end
, which was a hack to remove warnings. We can do it better now, and because this is only for signatures, the code generation issue what we had in simplify-type-conv-ignore-unused-warning doesn't apply.Update to follow
Ppx_core
evolution.
ppx_typerep_conv
- Update following
Ppx_core
andType_conv
evolution. - Add a README.
ppx_variants_conv
-
The
iter
function generated by ppx_variants_conv and ppx_fields_conv allowed one to give function which returned values of arbitrary types as iter function. This feature constraint these functions to return unit.N.B. the signature generated by the use of
@@deriving variants
(resp. fields) in interface already constrained the type to unit. Update to follow
Type_conv
evolution.
re2
Switched to PPX.
-
Add
Re2.Parser.any_string
combinator.There are no tests because
any_string
is constructed only from the tested API and there's almost no interesting properties of it that can be verified.
rpc_parallel
Switched to PPX.
Expose the
connection_timeout
argument in rpc_parallel. This argument exists inRpc_parallel_core.Parallel
, but it is not exposed inRpc_parallel.Parallel
.Allow custom handling of missed async_rpc heartbeats.
Give a better error message when redirecting output on a remote box to a file path that does not exist.
remove unncessary chmod 700 call on the remote executable
Give a clear error message for the common mistake of not making the
Parallel.Make_worker()
functor application top-level-
Make errors/exceptions in
Rpc_parallel
more observable- Make stderr and stdout redirection mandatory in order to encourage logging stderr
- Clean up the use of monitors across
Rpc_parallel
- Fix bug with exceptions that are sent directly to
Monitor.main
(e.g.Async_log
does this)
Add the ability to explicitly initialize as a master and use some subcommand for the worker. This would allow writing programs with complex command structures that don't have to invoke a bunch of
Rpc_parallel
logic and start RPC servers for every command.Add the ability to get log messages from a worker sent back to the master. In fact, any worker can register for the log messages of any other workers.
sexplib
- Switch code in
lib
subdir to ppx-style.
textutils
Switched to PPX.
Fixed a bug where the computation of cell heights could cause division by zero in some cases.
-
Expose the constructors of
Ascii_table.Align.t
so that we can writeColumn.create ~align:Left ...
instead of
Column.create ~align:Align.left
typerep
- Add whether record fields are mutable.
typerep_extended
Switched to ppx.
Kill the nonrec rewrite done by typerep. It is no longer needed since 4.02.2, we kept it only for compatibility with the camlp4 code.