Module Core__Iobuf

A non-moving (in the GC sense) contiguous range of bytes, useful for I/O operations.

An iobuf consists of:

All iobuf operations are restricted to operate within the limits. Initially, the window of an iobuf is identical to the limits. A phantom type, "seek" permission, controls whether or not code is allowed to change the limits and window. With seek permission, the limits can be narrowed, but can never be widened, and the window can be set to an arbitrary subrange of the limits.

A phantom type controls whether code can read and write bytes in the bigstring (within the limits) or can only read them.

To present a restricted view of an iobuf to a client, one can create a sub-iobuf or add a type constraint.

Functions operate on the window unless the documentation or naming indicates otherwise.

type seek = Core.Iobuf_intf.seek
include sig ... end
val sexp_of_seek : seek ‑> Sexplib.Sexp.t
type no_seek = Core.Iobuf_intf.no_seek
include sig ... end
val sexp_of_no_seek : no_seek ‑> Sexplib.Sexp.t
type (-'data_perm_read_write, +'seek_permission) t

The first type parameter controls whether the iobuf can be written to. The second type parameter controls whether the window and limits can be changed.

See the Perms module for information on how the first type parameter is used.

To allow no_seek or seek access, a function's type uses _ rather than no_seek as the type argument to t. Using _ allows the function to be directly applied to either permission. Using a specific permission would require code to use coercion :>.

There is no t_of_sexp. One should use Iobuf.Hexdump.t_of_sexp or sexp_opaque as desired.

include Core__.Import.Invariant.S2 with type (rw, seek) t := (rw, seek) t
type ('a, 'b) t
module Window : Core__.Import.Hexdump.S2 with type (rw, seek) t := (rw, seek) t

Provides a Window.Hexdump submodule that renders the contents of t's window.

module Limits : Core__.Import.Hexdump.S2 with type (rw, seek) t := (rw, seek) t

Provides a Window.Hexdump submodule that renders the contents of t's limits.

Provides a Hexdump submodule that renders the contents of t's window and limits using indices relative to the limits.

include Core.Iobuf_intf.Compound_hexdump with type (rw, seek) t := (rw, seek) t
type ('rw, 'seek) t
module Hexdump : sig ... end
module Debug : Core.Iobuf_intf.Compound_hexdump with type (rw, seek) t := (rw, seek) t

Provides a Debug.Hexdump submodule that renders the contents of t's window, limits, and underlying bigstring using indices relative to the bigstring.

Creation

val create : len:int ‑> (__t

create ~len creates a new iobuf, backed by a bigstring of length len, with the limits and window set to the entire bigstring.

val of_bigstring : ?⁠pos:int ‑> ?⁠len:int ‑> Core__.Import.Bigstring.t ‑> ([< Core__.Import.read_write ], _t

of_bigstring bigstring ~pos ~len returns an iobuf backed by bigstring, with the window and limits specified starting at pos and of length len.

forbid immutable to prevent aliasing

val of_string : string ‑> (__t

of_string s returns a new iobuf whose contents are s.

val sub_shared : ?⁠pos:int ‑> ?⁠len:int ‑> ('d_t ‑> ('d_t

sub_shared t ~pos ~len returns a new iobuf with limits and window set to the subrange of t's window specified by pos and len. sub_shared preserves data permissions, but allows arbitrary seek permissions on the resulting iobuf.

val set_bounds_and_buffer : src:([> Core__.Import.write ] as data, _t ‑> dst:('dataseekt ‑> unit

set_bounds_and_buffer ~src ~dst copies bounds metadata (i.e., limits and window) and shallowly copies the buffer (data pointer) from src to dst. It does not access data, but does allow access through dst. This makes dst an alias of src.

Because set_bounds_and_buffer creates an alias, we disallow immutable src and dst using [> write]. Otherwise, one of src or dst could be read_write :> read and the other immutable :> read, which would allow to write the immutable alias's data through the read_write alias.

set_bounds_and_buffer is typically used with a frame iobuf that need be allocated only once. This frame can be updated repeatedly and handed to users, without further allocation. Allocation-sensitive applications need this.

val set_bounds_and_buffer_sub : pos:int ‑> len:int ‑> src:([> Core__.Import.write ] as data, _t ‑> dst:('dataseekt ‑> unit

set_bounds_and_buffer_sub ~pos ~len ~src ~dst is a more efficient version of: set_bounds_and_buffer ~src:(Iobuf.sub_shared ~pos ~len src) ~dst.

set_bounds_and_buffer ~src ~dst is not the same as set_bounds_and_buffer_sub ~dst ~src ~len:(Iobuf.length src) because the limits are narrowed in the latter case.

~len and ~pos are mandatory for performance reasons, in concert with @@inline. If they were optional, allocation would be necessary when passing a non-default, non-constant value, which is an important use case.

val read_only : ([> Core__.Import.read ], 'st ‑> (Core__.Import.read'st

Generalization

One may wonder why you'd want to call no_seek, given that a cast is already possible, e.g. t : (_, seek) t :> (_, no_seek) t. It turns out that if you want to define some f : (_, _) t -> unit of your own, which can be conveniently applied to seek iobufs without the user having to cast seek up, you need this no_seek function.

read_only is more of an historical convenience now that read_write is a polymorphic variant, as one can now explicitly specify the general type for an argument with something like t : (_ perms, _) t :> (read, _) t.

val no_seek : ('r_t ‑> ('rno_seekt

Accessors

val capacity : (__t ‑> int

capacity t returns the size of t's limits subrange. The capacity of an iobuf can be reduced via narrow.

val length : (__t ‑> int

length t returns the size of t's window.

val is_empty : (__t ‑> bool

is_empty t is length t = 0.

Changing the limits

val narrow : (_seekt ‑> unit

narrow t sets t's limits to the current window.

val narrow_lo : (_seekt ‑> unit

narrow_lo t sets t's lower limit to the beginning of the current window.

val narrow_hi : (_seekt ‑> unit

narrow_hi t sets t's upper limit to the end of the current window.

Changing the window

One can call Lo_bound.window t to get a snapshot of the lower bound of the window, and then later restore that snapshot with Lo_bound.restore. This is useful for speculatively parsing, and then rewinding when there isn't enough data to finish.

Similarly for Hi_bound.window and Lo_bound.restore.

Using a snapshot with a different iobuf, even a sub iobuf of the snapshotted one, has unspecified results. An exception may be raised, or a silent error may occur. However, the safety guarantees of the iobuf will not be violated, i.e., the attempt will not enlarge the limits of the subject iobuf.

module type Bound : Core.Iobuf_intf.Bound with type (d, w) iobuf := (d, w) t
module Lo_bound : Bound
module Hi_bound : Bound
val advance : (_seekt ‑> int ‑> unit

advance t amount advances the lower bound of the window by amount. It is an error to advance past the upper bound of the window or the lower limit.

val unsafe_advance : (_seekt ‑> int ‑> unit

unsafe_advance is like advance but with no bounds checking, so incorrect usage can easily cause segfaults.

val resize : (_seekt ‑> len:int ‑> unit

resize t sets the length of t's window, provided it does not exceed limits.

val unsafe_resize : (_seekt ‑> len:int ‑> unit

unsafe_resize is like resize but with no bounds checking, so incorrect usage can easily cause segfaults.

val rewind : (_seekt ‑> unit

rewind t sets the lower bound of the window to the lower limit.

val reset : (_seekt ‑> unit

reset t sets the window to the limits.

val flip_lo : (_seekt ‑> unit

flip_lo t sets the window to range from the lower limit to the lower bound of the old window. This is typically called after a series of Fills, to reposition the window in preparation to Consume the newly written data.

The bounded version narrows the effective limit. This can preserve some data near the limit, such as an hypothetical packet header, in the case of bounded_flip_lo or unfilled suffix of a buffer, in bounded_flip_hi.

val bounded_flip_lo : (_seekt ‑> Lo_bound.t ‑> unit
val compact : (Core__.Import.read_writeseekt ‑> unit

compact t copies data from the window to the lower limit of the iobuf and sets the window to range from the end of the copied data to the upper limit. This is typically called after a series of Consumes to save unread data and prepare for the next series of Fills and flip_lo.

val bounded_compact : (Core__.Import.read_writeseekt ‑> Lo_bound.t ‑> Hi_bound.t ‑> unit
val flip_hi : (_seekt ‑> unit

flip_hi t sets the window to range from the the upper bound of the current window to the upper limit. This operation is dual to flip_lo and is typically called when the data in the current (narrowed) window has been processed and the window needs to be positioned over the remaining data in the buffer. For example:

      (* ... determine initial_data_len ... *)
      Iobuf.resize buf ~len:initial_data_len;
      (* ... and process initial data ... *)
      Iobuf.flip_hi buf;

Now the window of buf ranges over the remainder of the data.

val bounded_flip_hi : (_seekt ‑> Hi_bound.t ‑> unit
val protect_window_and_bounds : ('rwno_seekt ‑> f:(('rwseekt ‑> 'a) ‑> 'a

protect_window_and_bounds t ~f applies f to t and restores t's bounds and buffer afterward.

Getting and setting data

"consume" and "fill" functions access data at the lower bound of the window and advance lower bound of the window. "peek" and "poke" functions access data but do not advance the window.

val to_string : ?⁠len:int ‑> ([> Core__.Import.read ], _t ‑> string

to_string t returns the bytes in t as a string. It does not alter the window.

val to_string_hum : ?⁠max_lines:int ‑> (__t ‑> string

Equivalent to Hexdump.to_string_hum. Renders t's windows and limits.

module Consume : sig ... end

Consume.string t ~len reads len characters (all, by default) from t into a new string and advances the lower bound of the window accordingly.

module Fill : sig ... end

Fill.bin_prot X.bin_write_t t x writes x to t in bin-prot form, advancing past the bytes written.

module Peek : sig ... end

Peek and Poke functions access a value at pos from the lower bound of the window and do not advance.

module Poke : sig ... end

Poke.bin_prot X.bin_write_t t x writes x to the beginning of t in binary form without advancing. You can use X.bin_size_t to tell how long it was. X.bin_write_t is only allowed to write that portion of the buffer to which you have access.

module Unsafe : sig ... end

Unsafe has submodules that are like their corresponding module, except with no range checks. Hence, mistaken uses can cause segfaults. Be careful!

val crc32 : ([> Core__.Import.read ], _t ‑> Core__.Import.Int63.t
val bin_prot_length_prefix_bytes : int

The number of bytes in the length prefix of consume_bin_prot and fill_bin_prot.

val fill_bin_prot : ([> Core__.Import.write ], seekt ‑> 'a Bin_prot.Type_class.writer ‑> 'a ‑> unit Core__.Import.Or_error.t

fill_bin_prot writes a bin-prot value to the lower bound of the window, prefixed by its length, and advances by the amount written. fill_bin_prot returns an error if the window is too small to write the value.

consume_bin_prot t reader reads a bin-prot value from the lower bound of the window, which should have been written using fill_bin_prot, and advances the window by the amount read. consume_bin_prot returns an error if there is not a complete message in the window and in that case the window is left unchanged.

Don't use these without a good reason, as they are incompatible with similar functions in Reader and Writer. They use a 4-byte length rather than an 8-byte length.

val consume_bin_prot : ([> Core__.Import.read ], seekt ‑> 'a Bin_prot.Type_class.reader ‑> 'a Core__.Import.Or_error.t
module Blit : sig ... end

Blit copies between iobufs and advances neither src nor dst.

module Blit_consume : sig ... end

Blit_consume copies between iobufs and advances src but does not advance dst.

module Blit_fill : sig ... end

Blit_fill copies between iobufs and advances dst but does not advance src.

module Blit_consume_and_fill : sig ... end

Blit_consume_and_fill copies between iobufs and advances both src and dst.

I/O

type ok_or_eof =
| Ok
| Eof
include sig ... end
val compare_ok_or_eof : ok_or_eof ‑> ok_or_eof ‑> int
val sexp_of_ok_or_eof : ok_or_eof ‑> Sexplib.Sexp.t
val input : ([> Core__.Import.write ], seekt ‑> Core__.Import.In_channel.t ‑> ok_or_eof

Iobuf has analogs of various Bigstring functions. These analogs advance by the amount written/read.

val read : ([> Core__.Import.write ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> ok_or_eof
val read_assume_fd_is_nonblocking : ([> Core__.Import.write ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> Core__.Syscall_result.Unit.t
val pread_assume_fd_is_nonblocking : ([> Core__.Import.write ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> offset:int ‑> unit
val recvfrom_assume_fd_is_nonblocking : ([> Core__.Import.write ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> Core.Iobuf_intf.Unix.sockaddr
module Recvmmsg_context : sig ... end with type (rw, seek) iobuf := (rw, seek) t

recvmmsg's context comprises data needed by the system call. Setup can be expensive, particularly for many buffers.

val recvmmsg_assume_fd_is_nonblocking : (Core.Iobuf_intf.Unix.File_descr.t ‑> Recvmmsg_context.t ‑> Core.Iobuf_intf.Unix.Syscall_result.Int.t) Core__.Import.Or_error.t

recvmmsg_assume_fd_is_nonblocking fd context returns the number of context iobufs read into (or errno). fd must not block. THREAD_IO_CUTOFF is ignored.

EINVAL is returned if an Iobuf passed to Recvmmsg_context.create has its buf or limits changed.

val send_nonblocking_no_sigpipe : unit ‑> (([> Core__.Import.read ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> Core.Iobuf_intf.Unix.Syscall_result.Unit.t) Core__.Import.Or_error.t
val sendto_nonblocking_no_sigpipe : unit ‑> (([> Core__.Import.read ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> Core.Iobuf_intf.Unix.sockaddr ‑> Core.Iobuf_intf.Unix.Syscall_result.Unit.t) Core__.Import.Or_error.t
val output : ([> Core__.Import.read ], seekt ‑> Core__.Import.Out_channel.t ‑> unit
val write : ([> Core__.Import.read ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> unit
val write_assume_fd_is_nonblocking : ([> Core__.Import.read ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> unit
val pwrite_assume_fd_is_nonblocking : ([> Core__.Import.read ], seekt ‑> Core.Iobuf_intf.Unix.File_descr.t ‑> offset:int ‑> unit

Expert

module Expert : sig ... end

The Expert module is for building efficient out-of-module Iobuf abstractions.