# ExDataSketch v0.10.3 - Table of Contents Production-grade streaming data sketching algorithms (HLL, ULL, CMS, Theta, KLL, DDSketch, REQ, FrequentItems, MisraGries, Bloom, Cuckoo, Quotient, CQF, XorFilter, IBLT, FilterChain) with optional Rust NIF acceleration and XXHash3. ## Pages - [Changelog](changelog.md) - Core Concepts - [The Aggregation Wall](aggregation_wall.md) - [Distributed Merge Semantics](distributed_merge_semantics.md) - [Hash Strategy Internals](hash_strategies.md) - Getting Started - [Quick Start](quick_start.md) - [Usage Guide](usage_guide.md) - [Integration Guide](integrations.md) - [Livebooks](livebooks.md) - Tutorials: Cardinality - [HLL (HyperLogLog) Tutorial](livebook_hll.md) - [ULL (UltraLogLog) Tutorial](livebook_ull.md) - Tutorials: Frequency & Heavy Hitters - [CMS (Count-Min Sketch) Tutorial](livebook_cms.md) - [FrequentItems Tutorial](livebook_frequent_items.md) - [MisraGries Tutorial](livebook_misra_gries.md) - Tutorials: Quantiles - [KLL Sketch Tutorial](livebook_kll.md) - [DDSketch Tutorial](livebook_ddsketch.md) - [REQ (Relative Error Quantiles) Tutorial](livebook_req.md) - Tutorials: Set Operations - [Theta Sketch Tutorial](livebook_theta.md) - Tutorials: Membership Filters - [Bloom Filter Tutorial](livebook_bloom.md) - [Cuckoo Filter Tutorial](livebook_cuckoo.md) - [Quotient Filter Tutorial](livebook_quotient.md) - [CQF (Counting Quotient Filter) Tutorial](livebook_cqf.md) - [XorFilter Tutorial](livebook_xor_filter.md) - [FilterChain Tutorial](livebook_filter_chain.md) - Tutorials: Set Reconciliation - [IBLT (Invertible Bloom Lookup Table) Tutorial](livebook_iblt.md) - Production & Operations - [Persistence](persistence.md) - [Persistence Snapshots](livebook_persistence_snapshots.md) - [Windowing](windowing.md) - [Supervised Sketches](supervised_sketches.md) - [Telemetry Integration Guide](telemetry.md) - [Observability Guide](observability.md) - [Precompiled NIFs](precompiled_nifs.md) - Framework Integrations - [Streaming Sketches](streaming_sketches.md) - [Streaming Cardinality Estimation](livebook_streaming_cardinality.md) - [Broadway Integration](broadway_integration.md) - [Broadway Integration (Livebook)](livebook_broadway_integration.md) - [GenStage Integration](genstage_integration.md) - [GenStage Aggregation](livebook_genstage_aggregation.md) - [Flow Integration](flow_integration.md) - [Distributed Merges](livebook_distributed_merges.md) - Case Studies - [AI Token Stream Analytics](livebook_ai_token_analytics.md) - Interop & Compatibility - [Apache DataSketches Interop](apache_interop.md) - [Serialization Compatibility Contract (v0.8.0)](serialization_compatibility.md) - [FI1 Binary State Format](frequent_items_format.md) - Project History & Internals - [HLL Hot-Path Optimization](hll_performance.md) - [Membership-Filter Raw-Hashing Internals](filter_performance.md) - [v0.8.0 Architectural Summary](v0-8-0_architecture.md) - [Migrating to ex_data_sketch v0.8.0](v0-8-0_migration_notes.md) - [v0.11.0 Roadmap Preview](roadmap.md) ## Modules - [ExDataSketch](ExDataSketch.md): Production-grade streaming data sketching algorithms for Elixir. - [ExDataSketch.Binary](ExDataSketch.Binary.md): Public facade for the EXSK binary frame. - [ExDataSketch.Binary.CRC](ExDataSketch.Binary.CRC.md): CRC32C (Castagnoli polynomial, reflected, init 0xFFFFFFFF, xor-out 0xFFFFFFFF) — the checksum used by EXSK v2 frames. - [ExDataSketch.Binary.Header](ExDataSketch.Binary.Header.md): EXSK v2 binary frame header. - [ExDataSketch.Binary.Validator](ExDataSketch.Binary.Validator.md): Structured validation primitives for EXSK frames. - [ExDataSketch.Config](ExDataSketch.Config.md): Per-family default option overrides from Application config. - [ExDataSketch.DataSketches.CompactSketch](ExDataSketch.DataSketches.CompactSketch.md): Apache DataSketches CompactSketch binary codec for Theta sketches. - [ExDataSketch.DataSketches.KLLSketch](ExDataSketch.DataSketches.KLLSketch.md): Apache DataSketches KLL binary codec for `ExDataSketch.KLL`. - [ExDataSketch.DataSketches.Murmur3](ExDataSketch.DataSketches.Murmur3.md): Minimal MurmurHash3_x64_128 implementation for DataSketches seed hash computation. - [ExDataSketch.Hash.Metadata](ExDataSketch.Hash.Metadata.md): Shared hash + sketch metadata block. - [ExDataSketch.Hash.Murmur3](ExDataSketch.Hash.Murmur3.md): MurmurHash3_x64_128 (64-bit output). - [ExDataSketch.Hash.Validation](ExDataSketch.Hash.Validation.md): Compatibility checks for hash configurations across merging sketches. - [ExDataSketch.Hash.XXH3](ExDataSketch.Hash.XXH3.md): XXHash3 (64-bit) hash algorithm. - [ExDataSketch.Integration](ExDataSketch.Integration.md): Runtime dependency detection for optional integrations. - [ExDataSketch.SampleData](ExDataSketch.SampleData.md): Sample-data generators for the tutorial livebooks under `livebooks/sketches/`. - Sketch Algorithms - [ExDataSketch.Bloom](ExDataSketch.Bloom.md): Bloom filter for probabilistic membership testing. - [ExDataSketch.CMS](ExDataSketch.CMS.md): Count-Min Sketch (CMS) for frequency estimation. - [ExDataSketch.CQF](ExDataSketch.CQF.md): Counting Quotient Filter (CQF) for multiset membership with approximate counting. - [ExDataSketch.Cuckoo](ExDataSketch.Cuckoo.md): Cuckoo filter for probabilistic membership testing with deletion support. - [ExDataSketch.DDSketch](ExDataSketch.DDSketch.md): DDSketch quantiles sketch for value-relative-accuracy quantile estimation. - [ExDataSketch.FilterChain](ExDataSketch.FilterChain.md): Capability-aware composition framework for chaining membership filters. - [ExDataSketch.FrequentItems](ExDataSketch.FrequentItems.md): FrequentItems sketch for approximate heavy-hitter detection using the SpaceSaving algorithm. - [ExDataSketch.HLL](ExDataSketch.HLL.md): HyperLogLog (HLL) sketch for cardinality estimation. - [ExDataSketch.IBLT](ExDataSketch.IBLT.md): Invertible Bloom Lookup Table (IBLT) for set reconciliation. - [ExDataSketch.KLL](ExDataSketch.KLL.md): KLL (Karnin-Lang-Liberty) quantiles sketch for rank and quantile estimation. - [ExDataSketch.MisraGries](ExDataSketch.MisraGries.md): MisraGries sketch for deterministic heavy hitter detection. - [ExDataSketch.Quantiles](ExDataSketch.Quantiles.md): Facade for quantile sketch algorithms. - [ExDataSketch.Quotient](ExDataSketch.Quotient.md): Quotient filter for probabilistic membership testing with safe deletion and merge. - [ExDataSketch.REQ](ExDataSketch.REQ.md): REQ (Relative Error Quantiles) sketch for tail-accuracy quantile estimation. - [ExDataSketch.Theta](ExDataSketch.Theta.md): Theta Sketch for set operations on cardinalities. - [ExDataSketch.ULL](ExDataSketch.ULL.md): UltraLogLog (ULL) sketch for cardinality estimation. - [ExDataSketch.XorFilter](ExDataSketch.XorFilter.md): Xor filter for static, immutable probabilistic membership testing. - Stream Integration - [ExDataSketch.Stream](ExDataSketch.Stream.md): Stream-native sketch construction and reduction. - Dataflow Integration - [ExDataSketch.Broadway](ExDataSketch.Broadway.md): Broadway integration for sketch aggregation. - [ExDataSketch.Broadway.PeriodicAggregator](ExDataSketch.Broadway.PeriodicAggregator.md): A GenServer that periodically accumulates and flushes sketch data. - [ExDataSketch.Flow](ExDataSketch.Flow.md): Flow integration for parallel sketch reduction. - [ExDataSketch.GenStage](ExDataSketch.GenStage.md): GenStage integration for sketch aggregation. - [ExDataSketch.GenStage.SketchConsumer](ExDataSketch.GenStage.SketchConsumer.md): A GenStage consumer that accumulates events into a sketch. - [ExDataSketch.GenStage.SketchProducer](ExDataSketch.GenStage.SketchProducer.md): A GenStage producer that emits accumulated sketch snapshots on demand. - [ExDataSketch.GenStage.SketchStage](ExDataSketch.GenStage.SketchStage.md): A combined GenStage producer-consumer for sketch aggregation pipelines. - Persistence - [ExDataSketch.Storage](ExDataSketch.Storage.md): Behaviour, registry, and dispatching facade for ExDataSketch's persistence backends. - [ExDataSketch.Storage.CubDB](ExDataSketch.Storage.CubDB.md): CubDB-backed persistence for sketches. - [ExDataSketch.Storage.DETS](ExDataSketch.Storage.DETS.md): DETS-backed persistence for sketches. - [ExDataSketch.Storage.ETS](ExDataSketch.Storage.ETS.md): ETS-backed persistence for sketches. - [ExDataSketch.Storage.Ecto](ExDataSketch.Storage.Ecto.md): Ecto-backed persistence for sketches. - [ExDataSketch.Storage.Ecto.Migration](ExDataSketch.Storage.Ecto.Migration.md): Migration helper for creating the `ex_data_sketch_sketches` table. - [ExDataSketch.Storage.Ecto.Schema](ExDataSketch.Storage.Ecto.Schema.md): Ecto schema for the `ex_data_sketch_sketches` table. - [ExDataSketch.Storage.Mnesia](ExDataSketch.Storage.Mnesia.md): Mnesia-backed persistence for sketches. - Windowing - [ExDataSketch.Window](ExDataSketch.Window.md): A ring of tumbling sub-sketches for answering "in the last N" questions without a user-side timer. - Supervision - [ExDataSketch.Server](ExDataSketch.Server.md): A supervised, named, concurrently-updatable sketch process with optional windowing and snapshotting. - [ExDataSketch.Sketches](ExDataSketch.Sketches.md): A supervisor for starting `ExDataSketch.Server` processes per tenant or key at runtime, addressed by an arbitrary term rather than a compile-time atom. - Infrastructure - [ExDataSketch.Backend](ExDataSketch.Backend.md): Behaviour defining the computation backend for ExDataSketch. - [ExDataSketch.Backend.Pure](ExDataSketch.Backend.Pure.md): Pure Elixir backend for ExDataSketch. - [ExDataSketch.Backend.Rust](ExDataSketch.Backend.Rust.md): Rust NIF-accelerated backend for ExDataSketch. - [ExDataSketch.Codec](ExDataSketch.Codec.md): ExDataSketch-native binary serialization codec (EXSK format). - [ExDataSketch.Hash](ExDataSketch.Hash.md): Stable 64-bit hash interface for ExDataSketch. - [ExDataSketch.Sketch](ExDataSketch.Sketch.md): Behaviour implemented by every concrete sketch module. - [ExDataSketch.Telemetry](ExDataSketch.Telemetry.md): Structured telemetry event emission for ExDataSketch. - [ExDataSketch.Telemetry.Metrics](ExDataSketch.Telemetry.Metrics.md): Ready-made `Telemetry.Metrics` definitions for every ExDataSketch telemetry event. - [ExDataSketch.Telemetry.OpenTelemetry](ExDataSketch.Telemetry.OpenTelemetry.md): OpenTelemetry span integration for ExDataSketch telemetry events. - Dashboard - [ExDataSketch.LiveDashboard.Page](ExDataSketch.LiveDashboard.Page.md): A `Phoenix.LiveDashboard.PageBuilder` page listing every ExDataSketch telemetry event, its measurements, and the `Telemetry.Metrics` names `ExDataSketch.Telemetry.Metrics.all/1` derives from them. - Errors - [ExDataSketch.Errors](ExDataSketch.Errors.md): Error types for ExDataSketch. - Exceptions - [ExDataSketch.Errors.DeserializationError](ExDataSketch.Errors.DeserializationError.md): Returned when binary data cannot be deserialized. - [ExDataSketch.Errors.FilterFullError](ExDataSketch.Errors.FilterFullError.md): Raised by a bang function (`put!/2`, and so on) when the underlying filter has no capacity left for the insertion, e.g. `ExDataSketch.Cuckoo` exhausting its relocation budget, or an `ExDataSketch.FilterChain` stage reporting `{:error, :full}`. The non-bang counterpart returns `{:error, :full}` (or `{:error, :full, partial}`) instead of raising. - [ExDataSketch.Errors.IncompatibleSketchesError](ExDataSketch.Errors.IncompatibleSketchesError.md): Returned when attempting to merge sketches with incompatible parameters. - [ExDataSketch.Errors.InvalidChainCompositionError](ExDataSketch.Errors.InvalidChainCompositionError.md): Raised when a FilterChain stage composition is invalid. - [ExDataSketch.Errors.InvalidOptionError](ExDataSketch.Errors.InvalidOptionError.md): Returned when an option value is invalid. - [ExDataSketch.Errors.NotImplementedError](ExDataSketch.Errors.NotImplementedError.md): Raised when a stubbed function is called before its implementation is available. - [ExDataSketch.Errors.UnsupportedOperationError](ExDataSketch.Errors.UnsupportedOperationError.md): Raised when an operation is not supported by a structure. ## Mix Tasks - [mix ex_data_sketch.gen.migration](Mix.Tasks.ExDataSketch.Gen.Migration.md): Generates a migration for the `ex_data_sketch_sketches` table.