Sample-data generators for the tutorial livebooks under livebooks/sketches/.
Not part of the sketch API -- this module exists purely so each livebook's
"Sample data" section is a single function call instead of a duplicated
cache-path/generate/cache-write block. Every function here generates a
fixed shape of data (documented per function below) at a default size
matching what its livebook actually uses, and transparently caches that
default result under System.tmp_dir!(), so re-running a livebook (or a
whole verification sweep across all of them) after the first pass is
instant.
Every generator also accepts real size overrides (:count, :pool_size,
:half_count, or a couple of Theta/IBLT-specific keys -- see each
function's @doc), primarily so this module's own regression tests don't
need to generate millions of items to exercise it. Caching only applies
to the default (no-override) call -- passing any option other than
:backend skips the cache file entirely, so a small test run can never
read stale-shaped data from, or overwrite, the real livebook cache.
Randomized generators use the ExDataSketch.Backend.Rust NIF when it's
available, since the defaults are 1-2 million items; pass backend: ExDataSketch.Backend.Pure to force the Elixir fallback (used
automatically when the NIF isn't loaded). Deterministic range-based
generators (no randomness involved -- bloom_urls/1, cuckoo_sessions/1,
quotient_api_keys/1, xor_filter_domains/1, filter_chain_users/1,
theta_sets/1, iblt_keys/1) are plain Elixir only: a single linear pass
building formatted strings has nothing for a NIF to meaningfully
accelerate.
Summary
Functions
Bloom tutorial: inserted URLs plus an equal number of novel URLs to test the false-positive rate against.
CMS tutorial: page-view events, power-law distributed (a few pages dominate).
CQF tutorial: rate-limit-check events, power-law distributed.
Cuckoo tutorial: inserted sessions plus an equal number of novel sessions.
DDSketch tutorial: operation durations (ms) spanning several orders of magnitude (fast API calls, medium DB queries, rare slow jobs).
FilterChain tutorial: inserted users plus an equal number of novel users.
FrequentItems tutorial: search queries, power-law distributed.
HLL tutorial: events drawn uniformly from a visitor pool.
IBLT tutorial: two mostly-agreeing key sets simulating drifted replicas.
KLL tutorial: simulated latencies (ms), mostly fast with a long tail.
MisraGries tutorial: same shape as frequent_items_queries/1 (so the two
tutorials are directly comparable), cached separately.
Quotient tutorial: inserted API keys plus an equal number of novel API keys.
REQ tutorial: latencies (ms), tight and boring in the bulk with a rare, important tail.
Theta tutorial: two overlapping user-ID sets with a known true union/intersection.
ULL tutorial: events drawn uniformly from a session pool.
XorFilter tutorial: blocklisted domains plus an equal number of novel (safe) domains.
Functions
Bloom tutorial: inserted URLs plus an equal number of novel URLs to test the false-positive rate against.
Options: :half_count (default 500,000).
CMS tutorial: page-view events, power-law distributed (a few pages dominate).
Options: :count (default 2,000,000), :pool_size (default 10,000).
CQF tutorial: rate-limit-check events, power-law distributed.
Options: :count (default 1,000,000), :pool_size (default 50,000).
Cuckoo tutorial: inserted sessions plus an equal number of novel sessions.
Options: :half_count (default 500,000).
DDSketch tutorial: operation durations (ms) spanning several orders of magnitude (fast API calls, medium DB queries, rare slow jobs).
Options: :count (default 1,000,000).
FilterChain tutorial: inserted users plus an equal number of novel users.
Options: :half_count (default 500,000).
FrequentItems tutorial: search queries, power-law distributed.
Options: :count (default 1,000,000), :pool_size (default 5,000).
HLL tutorial: events drawn uniformly from a visitor pool.
Options: :count (default 2,000,000), :pool_size (default 500,000).
IBLT tutorial: two mostly-agreeing key sets simulating drifted replicas.
Options: :shared_count (default 200,000), :only_a_count (default 7),
:only_b_count (default 5).
KLL tutorial: simulated latencies (ms), mostly fast with a long tail.
Options: :count (default 1,000,000).
MisraGries tutorial: same shape as frequent_items_queries/1 (so the two
tutorials are directly comparable), cached separately.
Options: :count (default 1,000,000), :pool_size (default 5,000).
Quotient tutorial: inserted API keys plus an equal number of novel API keys.
Options: :half_count (default 300,000).
REQ tutorial: latencies (ms), tight and boring in the bulk with a rare, important tail.
Options: :count (default 1,000,000).
Theta tutorial: two overlapping user-ID sets with a known true union/intersection.
Options: :count_a (default 600,000), :count_b (default 600,000),
:overlap (default 200,000, must be <= min(count_a, count_b)). Set A is
1..count_a; set B starts overlap items before set A ends, giving an
intersection of exactly overlap items.
ULL tutorial: events drawn uniformly from a session pool.
Options: :count (default 2,000,000), :pool_size (default 300,000).
XorFilter tutorial: blocklisted domains plus an equal number of novel (safe) domains.
Options: :half_count (default 500,000).