Ask Alidade from Slack and Linear.
Engineering
Why we don’t sample your data
Sampling saves compute and costs accuracy where it matters most: small segments and rare events.
Jonas Ekberg
·
Many analytics tools read a slice of your events and scale the result up. It is fast and cheap, and for a large total it is usually close enough. The trouble starts when you ask about something small.
Where sampling goes wrong
Say a bug hits 0.5% of checkouts, about 50 sessions out of 10,000 a day. A 10% sample sees five of them, and on a bad day sees one. The estimate swings from half the true value to double it. That is exactly the kind of change an alert should catch.
Rare events, small segments and short time ranges are where decisions get made, and where a sample is weakest.
What we do instead
Alidade runs every query against the full table in your warehouse. We keep costs down by pre-aggregating hourly, partitioning by date and reading only the columns a query needs. We never do it by dropping rows.
In our tests on a table of 2 billion events, the hourly pre-aggregates cut scanned data by about 95%, and answers matched a full scan exactly.
The trade is a little more compute on your side. Most teams spend under 50 dollars a month on the extra queries, and Alidade shows the estimated cost before any heavy query runs.
