How we cut query latency by 94% with a columnar rewrite

The short version

Last quarter, our p95 dashboard queries took 3.2 seconds. Today they take 190 milliseconds. This is the story of the rewrite that got us there—not a heroic tale, but a series of unglamorous decisions that compounded.

We stopped storing data the way we queried it, and we stopped interpreting data at read time. Everything below is the long version: where the time went, what we changed, which ideas failed, and how we decided the rewrite was ready for production.

Where the time went

The first mistake was looking at the database as a single slow box. A query trace showed a different picture: storage scans, expression parsing, row materialization, and cache misses were each adding a little friction. No individual stage looked catastrophic, but the accumulated work dominated the request.

Our row-oriented format read complete records even when a chart needed three fields from a payload containing dozens. The execution layer then interpreted filter expressions repeatedly, while the cache was tuned for recurring dashboards rather than the varied, exploratory queries analysts run during an investigation.

The database was not slow. The way we asked it questions was expensive.

Move one: columnar chunks

We reorganized storage into sorted runs of roughly 100,000 rows, with values grouped by column and compressed using the encoding best suited to that column. Timestamps became delta-encoded, repeated dimensions used dictionaries, and numeric series stayed in tight contiguous buffers.

This layout changed the unit of work. A query could now read only the columns it needed, skip entire chunks outside its time range, and decompress values in batches instead of reconstructing records one field at a time.

Move two: vectorized execution

Our original operators were easy to reason about because they processed one row at a time. They were also wasteful. Every row crossed the same function boundaries, repeated the same type checks, and paid the same branching costs.

The new engine operates over vectors of values. Filters produce bitmaps, arithmetic operators consume contiguous arrays, and aggregations can stay in CPU-friendly loops. The improvement was not one clever instruction; it was fewer interpretive decisions per value.

Move three: predicate pushdown

Columnar storage only helps if the engine can avoid bringing irrelevant data into memory. We therefore pushed time bounds, equality filters, and partition predicates as close to storage as possible.

When a query asks for a service, region, and narrow time window, the storage layer now rejects chunks before they are decoded. The execution plan carries these constraints all the way down, so every layer does less work.

The dead ends

We spent two weeks tuning the cache before accepting that caching the wrong representation could not rescue the design. We also prototyped a more aggressive compression scheme that looked excellent in benchmarks but produced unacceptable CPU spikes during concurrent reads.

Another tempting direction was a complete rewrite of the query language. We resisted it. The largest gains came from making the existing language cheaper to execute, not from asking customers to relearn how to express a question.

How we measured the rewrite

Every change had to improve a representative query set rather than a hand-picked benchmark. We replayed dashboard traffic, ad-hoc investigations, long-range rollups, and high-cardinality group-bys against identical data snapshots.

We tracked p50, p95, and p99 latency separately, along with bytes read, decompression CPU, peak memory, and concurrency collapse. A faster single query was not a win if the system became unpredictable when twenty investigations ran together.

What the numbers taught us

The final 94% improvement was the result of removing layers, not inventing them. Column pruning delivered the largest step, vectorized operators made that step consistent, and predicate pushdown prevented the new layout from being undermined by an eager planner.

We also learned that tail latency is a design constraint, not a reporting detail. The median got better early. The work that made the system feel dependable happened later, when we eliminated pathological plans and bounded memory growth under load.

What we would do differently

We would have instrumented storage decisions earlier. For the first month, we could see query duration but not why a chunk was read or skipped. That made optimization feel like archaeology.

We would also define concurrency targets before choosing compression formats. A format that wins in isolation can lose in a real service when CPU, memory, and network are shared. The lesson is simple: optimize the whole request lifecycle, and keep the measurement close to the decision.

The principle we are keeping

None of this is novel. That was the point. The 94% came from removing layers, not inventing them—and from measuring every step so we knew exactly which abstractions we could afford.

Performance work becomes repeatable when it stops being a search for magic and becomes a practice of narrowing the amount of work each request is allowed to do.