Compressed logs are faster to search — seven formats measured at 1 GB and 3 GB

[Update, 16 October 2026] Speed today — re-measured with v1.9.0 “Full Spectrum”

The uvf and uvp numbers in this article come from the Wide Field development build (1.7.1.6, September 2026). The same files on the same Mac were measured again with the current version, v1.9.0 (14 October 2026).

Searching 3 GB for “東京”	This article (1.7.1.6)	Now (v1.9.0)
uvp after converting to .uwvz (from the 2nd search)	0.31–0.40 s	0.15–0.16 s (about 2× faster)
the same at 1 GB	0.14–0.16 s	0.10 s
uvp first run (building the .uwvz from plain text, search included)	3.59 s	3.84 s (about the same)
uvf searching the compressed file directly (first run, seven formats)	0.89–23.48 s	0.86–22.32 s (about the same; on par with rg -z in every format)

The article’s conclusion — compressed logs are faster to search than plain text — still holds with v1.9.0 (rg -z, 3 GB, first run: lz4 0.94 s, gz 1.32 s, zst 1.62 s, xz 1.67 s, br 2.49 s vs plain 3.31 s). Details: Benchmarks and the v1.9.0 “Full Spectrum” article.

Logs cannot stay as they are. Older ones get compressed and kept. When you pick the format, do you pick it by how small it gets and nothing else?

I compressed the same file seven ways and timed ripgrep searching each one for a single term. The size ordering and the speed ordering do not match. And for most formats, the compressed file was faster to search than the plain one.

Note: the uvf, uvp and .uwvz numbers in this article come from Wide Field (1.7.1.6), a development build measured before release. Wide Field was released as v1.7.3.6 on 29 September 2026 (update). Figures measured on the released version are on the Measured Results page. The format comparison itself (the rg -z numbers) uses the released ripgrep 15.2.0.

One term in 3 GB

One 2,937.7 MB text file and the same file in seven formats. The term is 東京; every number is the first run after dropping the cache.

uvf is the search command of the free UwView: it searches the compressed file with a built-in decoder, without calling any external command. Every format came out about the same as rg -z (under 1.5×). The last row is for reference: the paid uvp after it has turned the file into .uwvz (its own compression format plus index), searching the second time on (more in the last section).

The five below plain (br, xz, zst, gz, lz4) beat plain text. So does .uwvz in the last row, from the second search on. For the five, the reason is simple: there is a fifth to a ninth as much to read from disk. The time saved on reading is larger than the time spent decompressing. This external SSD reads at about 950 MB/s, so 2.9 GB takes three seconds just to get off the disk. 318 MB takes 0.3 s.

The two above plain (bz2, lzma) lose to plain text. For lzma and bz2, decompression itself cannot keep up — bz2 decompresses at 118 MB/s here.

At 1 GB, the same ordering

The same data at 979.2 MB, read the same way as the 3 GB table.

The size ordering (bz2 < xz < lzma < br < gz < zst < lz4) and the speed ordering are the same at both sizes. Three times the data, roughly three times the seconds — it scales the way you would hope.

Time to compress

To store a log compressed you first have to compress it. If logs arrive every day, so does this cost.

More time does not buy a smaller file. bz2 spends 135 s to reach 8.5%; xz spends 50 s to reach 8.6% — a third of the time for the same size. And zst finishes in a tenth of the time gz takes (1.94 s vs 19.64 s), at almost the same size (11.8% vs 10.8%). zst and lz4 compress about as fast as the file can be read. xz takes 50 s even compressing in parallel; bz2 and lzma take over two minutes.

The bottom row, .uwvz, is uvp‘s own compression format plus index: the time to build it from the plain file (building plus the first search, Wide Field).

xz, lzma and brotli were run lighter than their defaults (the defaults are -6 for xz and lzma, 11 for brotli). At the defaults they would get somewhat smaller and compress more slowly still.

Choosing on all three

Compression is paid once, size for as long as you keep the file, search every time you look. Here are the three together.

• If in doubt, zst. It compresses in a tenth of gz’s time, and is about the same size and speed to search. It is near the top on all three and has no weak spot.
• If you do not know what the reader has, gz. Everything can decompress it. The price is ten times zst’s compression time.
• For long-term storage, xz. Spend the time once, keep the smallest file, and it is still faster to search than plain text. Block splitting is required: this xz has 245 blocks and the external xz command decompresses them in parallel. A single-block xz (written with -T1) is far slower to search.
• For a firehose of logs, lz4. Compression is instant and search is the fastest — but at 18.1% it is not for long-term storage.
• br is not for logs. It compresses as slowly as gz, is only slightly smaller, and takes twice as long to search. It was made for serving web content.
• Skip bz2 and lzma. Slow to compress, slow to search, and no smaller than xz. If you already keep logs as bz2, repacking them as zst or xz is worth it.

How it was measured

• Files: OpenStreetMap Japan as XML, cut to 979.2 MB and 2,937.7 MB. The 3 GB one is the same content concatenated three times; the repeat interval is longer than each format’s window, so the ratios stay close to the 1 GB ones
• Term: 東京 (1,145 hits at 1 GB, 3,435 at 3 GB)
• Settings: gzip -6, bzip2 -9, xz -3, lzma -3, zstd -3, lz4 -1, brotli -q 5 (defaults for gz, bz2, zst and lz4; lighter than default for xz, lzma and brotli). xz 5.6 and later compress in parallel by default, producing several blocks (82 at 1 GB, 245 at 3 GB)
• Compression times were measured once, without dropping the cache, and are for reference
• Each measurement: sync → drop the cache → wait 10 s → first run (cold) → the repeat right after (hot)
• ripgrep 15.2.0 with -z. rg -z shells out to the external tools (gzip, bzip2, xz and so on) to decompress
• Mac (Apple M4, 10 cores), external USB SSD. Seconds vary by machine
• Every measurement was checked against the plain-text search for the same hit count (20 items each, 0 mismatches)

What this looks like in our own tool

From here on, this is about the product I make. UwView’s search command decompresses with built-in decoders instead of shelling out. I measured whether that shows.

The free uvf is even with rg -z on every format. Over the 3 GB pair of runs it is 1.29× on gz, 1.26× on lzma, 1.15× on br, 1.09× on lz4, 1.07× on zst and 1/1.07 on xz — all under 1.5×. Not going through a pipe should help, but not by enough to call it a difference, and on xz, its weakest format, it falls just short of the external xz’s parallel decompression.

The paid uvp converts a compressed file once into .uwvz (its own compression format plus index) and searches that from then on — the last row of the table above. Searching takes 0.31–0.40 s regardless of the original format. Stored as bz2 or stored as xz, after the conversion it is the same. The first search, conversion included, also landed within 1.5× of rg -z for every format (gz 1.82 s vs 1.36 s, bz2 24.41 s vs 24.93 s, and so on). The format only matters once, during that conversion (24.41 s from bz2, 1.34 s from lz4).

So this is less “a faster search” than a different way to keep the file. Leave it compressed and decompress on every search, or convert it and stop decompressing. The first is smaller; the second is faster to search. At 11.2%, the .uwvz lands about where zst does — the same storage cost, with search times in the third of a second range. The original can be deleted; -extract brings it back.

Whether the conversion pays depends on how often you search. With bz2 you save 24.5 s per search, so it pays on the first one; with gz the gap is 0.96 s, so it pays on the second.

These numbers are from Wide Field (1.7.1.6), a development build measured before release. Wide Field was released as v1.7.3.6 on 29 September 2026 (release notes). The free uvf searches all seven formats as they are; converting to .uwvz and searching that is uvp, part of the paid UwView Pro.
The free UwView is on GitHub Releases. On a Mac, brew install --cask amru195704/uwview/uwview; on Windows, Scoop (scoop bucket add uwview https://github.com/amru195704/scoop-uwview, then scoop install uwview).