“The first question is about 20% slower than ripgrep” — that is what I was going to write. uvp, the UwView Pro command, builds a .uwvz (compressed copy plus index) on the first search. Of course building costs time; in exchange, every question after that is fast. That is the tool.
Before putting “20%” in an article, though, I measured what that 20% was made of. Half of it turned out to be a loss we did not have to take.
Results
One file of 51.25 GB. .uwvz deleted first, cold after sudo purge, median of three runs (output to /dev/null).
| 50 GB, first question | Time | What it does |
|---|---|---|
| ripgrep 15.2.0 | 54.94–55.81 s | searches while reading |
uvp v1.6.6 |
65.7 s | builds the .uwvz in 58.6 s → searches the new .uwvz again, ~7 s |
uvp v1.6.6.1 |
59.04 s | searches while building (build 58.6 s; 0.03 s to emit the results) |
(for reference) uvp second question |
6.6 s | searches the .uwvz; 8.4× ripgrep |
65.7 s → 59.04 s: 6.7 s faster. The gap to ripgrep went from about 20% to 1.06–1.07× (6–7%).
What was going on
Through v1.6.6, the first question ran in two steps:
- Read the original file and build the
.uwvz(58.6 s) - Read the freshly built
.uwvzand search it (~7 s)
That “~7 s” looked familiar. It is almost exactly the 6.6 s of the second question at the bottom of the table.
Naturally — step 2 was a second question. The first question had a whole second question inside it.
Step 1 reads the original file from start to end, yet never looks at the search term while doing so. Only after it finishes does it open the .uwvz it just wrote and search again. Build it the straightforward way and this is what you get: keep “build the index” and “search the index” as separate parts, and the first question is just the two glued together.
What v1.6.6.1 does
It looks at the search term while it reads.
As each line is packed into the .uwvz, the same line is checked against the term. By the time the read finishes, every hit is already in hand; all that is left is to print them, which takes 0.03 s.
$ uvp japan-latest.osm '東京' > /dev/null
creating japan-latest.osm.uwvz while searching …
created japan-latest.osm.uwvz while searching in 58.6s
On the first question it tells you it is building and searching together (from the second question on the .uwvz already exists, so you will not see this).
Building still takes 58.6 s. Nothing about compression or indexing changed. Only the second search went away, and its 6.7 s came straight off the first question.
The window (GUI) already worked this way: v1.6.6 “First Light” made it start searching without waiting for the index to finish. This time the command caught up with the window.
Why it is still 6–7% slower
59.04 s is a read rate of 828 MB/s. The free uvf, which searches the same file without building a .uwvz, reads at 962 MB/s — so this is 86% of that.
The rest is compressing, and writing a 5.74 GB .uwvz. That cannot be cut: cut it, and the second question no longer comes back in 6.6 s.
So the first question’s delay is now entirely the price of the second question. Through v1.6.6 there was an unnecessary second search on top of that price.
If the output goes to a file, the gap widens a little. In the test script, which writes the hits to a file and checks them, the first question took 3.74 s at 3 GB, 12.18 s at 10 GB and 63.55 s at 50 GB (1.13×, 1.09× and 1.14× ripgrep’s time). The 4.5 s over the 59.04 s above is the writing and checking.
Which one to use
- Looking at a file once → the free
uvfis enough. It does not build a.uwvz, so it is as fast as ripgrep (benchmarks). - Coming back to the same file again and again →
uvp. Pay 6–7% on the first question and from the second question on it searches 50 GB in 6.6 s.
What changed in v1.6.6.1 is that the price is now only the part you actually have to pay.
How it was measured
- File: OpenStreetMap Japan as XML, one file of 51,254,526,392 bytes (
japan-latest.osm) - Term:
東京(94,979 hits) - First question with the
.uwvzdeleted, cold aftersudo purge. Median of three runs with/usr/bin/time -p(59.04 / 59.02 / 60.07 s). Output to/dev/null - ripgrep 15.2.0’s 54.94–55.81 s is the range measured on the same machine, same file, same conditions
- v1.6.6’s 65.7 s and its breakdown (build 58.6 s + searching again) are the median of three runs under the same conditions
- Mac M4 / 32 GB / external USB SSD (raw read about 950 MB/s). Seconds vary by machine
UwView Pro is here ($129 one-time or $9/month, with a 14-day free trial).
The free UwView (uvf) is on GitHub Releases; on a Mac, brew install --cask amru195704/uwview/uwview, and on Windows, Scoop (scoop bucket add uwview https://github.com/amru195704/scoop-uwview, then scoop install uwview).
