Where the Second Query on 10 GB Got Shorter — Full Spectrum Part 4: 0.73s Against ripgrep’s 13.54s on a Regex With No Required Literal, While the Two-Word Join Stays Level at 1.24x

Technical Guide

A mechanism that pays off at 3 GB does not automatically pay off at 10 GB.

The line skeleton and the block marks from the previous two parts were both designed against 3 GB measurements. Run the full suite over 10 GB and 100 million lines, and four queries were still slow on the second pass. This part is about where those four got shorter, what stayed merely level, and how a second search term inside a ±N line window ended up being matched.

The short version: 0.73 s against ripgrep’s 13.54 s on a regex with no required literal, 2.51x on the two-pass total across 30 items, and a ±N two-word join that stays level at 1.24x while the single-pass multi-term scan got dropped

The file is 10 GB — 10,255,315,281 bytes, 100 million lines of OpenStreetMap XML. I call it “10 GB” to match earlier articles, but the primary figure is the byte count: SI notation (÷1000³) shows 10.26 GB and GiB (÷1024³) shows 9.55 GiB. Same file either way.

In the v1.9.0 suite (Mac, 30 items, 27 matched, zero mismatches) the second-pass (hot) figures came out like this.

  • A regex with no required literal ([0-9]{3}-[0-9]{4}", 24,988 matching lines): ripgrep 13.54 s against uvp 0.73 s — 18.55x
  • -E -v (lines not matching ^ +<, zero matches): ripgrep 3.57 s against uvp 0.13 s — 27.46x
  • -tail 10 (last ten matches): rg piped into tail 0.98 s against uvp 0.40 s — 2.45x
  • A plain search (11,393 matching lines): ripgrep 0.98 s against uvp 0.43 s — 2.28x
  • Across 30 items the two-pass total (cold plus hot) came to 2.51x

The losses first. The ±N line two-word join only reached 1.24x even with the new method — under 1.5x, which I call level, with the new method on the faster side. On 3 GB that is 0.713 s to 0.575 s. I also built and measured a scan that searches for every term in a single decompression pass. It was not faster (0.564 s to 0.579 s), so it never shipped. When the second term looks likely to match more than a million lines, the new method is skipped and the older one runs instead. The first pass is still slower in uvp because the index is being built — 10.96 s against 12.72 s, which is 1/1.16 and level — and the aggregation query is marginally faster in ripgrep too, 0.91 s against 1.03 s (1/1.13, level).

uvf and uvp are the search commands of UwView, which I build (uvf is the free edition, uvp is UwView Pro). The figures here were measured on v1.9.0 “Full Spectrum” (released 14 October 2026, build 26.10.13.22) on a Mac (Apple M4, 10 cores, 32 GB of memory, external USB SSD) on 13 October 2026. The before-and-after ratios (1.8.3.2 against 1.8.3.6) come from development builds. v1.9 carries the pet name “Full Spectrum” because it brings Shift_JIS, EUC-JP and UTF-16 files into the index as UTF-8, so encoding and file type no longer change how fast a search runs.

The problem: four queries that were still slow on the second pass over 10 GB

Right after the skeleton and the marks went in, the 10 GB suite showed a first pass that had barely moved (11.79 s to 11.84 s) and four second-pass queries that were still heavy. The table below comes from development builds 1.8.3.2 and 1.8.3.6 — the v1.9.0 suite has no “before” condition, so seeing the change requires putting those two builds side by side.

Second pass over 10 GB (hot) 1.8.3.2 (s) 1.8.3.6 (s)
-E -v 1.69 0.25
-C 3 (second term within ±3 lines) 2.68 1.82
-tail, -out 0.93 0.39–0.42
Regex with no required literal 1.74 0.72

They were slow for four different reasons. -E -v asks for lines that do not match, so the block marks cannot help — marks only ever work in the direction of skipping blocks that contain something. -C 3 found the first term and then re-read the surrounding lines to look for the second. -tail only needs the last ten matches but was scanning everything. And a regex with no required literal gives the narrowing step nothing to work with.

-E -v and the unprefixable regex — the line skeleton held up at 10 GB

Those two were solved by the line skeleton from Part 2, unchanged. It keeps a small dictionary of line shapes, runs the pattern against that once, and leaves a block closed when it is settled that nothing in it can match. Unlike the marks, it works on the “contains none” side too, so -v can use it.

Searching 10 GB for lines that do not match ^ +< returns nothing at all. The skeleton establishes that up front, so uvp’s second pass takes 0.13 s against ripgrep’s 3.57 s — 27.46x. The first pass is 0.22 s against 11.09 s, which is simply the difference between re-reading the original file and not re-reading it.

The unprefixable regex takes the same route. [0-9]{3}-[0-9]{4}" yields no required literal, so the block marks from Part 3 are unavailable. The skeleton alone brings it to 0.73 s against ripgrep’s 13.54 s — 18.55x. The free edition, which builds no index, took 3.82 s, still 3.54x faster than ripgrep here. ripgrep barely improves between passes on this query (14.29 s cold, 13.54 s hot), which is why the gap is so wide.

-C 3 — stop re-reading, search the second term once, match on line numbers

This is the new mechanism in this part. term1 -C N term2 returns lines where the second term appears within N lines of the first. The old implementation re-read the ±N window for every hit on the first term. With many hits, that re-reading piles up.

The new one searches the whole file once for the second term and matches it against the first term’s ±N windows by line number (-v included). The re-reading disappears; a full scan for the second term appears in its place. Which one wins flips depending on hit counts, so a decision has to be made per query.

The input to that decision is a sample: eight places in the file, 4,096 lines each. From that the second term’s hit count is estimated, and:

  • Above roughly a million lines, use the old method (adjustable with UVP_JOIN_MAX)
  • If the skeleton and marks already narrow the first term to under half the file, use the old method — a narrow range is faster even with re-reading
  • UVP_NO_JOIN=1 forces the old method always, for cross-checking and measurement

The million-line threshold came from measuring both sides of it. At 250,000 and 500,000 a word matching 930,000 lines fell back to the old method and dropped to 0.81x–0.89x. At two million, a word matching 20 million lines dropped to 0.75x. Missing in either direction costs something, so the threshold sits between them.

Figures below: 3 GB, 1 MB blocks, extra sections present, second pass, median of three runs.

Query Old method (s) New method (s) Ratio (x)
k="amenity" -C 3 k="name" 0.713 0.575 1.24
Same with -v 0.749 0.590 1.26
highway (930k lines, estimate routes it to the old method) — — 0.99
<tag (20M lines, same) — — 0.98
<nd (54M lines, same) — — 0.97

1.24x and 1.26x are level. They do not reach 1.5x, so they are not a difference I can claim. The new method is on the faster side, but that is all. With 4 MB blocks and no extra sections the same query goes 0.858 s to 0.647 s (1.32x) — level again. The three queries that the estimate sent to the old method land at 0.97x–0.99x, meaning only the cost of estimating is visible.

In the 10 GB suite, k="amenity" -C 3 k="name" (66,778 matching lines) took 2.22 s cold and 1.85 s hot. ripgrep has no equivalent for this query — rg -C prints surrounding context rather than selecting lines by what is in that context. For reference, the nearest ripgrep shape, two terms on the same line via two piped rg calls (2,659 matching lines), gives a second pass of 1.03 s against uvp’s 0.90 s — 1.14x, level.

What got dropped: searching every term in one decompression pass

-seq (lines where terms appear in a given order) searches the whole file per term and matches positions against each other. Decompression is the expensive part, so searching for every term during a single decompression ought to be faster. I built it and measured it.

The result was 0.564 s to 0.579 s. Not faster. On this Mac the matching work costs more than the decompression, so searching for two terms together took as long as searching twice. All it added was complexity, so it came out. -seq still uses the original design: 0.56–0.60 s on 3 GB, and 2.22 s cold / 1.84 s hot on 10 GB.

The premise — that decompression dominates — simply did not hold for this data on this machine. Shipping it unmeasured would have left the complexity and none of the gain.

The numbers against ripgrep (v1.9.0 suite, 10 GB)

The opponent is ripgrep 15.2.0, with identical queries and verified identical match counts. Cold runs follow a cache purge and a ten-second wait; hot runs follow immediately. The plain search’s cold time is measured after deleting the .uwvz, so it includes building the index.

Query Matches (lines) ripgrep-equivalent cold (s) Same hot (s) uvp cold (s) uvp hot (s) Hot ratio (x)
Regex with no required literal 24,988 14.29 13.54 1.29 0.73 18.55
-E -v (^ +<) 0 11.09 3.57 0.22 0.13 27.46
-v (=") 1,224,742 11.08 2.62 1.41 1.09 2.40
-tail 10 10 11.49 0.98 2.54 0.40 2.45
Plain search 11,393 10.96 0.98 12.72 0.43 2.28
Aggregation -uniq (top 20) 20 10.95 0.91 1.42 1.03 1/1.13

The -tail comparison is against rg piped into tail -10. uvp reads from the end, so even its first pass is 2.54 s against 11.49 s.

Summed over 30 items: cold 320.52 s against 129.36 s (2.48x), hot 76.18 s against 28.85 s (2.64x), and a two-pass total of 396.70 s against 158.21 s — 2.51x.

Losses: queries that stayed level, the first pass, aggregation, and the fallback

1. The ±N two-word join is level. 1.24x and 1.26x. Removing the re-reading and still not reaching 1.5x means the full scan that replaced it costs about as much as the re-reading did.

2. When the second term is common, the new method is not used. Above roughly a million lines it falls back. So the mechanism is strictly for sparse second terms; words like highway or <tag get nothing from it, only the sampling overhead — 0.97x to 0.99x.

3. The single-pass multi-term scan was dropped. 0.564 s to 0.579 s. The time spent building and measuring it does not come back.

4. The first pass is faster in ripgrep. The plain search is 10.96 s against 12.72 s cold (1/1.16, level), the difference being the index build. The free edition, which builds nothing, came in at 10.18 s cold and 0.89 s hot — close to ripgrep (1.08x and 1.10x).

5. The aggregation query’s second pass is faster in ripgrep. -uniq with a top-20 cut is 0.91 s against 1.03 s (1/1.13, level); -sort count and -sort value are 1/1.11 and 1/1.12. Nothing shrank here.

6. The batched-read fix has not been re-measured cold. Moving to 1 MB blocks and reading the extra sections made the 10 GB cold times for -i, regex and aggregation 0.15–0.2 s slower. The fix reads the mark tables in 8 MB batches instead of per block, and only loads the skeleton and mark sections for queries that use them — but re-measuring cold needs sudo for the cache purge, so it is unmeasured on my side. The next full suite will confirm it.

Correctness: byte-identical output was the condition for shipping it

Changing how a join is computed can change which lines come out, and a fallback means two code paths. Both paths producing the same answer is the premise, not a nice-to-have.

  • Byte-identical output: the new method and UVP_NO_JOIN=1 (the old one) were compared byte for byte across -C, -v, -i, -E, -head, -uniq, a chained second -C, bundled indexes, CRLF and --csv.
  • On a real build: from the test dmg, k="amenity" -C 3 k="name" over 3 GB produced the same output as UVP_NO_JOIN=1.
  • The path is visible: the CLI reports which route a stage took. When the join is used it prints that the stage matched the previous stage’s ±N window against a full-file search for the term by line number. A tool that switches methods for speed is not trustworthy if you cannot see which method ran.
  • Full suite: 27 items matched over 10 GB with zero mismatches. The automated tests pass at 1,068 cases for Pro.Core and 372 for the UI.

The general lesson: “removing expensive work” only counts once you have measured what replaced it

Three things came out of fixing these four queries.

Measure what you removed and what you added, side by side. The join stayed at 1.24x because the new full scan was about as heavy as the re-reading it replaced. Looking at only one side invites the conclusion that removing re-reads must have made things faster. Put both numbers on the page and “level” becomes obvious.

Pick a threshold by measuring past it in both directions. The million-line cutoff exists because 250,000 and 500,000 dropped to 0.81x–0.89x and two million dropped to 0.75x. A threshold chosen by feel tells you nothing about what it costs to be wrong on either side.

Premises are machine- and data-specific. “Decompression is expensive, so do it once” did not hold for this XML on this Mac. It may well hold elsewhere — but not shipping it until it does means less to unwind later.

In uvp

uvp japan.osm 'k="amenity"' -C 3 'k="name"'     # second term within ±3 lines
uvp japan.osm 'k="amenity"' -C 3 'k="name"' -v  # second term absent from ±3 lines
uvp japan.osm -seq 'k="amenity",k="name"'       # terms appearing in this order
uvp japan.osm '[0-9]{3}-[0-9]{4}"' -E           # no required literal; narrowed by the skeleton
uvp japan.osm '^ +<' -E -v                      # non-matching lines; the skeleton applies
uvp japan.osm 'Tokyo' -tail 10                   # last ten matches, read from the end
UVP_NO_JOIN=1 uvp japan.osm 'k="amenity"' -C 3 'k="name"'  # always the old method
uvf japan.osm '[0-9]{3}-[0-9]{4}"' -E           # the free edition builds no index

The join threshold is set with UVP_JOIN_MAX (for measurement). Exit codes follow grep: 0 for matches, 1 for none, 2 for an error or an incomplete result. The free edition is on GitHub Releases (brew install --cask amru195704/uwview/uwview on macOS, Scoop on Windows, Snap on Linux).

  • Full Spectrum Part 1 — bringing Shift-JIS, EUC-JP and UTF-16 into the index as UTF-8: https://uvp.y42u.net/en/blog/uwview-fs01-sjis-utf8-index-en/
  • Full Spectrum Part 2 — skipping whole blocks of non-matching lines with the line skeleton: https://uvp.y42u.net/en/blog/uwview-fs02-line-skeleton-en/
  • Full Spectrum Part 3 — knowing which blocks not to open: https://uvp.y42u.net/en/blog/uwview-fs03-block-marks-en/
  • The v1.9.0 “Full Spectrum” release post (what the whole version changed): https://uvp.y42u.net/en/blog/uwview-v190-full-spectrum-en/
  • Extracting a required literal from a regular expression: https://uvp.y42u.net/en/blog/uwview-fl04-regex-required-literal-en/
  • Where ripgrep wins, collected in one place: https://uvp.y42u.net/en/blog/uwview-wf14-where-ripgrep-wins-en/
  • The newline counting that took 40% of the second query: https://uvp.y42u.net/en/blog/uwview-wf12-newline-hot-en/
  • The previous series, wrapped up (multiple files, seven compression formats, zip, pbf): https://uvp.y42u.net/en/blog/uwview-wf15-wide-field-recap-en/
  • Every measurement, in one table: https://uvp.y42u.net/en/benchmarks-en/
  • Source code on GitHub: https://github.com/amru195704/UwView

And if you keep searching the same huge logs for the same handful of terms, UwView Pro makes the second query a step faster — a persistent index, search inside the compressed cache, and roughly a ninth of the storage (every OS, one-time purchase or monthly subscription, 14-day free trial; see the pricing page).

From the developer: my apps, Kindle books and public projects are listed at GitHub: amru195704.

Copied title and URL