Three Fixes in the Free Edition — Full Spectrum Part 5: Writing Out Matching Lines Goes From 4.70 s to 2.68 s and 28 GB of Allocation to 0.4 GB, Yet Stays Level With ripgrep’s 2.83 s

The first four parts of this series were all about the index in the paid uvp. The free uvf builds no index, and it had a different kind of slowness left in it.

Reading through a full test run, I found one query where only the free edition lost to ripgrep. Chasing it down, the cost turned out not to be in the matching at all — it was in writing the matching lines out. Two more fixes came out of the same pass. All three shipped in v1.8.1.9 “Finder Scope” and are included in v1.9.0 “Full Spectrum”.

The short version: writing out goes 4.70 s to 2.68 s with allocation 28 GB to 0.4 GB, but ripgrep’s 2.83 s leaves it level at 1.06x; the lookahead query becomes 1.65x faster than ripgrep; and the ** listing fix is level at 1.38x

The losses first.

Before the fix, a 3 GB file searched with \d+\.\d{6,} and -H was 1.85x faster in ripgrep. While running it, uvf allocated up to 28 GB. After the fix it is 4.70 s to 2.68 s, with allocation 28 GB down to 0.4 GB. And ripgrep still takes 2.83 s — 1.06x, which is under 1.5x and so level, with uvf on the faster side. It caught up. It did not win.

The three fixes side by side:

• Writing out matching lines (3 GB, \d+\.\d{6,}, -H): 4.70 s to 2.68 s (ripgrep 2.83 s, 1.06x and level). Allocation 28 GB to 0.4 GB
• A positive lookahead (?=…) (Linux kernel, 60,000 files): 1.52 s to 0.57 s (ripgrep 0.94 s — 1.65x faster). A single file with many candidate lines is handed to the window app: 4.68 s to 2.18 s
• Expanding folders with ** (a 60,000-file copy): listing construction 0.33 s to 0.24 s (1.38x, level). And it does not apply to a real src/linux tree with a .git directory in it

uvf and uvp are the search commands of UwView, which I make (uvf is the free edition, uvp the paid UwView Pro). All three fixes here shipped in v1.8.1.9 “Finder Scope” and are included in v1.9.0 “Full Spectrum”. The v1.9 line is nicknamed Full Spectrum because it brings Shift_JIS, EUC-JP and UTF-16 files into the index as UTF-8, so text searches at the same speed whatever the encoding or file type. Times were measured on one Mac (Apple M4, 10 cores, 32 GB, external USB SSD) on 6–7 October 2026. The 4.70 s to 2.68 s figure is the one published in the v1.8.1.9 release notes; the lookahead and ** figures were measured on development builds (test builds 1.8.1.2 and 1.8.1.3). The ripgrep version is not recorded in these measurements, so I am not quoting one.

The problem: the matching was fine, the output was not

The v1.8.1 test summary had a section called “things that look off” — items that were slower than the previous release, or losing to ripgrep. One of the four was uvf running \d+\.\d{6,}.

That expression finds numbers with six or more digits after the decimal point. The target is 3 GB of OpenStreetMap XML, where almost every line carries a latitude or longitude. So 28.3 million lines match. As queries go, it is an extreme one: nearly the whole file is the answer.

The other three turned out not to be version changes at all. The free edition’s second pass over the kernel looked 10% slower, but measuring three releases alternately gave 10.98, 10.96 and 10.95 s across five queries — and ripgrep had slowed by the same amount (7.46 s to 7.91 s). A 3 GB first pass of 11.87 s re-measured at 3.09 s (ripgrep 3.31 s) and never reproduced. The paid edition’s 13.88 s including index build was 6.1–6.4 s on a warm machine regardless of version.

Only one of the four was real. And it was not in the regex engine.

Fix 1: stop building a string for every matching line

The old code built three strings per matching line on its way out: one to attach the line number, one for the file name, one for the newline. Each of those took a fresh allocation.

For one line, nobody notices. For 28.3 million lines, total allocation reached 28 GB on a machine with 32 GB of physical memory. The search itself had already finished; the time was going into handing the answer over.

The fix is dull. Write straight into one reused buffer and stream it out. No per-line strings.

• 3 GB, \d+\.\d{6,}, -H: 4.70 s to 2.68 s (1.75x faster than before)
• Allocation: 28 GB to 0.4 GB — one seventieth
• Output is byte-for-byte identical to the old code across ten combinations of flags

What matters here is how narrow the win is. Queries with few matches, and multi-file targets like the 60,000 kernel files, ran at exactly the shipped speed. Per-line allocation only piles up when the lines pile up. “Wasting memory” and “being slow” overlap under one condition, not in general.

Fix 2: narrow by the required word inside a positive lookahead

The second fix is in the regex path. An expression like ^(?=.*User: …)(?=.*Action: …)(?=.*Time: …) uses positive lookaheads to mean “lines containing all three”.

The old code applied that expression to every line. The machinery for pulling a required literal out of a regex and narrowing with it had been there for a while, but it never looked inside a lookahead.

The reason it safely can is that matching happens one line at a time and . does not match a newline. Whatever is inside the lookahead must appear in the matching line itself. So applying the expression only to lines containing Action: is enough.

• Linux kernel, 60,000 files: 1.52 s to 0.57 s (2.67x faster than before). ripgrep takes 0.94 s, so this is 1.65x faster
• A single-term query also moved, 0.64 s to 0.56 s (1.14x, level)

Negative lookaheads and lookbehinds are still left alone. You cannot derive a word that must be present from a condition about absence. When the inside of a lookahead cannot be read, the literals from the rest of the expression are used exactly as before. To check that the conservative side holds, a property test — matching lines always contain the extracted literal — now runs over random expressions that mix lookaheads and lookbehinds.

One file with many candidates goes to the window app

Adding that narrowing had a side effect. A 1 GB single log where about 3.7 million lines were candidates no longer counted as an “apply to every line” query, and ran 4.68 s on the lookahead path.

The uvf binary is compiled ahead of time so that it starts fast, which means it has no way to compile a regular expression at run time. The window app does. So a single file with a large number of candidate lines is handed over to the window app.

• 1 GB single file, about 3.7 million candidates: 4.68 s to 2.18 s (2.15x faster)
• The decision comes from sampling the first 8 MB, extrapolating the ratio to the whole file, and handing over above 300,000 lines
• Bundled indexes (the 60,000 kernel files) are never handed over
• Simple expressions (User: (\S+)) are never handed over either. Measured at 0.93 s against 1.02 s — handing over is the slower direction

That last line is why the rule is restricted to lookahead expressions. “Being able to compile is faster” held only while the expression was expensive.

Fix 3: read directory listings ahead when expanding **

The third fix is in file expansion. Writing 'linux/**' walks the tree below and builds a file list. The old code read each directory as the walk reached it.

Now the listings below are fetched as soon as a directory is found. Only the listing is read ahead; the exclusion rules are still applied by the walker, in the same order. The result is unchanged.

• A 60,000-file copy of the kernel: listing construction 0.33 s to 0.24 s (1.38x, which is under 1.5x and so level, with the prefetch on the faster side)

And this fix does nothing on a real source tree. Below any directory holding .ignore, .gitignore or .git, it does not prefetch deeply; if the starting directory or anything above it carries exclusion rules, it does not prefetch at all. That avoids reading directories in the background that are about to be excluded. On a real src/linux with its .git, speed is exactly what it was. The gain showed up only on a copy with .git removed.

Dropped

Handing masking (-mask) to the window app was built, measured and left out: 1.97 s against 2.08 s on a 1 GB file with 2.3 million lines. It was not faster. The same tool as fix 2, pointed at a job where it did not pay.

Skipping the second read during listing construction (the one that counts links) was dropped too — removing it saved only about a tenth, and there is no cheap way to recognise a link.

The numbers, all three together

Units are in the headers. Of the before-and-after ratios, only the write-out row comes from the v1.8.1.9 release notes; the other two were measured on development builds.

Only the top two rows can be set against ripgrep. Writing out is 2.68 s against 2.83 s — level, with uvf on the faster side. The lookahead query is 0.57 s against 0.94 s, making uvf 1.65x faster. The lower three have no ripgrep counterpart at all — handing work to a window app, prefetching for **, and masking are not things ripgrep does — so those compare against the previous uvf instead.

Allocation went from 28 GB to 0.4 GB. ripgrep’s memory use was not measured, so there is no figure to set beside it.

The losses: caught up only, narrow conditions, nothing on a real tree

1. Writing out merely caught up. 2.68 s against 2.83 s is 1.06x, inside level. Erasing a 1.85x deficit is real, but it cannot be written up as a win.

2. The write-out fix only pays off when most lines match. Queries with few matches, and the 60,000 kernel files, run at the shipped speed. Two seconds appear only at something as extreme as 28.3 million matching lines.

3. The ** prefetch does nothing on a real tree with .git. The 0.33 s to 0.24 s figure comes from a copy with .git removed, and it is 1.38x — level. The conditions where this fix helps may only exist in test data.

4. Negative lookaheads and lookbehinds cannot narrow anything. A condition about absence yields no word that must be present, so those expressions run at the old speed.

5. The hand-over decision is an estimate. The ratio from the first 8 MB is extrapolated to the whole file, so a file whose lines change shape halfway through will fool it. A wrong guess only hands over work that did not need handing over (or fails to), so the answer never changes — but the timing stops being predictable.

6. Masking was dropped. 1.97 s to 2.08 s. The time spent building and measuring it does not come back.

Correctness: byte-identical output before any of it shipped

All three fixes are meant to change speed and nothing else, so byte-identical output was the condition for shipping them.

• Writing out: identical to the old code across ten flag combinations. Automated tests: 765 in UwView.Core.Tests (9 new) and 899 in UwView.Pro.Core.Tests, all passing.
• Lookahead narrowing: identical across eight cases covering both the 1 GB single file and the 60,000 kernel files. A property test — matching lines always contain the extracted literal — was added over random expressions mixing lookaheads and lookbehinds.
• ** prefetch: only listings are read ahead; exclusion is decided by the walker in the same order as before. Nothing about the decision sequence changed, so nothing about the result can.
• Hand-over decision: checked that handing over and not handing over produce the same output. The decision not to hand over -mask was made by the same comparison.

What travels: the slow part is not always the part that searches

Three things stayed with me.

Measure the output path too. \d+\.\d{6,} was slow in the writer, not the matcher. When you are speeding up a search tool, the instinct is to look only at how the expression is applied. A per-line allocation is invisible until the lines get numerous.

Allocation totals are a clue about time. Once 28 GB showed up, “three strings per matching line” was a short step away. Seconds alone do not tell you where to look.

Write down the condition along with the win. The write-out fix helps only when matches are dense; the ** prefetch only without .git; the hand-over only for lookahead expressions. None of them is “faster, always.” Remembering “we made it faster” without the condition means that next time it does not show up, you will not know why.

How you write it with uvf (and the same with uvp)

uvf japan.osm '\d+\.\d{6,}' -E -H                # write out matching lines with file names
uvf 'linux/**' '^(?=.*User: )(?=.*Action: )' -E  # stacked positive lookaheads, narrowed from inside
uvf 'linux/**' 'User: (\S+)' -E                  # a simple expression, never handed over
uvf 'linux/**' --files                            # just the expanded file list
uvp japan.osm '\d+\.\d{6,}' -E -H                # the paid edition, same syntax, with the index
uvp 'linux/**' '^(?=.*User: )(?=.*Action: )' -E  # the same expression, index-backed

The syntax is the same in both editions. Exit codes follow grep: 0 for matches, 1 for none, 2 for an error or an incomplete result. The free edition is on GitHub Releases (brew install --cask amru195704/uwview/uwview on macOS, Scoop on Windows, Snap on Linux).

Links

• Full Spectrum Part 1 — bringing Shift-JIS, EUC-JP and UTF-16 into the index as UTF-8: https://uvp.y42u.net/en/blog/uwview-fs01-sjis-utf8-index-en/
• Full Spectrum Part 2 — skipping whole blocks of non-matching lines with the line skeleton: https://uvp.y42u.net/en/blog/uwview-fs02-line-skeleton-en/
• Full Spectrum Part 3 — knowing which blocks not to open: https://uvp.y42u.net/en/blog/uwview-fs03-block-marks-en/
• Full Spectrum Part 4 — where the second query on 10 GB got shorter: https://uvp.y42u.net/en/blog/uwview-fs04-10g-second-search-en/
• The v1.9.0 “Full Spectrum” release post (what the whole version changed): https://uvp.y42u.net/en/blog/uwview-v190-full-spectrum-en/
• Extracting a required literal from a regular expression: https://uvp.y42u.net/en/blog/uwview-fl04-regex-required-literal-en/
• Where ripgrep wins, collected in one place: https://uvp.y42u.net/en/blog/uwview-wf14-where-ripgrep-wins-en/
• Every measurement, in one table: https://uvp.y42u.net/en/benchmarks-en/
• Source code on GitHub: https://github.com/amru195704/UwView

And if you keep searching the same huge logs for the same handful of terms, UwView Pro makes the second query a step faster — a persistent index, search inside the compressed cache, and roughly a ninth of the storage (every OS, one-time purchase or monthly subscription, 14-day free trial; see the pricing page).

From the developer: my apps, Kindle books and public projects are listed at GitHub: amru195704.