Our Index Now Holds Shift-JIS, EUC-JP and UTF-16 as UTF-8 — Full Spectrum Part 1: 5.4x to 38.8x Over ripgrep -E From the Second Query, 1.2x to 1.5x Behind on the First

The conversion ran every single time.

Searching a 3 GB Shift-JIS file meant decoding 3 GB before a single hit could be counted. Then again on the second query. Then again on the third.

We had an index. So why was the decode still there? The answer was one line of design: what encoding the index itself is stored in.

Up front: 5.4x to 38.8x from the second query, but ripgrep is still 1.19x to 1.51x faster on the pass that builds the index

As of v1.9.0, a .uwvz (UwView Pro’s compression format plus index) always stores its text as UTF-8, even when the source file is Shift-JIS, EUC-JP or UTF-16. The conversion happens once, while the index is built; from the second query on, searching a Shift-JIS file costs the same as searching UTF-8. Over cold plus hot on a Mac, Shift-JIS at 2,887 MB came out 5.44x to 29.11x faster than rg -E shift_jis, EUC-JP at 2,887 MB 5.38x to 23.26x, and UTF-16LE at 5,765 MB 8.06x to 38.79x. Across seven query patterns and three encodings — 21 queries — the output matched ripgrep byte for byte, with identical exit codes.

The losses first. The pass that builds the index is slower than ripgrep: 4.93 s vs 3.35 s on Shift-JIS (ripgrep 1.47x faster, which is under 1.5x, so I call that level), 5.01 s vs 3.32 s on EUC-JP (ripgrep 1.51x faster), 7.76 s vs 6.53 s on UTF-16LE (level). On top of that, -extract — writing the file back in its original encoding — regressed to 1/2.29 of the previous build partway through development and had to be rebuilt as a parallel pass. And EUC-JP supplementary kanji (JIS X 0212) still cannot be read.

uvf and uvp are the search commands of UwView, which I make (uvf is the free build, uvp is the paid UwView Pro). The numbers here come from v1.9.0 “Full Spectrum” (released 14 October 2026, build 26.10.13.22), measured on a Mac (Apple M4, 10 cores, 32 GB, external USB SSD) on 14 October 2026. v1.9 carries the pet name “Full Spectrum” because Shift-JIS, EUC-JP and UTF-16 files are now stored as UTF-8 inside the index, so encoding and file type no longer change how fast a search runs.

The problem: an index that still decoded 3 GB on every query

uvp builds a .uwvz on the first query and searches inside it afterwards. In the previous line of builds (v1.8, “Finder Scope”), that .uwvz kept the source encoding. A Shift-JIS file produced a Shift-JIS index.

Which means every query had to do one of two things: encode the pattern into Shift-JIS, or decode the bytes it read into UTF-8. Either way, work proportional to the data landed before any match could be decided — on the second query exactly as much as on the first. The index had removed the second read of the file; it had not removed the second decode.

Measured on the previous build (1.8.1.9), seven query patterns over a 2,887 MB Shift-JIS file took 2.53 s to 4.41 s each across two runs. The same content in UTF-8 finished in about 0.3 s. That gap is the decode.

There was a worse symptom hiding behind it. Searching a 6 GB UTF-16 file in its original encoding failed with Specified argument was out of the range (exit code 2). Files up to about 1,000 MB were fine, which is exactly why nobody noticed; 1.8.1.9 and 1.8.3.14 both had it. The carry-over bytes of a character split across read boundaries could exceed the read window.

The cause: the original encoding was there to protect byte offsets

Storing UTF-8 from the start would have been the obvious choice. The reason it wasn’t is not speed — it is offsets.

The index records which byte of the source file each line starts at. That is what lets the GUI open the original file and accept an index a CLI run built for it, and what lets -extract reproduce the original file. Change the encoding and every offset moves: a two-byte Shift-JIS character becomes three bytes in UTF-8, so the same line no longer starts at the same byte.

Inject an index with shifted offsets into a view of the original file and the lines are wrong. So “don’t convert” was the default. Changing it meant closing every path where the two kinds of offset could meet.

The fix: version 4 of the format is always UTF-8, and always opened on its own

Four changes.

1. A .uwvz is always UTF-8 (format version 4). Version 3 and earlier are not read at all; if the file won’t open, it is rebuilt. An option to keep the original encoding (--keep-encoding) was written and then removed. Making it selectable meant two kinds of index, each with different speed and different restore guarantees, and a set of paths where the GUI and the CLI could disagree. One path is easier to explain and far easier to test.

2. A converted .uwvz is always opened on its own. The paths that would hand it to a view of the original file — injecting line offsets, searching the original file through the index, resuming a partial index, appending to a bundle — all verify length and timestamp first, and the header timestamp is written as 0, so verification fails by construction. It behaves like opening a gz file.

3. Decode line by line. This was the subtle one. The default CP932 decoder in .NET turns 0x82 0x0A — the lead byte of a two-byte character followed by a newline — into a single character, swallowing the newline. Decode the file as one stream and you end up with fewer lines than the file has, and every line number after that point is wrong. When the newline count changes, the decoder re-reads that region line by line so the numbering matches the source.

4. Broken bytes become the middle dot. The specification said U+FFFD. But the viewer and the search already show “・” for undecodable bytes, because that is what CP932 does by default. Storing the same character in the index means search output does not change by a single byte from the previous build. Here “same as before” beat “more correct”.

One promise was deliberately weakened. -extract no longer guarantees identical bytes — it guarantees identical characters. Characters with two code points (the NEC-selected IBM extensions, such as ED40) come back as the standard code Windows uses (FA5C). Dropping the verification pass that compared bytes on the way in took index construction from 9.67 s to 5.33 s (measured on a development build; the build that kept the original encoding took 5.01 s, so this is level with it). On real work data — 123 Shift-JIS CSV and text files, 213 MB, from a CAD conversion job — all 123 restored byte-identically.

The numbers

The data is OpenStreetMap XML (2,892 MB, UTF-8) converted with iconv to Shift-JIS, EUC-JP and UTF-16LE. The comparison is ripgrep 15.2.0 with -E <encoding>, so it decodes the same text correctly. Cold means the cache was purged and ten seconds elapsed; hot is the run immediately after.

The pass that builds the index (delete the .uwvz, run one query)

Nothing surprising: building an index costs time. For a single query on a file you will never search again, rg -E is the better tool.

From the second query on (Shift-JIS 2,887 MB, cold plus hot)

A7 is an outlier for a different reason, and it is not the encoding. v1.9.0 also skips whole blocks of lines that cannot contain a hit, which is exactly what a query like A7 benefits from. The encoding work explains A1 through A6 — 5.44x to 9.25x. The block-skipping mechanism gets its own article next in this series.

Across all three encodings, two runs combined: Shift-JIS 5.44x to 29.11x, EUC-JP 5.38x to 23.26x, UTF-16LE 8.06x to 38.79x. Excluding A7, the range is 5.38x to 16.50x. UTF-16 looks best simply because the source is twice the size (5,765 MB) and ripgrep reads all of it every time, while the index is 262.3 MB.

On a hot cache, the encoding disappears

This is the result I actually care about. Once the index holds UTF-8, whether the file started as Shift-JIS or UTF-8 no longer shows up in the time. The conversion is confined to the build.

What still loses

-extract got slower before it got faster. Converting UTF-8 back to the source encoding in a single stream took 7.88 s for the 2,887 MB Shift-JIS file, against 3.44 s on the previous build, which never converted at all — the old build 2.29x faster. The fix was to decompress and convert in 8 MB chunks in parallel and write them in order, with chunk boundaries aligned to UTF-8 character boundaries (each chunk reads three extra bytes ahead; the continuation bytes at the head belong to the previous chunk). On a hot cache that came down to 1.03 s. In the v1.9.0 run, -extract took 3.52 s for Shift-JIS, 3.51 s for EUC-JP and 6.71 s for UTF-16LE, including writing 3 GB and 6 GB out cold, and every restore matched the source byte for byte.

EUC-JP supplementary kanji (JIS X 0212) cannot be read — they come back as “・”, and the test data is generated with them removed. The help text says so.

A restore is not guaranteed to be byte-identical. Characters with two code points come back with the standard one, and undecodable bytes come back as the middle dot. If you verify archives by hash, this matters; -info reports what was converted.

The general point: an index can absorb the decode, not just the read

When you put an index in front of a scan, the first thing you remove is the second read. That part is easy to see.

What is easy to miss is the per-query transformation sitting behind the read. Character decoding, newline counting, normalisation, case folding — none of these are “reading”, they are “turning bytes into characters”, and most of them can be done once, at index time, with the result stored in the index.

The price is that your offsets no longer match the source file. So the work is not the conversion itself; it is making sure a converted index can never meet a code path that expects source offsets — which usually means collapsing to one path and sealing the others. Dropping the option to keep the original encoding was not a performance decision. It was a decision about how many paths I was willing to test.

The same thing on the command line

uvp japan.sjis 東京                   # first query: build the UTF-8 .uwvz, then search
uvp japan.sjis 東京                   # second query on: search the index, no decoding
uvp japan.sjis -i HIGHWAY             # case-insensitive
uvp japan.sjis 'k="highway[^"]*"' -E  # regular expression
uvp japan.sjis '="' -v                # non-matching lines
uvp japan.sjis -info                  # what the index holds, and the source encoding
uvp japan.sjis -extract back.txt      # write it back in the original encoding
uvf japan.sjis 東京                   # the free build keeps no index and reads each time

Exit codes follow grep: 0 for hits, 1 for no hits, 2 for an error or an incomplete result. The free build is on GitHub Releases (brew install --cask amru195704/uwview/uwview on macOS, Scoop on Windows, Snap on Linux).

Links

• The v1.9.0 “Full Spectrum” release post: https://uvp.y42u.net/en/blog/uwview-v190-full-spectrum-en/
• Where the previous series landed (many files, seven formats, zip, pbf): https://uvp.y42u.net/en/blog/uwview-wf15-wide-field-recap-en/
• How the index got down to reading the file once (First Light recap): https://uvp.y42u.net/en/blog/uwview-fl13-first-light-recap-en/
• Counting newlines was 40% of a cached query: https://uvp.y42u.net/en/blog/uwview-wf12-newline-hot-en/
• All measured data (benchmark list): https://uvp.y42u.net/en/benchmarks-en/
• Source code (GitHub): https://github.com/amru195704/UwView

And if you keep old Shift-JIS logs or huge text files around and want the second query to be fast, try UwView Pro: a persistent index, search over the compressed cache, and roughly 1/9 of the original size (all platforms; one-time purchase or monthly plan, with a 14-day trial).

From the developer: a full list of my apps, Kindle books and public projects is on GitHub: amru195704.