Everything We Measured on Huge Files — Four Boundaries Where the Numbers Flip

The ceiling was 950 MB/s.

That “N seconds” figure — at what file size, and on which attempt?

Change either one and the winner changes, on the same tool and the same file. Laying this series’ measurements side by side, four boundaries come into focus.

This is the last part, so here they are as a way of reading numbers: what “maximum supported size” does not tell you, the design that shows only the head until the index finishes, why read count beats raw speed, and the huge text files that aren’t logs.

Up front: UwView reads a 258.68 GB, 4.5-billion-line single file all the way to the tail while the index is still building (5 min 28 s to finish it — about 789 MB/s). Pro saves the index and the compression, so the second open comes back instantly with line numbers attached (0.02–0.07 s measured on a 47.73 GB text file). But on opening alone we lose to klogg — 52.55 s against 100.6 s at 51 GB. All figures come from one specific setup; results vary. Details at the end

Every number here is from one machine and one file. Disk type, connection, filesystem and page-cache state change them substantially, so please don’t read them as a general performance ranking.

---

1. “Maximum supported size” is not the size you can investigate

Situation

Someone hands you a 48 GB file. Your editor’s spec sheet says “unlimited file size.” Fine, you think, and open it.

It opens. The head appears.

Then you try to jump to the tail, and you can’t. What’s reachable is line 18,951. The file has about 892 million lines.

Why it happens

“Can open” means different things in different products.

A spec sheet’s maximum size almost always means “can begin loading without crashing.” How much of the file you can operate on while it loads is a separate question — and that one isn’t printed.

Measured numbers, from opening a ~48 GB, ~892-million-line OpenStreetMap Japan extract on a Windows laptop (Core i7-1165G7, 16 GB RAM, internal SSD). In an editor built to load the whole file before letting you work, the progress went:

• 1 min 14 s: 29%
• 2 min 35 s: 50%
• 4 min 06 s: 100% ← only now can you move anywhere in the file
• Index build finished at 246 s

Read this as “how an ordinary editor behaves.” Some products ship a dedicated huge-file mode, and we have not measured what happens with that enabled — so we are not naming a product here. The point is not which editor is better; it is which metric you are reading.

So the spec sheet’s “unlimited” and those 4 minutes 6 seconds are two unrelated metrics — and it’s the second one that decides how the investigation feels (Part 1).

What general tools do, and where they stop

If you don’t want to wait, don’t open the whole thing.

# Get a rough idea of size and line count (an exact count needs a full scan)
ls -l osm-japan.osm
wc -l osm-japan.osm    # minutes, at 48 GB

# Look at the tail only
tail -n 200 osm-japan.osm

# Cut out just what you need
split -b 2G osm-japan.osm part-

That gets you to “looking.”

Three limits.

One, split multiplies the original. Slicing 48 GB into 2 GB pieces means 24 files and 48 GB of free space. And the cut points ignore content, so they land inside XML tags and records.

Two, tail shows you the tail but can’t search. If the answer lives outside those 200 lines, you’re back to the whole file.

Three, the spec sheet can’t tell you. Comparing “maximum size” and “maximum lines” columns never produces 4 minutes 6 seconds. Measuring on your own file is the only way right now.

---

2. Only the head is visible until the index finishes

Situation

You open a 50 GB log in a log viewer. A progress bar appears. The scrollbar thumb slowly shrinks.

What you want is the tail. That’s where the incident is.

Until it finishes, what’s at the tail is unknown.

Why it happens

Jumping to line N requires a table of where line N starts — an index.

Text files have no record boundaries. Until newlines are counted, the position of line 100,000,000 isn’t known. That leaves three designs.

• Build it all, then show anything — line numbers are exact; you wait
• Don’t build it, read sequentially — instant, but no line numbers and no sense of position (less is close to this)
• Build it in the background while showing the whole file — no waiting, but harder to implement

Most viewers pick the first, which is why “only the head until the index finishes” is so common.

The important part: most of that wait isn’t laziness. Opening a 51.25 GB file in klogg takes 52.55 s. As a read rate that’s 930 MB/s — and a raw read of the same disk runs at about 950 MB/s. klogg is using essentially all of it.

There’s no speed left to reclaim. The only thing left to change is whether you wait — that is, the design.

What general tools do, and where they stop

To see the tail first:

# Read backwards from the end
tac huge.log | head -n 500

# less reads the whole file to reach the end (G makes you wait)
less +G huge.log

# Grab the last bytes without counting lines
tail -c 5000000 huge.log | less

You get the tail.

Three limits.

One, line numbers no longer match the original. The first line tail -c gives you starts mid-line, so counting it as line 1 shifts everything. That can’t go in a report (Part 24).

Two, “near the tail” and “30 minutes before the tail” are different problems. You want to walk back by time, and all you have is a byte offset (Part 7).

Three, neither colour rules nor saved queries exist there. You can reach the tail quickly, but you can’t bring a way of looking with you (Part 22).

---

3. Read count beats raw speed

Situation

You’re chasing 5xx in an nginx access log. The routine is the usual one.

1. grep the 5xx to get your bearings
2. Open the same file in a viewer
3. Search in the viewer to reach that timestamp
4. Read the surrounding lines

Each step takes tens of seconds. And yet an hour has gone by.

Why it happens

You’re reading the same file from the top, over and over.

At 50 GB and 950 MB/s, one full scan has a floor of roughly 54 seconds. Step 1 reads once, step 2 reads again, step 3 reads a third time: about 2 minutes 40 seconds right there. No amount of polishing the commands reduces the count.

The measurements show the same shape. Time to “find it and read it on screen,” at 51 GB:

	51 GB
klogg (open 52.55 s + search 55.59 s)	108.14 s

klogg reads the file once to open it and once more to search it. Each pass is running near 930 MB/s, so nothing here is slow. Two reads simply cost twice.

This is what a single-command benchmark hides. Shaving 10% off grep matters less than going from two reads to one (Part 15).

What general tools do, and where they stop

The standard move is to shrink first, then hand it on.

# Pull just the 5xx, carrying the original line numbers
grep -n -E ' (5[0-9]{2}) ' access.log > 5xx.txt

# Look at counts and time distribution (one read of the original so far)
awk '{print $4}' 5xx.txt | cut -c2-15 | uniq -c | sort -rn | head

# Read around a line of interest, in the original's coordinates
awk 'NR>=41284440 && NR<=41284470 { printf "%d\t%s\n", NR, $0 }' access.log

It works. Carrying line numbers with grep -n lets everything downstream point back at the original.

Three limits.

One, that last awk reads the whole file again. Knowing the line number doesn’t give you a way to jump to it with standard tools. Another 54 seconds every time you want to see a bit more.

Two, intermediate files accumulate. 5xx.txt is not the original. Pass it around and nobody can tell which file and which range it came from (Part 24).

Three, pipes don’t reduce the count. grep | awk | sort is one read, but a new question starts again from the top. A full scan per question is the structure of this route.

---

4. Huge text isn’t only logs

Situation

You need to look at map data. You take OpenStreetMap’s United States extract (PBF, ~11 GB) and expand it to XML with osmium cat. About 15 minutes later:

us-260726.osm
258,679,440,228 bytes  =  258.68 GB (240.9 GiB)

One XML file, not split. Line count: 4,509,830,821 (4.5 billion). It isn’t a log, so nothing rotates it.

Why it happens

Interchange formats aren’t designed for a human to peek into the middle.

Logs rotate and get cut by date. Map data, SQL dumps, NDJSON and simulation output mean something precisely because they’re one piece, so they don’t get cut. Expanded, they run to hundreds of gigabytes.

• XML tends toward one element per line — 4.5 billion lines is the flip side of lines being short
• Purpose-built tools assume processing — osmium, xmlstarlet and jq are strong at converting and extracting, and weak at just looking first (Part 9)
• split breaks the structure — cut by byte count and you cut inside a tag

Measurements at this size, on a Mac: a full scan of 258.68 GB to finish the index took 5 min 28 s (328 s) — about 789 MB/s. A full-text search for “New York” after the index completed took 34.8 s, 100,492 hits.

What general tools do, and where they stop

To look at the raw thing, stream it.

# Check the structure from the head
head -c 200000 us-260726.osm | xmlstarlet fo 2>/dev/null | head -60

# Count one kind of tag (one full scan)
grep -c '<way ' us-260726.osm

# Get your bearings without converting anything
grep -n -m 5 '"New York"' us-260726.osm

You can make judgements.

Three limits.

One, every check costs a full scan. Counting lines alone takes minutes; a new question takes minutes again.

Two, the look before jq or osmium gets skipped. Guess the schema wrong and you find out half an hour into the job.

Three, you can’t go back to the PBF. The original 11 GB is compressed binary, so reading it as text requires expansion, and expansion means 258 GB. “Read it while it stays small” isn’t on the menu on this route (Part 8).

---

What the four had in common

Measurement	The number	What it says	What flips it
Opening 48 GB	4 min 06 s to full access; index 246 s	“Supported size” guarantees nothing about start time	File size (the gap disappears when small)
Opening 51 GB	52.55 s = 930 MB/s (raw read ~950 MB/s)	Most of the wait is physics, not speed	Design (wait, or build in the background)
Find and read	108.14 s (open + search = two reads)	Read count matters more than seconds	Number of questions (one question, small gap)
Indexing 258.68 GB	328 s = ~789 MB/s; 4.5 billion lines	A structure that costs a full scan per check	Whether there’s a second time (does the index persist)

All four have the same equation underneath.

One full scan = file size ÷ disk read rate.

At 50 GB and 950 MB/s, about 54 seconds. No tool removes those 54 seconds. Which is why only numbers carrying these three things mean anything in a comparison:

• Size — a 3 GB result inverts at 50 GB. Fitting in memory or not is a different sport
• Count — for a single question, the side that builds an index loses. It only pays when you ask the same file twice or more
• Disk — internal SSD, external USB SSD and a network share don’t even have the same bottleneck (Part 17)

Turn that around and there are only two places left to win: not making you wait (show the whole file before the scan completes) and making the second time free (keep the index).

Back to “the ceiling was 950 MB/s.” That wasn’t a concession — it was the premise you design against. Twenty-five parts, and that’s the single point behind all of them.

---

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls and searches huge text files from the moment you open them. It never loads the whole file into memory, so files larger than RAM open fine. The index is built in the background, and line numbers appear once it’s done.

• The whole file is visible before the index finishes: straight at chapters 1 and 2. Jump to the tail immediately after opening, check the 50% mark, start searching. On the 258.68 GB, 4.5-billion-line XML, the contents were readable with the index 10% built (the 258 GB measurements)
• The same 48 GB, compared: index built in 74 s (EmEditor Professional 26.2.5 on the same machine: 246 s); full-text search for “東京” in 12.5 s (against 160 s). Windows 11, Core i7-1165G7, 16 GB RAM, internal SSD, x64 builds on both sides. EmEditor was run without its Large File option — we have not measured what happens with it enabled (the 48 GB comparison)
• Colour rules and saved search results: chapter 3’s 5xx work lives here. Follow the clusters by eye with a status-code preset, then write the hit list to a separate file with the original line numbers (tracing 5xx)
• Everything runs on your own machine. The file you’re investigating is never sent to an outside service

Beyond that is UwView Pro territory.

• The index and compression are saved as .uwvz: chapter 4’s “a full scan per check” goes away. A file you’ve opened once reopens with line numbers still attached in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup)
• Storage drops to about 1/9: for the 258.68 GB file, the .uwvz sidecar was 28.61 GB. It stays searchable while compressed, so nothing has to be expanded to be read (Part 8)
• Search up to ~9× faster from the second time on (via the compressed cache; measured, conditions apply). This is where chapter 3’s “a full scan per question” stops applying
• Drill-down search keeps the original line numbers however far you narrow (drill-down search)

Honestly: on opening alone, we lose.

• Just opening a 51.25 GB file was faster in klogg on v1.6.5 — 52.55 s against 100.6 s. v1.6.6 turned that over: 50.44 s (and 2.99 s at 3 GB, 10.13 s at 10 GB — faster than klogg at every size), reading at 967 / 966 / 969 MB/s, the speed of the medium itself. The whole run also improved, because the search now starts without waiting for the index: 53.7 s, about 2× klogg’s 108.14 s and close to the uvf … -open command (50.76 s on v1.6.6, about 3 s apart) (the v1.6.6 measurement)
• A complete first read and a complete index take physical time. 328 s for 258.68 GB isn’t fast; it’s just readable while it scans
• The first search costs one pass over the raw file. From v1.6.6 it runs while the file is opening, though, so you are not waiting for a separate scan once it is open — 53.7 s end to end at 50 GB. Instant results still start from the second search (via the compressed cache)
• It doesn’t parse structure. Formatting or querying XML and extracting from JSON are jobs for xmlstarlet and jq, and this doesn’t replace them
• It handles text. Compressed binaries like PBF are out of scope, so expansion comes first
• Every figure here is from one specific setup. None of it guarantees the same result for you

The same things, from the command line

v1.6.0 added a uvp command (uvf in the free version). It uses the same .uwvz as the GUI, so an index built on the CLI still counts in the GUI. Mapped onto this article’s four chapters:

# Ch.1: before you open and wait, just confirm the term is in there
uvp osm-japan.osm '東京'

# Ch.2: read a tail-side anomaly in context, with line numbers
uvp huge.log 'FATAL' -C 5

# Ch.3: get the 5xx distribution in one read (the index pays from question two)
uvp access.log -uniq ' (5[0-9]{2}) ' -head 20

# Ch.4: write out only the relevant spot from a 258 GB-class file
uvp us-260726.osm '"New York"' -C 2 -out survey/ny.txt.gz

# Line-number jumps, colour rules and encoding switching are GUI features. Hand it over here
uvp access.log ' 503 ' -open

Exit codes follow grep — 0 = found, 1 = not found — plus 2 = stopped at a limit (unlimited by default; only when you set -limit N). if uvp access.log ' 503 '; then works as written.

Honestly: on the first question uvp is a little slower than ripgrep — it builds its .uwvz (a compressed cache plus index) first, 59.0 s against 54.9–55.8 s at 50 GB, 6–7%. In v1.6.6.1 it searches while building the index instead of re-reading it afterwards, which cut 6.7 s. It starts paying from the second question: the same 50 GB search takes 6.6 s against ripgrep’s 55–57 s — 8.4×. On a small file such as 3 GB the two are level, because an index has little to earn there. The free uvf is level with ripgrep at every size (measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

This is part 25 and the last one. Thank you for reading. Next time you see a huge-file comparison, just check whether size, count and disk are all stated. That alone tells you whether the number applies to your situation.

• And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

• Part 1 — Four standard tools that sink under a huge file: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
• Part 7 — Four ways to read timestamps as a weapon: https://uvp.y42u.net/en/blog/uwview-ps07-timestamp-driven-triage-en/
• Part 8 — Compressed storage and searchability at once: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
• Part 9 — Four times you read structured data raw: https://uvp.y42u.net/en/blog/uwview-ps09-read-raw-structured-data-en/
• Part 15 — Four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
• Part 17 — Four frictions between dev environments and logs: https://uvp.y42u.net/en/blog/uwview-ps17-dev-env-log-friction-en/
• Part 22 — Four habits of a team that can hand it over: https://uvp.y42u.net/en/blog/uwview-ps22-shareable-log-investigation-en/
• Part 24 — Four habits for leaving the original untouched: https://uvp.y42u.net/en/blog/uwview-ps24-never-modify-the-original-en/
• 48 GB, 892 million lines, measured against EmEditor: https://uvp.y42u.net/en/blog/uwview-emeditor-48gb-comparison-en/
• Opening 258.68 GB and 4.5 billion lines: https://uvp.y42u.net/en/blog/uwview-osm-usa-258gb-en/
• Tracing 5xx errors in access logs as one thread: https://uvp.y42u.net/en/blog/uwview-access-log-5xx-workflow-en/
• Benchmarks at three sizes: https://uvp.y42u.net/en/blog/uwview-pro-benchmark-3sizes-en/
• Drill-down search: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
• We shipped a uvp command (measured against ripgrep): https://uvp.y42u.net/en/blog/uvp-cli-release-vs-ripgrep-en/
• Source code (GitHub): https://github.com/amru195704/UwView

---

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

---

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Every figure described as measured comes from one specific setup and one specific file, and none of it guarantees the same result elsewhere; disk type, connection, filesystem, page-cache state, concurrent processes, and each product’s version and settings change outcomes substantially. Figures for other products were measured on my own machine and do not generally represent those products’ performance. Each of them has many features and strengths this article does not cover. Command examples may need adjusting for your environment (GNU vs BSD, differences among awk, grep, split, xmlstarlet and osmium, and your shell). Always confirm option names and defaults against your local man. OpenStreetMap data is provided under the ODbL; follow its licence terms when you use it. How logs may be handled or moved is governed by your organisation’s policy, your client’s, and applicable law. If you spot an error, please leave a comment and I’ll check and correct it.