My self-made large-file text viewer “UwView,” which had advertised a “maximum of 200 million lines,” recently opened a 51GB, roughly 890-million-line file — an XML conversion of the entire OpenStreetMap Japan dataset — without any trouble. The total line count matched wc -l exactly, without a single line of discrepancy, and jumping to the very last line (line 890 million) took 0.006ms.
That got me thinking. What’s the theoretical upper limit on the number of lines it can open? Curious, I worked through my own code’s data types and structures to calculate the ceiling, step by step.
Repository: amru195704/UwView (PolyForm Internal Use License 1.0.0)
The short answer first: the type limit is about 9.2 quintillion lines
UwView tracks both line numbers and byte offsets as signed 64-bit integers (.NET’s long) throughout. So the ceiling imposed by the data type itself is:
long.MaxValue = 9,223,372,036,854,775,807 (≈ 9.2×10^18 lines = about 9.2 quintillion lines)
That’s already a number with no real-world relevance, but the story doesn’t end there. In practice, several walls appear well before you’d ever reach it.
Wall #1: File size
Since byte offsets are also long, the largest file UwView can handle is roughly 8 EiB (exbibytes). And because the smallest possible line is just a single newline byte (\n), the “upper limit on line count” and the “upper limit on file size” both land in the same 8 EiB neighborhood, essentially coinciding. Of course, in the real world, the file system’s own maximum file size kicks in first (ext4 caps out at 16TB per file; APFS and NTFS allow several EB, and so on).
Wall #2: The index structure
To keep memory usage low, UwView uses a sparse index that records a byte position only once every N lines (the default is N=256). These checkpoints are held in a List<long> — backed internally by an array — but .NET arrays are capped at about 2.14 billion elements (Int32.MaxValue).
That means, at the default setting, the index array hits its ceiling at the following line count:
2.14 billion × 256 ≈ 5.5×10^11 lines (about 550 billion lines)
This can be relaxed by increasing blockLines (N), but the index’s memory footprint becomes the binding constraint before that. Index size can be roughly estimated as “line count ÷ 256 × 8 bytes”; in the real-data measurement article, 890 million lines produced a 26.6MB index. Extrapolating that ratio out to 550 billion lines would require about 17GB of RAM for the index alone — and in practice, this is roughly where the real-world limit sits.
Wall #3: The real wall is “capacity”
To sum up, there are three layers:
| Stage | Limit | Does it actually bind? |
|---|---|---|
| Limit imposed by the data type | About 9.2×10^18 lines (about 9.2 quintillion lines) | Effectively unreachable |
| Effective limit imposed by the structure | About 550 billion lines by default (can be extended by changing settings) | Index RAM becomes the binding constraint first |
| The real-world practical wall | Storage capacity | This is the realistic upper limit |
The single biggest factor in the 890-million-line test was exactly this: “file capacity binds before line count does.” The 51GB file didn’t fully fit in cache, and random reads dropped to millisecond-scale — but nothing broke. That’s the natural consequence of a design that never keeps the whole file resident in memory (mmap), renders only the visible lines, and looks up line positions through a small sparse index.
So what’s the actual real-world limit?
Reaching the type-imposed ceiling of 9.2×10^18 lines would require an 8 EiB-class file, which is far beyond anything a human being could realistically provision as storage. In practice, “the largest file you can actually get your hands on” is the real limit of the viewer.
If anyone out there happens to have an extremely large text file (hundreds of GB to TB scale) sitting around, I’d genuinely love to hear from you if you try opening it in UwView and find out where the limit actually is.
Summary
- UwView’s line numbers and byte offsets are 64-bit integers (
long), giving a type-imposed limit of about 9.2 quintillion lines — but that’s effectively unreachable in practice. - Well before that, the sparse index’s array element limit (about 550 billion lines by default) and index memory usage become the binding constraints.
- Even past the index wall, what ultimately matters is the simple reality of storage capacity.
- The conclusion confirmed in the 890-million-line real-data test — that file capacity binds before line count does — holds up exactly the same way in this theoretical calculation.
Related articles
- I Opened a 51GB, 890-Million-Line Real-World File (OSM Japan) in UwView — “Apology: The 200-Million-Line Claim Was Wrong”
- UwView Goes Open Source on GitHub — Renewing a Windows 95-Era Large-File Text Viewer for 200 Million Lines and All Platforms
Sources
- amru195704/UwView (GitHub) https://github.com/amru195704/UwView
- Int64.MaxValue Property (.NET) https://learn.microsoft.com/dotnet/api/system.int64.maxvalue
- Array class limits (.NET) https://learn.microsoft.com/dotnet/api/system.array
Premium version: UwView Pro (on sale now, Windows/macOS/Linux)
This is the commercial version: “instantly view and search massive files at any time; the index is saved so that from the second open onward it opens instantly with line numbers already shown; searches up to about 9× faster; stores logs at about 1/9 the size and opens them as-is.” One-time purchase $129 / $9 per month.
→ https://uvp.y42u.net/en/pro-en/
(UwView / UwView Pro let you view and search the entire file the moment you open it, even on the very first open. The index is built in the background, and line numbers are displayed once it’s complete. UwView Pro saves the index and compression, so from the second open onward it opens instantly with line numbers already shown. Other viewers show only the head until indexing finishes, whereas UwView lets you view and search the whole file instantly from the first open.)
From the developer: a list of my apps, Kindle books and open-source projects is on GitHub: amru195704.
A note
The information in this article is provided for reference purposes only, and its accuracy or completeness is not guaranteed. If you notice any errors or inaccuracies, please let us know in the comments and we will review and correct them.
コメント