Eight Minutes for zgrep, Every Single Time — Keeping Logs Compressed and Still Searchable

zgrep hasn’t returned in eight minutes.

And this is the third time today. Same archive, different search term. Eight minutes on the first run is the price of admission. Why is the third run also eight minutes?

Compression solves the capacity problem cleanly. The day after it does, a different problem begins: everything you made smaller became slower to read. We tend to accept that as the cost of archiving and stop thinking about it.

What follows are four situations you meet after the logs are compressed: the cost of repeated zgrep, logs parked in cloud storage, pulling one file out of a backup, and detecting corruption in storage. Each has workarounds. And all four run into the same single assumption.

Up front: UwView Pro keeps a log at roughly 1/9 its size and opens it without unpacking — and from the second open onward it reopens with line numbers in about 0.02–0.07 s (measured on a 47.73 GB text file; ratio and speed vary with content and environment — details at the end)

1. The third zgrep costs exactly as much as the first

Situation

Six months of history sits in app-2026-03.log.gz. An incident recurs, and you need a specific request ID out of that archive.

zgrep 'req-8f21ac' app-2026-03.log.gz

Eight minutes later, three hits. You want context, so you retype it with -C 20. Eight more minutes. You want to try a different ID. Eight more.

Why it happens

The decompressed data is thrown away every single time.

Part 4 covered the structural point that compression takes away the ability to seek. The problem here is one step past that: what it costs to read the same archive repeatedly.

zgrep is a wrapper that pipes gzip -dc into grep. Every search triggers a full decompression; the bytes flow through the pipe, grep evaluates them, and then they’re discarded. The next search redoes the identical work from zero.

The bottleneck also swaps places. Grepping raw text is I/O-bound. Grepping a compressed file reads far fewer bytes but becomes CPU-bound on decompression — gzip decompresses at a few hundred MB/s, while a well-tuned grep runs at GB/s. Faster disks don’t shorten this wait.

And investigation is repetition by nature. You change the term, widen the context, check from another angle. Decompression cost accumulates linearly with every one of those passes.

Working with general-purpose tools, and where it stops

Parallelism, and moving to a format that decompresses faster:

# Search pre-split archives in parallel
ls app-2026-03.*.gz | xargs -P 8 -I{} sh -c 'zcat {} | grep -H "req-8f21ac" || true'

# zstd decompresses much faster than gzip at comparable ratios
zstd -dc app-2026-03.log.zst | grep 'req-8f21ac'

# Decompress once, then work on the plain file
zcat app-2026-03.log.gz > /tmp/work.log     # eight minutes; every search after is fast
grep -C 20 'req-8f21ac' /tmp/work.log

Moving to zstd is a straightforward win. So is the third approach — if you’re going to search three times, one decompression plus three fast greps beats three decompressions.

Three limits. First, you need somewhere to put the expansion. If you stored at 1/9, unpacking wants nine times the space. You compressed because disk was tight, and reading demands the original size back — the same wall as in Part 4.

Second, /tmp/work.log is not the original. Line numbers you cite in an incident report are line numbers in a derived file. Re-expanding should reproduce them, but nothing guarantees that except your own care.

Third, the temp file you meant to delete doesn’t get deleted. “Still investigating” turns into several days, and you find out when the disk-usage alert fires. The temporary expansion eats the benefit of compressed storage.

2. Checking an evidence log in the cloud without pulling it back

Situation

Retention-mandated logs live in object storage. They’re large, so they’re in a cheap tier — infrequent access, or an archive class.

A question arrives: “Do you have the log for this ID on that day?” You only need to know whether it exists. But an object in the archive tier has to be restored first: submit a request, wait hours, then read.

Why it happens

Cloud storage bills “putting” and “reading” separately, and cheapness is inversely proportional to retrievability.

Lower storage classes cost less to hold but add latency and retrieval charges when you want the data back. The moment you choose “store it cheaply,” you have given up “check it casually.” That’s the design, not an accident.

Compression helps on the storage side. Text logs shrink well, so compressing before upload cuts the stored volume substantially — in one measured case a 48 GB log became 5.3 GB, about 1/9 (varies with content and settings). Less stored volume means proportionally less storage billing, so compression is an unambiguous win against the storage line item.

On the retrieval side it works against you. Checking the contents means pulling the whole object and expanding it: a request to confirm one line turns into a full-object retrieval. Range requests exist at the storage layer, but fetching bytes from the middle of a compressed file gives you nothing you can decompress.

Working with general-purpose tools, and where it stops

# Plain text: fetch a byte range
aws s3api get-object --bucket logs --key 2026-03/app.log --range bytes=0-1048576 head.log

# Compressed: partial expansion works from the head
aws s3 cp s3://logs/2026-03/app.log.gz - | gzip -dc 2>/dev/null | head -100

# Upload in chunks so you can retrieve only the chunk you need
split -b 1G app.log chunk_ && gzip chunk_* && aws s3 sync . s3://logs/2026-03/

The third one — chunk, then compress — is the pragmatic compromise most teams land on.

Three limits. First, chunking breaks the original. Which line of the source is line 1,203 of chunk_ac.gz? The unit of “one log file” is gone, and a human is now doing the arithmetic.

Second, you can’t tell which chunk holds it until you fetch it. Time-ordered data gives you a decent guess; searching by ID does not. You end up retrieving every chunk to find the one you wanted — which is exactly what the chunking was supposed to avoid.

Third, the moment it lands locally, it stops being a log you parked in the cloud. Managing, deleting, and access-controlling that copy is now your problem, on your machine. Evidence you deposited somewhere safe gets duplicated and scattered every time someone checks it.

3. Restoring one file from a backup just to look inside

Situation

“Do we still have last November’s batch logs?” You do — inside a monthly tar archive of a few hundred gigabytes.

You want one file out of it, and honestly you want to read a handful of lines. You start a restore job and it asks you to confirm free space at the restore target.

Why it happens

Backups are optimized for restoring everything, not for peeking at one thing.

tar is short for tape archive, and it assumes sequential access accordingly. You can name a single file, but it reads the archive in order until it reaches that position — and if the archive is a .tar.gz, everything ahead of your file must be decompressed on the way. A file near the end of a few hundred gigabytes is, in practice, a full read.

Incremental backups add another layer: restoring one day means applying the last full backup plus the increments in order. Reading one file requires reconstruction across multiple archives.

And checking is never one round. You restore, open it, and it’s not the file you meant. You want the neighbouring date. Another restore job. Each round trip costs tens of minutes to hours.

Working with general-purpose tools, and where it stops

tar -tzf backup-2025-11.tar.gz | grep batch      # list contents (full decompression runs)
tar -xzf backup-2025-11.tar.gz path/to/batch.log # extract one file (expands everything before it)

# Uncompressed tar lets you build an index and seek directly
tar -tvf backup.tar > backup.index
dd if=backup.tar bs=512 skip=$OFFSET count=$BLOCKS of=batch.log

The last approach genuinely works — uncompressed tar has computable block boundaries, so a known offset is a direct read.

Three limits. First, it only works uncompressed. If you compressed the backup for capacity, this option isn’t available. “Small and unreadable” versus “large and readable” shows up here too.

Second, what you extract is still one enormous file. If batch.log is 40 GB, you’re back to the problem of opening it. Restoring was only the entrance.

Third, the granularity is wrong. The smallest unit a backup system offers is a file. The smallest unit you actually want to inspect is a line.

4. Knowing that nothing rotted while it sat there

Situation

You pull a three-year-old log off the NAS. gzip -dc stops partway with an error. The file is corrupt.

You can’t tell when it happened — bad on write, a flipped bit in storage, a truncated copy. The worst part is that nobody noticed until today.

Why it happens

Nobody reads archived logs.

Most retained logs are never opened again. Never opened means corruption is never detected. Silent data corruption is a low-probability event, but across years and tens of terabytes it isn’t a probability you can wave away.

Compression amplifies the damage. One flipped byte in plain text ruins one character; a compressed stream decodes with reference to what came before, so one damaged spot makes everything after it unreadable. Compressing for capacity makes each failure more expensive.

There’s also this: corruption and tampering are technically indistinguishable. What an audit or an incident response needs isn’t “it isn’t broken” but “it is identical to what was stored.” That can’t be constructed after the fact. If you didn’t record it at storage time, there’s no way to prove it later.

Working with general-purpose tools, and where it stops

sha256sum app-2026-03.log.gz > app-2026-03.log.gz.sha256   # record at storage time
sha256sum -c app-2026-03.log.gz.sha256                     # verify periodically
find /archive -name '*.sha256' -exec sha256sum -c {} +      # verify in bulk

gzip -t app-2026-03.log.gz          # gzip's built-in CRC32 integrity check
par2 create -r5 app-2026-03.log.gz  # 5% redundancy: recover from minor damage

With par2 you get repair as well as detection. Checksumming filesystems like ZFS or Btrfs are another route.

Three limits. First, verification is itself a full read. Verifying tens of terabytes monthly costs someone that I/O and that time — a recurring read of your entire archive purely to confirm nothing changed.

Second, the practice doesn’t survive. A sidecar .sha256 has to travel with the file through every move and copy. Anything managed as a separate file eventually gets separated.

Third, verifying and reading are different activities. As long as “verification day” and “reading day” are distinct, the situation where you learn about corruption on the day you finally open the file three years later hasn’t improved. What you want is verification as a side effect of opening, and that requires the archive format itself to carry a checksum.

What the four had in common

Read columns two and three side by side and the trade becomes explicit. We are selling accessibility to buy capacity. And nobody signs off on it — the deal closes the moment you type gzip.

Collect the stopping points and three requirements fall out.

• Read and search while compressed, without expanding. The intermediate expansion step is what creates the repeated cost, the free-space requirement, and the derived temp file. Remove the step and all three go with it.
• Seek by position while compressed. “Just show me this one line” needs random access in the compressed format itself. Chunking is a hand-rolled approximation of that.
• Fold integrity checking into the act of opening. If the archive carries its own checksum and verifies on open, you no longer need to schedule a verification day.

All three are satisfiable at once if the compressed format is designed for reading rather than for storing. Which also explains why gzip and tar don’t satisfy them: they were built to ship and to shelve, not to read.

Back to the third eight-minute zgrep. zgrep isn’t slow. Throwing away eight minutes of work, every time, is slow. Keep the work and the second run is a different conversation.

The tool I use

UwView (free), which I develop, is a viewer that displays, scrolls, and searches a huge text file from the moment it opens. The index is built in the background, and line numbers appear when it completes (most other viewers show only the head until indexing finishes). Nothing is split, so the original stays a single, unmodified file. Opening the 40 GB batch.log you restored in section 3, directly, is this part of the story.

What addresses all four situations above is the compressed cache in UwView Pro. The index and compression are saved as a .uwvz, so a log can be stored at roughly 1/9 its size and opened without unpacking (48 GB → 5.3 GB in one measurement; the ratio depends on content). Section 1’s “throw away the expansion every time” stops being possible.

• Instant with line numbers from the second open: on a 47.73 GB text file, reopening measured 0.02–0.07 s (varies by environment). Opening the same archive three times in a day is no longer a waiting problem.
• Search through the compressed cache: under the same measurement, string search ran about 9× faster (second open onward, via the compressed cache) — fewer bytes read, and no repeated expansion.
• Checksummed: the archive carries its own checksum, so corruption in storage is detectable. Section 4’s stray sidecar hash problem doesn’t arise.
• Drill-down search: narrow a result set by a second term while still compressed, with term (count) on each tab so per-stage counts survive. → the drill-down article

To be straight about the boundary: UwView does not open .gz or .tar.gz directly. It helps when you expand the original once, open it in UwView, and keep and read it as a .uwvz from then on. That is a proposal to replace your gzip archives, not to read gzip archives faster. If you need to keep the existing archives as they are, the move to zstd from section 1 is the more sensible answer. It also does nothing about cloud storage classes or retrieval fees (section 2) — only about making the uploaded file smaller. Note that the free UwView covers single-stage search with ±1 context and result export; drill-down, sequence search, the variable ±N, and the saved index and compression are Pro features.

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

• Part 1: Four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
• Part 2: Four ways to trace causality in logs: https://uvp.y42u.net/en/blog/uwview-ps02-log-causality-tracing-en/
• Part 3: Four character-encoding traps: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
• Part 4: Deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
• Part 5: Four ways to live with development logs: https://uvp.y42u.net/en/blog/uwview-ps05-debug-log-practices-en/
• Part 6: The first hours of an intrusion investigation: https://uvp.y42u.net/en/blog/uwview-ps06-intrusion-triage-auth-logs-en/
• Part 7: Four ways to use timestamps as evidence: https://uvp.y42u.net/en/blog/uwview-ps07-timestamp-driven-triage-en/
• Drill-down search — from 100,492 hits to 3 in two clicks: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
• Opening a 48 GB file isn’t the same as investigating it: https://uvp.y42u.net/en/blog/uwview-emeditor-48gb-comparison-en/
• Source code (GitHub): https://github.com/amru195704/UwView

From the developer: A full list of my apps, Kindle books and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Compression ratios, decompression speed and search speed vary widely with log content, compression algorithm and level, CPU, and storage. The measured figures cited are examples from one specific environment and are not a guarantee of the same result. Cloud storage classes, retrieval mechanisms and billing models differ by provider and change over time — please check your own contract. Command examples may need adjusting for your environment (GNU/BSD, shell, the implementation and version of tar, gzip and zstd, cloud CLI version, and so on). If you spot an error or an inaccuracy, please let me know in the comments and I’ll check and correct it.