Automating Log Retention — Writing Generation Rules and a Housekeeping Policy That Survive

Three years of logs were gone.

That retention rule you automated — when did you last look at it?

A rule nobody revisits keeps working quietly, in whatever direction it was pointed on the day it was written.

Log retention usually starts with “let’s write a logrotate config.” And usually stops there. The generation count stays whatever somebody picked. The compression boundary stays a rule of thumb. Deletion happens once a year, when someone remembers. Four scenes here — automating generations, archives rotting on a slow disk, the housekeeping decision, and the invisible cost of waiting for files to open — each starting from why the rule goes hollow.

Up front: UwView Pro keeps archives at roughly 1/9 and searches them in that compressed form, and a file you’ve opened once reopens with line numbers still attached in 0.02–0.07 s (measured on a 47.73 GB text file; one specific setup, results vary). Once “compressed means unreadable” stops being true, a retention rule is a two-way choice — raw or compressed — instead of a three-way one that ends in deletion (details at the end)

This article is about designing retention rules. How long you must keep logs, whether you may delete them, and what you may copy off-site are governed by your organisation’s policy, your client’s, and applicable law.

1. The logrotate config was written. Nobody has looked since

Situation

You wrote a retention rule when the server was built.

/var/log/app/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
}

Thirty generations. A month. It was enough at the time.

Two years later, the log volume is 8× what it was. rotate 30 is unchanged. You think you’re keeping a month; you’re keeping 70% of the disk. Nobody has noticed.

Why it happens

Because the rule is written in days, not in bytes.

rotate 30 guarantees a count, nothing else. Change the daily volume and the footprint changes with it — and log volume only ever moves one way.

The hollowing-out continues in two more ways.

• Nobody remembers what delaycompress was for. Leaving the most recent generation uncompressed is a concession to processes still holding a file handle. Most environments have already solved that with copytruncate or a postrotate hook, so the concession survives with no reason left behind it. If that generation is tens of GB, it is sitting there raw for nothing
• Only deletion stays manual. rotate removes what exceeds the count — but anything copied elsewhere is outside its world. Your NAS and your external drives accumulate files logrotate has never heard of

The general-tool approach, and where it stops

Watch bytes rather than days. That’s step one.

# What each generation actually costs
du -sh /var/log/app/*.log* | sort -h | tail -20

# Find the ones that are large even after compression
find /var/log/app -name '*.gz' -size +1G -printf '%s\t%p\n' | sort -rn

# Cut on total size instead of on days
#   in logrotate: pair size 2G / maxsize 2G with rotate
du -sb /var/log/app | awk '{ if ($1 > 200*1024*1024*1024) print "over 200GB" }'

maxsize works. Paired with daily, you get “rotate after a day or past 2 GB,” which is two conditions instead of one.

Three limits.

First, logrotate doesn’t know which files matter. Its only inputs are a date and a number. The file that carried the incident and the file from a quiet Sunday are deleted by the same rule.

Second, you have no basis for the threshold. The “one month” in “keep a month” is somebody’s intuition. Nothing records how far back anyone actually reads, so you can’t tell whether 30 should be 15 or 90.

Third, verifying a compressed generation is expensive. To check whether rotate 30 is right, you have to open generation 20 and confirm you no longer need it. That check is heavy, so nobody performs it, and the number survives untested (part 4 covered the decision criteria themselves).

2. The archive on the external drive is effectively dead

Situation

Old logs live on an external drive. Capacity is fine.

You need something from six months ago. You plug the drive in, open the folder, double-click a 40 GB file.

The progress bar doesn’t move. Ten minutes later, coffee made, it’s still loading.

Why it happens

On a slow disk, any tool designed to read the whole file converts its design directly into your waiting time.

Spinning disks and USB-attached drives run at a fraction of an internal SSD even on sequential reads. That’s where the tool’s assumptions start to bite.

• A tool that loads into memory costs size ÷ throughput. 40 GB at 100 MB/s is six and a half minutes, and you can’t interrupt it halfway
• A tool that indexes first reads exactly the same bytes. You see only the head until it finishes, so it feels identical
• Decompress-then-open pays twice. gunzip to a working disk, then open. Transfer, decompression and load — three rounds of the same wait
• And you need somewhere to decompress to. If it’s at 1/9, expansion needs nine times the free space. You archived it because the drive was full; now you can’t expand it there

The general-tool approach, and where it stops

Pull out only what you need, without expanding.

# Search inside without expanding (but it expands everything as it reads)
zgrep -n 'OutOfMemoryError' /mnt/archive/app-20260317.log.gz | head -20

# Peek at the head to work out the format
zcat /mnt/archive/app-20260317.log.gz | head -100

# Cut out just the range you want
zcat /mnt/archive/app-20260317.log.gz | sed -n '41284451,41284491p;41284492q'

# Measure the read speed itself, so your estimates have a basis
dd if=/mnt/archive/app-20260317.log.gz of=/dev/null bs=1M count=2000 status=progress

zgrep solves the capacity problem. Searching without staging an expanded copy is worth a lot.

Two limits.

First, zgrep expands the whole thing every time. gzip has no block-level random access, so reaching the tail of a 40 GB archive means expanding everything before it and throwing it away. Every single query pays full cost — narrow with two terms and you pay twice (part 8).

Second, on a slow disk that full pass dominates. Reading from the platter costs more than the decompression maths, so more CPU buys you nothing. The only lever left is reading fewer bytes.

3. Housekeeping runs on whoever is in the room

Situation

Once a year, you review storage.

“This 2023 folder — can we delete it?”

Nobody will say yes. No one can recall an occasion where deleting hurt, but no one can rule it out either. So it stays another year. And you have the same conversation next year.

Why it happens

Because “safe to delete” rests on human memory rather than on any attribute of the data.

When housekeeping stalls, three things are actually missing.

• You don’t know what the log is. The name says app-20230412.log.gz and nothing else. Access log? Batch run? You’d have to open it — and opening is heavy, so you don’t
• You don’t know whether it was ever examined. A file someone checked and cleared sits next to a file nobody has opened, wearing the same face. The first is safe to delete and the second isn’t, and that distinction lives nowhere on disk
• You don’t know whether it’s under a retention obligation. Files held for audit or contract are mixed in with files kept “just in case.” Deleting the first is an incident; keeping the second is waste (part 20)

Housekeeping is stalled, in other words, because the act of looking is expensive. The information needed to decide is only available by opening the file.

The general-tool approach, and where it stops

Write the deciding facts outside the files, into a ledger.

# Build a ledger: path, size, mtime, and a snippet of the first line
for f in /mnt/archive/*.gz; do
  printf '%s\t%s\t%s\t%s\n' \
    "$f" "$(stat -c %s "$f")" "$(stat -c %y "$f" | cut -d' ' -f1)" \
    "$(zcat "$f" | head -1 | cut -c1-80)"
done > inventory.tsv

# Pick out things nobody has read in a year (only if atime is live)
find /mnt/archive -name '*.gz' -atime +365 -printf '%s\t%p\n' | sort -rn | head

# Mark "already examined" in the ledger rather than on the file
awk -F'\t' '$2 > 10*1024^3 { print $1 }' inventory.tsv > big-files.txt

The ledger works. Build it once and the next review is only the delta.

Three limits.

First, even zcat | head -1 expands the first block. With a few hundred archives, building the ledger takes a night — and the first line tells you the format, not what happened that day.

Second, atime can’t be trusted. Most systems run relatime or noatime, so reads leave no trace. You want “when did anyone last look at this,” and that’s precisely what isn’t recorded.

Third, the ledger drifts from the files. It lives outside them, so every move, rename or re-compression has to be reflected by hand. Skip that, and next year someone says the ledger is stale — and you rebuild it from scratch.

4. Have you ever measured the waiting?

Situation

Someone asks for an estimate. “How long will the log investigation take?”

You say thirty minutes. It takes two hours. Afterwards you can’t account for the missing ninety.

Because you can’t account for it, you give the same estimate next year.

Why it happens

Because waiting isn’t recorded as work.

The two minutes spent opening a file never reach your task tracker. What reaches it is one block called “log investigation,” and nobody has measured what’s inside. Unmeasured things don’t get improved.

In practice it accumulates like this.

• Small each time, many times. Two minutes × ten reopens is twenty minutes. Two such days a week is close to three hours a month (part 10)
• Context breaks in the gap. Two minutes is long enough to check another tab. Coming back costs additional time to remember what you were chasing. The real cost of a wait exceeds the wait
• “It’s expensive, so I won’t check” creeps in. The moment you think “I’d like to test that hypothesis too, but that’s another two minutes,” the check gets dropped. Dropped checks leave no record and reappear later as rework
• There’s nothing to measure with. time measures commands. The wait on a GUI open, the stutter in a scroll, the delay before search results land — none of it is recorded anywhere

The general-tool approach, and where it stops

Measure at least the command side.

# What one pass actually costs (three runs)
for i in 1 2 3; do
  /usr/bin/time -f '%e sec  %M KB' grep -c 'ERROR' /mnt/archive/app.log
done

# Estimate from bytes read and throughput
#   40 GB at 100 MB/s = ~410 s. Ten times a day = 68 minutes
echo $(( 40*1024 / 100 * 10 / 60 )) min/day

# Recover session wall-clock from timestamped shell history
HISTTIMEFORMAT='%F %T ' history | tail -40

You can build the formula. Bytes read ÷ throughput × number of passes gives you the total wait, roughly.

Two limits.

First, you don’t know the “number of passes.” Nobody counts their reopens. It feels like three; it’s often ten. Without a denominator, you also can’t measure any improvement.

Second, measuring leads nowhere. Suppose you establish that tens of hours a year go into waiting. The general-tool side has no move to make: buy a faster disk, or split the file. The first is budget; the second means damaging the original. Measurement that can’t change anything gets abandoned.

What the four had in common

In all four, what’s stalled is a decision. It isn’t that people can’t write rules — it’s that verifying a rule costs too much to do.

The right-hand column rhymes because all four hit the same wall: opening what you archived, to check it. Whether 30 generations is right requires opening generation 20. Whether a folder can go requires seeing inside. Whether a change helped requires reopening and timing it. Because checking is expensive, the rule freezes on the day it was written.

Which leaves exactly one condition.

• Open it cheaply, in the form you stored it — without expanding, on a slow disk, reading only the part you want

Satisfy that and the order of operations changes. Today’s design says “compressed means unreadable, so anything I might read stays raw.” Raw eats space; space forces deletion decisions; deletion decisions are heavy, so housekeeping stalls. If compressed material stays readable, the entrance to that chain disappears.

Back to the three years that were gone. Nobody made a mistake. A rule written on one day simply kept working for three years, with nobody ever looking inside.

The tool I use

I build UwView (free), a viewer that makes huge text readable, scrollable and searchable from the moment it opens. It doesn’t pull the file into memory, so files larger than RAM open fine. Indexing runs in the background and line numbers appear when it completes (most viewers show you only the head until indexing is done). It never splits or extracts, so the original stays one file, unmodified.

• Files on slow disks are readable from the first moment. Part 2’s stuck progress bar doesn’t happen: display begins before the whole file has been read, so checking a format or getting your bearings costs no wait. Note that line numbers don’t appear until indexing completes — scrolling and searching work throughout
• No free space needed for expansion. You don’t have to unpack a .gz onto a working disk first, so a full archive drive is still inspectable
• Switch encodings without reopening (UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16, auto-detected). Older generations written in a different encoding are readable without producing a converted copy
• Everything runs on your own machine. Archived logs are never sent to an external service

Beyond that is UwView Pro.

• Store at roughly 1/9 and search it in that form. Parts 2 and 3, directly. Once “compressed means unreadable” is false, a retention rule becomes raw or compressed rather than raw / compressed / deleted — and deferring the deletion decision is exactly what makes the housekeeping conversation lighter (part 8)
• The index and compression are saved. From the second open onward the file comes back with line numbers, instantly (0.02–0.07 s measured on a 47.73 GB text file; one specific setup, results vary). Part 4’s ten reopens a day stop accumulating on this side of the first open
• Drill-down search. Useful for part 3’s ledger: ask the same archive repeatedly and every question after the first uses the saved index. The original line numbers survive to the last stage (drill-down article)
• Save the queries, resume tomorrow. An annual review is largely re-applying last year’s conditions; here the conditions persist as artefacts (archive × session restore)

Stated plainly: UwView is not a log management system.

• It doesn’t rotate, delete or archive anything. It doesn’t replace part 1’s logrotate — it’s the tool on the verifying side of that config
• There’s no retention-period tracking and no compliance ledger. Part 3’s inventory still has to be built separately
• No aggregation, no dashboards, no alerting. Nothing automatically records part 4’s waiting, either
• It does not search .gz directly at speed. The speed arrives after the file has been rebuilt once as a .uwvz; the first pass costs a full read
• It handles text. Binary dumps and database files are out of scope

The same things, from the command line

v1.6.0 added a uvp command (and uvf for the free build). It uses the same .uwvz as the GUI, so an index built from the shell is already there when you open the file in the app. Mapped onto this article’s four parts:

# Part 1: before deleting generation 20, confirm what's actually in it
uvp /var/log/app/app.log.20 'ERROR'

# Part 2: question an archive on a slow disk without expanding it
uvp /mnt/archive/app-20260317.log.uwvz 'OutOfMemoryError' -C 5

# Part 3: get the breakdown of what a file contains, as housekeeping material
uvp /mnt/archive/app-20260317.log.uwvz -uniq '\[([A-Z]+)\]' -head 20

# Part 3, continued: write the evidence for the decision next to the ledger, compressed
uvp /mnt/archive/app-20260317.log.uwvz 'FATAL' -C 3 -out inventory/20260317-fatal.txt.gz

# Part 4: measure what the second pass really costs (the first pays for the index)
time uvp /mnt/archive/app-20260317.log.uwvz 'ERROR' 'timeout'

Exit codes are grep’s — 0 found, 1 not found — plus 2 when a limit is hit and the output is cut off (no limit by default; only when you set one with -limit N). if uvp app.log.20 'ERROR'; then works as written, so part 1’s pre-deletion check drops straight into a housekeeping script.

Stated honestly: on the first question ripgrep is 15–20% faster, because uvp builds its index first. On a 3 GB file that fits in RAM, rg stays ahead on the second question too. uvp pays off past 10 GB, when you ask the same file more than one question (the measurements; Mac M4, external USB SSD, OpenStreetMap XML — one setup, results vary).

What it does is let you check the contents in the form you stored them, point at them with the original’s own coordinates, and leave the check in a shape you can run again. If your retention rule has been frozen since the day it was written, start by opening generation 20.

• And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

• Part 1 — Four standard tools that sink under a huge file: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
• Part 4 — Delete, keep or compress: the criteria: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
• Part 8 — Compressed storage and searchability at once: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
• Part 10 — Four things to set up for 2 a.m. you: https://uvp.y42u.net/en/blog/uwview-ps10-oncall-night-preparation-en/
• Part 15 — Four limits of command-line craft: https://uvp.y42u.net/en/blog/uwview-ps15-cli-craft-limits-en/
• Part 16 — Four principles for logs as evidence: https://uvp.y42u.net/en/blog/uwview-ps16-log-as-evidence-en/
• Part 20 — Surviving an audit by design: https://uvp.y42u.net/en/blog/uwview-ps20-audit-log-retrieval-en/
• Part 22 — Four habits of a team that can hand it over: https://uvp.y42u.net/en/blog/uwview-ps22-shareable-log-investigation-en/
• Reopen it tomorrow right where you left off (archive × session restore): https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
• Drill-down search: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
• We shipped a uvp command (measured against ripgrep): https://uvp.y42u.net/en/blog/uvp-cli-release-vs-ripgrep-en/
• Source code (GitHub): https://github.com/amru195704/UwView

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The log excerpts, line numbers, file sizes, generation counts and time estimates are illustrative and do not refer to any real system or engagement. Command examples may need adjusting for your environment (GNU vs BSD, differences among stat, find and time, your shell, and your logrotate version). Always confirm option names and defaults against your local man. logrotate behaviour depends on your configuration and on the application itself, and whether atime is recorded depends on mount options. Figures described as measured come from one specific setup and are not a guarantee of the same result; disk type, connection, filesystem, fragmentation, encryption, page-cache state and concurrent processes change outcomes substantially. Retention periods, deletion and off-site copying of logs are governed by your organisation’s policy, your client’s, and applicable law. If you spot an error, please leave a comment and I’ll check and correct it.