Huge Files

Technical Guide

Everything We Measured on Huge Files — Four Boundaries Where the Numbers Flip

The ceiling was 950 MB/s. That "N seconds" figure — at what file size, and on which attempt? Change either one…
Technical Guide

Never Write to the Original — Four Habits for Working With Logs

You opened it. It grew by three bytes. That log file — is it still the same file after you read it? Operations…
Technical Guide

Automating Log Retention — Writing Generation Rules and a Housekeeping Policy That Survive

Three years of logs were gone. That retention rule you automated — when did you last look at it? A rule nobody…
Technical Guide

Log Investigation Lives in One Person’s Head — Four Habits of a Team That Can Hand It Over

The handover note was three lines long. The query you built yesterday — could anyone else rebuild it right now…
Technical Guide

The Morning the Monthly Batch Died — Four Realities of Running a Data Pipeline

line 48,213,996. That's all you got. Can you look at that line right now? The pipeline hands you a coordinate.…
Technical Guide

Only the Line Number Survived — Four Techniques for Inspecting Huge Data

It died at line 8,342,119. That is the entire lead. The file is 12 GB and the migration sign-off is the day af…