While measuring how my search engine scales with thread count (results here), I threw away my entire first data set. Here’s why — and if you ever benchmark anything on a fanless laptop, this is not someone else’s problem.
The first data set was clean, and wrong
I measured 1 through 10 threads, in order. The curve looked beautiful, including a shiny record: “11 threads, fastest at 5.2 seconds.” Right before writing the article, I re-ran the same conditions as a sanity check.
The same conditions came back up to 54% different. 4 threads: 11.0s became 9.0s. 11 threads: 5.2s became 8.0s. The record run did not reproduce.
No rogue background process. The culprit was the chassis.
The culprit: thermal history on a fanless machine
The test machine is a MacBook Air (Apple M4) — fanless. Under sustained load it heats up and thermal-throttles. Which means: every measurement depends on what you measured just before it.
The nasty part is that this is not random noise — it’s a systematic error. Sweep 1→2→…→10 in order and the later thread counts always run on a hotter chip. The high-thread results are penalized, consistently, in the same direction. Averaging repeated sweeps doesn’t remove it, because the bias repeats too. That “fastest 5.2s at 11 threads”? A single lucky run on a cold chip. The cleaner the graph, the more suspicious you should be.
The fix: round-robin
Stop measuring each thread count in a block. Instead: sweep 1-16 threads once, repeat the whole sweep for 4 rounds, and take the per-thread-count median across rounds. Every thread count now experiences cold and hot states equally; thermal history no longer piles onto specific configurations.
The corrected protocol (128 measurements) produced the companion article’s conclusion: a plateau at 8 threads, 3.0x. The 5.2s ghost vanished; 7.4s became a reproducible floor.
How do you know the fix worked?
Claiming “the new numbers are right” needs evidence. Mine: two search patterns with hit counts differing by more than an order of magnitude (10,967 vs. 722 hits) produced nearly identical scaling curves. If heat or any other disturbance were still leaking in, there’s no reason two independent series would line up that well.
I kept the first CSV for reference but decided it never appears in an article. Publish a retracted number “for reference” and someday someone will cite it.
Lessons
- On a fanless machine, measurement order leaks into results. A sequential sweep systematically penalizes the later configurations
- Distrust beautiful single records. One cold run masquerades as a best case
- Round-robin + median distributes cold and hot fairly across all configurations
- Validate with independent series, not repetition. Re-running the same biased protocol reproduces the same bias
The benchmark’s enemy wasn’t a rival product. It was my own chassis.
And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, $129 one-time / $9 per month).
From the developer: a list of my apps, Kindle books and open-source projects is on GitHub: amru195704.
A note
The information in this article is provided for reference purposes only, and its accuracy or completeness is not guaranteed. Feature availability reflects each version as of the time it was checked, and may change in future updates. If you notice any errors or inaccuracies, please let us know in the comments and we will review and correct them.
コメント