1EQQ8ZzGD

up errors properly (a full disk situation was masking them,
if my memory serves), our validation software also wasn't
dishing up errors properly (that was written very hastily, and
without a clean state to start from, was missing several important
error conditions), and several other factors were compounding
our lack of concrete, verifiable information.
The only prospect then was to roll back about two months
to the last full backup that we thought might work. In that
meeting, Galyn mentioned she might have a copy at her
house. So we went home to get that machine, and you can
watch the video for how that went . . .
With Galyn's machine now back in the building, we duped
that data immediately, then set about the task of trying to
verify and validate this tree, which we thought might be about
two weeks old. We compared Galyn's restoral with a much
older one (from two months prior) and couldn't determine
a clear winner, there were too many inconsistencies. So, instead,
we set about the task of assembling what effectively
amounted to a new source tree, by hand, one file at a time.
The total number of files involved was well into the six figures,
but we'll round down to 100,000 for the sake of the rest
of this discussion to make the math easier.
We identified the files that hadn't changed between the
two, and took those straight away. Then there were the files
that were on Galyn's but not on the older one; we took Galyn's
and assumed they were new. Then there were files that were
on the older one but not on Galyn's; we put those in the "hand
check" pile, since it is unusual for files to be deleted within a
production source tree, and we were suspicious of those deletions.
Then there were the files that were different across the
two backups, those also went into the "hand check" pile along
with any files that were touched more recently than Galyn's
version.
Given that, we had something like 70,000 files that we felt
good about, and we poured those into a new source tree. For
the remaining 30,000 files, it was all hands on deck.
We checked things across three partially complete, partially
correct trees . . . the two month old full backup (A), Galyn's
(B, which we thought was the best one), and another
cobbled together tree (C) from the stray files left around from
249