PC Magazine July 2017

CROWDSOURCING VS. DRIVE TESTING
There are a lot of “fastest” awards. They’re all correct, according to their own
testing and methodology, and they all have something interesting to say.
In testing, the main division is between crowdsourcing and drive testing.
Crowdsourcing, which is done by Mosaik/Sensorly, Nielsen, Ookla Speedtest,
and OpenSignal, relies on users to run speed tests on their own devices. With a
big enough crowd, you can get a good picture of a network.
Crowdsourcing is always happening, so it’s up to the minute. And it uses a great
range of devices, so you can tell the difference between them. We use Ookla’s
crowdsourced data in our analysis showing how the Galaxy S8 is faster than
earlier phones, for instance.
But crowdsourced apps often can’t tell whether a test is indoors or outdoors,
which make for very different results. They may not do a good job of finding
dead zones, if their users don’t run tests in places which obviously have no
signal. They may have bigger crowds with some carriers or in some cities. And
they leave open the possibility that people using one carrier might be using
better phones, in better weather, on a less congested day than people on
another carrier.
Drive testing is what we do, along with Root Metrics and P3. This kind of testing
lets us compare carriers using the same device, in the same place, at the same
time. In this way, we can eliminate variables and map out coverage on our
route. It lets us make sure we have as much data as we want in each city, so we
feel confident in our results.
But drive testing takes enough work that it isn’t happening continuously in
every city. It won’t show you the performance of phones you don’t drive with.
And it can cover only the routes you drive along.
Methodology-wise, we balance six different elements for our speed score. Other
studies may focus on downloads, use a different measurement of latency, or (in
Nielsen’s case) use a totally idiosyncratic attempt to measure the speeds coming
into various mobile apps. We think our balance makes the most sense, but we
also respect the different decisions others have made.
PC MAGAZINE DIGITAL EDITION I SUBSCRIBE I JULY 2017