Not all speedtests are created equal

Our engineers have taken a moment to help us write this blog post. The moral of the story will be “Not all SpeedTests are created equal,” but a longer definition should be “Don’t just measure your speed based on a speedtest, or at the very least compare multiple tests over multiple servers.” Before we dig into it, I want to show you 3 SpeedTests done within 5 minutes, on the same connection using 3 different testing tools. Just look at how things are reported different by each, and actually how none of them are “wrong”, you just have to understand the factors behind a speedtest, and understand it will not match your package 100% – ever, and we will explain why. (Test #1 – SpeedTest.net, Test #2 – NPerf.com, Test #3 – Google SpeedTest – All conducted on a 50Mbps down, 10Mbps up modem).

Web-Based SpeedTest, and Android/iOS speedtest are not accurate by nature. They will give some good information but are in no way used to measure actual full-line / full-service potential. Below we will discuss why. We will also discuss “test-a-holics” and some other talking points. We want everyone, not just our customers, to understand what a speedtest really is, how to conduct a reliable test, and how to get the most out of their connection.

Other Network Traffic

None of the speedtest options that exist take into consideration what is currently going on, in your network, and background network traffic. On too many occasions to count, we have customers who will submit a network issue report and upon investigating we find that the culprit is not actually a network or hardware issue but instead “unknown” traffic that is using a large amount of their bandwidth. By default, and for security, we can not see what traffic is going across your network. However, we can enable some special monitoring when investigating issues and the most common culprits are updates. Microsoft Windows, Play Store, Apple Store, Xbox/Play Station, etc. Most people do not understand that if a device is on, but not being used actively, it can be using the network heavily in the background. In fact, many platforms are designed to only update when the device thinks it will not be used heavily. However devices can not account for other network use at the same time.

What this means, is that when the computer, phone, or gaming system you are not using starts to update while you are watching a Netflix movie, for example, on another device, if the updates cause the device to saturate the line of service and use all the available bandwidth, your Netflix will buffer or go to a lower quality. This has nothing to do with your ISP’s network, and is not a network issue. It is the same as if you tried to wash clothes and take a shower at the same time – the water pressure and flow rate will be less to the shower while the washing machine is filling up or rinsing. The water company is not delivering any less water to your house, you are just using the maximum amount that the pipe can supply. Your internet is no different, however instead of a physical limit like the pipe size, it is a limit based on the plan you have purchased. The only way to get a full network speedtest is to turn all other devices off, however even then there are other technical factors like your devices connection type (Wired/Wireless), the server you are testing against, networks between your ISP and the server, the type of test and data used, connection overhead, and the list goes on and on. Typically a speedtest will show 85-95% of actual capacity, even when nothing else is using the network.

Length of SpeedTest

A typical speedtest only last 10-15 seconds, and that is a minuscule amount of time. When data flows from a server to your device, it does not flow in an even steady pace. The data is comprised of packets, and packets come and go in various rates. Combined with the fact that your 10-15 second speedtest may been tested at the same time another request was made, you are going to have inaccurate results. To get a more accurate speedtest, you would need to measure a much longer period of time like 10-15 minutes. Currently there are no “easy one click” testing tools that offer this. You would need to read up on iPerf and setup your own test. We can provide you with an iPerf server to test against if you wish to do this, just open a support ticket.

Type of connection (WiFi vs. Wired)

If you are testing your connection speed on a mobile phone or device connected to the modem through WiFi, your test results are essentially pointless, especially if trying to use the speedtest to find a connection issue. WiFi has interference, WiFi has other local network and broadcast traffic going on, and if you are far away from the modem – there is a high likelihood that your Internet Service is actually faster than the speed at which your WiFi is transmitting. For example, if your device shows it is connected to WiFi and the WiFi max rate is 72Mbps – real world throughput will be half of that, same as if it shows 36Mbps for the WiFi throughput. So the BEST internet speedtest you will get will be around 12-15Mbps when really, your service is capable of a lot more. If you have bad WiFi performance, then you will need WiFi extenders or Wired Connections. Always check your WiFi signal and WiFi transfer rate under your devices advanced WiFi settings / information. For instance, if a device next to the modem or wired directly into the modem is working great, but a phone 4 rooms away is buffering – the WiFi is the problem, NOT the internet connection.

Server Location / Intermediate Networks

When you use an Internet SpeedTest never just test against the automatically selected SpeedTest server, go to the server list and find the server you know to be close to you, and then test 3 other closest servers and average the results. IP based location is a joke, and often you will find yourself running a test against a server hundreds or thousands of miles away in a major city.  At that point, your SpeedTest is having to exit the Internet Service Provider, travel a vast distance over networks not controlled by the ISP (that may be saturated or congested), and then fight in a virtual line of other testers. It is a recipe for flawed results. Sure, it can give a baseline, but it is never “on the money”. Additionally, if a network between your ISP and the destination network is congested or over sold, that is not something your ISP can control. At that point, all they can do is open a ticket with that network and ask them to look into it. If a link is saturated, until the other ISP upgrades service, there is nothing that can be done to resolve the issue. We have for instance seen this multiple times with Direct TV Now, where the Direct TV Now network is overwhelmed, and can not handle the requests, which will cause video to buffer or be lower quality – and this has nothing to do with our service or any other ISP’s service – the only people that can resolve it are Direct TV.

Method of testing metric

Internet traffic passes in packets, and over multiple protocols and standards. Each of them have different overhead that can limit, or slow data packets. A test over TCP / HTTP / Socks will all act and perform different. They will never give the same result.

In closing, there are too many things to count that can impact speedtest results, and not just in a negative way, speedtest can also be overly compensated or exaggerated by some testing tools, or services making the end user think they are getting better service than they really are. At the end of the day, a SpeedTest is an estimated throughput, and not an actual throughput. If you need an accurate test, start looking into iperf, or other methods, but most cost thousands in equipment and are not practical for end users. The Internet is a highway, and the flow will always be changing, and taking different routes. Never expect speeds to be consistent, unless you have a dedicated business line of service over a layer2 connection. Otherwise, it will not be consistent. If you are a “test-o-holic” and test your connection constantly, and complain every time the speed drops a few Mbps – you will always be disappointed, and you will never be happy – that is just the cold hard truth.