Performance Monitoring FAQ
The Performance Benchmark was born out of one main requirement; to create greater transparency within the broadband market.
Currently the average consumer is at a loss in a sea of acronyms, stats and technical jargon without really knowing or understanding how to measure what it is they are getting or how to improve it. Through this project the FCC and SamKnows hope to level the playing field and make the industry more accessible, open and accountable, by giving both consumers and ISPs access to the same broadband performance data.
We're on the lookout for US broadband consumers to join our community, if you're interested please click here and fill out our short registration form.
Still curious? Check out the FAQs below.
- What is the "FCC Samknows Broadband Community"?
- Why are you doing it?
- What tests are performed?
- Which ISPs are you including in the tests?
- How is this different to speed test results being published?
- How is the Whitebox connected to my network?
- Can I use the Whitebox to connect directly to the internet?
- Does the Whitebox provide a wireless network?
- Who manufactures the Whitebox?
- What effect will this have on my monthly download cap?
- Will this affect my usage of my Internet connection?
- How does this affect my security and privacy?
- Can Community members see their own individual results?
- How long will the testing period last?
- Do I get to keep the Samknows unit after the testing period has elapsed?
- Does SamKnows make use of any GPL licenced software, and are the sources available?
What is the "FCC Samknows Broadband Community"?
Originally launched as the Performance Monitoring Network the FCC SamKnows Broadband Community is a unique system that aims to gather and report statistical data on the performance of broadband providers across the globe. Being able to compare broadband on price, headline speeds (which are still the subject of much controversy!), anecdotal discussion and speed tester results just isn't enough - the Community, with your help, aims to put the information back in the public's hands.
We'll be deploying our rather lovely SamKnows "White boxes", an asset to any home, filled with some clever technology to volunteers all around the country. Once connected to your home network the unit will perform a series of tests at regular intervals during the day, every day of the year, with little to no involvement from you, simple! The results of these tests are then fed up to our reporting engine and combined with the results of others on the same ISP to form a national view of how that ISP is performing. We're even working towards future deployments that will allow us to drill down to regional and/or product performance.
This project goes far beyond running just speed tests; our White Boxes are lean, mean, broadband testing machines periodically checking latency, packet loss, DNS query times and failures, web page loading times, as well as the obligatory suite of speed tests. Additional tests are also in development too. The goal is to provide a statistically sound measurement of overall broadband quality.
Why are you doing it?
We're striving for greater transparency within the broadband market. There's simply nothing else like it out there anywhere (to the best of our knowledge anyway!). The statistics will provide a whole new means of looking at broadband services, levelling the playing field and making the industry more accountable and work harder for the consumer.
Presently one might look at a 16Mbps headline speed and assume that it means the service will be good for online gaming. An incorrect assumption is being made here - raw speed is far less important than many think. Latency and packet loss are far more important for online gaming. Our network will measure both of these factors (and many more) across all of the ISPs – greater insight into how the ISP’s perform, more insightful information for the consumer.
Just like Batman (kind of) we've taken it upon ourselves to do this because nobody else has! We think it'll generate some fantastically interesting data, which we intend to share with you.
What tests are performed?
At present we are performing the following tests:
- Multi-threaded HTTP download speed test
- Multi-threaded HTTP based upload speed test
- Availability of the connection
- Latency (both ICMP and UDP)
- Packet loss (both ICMP and UDP)
- DNS query resolution time
- DNS query failure rate
- Web page loading time
- Web page loading failure rate
- Video streaming performance
The above tests are performed against a cluster of test servers, hosted all around the US. The web browsing test is performed against 10 real US-based websites.
We are always developing new tests and can remotely upgrade the existing units with new testing profiles once they are ready for production use. Pretty comprehensive I think you'll agree!
Which ISPs are you including in the tests?
We are looking for customers of all ISPs to participate. Note that we will only publish statistics for an ISP when we have a reasonable sample size or can be certain that our results are not merely an anomaly.
How is this different to speed test results being published?
It's completely different! The only similarity is that our testing suite incorporates speed tests, amongst many other things.
Speed testers were written with the aim of helping individual users diagnose speed problems with their connections. Whilst results taken en-mass from them are useful for drawing very general conclusions, their usefulness does reduce when you consider:
- They consider speed alone, and only for your specific machine. They tell you nothing about how a VoIP call would work, or how good your web browsing experience would be.
- Environmental factors (e.g. using a wireless connection, having other Internet users on the network, etc) all impact results.
- They're run on an ad-hoc basis. If you want to compare results at different times of the day then you need to run it yourself every single time.
That's not to take anything away from speed testing websites; they're a very useful tool for diagnosing an individual connection's maximum upload/download speed. But can we draw meaningful and accurate statistics when looking at more than just speed? Not to the extent we’re aiming to achieve with our project.
How is the Whitebox connected to my network?
The Whitebox should be placed in between your existing router and your networked computers. Any devices that connected via ethernet cable to your existing router should instead connect to the Whitebox. This ensures that the SamKnows device is always aware of the network being used and will never run tests at a time when you require your full bandwidth to be available.
Can I use the Whitebox to connect directly to the internet?
No. The SamKnows whitebox is not a broadband modem or router. It operates as a bridge and connects to your existing router.
Does the Whitebox provide a wireless network?
No. In the FCC project, the Whitebox does not provide a wireless network, you would use your existing router to continue to provide wireless access in the home. The wireless capabilities of the device are used purely to passively monitor nearby wireless networks for traffic, ensuring that no tests are run when the line is active.
Who manufactures the Whitebox?
The Whitebox is a TP-Link router running our custom firmware. It runs as a bridge, meaning that it does not provide any IP addressing and will pass network traffic straight through its ports without causing any conflicts or requiring you to reconfigure your network addressing. SamKnows provides the software.
What effect will this have on my monthly download cap?
Our units involved in the FCC project transfer a large amount of data, which varies according to the speed of your connection. The usage on a 10Mbps connection will be around 20GB/month, and will likely be around 60GB on a 50Mbps connection. The amount that's downloaded is speed dependant (so a slower connection will use less traffic than a faster connection).
If you're on a product with a low cap then we'd advise against signing up, or at least informing us beforehand so that we can apply a different testing profile (we can run smaller speed tests on a per-device if necessary).
Will this affect my usage of my Internet connection?
No. Our tests are designed to be non-intrusive. The vast majority of the tests that take place throughout the day are very minor and would have no effect on normal Internet activity.
Speed tests are obviously more intensive, but before running these our units will check how busy your wired and wireless networks are. If they are transferring more than 64kbps downstream or 32kbps upstream then the tests will be deferred for 1 minute and then the process is attempted again and repeated for up to 5 minutes.
How does this affect my security and privacy?
We do not collect any personal browsing data that passes through the device. The Whitebox checks your bandwidth usage to decide whether to postpone the tests - for example, if you’re downloading a large file - in order to prevent any skewed test results. The Whitebox simulates the act of browsing to common websites in order to produce a metric for browsing speed rather than use your own browsing.
Testing information uploaded from the unit to our servers contains no information about you whatsoever. Furthermore, all such communications are encrypted, ensuring that results cannot be tampered with en-route.
Your individual unit's test results will be available to you alone. Your unit's results will also be aggregated with others from the same ISP to form a larger average set of results that can be viewed publicly.
We have absolutely no intention of doing anything that may adversely affect your privacy or security. If you have any concerns please feel free to contact us to discuss them.
Can Community members see their own individual results?
You know it! Once signed up and operational you will be sent an email with a link to your statistics page. If you don't receive the email, please contact us and we'll sort this out for you.
Below are some samples screenshots of some of the statistics that you will be able to access for your individual connection.
How long will the testing period last?
The initial testing period is scheduled to last three years.
Do I get to keep the Samknows unit after the testing period has elapsed?
Does SamKnows make use of any GPL licenced software, and are the sources available?
This product makes use of GPL licenced software. Complete source code for all GPL components and those that link to GPL licenced libraries are available at https://files.samknows.com/~gpl/