Date By Shlomi Gian Tags mobile / performance measurement / analytics performance

When speaking of content delivery over the internet, practitioners in the field usually break the "delivery path" into three "miles":

  • The "first mile" is the one connecting the origin to closest CDN server

  • The "middle mile" is in the cloud, connecting the server edge to the server that is closest to the client

  • The "last mile" is the one connecting to the end user

When speaking of mobile clients, the path is identical except that the last mile is over a wireless connection (Cellular or WiFi) this presents unique content delivery challenges.

While network conditions of the first and middle miles are fairly stable, this is not the case for the last mobile mile where latency, throughput, packet loss, RF signal and other factors are highly variable across devices and networks. Measuring the performance of content delivery over the last mobile mile can be expensive, inaccurate and painful. Improving content delivery performance over the last mobile mile in order to improve business KPIs is even harder.

In this short post I will try to share my experience in measuring mobile performance based on the years I spent in companies such as Akamai, Cotendo, Keynote Systems and DeviceAnywhere.

CDN solutions were introduced almost two decades ago in order to overcome latency on the public Internet. Companies like Akamai and Inktomi started caching static content on servers located closer to the end user (at the edge of the Internet) in an attempt to reduce the time it takes to download popular content and avoid long round trips to the content (aka "Origin") server. As popularity of these services increased the demand for 3rd party measurement service grew and companies like Keynote System and Gomez emerged. Measuring website performance was a relatively simple task with two common approaches:

  1. Synthetic Monitoring using test servers in data centers running http based test scripts.
  2. Real User Monitoring where real user PCs were used to run similar scripts.

Regardless of the approach the testing technology was pretty basic and only required a standard browser and a scripting environment. The common test variables were: location, browser type and network type. A few hundreds tests were needed in order to get an idea of a given website performance. That was not an issue since cheap PCs, located in a few fixed locations around the world, were able to execute processes/connection at a marginal cost.

Then came the mobile web age and few things changed:

  • New operating systems showed up (Brew, Nokia, MSFT, RIM followed by iOS and Android)
  • Network access was possible through hundreds of wireless carriers with different network technologies (e.g. 2G, 3G, LTE, WiFi) and spectrum characteristics
  • Bandwidth, latency and packet loss were constantly changing with high variability

Now a few hundred tests per device were required in order to get an idea of a mobile site performance. Unlike PCs, cellular modems and phones did not allow multiple scripts to run simultaneously and this resulted in longer and more expensive measurement efforts.

Performance: Mobile vs Web

Then came mobile apps and things got even more complicated. Now it was no longer possible to use cellular modems for performance measurement and tests became OS dependent. Scripting was not a viable method for "driving" applications continuously and new approaches that required hardware integration were introduced (see services such as DeviceAnywhere and Perfecto Mobile). Unfortunately, these solutions are neither simple nor inexpensive.

In most recent years in light of mobile dominance (Per Google: "more searches are now on mobile than desktop") and the challenges highlighted above, Real User Monitoring services became the more affordable method to collect high volume performance measurement at low cost. For mobile applications this requires an SDK integration with companies such as New Relic and Apteligent active in this space.

Most recently PacketZoom released a free SDK That provides much of this information for free. Built into this SDK are analytics capabilities as well as an A/B testing capabilities that saves the need for 3rd party services when evaluating mobile content delivery services. This SDK, meant for app developers, provides some really detailed RUM based metrics (including important business KPI's like session length, number of URL's dowloaded etc.) at absolutely no cost, and forever!! Check it out and let us know what else you'd like to see. We love hearing from our users.


Comments

comments powered by Disqus