Monday, September 15, 2008

Two CDN solutions we tried for Teamness

CDN (Content delivery network), as Wikipedia states, is a group of computers spread across the Internet who collaborate together to deliver content. In this post I'll tell you a bit about our experience trying to setup a CDN for Teamness.

Image by GustavoG

Why CDN?


The benefit of using CDN when you're offering a live web application is that you can delegate your static data, like images, movies, client scripts and such to be delivered from different points in the world. This translates in higher download speed for your users. When someone requests a page from your application, she/he's redirected to the nearest server in the CDN to have the static content delivered.

We are strong believers in the concept of Try-Before-You-Buy. That's why we setup a demo account for Teamness and it's also a reason why we're offering a free plan. We applied for trial versions from two offerers: Panther Express and EdgeCast.

Panther Express

They advertise having under 40ms delay for 95% of the computers in the world, but a problem for us was that Panther Express don't deliver content over HTTPS, or at least they didn't until the end of May 2008, when we were discussing with them.

We obtained a trial setup for one week, but the fact that we didn't get HTTPS content was a major issue.

EdgeCast

We signed a Master Service Agreement and a Service Order for a 4-days trial. They mentioned that they don't support CNAME’s for HTTPS, but only for HTTP.

After trial thoughts

Our intention was to sign up for a trial version, so we can run some tests and see what's the improvement of using a CDN service. We made different tests, adjusting the number of concurrent users and using also SSL content for EdgeCast case, but we did't obtain better results with CDN than with our server setup at SoftLayer.

Both Panther Express and EdgeCast asked for some sample URLs to check HTTP headers, so if you want to test a CDN solution before releasing, it's better to have your application online already. This is also helpful for doing comparative tests between your application with and without CDN enabled.

We will come back to evaluate more CDN offers in the future. In the mean time, our hosting provider SoftLayer started to offer CDN services and there is also another affordable solution we found at CacheFly. Here you can find a CDN guide with services ranked by approximate annual revenue, via CDN Evangelist.

7 comments:

Brent said...

Sorin,

My name is Brent Matschke, I'm the marketing manager here at EdgeCast. I was happy to read in your blog entry that you tried out our CDN service but dispoointed to read that you found the performance of your site was not improved by the user of our CDN.

May I ask how you measured performance? Here at EdgeCast, we use Gomez tools to measure our performance and, by our measurements, we see our network performing as well or better than our competitors and much better (on a global basis) than all simple single-source hosting solutions.

I'm curious to learn more about your test. Thank you very much,

Brent

Sorin Ostafiev said...

Hey Brad,

I think we knew before trying the CDN what the answer will be, that it will not be justified for us to use one, but we wanted to give it a try to see if we are wrong.

We thought that a CDN will not make sense for us for a few good reasons:

- We have a small number of resources in our application(s). On the public site a few megabytes and on the private site around several tenths of megabytes, but in fact only a few megabytes are actually requested on a regular basis (the rest are in some heavy js libraries, like dojo that are rarely used)

- A lot of optimizations to gain speed, to enumerate a few:
* the .css or the .js files are packed in one big file
* for browsers that are supporting inline images (e.g. Firefox, IE8) we are delivering the images directly from .css without needing to deliver the images by spending additional expensive requests to the server
* the resources are heavily cached on the client so mainly the client will experience the delay at the first request; they are also delivered compressed

- Because of the application’s business, the number of resources that can be publicly shared is constant, it’s not growing over time

- CDNs are in general expensive, at least for our free product, but we agreed that it might worth the money only if the results are spectacular

- Our hosting provider (Softlayer) has pretty good network connections and we didn’t have any issues regarding the network performance. You can see here that the average response time is pretty good

I don’t remember exactly how the setup of the tests was, but I remember we use WCAT and a combination of some scripts on top of WGET. We tested from 3 locations (Los Angeles/CA/US, Bucharest/Romania and Stockholm/Sweden) from Windows and Linux boxes during a few days (we had tests that were running continuously for many hours). We analyzed the results on the clients but also the IIS logs records or the EdgeCast stats and we tried to isolate and understand as much as possible the CDN performance.

The tests we did confirmed our initial assumptions that the gained speed is marginal compared to the money invested; the improvement is so low because of the nature of the application and the quantity of accessed resources. The conclusion was that there is a very small improvement (something around 5%) but only in a few circumstances (e.g. at a cold page reload).

Sorin

Brent Matschke said...

Hey Sorin,

Thanks for the detailed response to my question about how you did your testing. Sounds like you're well on top of your project and you really know what you're doing. I spoke to some of our more technical people here at EdgeCast (I'm in marketing and not the most techy person here) and here are a few comments and questions they came back with.

1) It’s better to test from a wider array of locations. This is where Gomez testing really gives you a complete answer to how well your site is performing from a global perspective. Gomez has thousands of end users in their network so their results eliminate any aberrations or skewing that can result if you just test from a few locations.

2) Did you have us enable gzip/support for compression? We do support gzip for compression but you have to enable that feature.

3) Did you load his content to the edge before testing? Maybe the testing was pulling a lot of content from his origin. It sounds to me like you probably did load your content to the edge, you clearly understand what you're doing, but it's always something to double check.

4) It seems that you tested when we did not support cnames for HTTPS. We do now.

All for now, this forum is great, thanks for sharing.

Brent Matschke, Marketing Mgr EdgeCast Networks, We Power Digital Media Content Delivery

Sorin Ostafiev said...

Hi Brent,

2. Yes, as far as I remember it was with HTTP compression enabled.

3. We didn’t upload the content on the origin, but we always had a warm up period (e.g. 1min) while load testing that we ignored from the results. We also double check by watching the web server’s logs on the origin to make sure that there are no hits during the testing period (excepting the warm up).

4. Our public website is almost entirely over HTTP so a CDN can help to speed up the experience for a visitor on a cold load, but I didn’t think that the CDN can help much for the the subsequential pages visited by that visitors because most of the resources are already cached on the client.
OTOH, on the private website, because a part of the accounts are accessed entirely over HTTPS (e.g. customers with greater security demands) a CDN can play a great role since the secure content is (usually) not cached on the client. But, at that time, discovering that you are not delivering HTTPS content scored quite negative for us when we were deciding if we need a CDN or not.

Cheers,
Sorin

Brent Matschke said...

Hi again Sorin,

I understand, your application resides pretty much entirely in the client's cache after the initial page load. A CDN certainly isn't going to help much after that. It's too bad we did not support cnames for HTTPS at the time you ran your tests. Try us again some time, we're here in sunny LA when ever you're ready.

Change of subject, are you planning on attending Digital Hollywood next week? http://www.digitalhollywood.com/

If so, look for me at the pool side mixer at 5:00 Monday evening, I'll have an EdgeCast badge on.

take it easy, later,

Brent

Sorin Ostafiev said...

Hi Brent,

Exactly, pretty much everything is heavily cached on the client and because the public content is not increasing over time a CDN makes not much sense in such a circumstance. However, for the private part over HTTPS, I think a CDN might increase the user experience and maybe in the future we’ll give another try.

Thanks for asking, but I’ll not attend the Digital Hollywood conference. I’m living in Sweden and LA is quite far :)

Have a nice weekend,
Sorin

spenser said...

Hi Sorin,

Did you ever think about rolling your own? You have much more control that way.

http://geo.dnsmasq.com

Spenser