Network latency experiments

Recently a series of blog posts by Jim Gettys has started a lot of interesting discussions and research around the Bufferbloat problem. Bufferbloat is the term Gettys’ coined to describe huge packet buffers in network equipment which have been added through ignorance or a misguided attempt to avoid packet loss. These oversized buffers have the affect of greatly increasing latency when the network is under load.

If you’ve ever tried to use an application which requires low latency, such as VoIP or a SSH terminal at the same time as a large data transfer and experienced high latency then you have likely experienced Bufferbloat. What I find really interesting about this problem is that it is so ubiquitous that most people think this is how it is supposed to work.

I’m not going to repeat all of the details of the Bufferbloat problem here (see bufferbloat.net) but note that Bufferbloat occurs at may different places in the network. It is present within network interface device drivers, software interfaces, modems and routers.

For many the first instinct of how to respond to Bufferbloat is add traffic classification, which is often referred to simply as QoS. While this can also be a useful tool on top of the real solution it does not solve the problem. The only way to solve Bufferbloat is a combination of properly sizing the buffers and Active Queue Management (AQM).

As it turns out I’ve been mitigating the effects of Bufferbloat (to great benefit) on my home Internet connection for some time. This has been accomplished through traffic shaping, traffic classification and using sane queue lengths with Linux’s queuing disciplines. I confess to not understanding, until the recent activity, that interface queues and driver internal queues are also a big part of the latency problem. I’ve since updated my network configuration to take this into account.

In the remainder of this post I will show the effects that a few different queuing configurations have on network latency. The results will be presented using a little utility I developed called Ping-exp. The name is a bit lame but Ping-exp has made it a lot easier for me to compare the results of different network traffic configurations.

Continue reading

The end of Google Wave

On May 28th, 2009 Google announced Wave to much fanfare. Wave was going to change the world by merging blogs, wikis and IM and finally replacing email as the digital world’s main collaborative tool.

On August 4th, 2010, 14 months later, Google announced they have stopped working on Wave due to a lack of users.

Despite my interest in the underlying technology, I never used Wave for anything productive. The main reason for this is that I didn’t know anyone else using it regularly. Of course this is the catch-22 faced by every new communications technology and it is at the heart of Wave’s failure. The underlying problem is that Wave didn’t add enough value for users working on a document alone. Wave’s revolutionary power would have eventually come from its collaborative features but the problem with focusing on these aspects early is that they do not add any value until there are users to collaborate with. In episode #68 of This Week in Startups, Marco Zappacosta defines the amount of value a service brings to users without network effects as network independent value (NIV). Wave had next to zero NIV.

It didn’t have to be this way. There are several aspects of Wave that could have been focused on to create a superior or at least unique single user experience. The most obvious would be to have made basic document editing and management a better experience. Or they could have focused on use cases not supported by other document systems. The Wave API allows for third party applications (bots) to contribute to documents at the same time as users. This could have been used to build bots which automate tasks that are time consuming or annoying in Word or Google Docs. On the collaborative side, Google could have concentrated on a specific, practical use case such as a shared white board (there are several bots which do white-boarding) or built interesting applications such as Gravity by SAP.

Wave’s failure saddens me because Google was really doing this right. They were publishing the protocol specifications and most of the source code. Even more importantly they architected Wave based on a distributed/federated model which allowed for Wave servers to exist within every organization just like email servers do. This is a much harder problem than a centralized, all Google architecture but it is critical that single organizations do not control (what could have become) a core Internet protocol.

One also has to wonder where the revolutionary innovation that is required to replace email will come from if even Google gives up after only 14 months. Wave represents a large change that requires time to diffuse and for infrastructure to be built up around it. It is completely unreasonable to expect that Wave would have had large success in just 14 months. One has to feel for the Wave team whom it seems were given the impossible mission of changing the world in 14 months.

I believe that years from now we’ll look back on Google Wave and realize that it was closer to the solution than we thought. One of the key features that makes me believe this is the bot API. The idea of allowing third party applications equal access to a live document is very powerful and could spawn huge amount of innovation. For example, there is no competition to Microsoft’s grammar checker in Word or Google’s spell checker in Google docs. There cannot be as these are functions which are part of the application. Now imagine a world where a document system like Wave is the norm. Any user could select which spell checker to use just by adding a different bot to the Wave. I believe this flexibility would spawn competition that would drive a great deal of innovation. This speaks to the power of decentralized systems such as Wave.

There may also be an opportunity for a startup or open source project to take the Wave ecosystem and run with it. A lot of the hard work has already been done.

Blackberry Torch

I really hope that RIM has a successful device in the Torch and Blackberry OS 6. It would be such a sad story for technology in Canada if RIM continues to ride the slow train to irrelevance.

That said, what is the deal with having a touch screen, track pad and a keyboard?

This screams weak, unprincipled design. Take a stand! Lead instead of trying to mash together the best of every device on the market into some Franken-input system.

Next is Now

Just stumbled on this video created by Rogers.

One of a few good quotes:

“10 years ago it took 72 hours to download Godfather… – Today it takes 10 minutes – It still takes 3 hours to watch”

ChangeCamp London

Thanks to the organizers and participants involved with ChangeCamp London yesterday. It was amazing to see such a strong turnout of people interested in making London better. I hope everyone got as much out of it as I did.

For anyone who couldn’t attend, you can get a feel for the event by looking at the #ccldn tweets and following up with the actions which will be posted on the website.

Bank websites

It boggles my mind that the banks haven’t figured out how to make their websites more useful than they are. Why are these sites limited to online versions of bank tellers?

Here’s what I want my bank’s site to provide:

  • Allow me to categorize or tag every transaction. I want to mark that Subway is “Eating out”.
  • Learn from previous transactions and automatically suggest classifications for me. If I marked “89328374 Ontario Inc.” as car repairs two months ago there is a good chance it goes into the same category this month.
  • Summarize the totals for each category and show trends. Am I spending more on eating out every month?
  • Allow me to set targets or thresholds for each category and send me notifications if I cross them. If my budget is $300/month for eating out I want to know when I’m beyond that.

The banks already have access to all of my account activity and I don’t want to provide a third party, especially one outside of Canada, with my online banking credentials to get this functionality.

Why would a bank build this?

  1. It provides an incentive to move all of your accounts to one bank.
  2. I’d switch banks to get these features so I’m sure others would too.

Books

I recently finished reading a couple books that are worth mentioning.

The first is Cory Doctorow’s Little Brother. Don’t let the teen categorization discourage you, this is a book any technology geek will love. The best one line summary I can think of is that this book is a modern version of Nineteen eighty-four with the addition of a little teen rebellion. The story and characters are very well written and interwoven are decent introductions to important security techniques such as public key encryption and onion routing. Targetting this book at teen readers was a brilliant idea. I can’t think of a better way to help them understand what a security state would feel like.

The second book is Who Controls the Internet?: Illusions of a Borderless World by Jack Goldsmith and Tim Wu. No, this isn’t some kind of conspiracy theory book. The answer put forward by the authors is basically that territorial governments control the Internet despite the Utopian non-governmental ideal that many technologists associate with the Internet. The source of this control comes from the fact that governments are alone (mostly) in their ability to use physical coersion. While this in of itself isn’t that earth shattering I found the suggestion that online businesses such as eBay are reliant on the excercise of this coersive power very interesting. Even if the Internet isn’t that interesting to you this book is worth reading for the discussion of the role of government in commerce.

I’ve also recently read Switch and Linchpin which both deserve short reviews but that will have to be some other time.

Django/mod_wsgi on Fedora 12

I recently deployed a Django application with mod_wsgi on my server which runs Fedora 12. Since this required a bit more configuration than a standard Apache virtual host I thought it might be useful to document the configuration for others.

SELinux

While SELinux can be a little annoying if you don’t understand how it works it is a very powerful security layer that should not be disabled. In order to get the Django/mod_wsgi application working I had to modify a couple of SELinux booleans which give Apache extra permissions.

setsebool httpd_tmp_exec on
setsebool httpd_can_network_connect on

mod_wsgi configuration

The default configuration tries to create the mod_wsgi sockets in a directory that SELinux does not allow Apache access to. You can change this by adding the following line to /etc/httpd/conf.d/wsgi.conf.

WSGISocketPrefix run/mod_wsgi

Apache virtual host configuration

Below is the Apache virtual host configuration. Note that I have chosen to use mod_wsgi’s daemon mode and processes instead of threads because some of the libraries I’m using are not thread safe.

<VirtualHost *:80>
 ServerAdmin dan@example.com
 DocumentRoot /home/vhosts/example.com/
 ServerName www.example.com

 Alias /robots.txt /home/vhosts/example.com/example/web/static/robots.txt
 Alias /favicon.ico /home/vhosts/example.com/example/web/static/favicon.ico

 # Static files.
 Alias /static /home/vhosts/example.com/example/web/static

 # Admin static files.
 Alias /media /home/vhosts/example.com/dependencies/Django-1.2.1/django/contrib/admin/media

 WSGIScriptAlias / /home/vhosts/example.com/example/web/example/django.wsgi
 WSGIDaemonProcess example.com processes=15 threads=1 display-name=%{GROUP}
 WSGIProcessGroup example.com

 ErrorLog logs/example.com-error_log
 LogFormat "%a %l \"%u\" %t %m \"%U\" \"%q\" %p %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" custom_log
 CustomLog logs/example.com-access_log combinedio
</VirtualHost>