Robin Clarke – Life and Tech

I have a Garmin Legend HCx and have recently mounted it on my bicycle again.  I’ve even got local maps on it, so it can even do navigation by following the road, but the options for route planning on the Garmin are somewhat limited…  While Google Maps still has some shortcomings with its directions for walking or cyclists, it has one great advantage: you can grab and drag the route as you please to quickly make a path just the way you like it.  But how can you get that route to your GPS?

Here’s how:

• In Google Maps, get the directions you want.  Drag the path, click till you’re happy.
• In the top-right of the screen you’ll see the “Link” link – click on that, and copy the URL to your address bar

• Now add the string “&output=kml” to the end of the url, and hit enter

• A download dialogue should now pop up asking you where you want to save the file maps.kml – save it
• Now use gpsbabel to transfer the track to your GPS.  For my Garmin it works like this:

gpsbabel -i kml -t -f maps.kml -o garmin -F usb:0

And that’s it… 🙂

I recently had the opportunity to review web monitoring services.  I was looking specifically for a service which would test a web application, as if a real user was testing it: including DHTML/AJAX, in a real browser.  There are a number of services such as serverguard24, and uptrends which do allow transactions (a complex sequence of requests), but they don’t run their macros in a real browser, and so aren’t enough like a real user.  Then I found alertfox!

Alertfox

To use AlertFox, you first have to install the iMacros plugin for your browser (available for Firefox and IE).  This makes recording a macro (sequence of clicks and data entries) a piece of cake.  You can choose to have clicks recorded by the id of the element you are selecting, or by the X/Y coordinates.  Once you have recorded a transaction (you may need to edit the script to replace session ids with wildcards (*)), you can create a “sensor” with it on the AlertFox site.  A Sensor is a rule set including the macro which should be carried out, how often it should be executed, in which browser, and what the acceptable time is that this transaction should take.  You can even define from which location the test should be carried out (“Europe Zone” and “World Zone” connections both currently come from servers hosted by serverloft in Germany, “US Zone” connections come from theplanet servers in Houston Texas).

When things go wrong: If the transaction cannot be completed (server is too slow, a page in the sequence is not as expected, …), you immediately get an email warning with a link to a screenshot.  This is great!  Setting up a host of these sensors I feel like I’ve got a horde of tireless monkeys working day and night clicking, typing, and making screenshots.  On your AlertFox dashboard you have a good overview of all your sensors, when they were tested, and what the results were each time.

One small annoyance: you often won’t be able to use a macro recorded in Firefox for IE – there are small differences in how they recognise tags which sometimes break cross compatibility.  Solution: record the same transaction twice – once in Firefox, and again in IE.

Why outsource monitoring?

You might wonder what the point is of outsourcing your web application monitoring: surely you can monitor server load in-house.  This will cover many causes of downtime, but what if your server is running fine, but network interruption has made it unavailable from outside?

So you do need to have monitoring from a remote server… Maybe you have another server and can make both of your servers monitor each other, or maybe you want to use a basic monitoring service which makes simple http requests to test if your server is online.  This is better – you will know if your server goes offline, but that doesn’t cover some of the most common problems which could occur…

One of the typical errors on a web application which won’t be caught by any of the above methods is your payment system breaking.  A software error, or a glich in the payment fulfillment provider… it will allow users to visit your site, put items in the shopping cart, but not complete the checkout.  Your burning marketing money bringing them to your site, and no cash is coming in.  The only way to be sure your site is working is to test real transactions in a real browser, but … monkeys aren’t that clever, and employing slave labour will probably get you in trouble.

If you are an online marketer it is quite likely that you are responsible for driving traffic to a site, monitoring and optimising your campaigns without having any control over the site you are working for.  If the site goes down or a section is broken you want to know asap so that you can inform your customer and stop burning marketing budget on a broken site.

What does it cost?

They have 4 packages ranging between free (only one Macro, and that only on Firefox) and $199.  I used the PRO2 version for $99/month which is the cheapest version which allows testing in IE and Firefox.

Links

I spent all my time talking about AlertFox because it’s the only one which fulfils my requirements, but there are many other monitoring services which may be up to your needs if you don’t need application level monitoring (DNS, HTTP, FTP, …), and here’s a list of some of the others I looked at:

• Serverguard24
• Alertra
• Uptrends
• Wormly
• Alertsite
• Site24x7
• Dotcom-monitor
• Pingdom
• Webmetrics

A typical problem which many web service providers have is that when they transfer a website from one hosting to another, there is an inevitable period of time when the application has been transferred, but the DNS changes have not propagated, and so some users will experience the site as being offline.  You can reduce the effects by reducing the TTL for your domain prior to the switch, but I prefer this method.

The key to this method is Varnish.  Varnish is a state of the art reverse proxy, and you should be using it to accelerate your web applications, even if you don’t need to switch service provider.  Explaining Varnish in detail goes beyond the scope of this howto, but in summary, clients connect to the Varnish proxy on the server, which (based on a vcl configuration file) either servers from its cache, or forwards the requests to backend servers.  This can allow you to have a farm of web servers all with one frontend, or just take the load of your backend servers by efficiently serving static content .  Varnish also allows complex filtering, and this is what makes the transition from one service provider to the next a piece of cake.

For the following I’ll use the terms “old” as your old ISP server, and “new” for your new ISP server.

Here’s the basic steps:

• Set up Varnish in front of your web server redirecting all requests to the local backend.
• Add a new backend in your Varnish configuration, defining your new IP as the host.
• For each application (typically a domain):
  • Copy files and databases from old to new
  • Install the files and databases on new.
  • Test the new server (make a static entry for the domain(s) in your hosts file)
    • Are all required modules installed?
    • Is everything working as expected?
  • Add a rule in your Varnish configuration that will forward all requests to this domain to your new server.
  • If you have a very active site, you may still have a few seconds of downtime in this step…
    • Put the web application into maintenance mode
    • Dump the database from old again (to get latest changes since your testing)
    • Load the database into new
    • Reload the varnish configuration to activate the forwarding to the new server
    • On the new server, take the application out of maintenance mode
  • Now all requests which hit your old server for this domain should be invisibly forwarded to your new ISP
  • Check that everything is working
  • Change the DNS entries for the domain – as this propagates, less traffic should be hitting your old ISP
• When you see no more traffic hitting your old server, you can turn it off.

Using this procedure, you can comfortably move any number of applications with minimal downtime, and enough time to test each one as you go.  It also will allow you to transfer applications where you do not have control of the DNS entries: you can do the transfer, and then inform the domain owner to change the ANAME entrie for their domains.

Some situations where this might cause problems:

• Your application has IP based filters or bot-blocking active – if so your new server will be getting a lot of traffic from one address (old server)
• Your application is not compatible with Varnish?  Some web applications (Squirrelmail for example) generate many unnecessary or duplicate HTTP headers.  Varnish chomps everything after the first 32 lines of HTTP headers, which in turn can cause the application to fail if something important was after the 32nd header…