A personal (but techy) project I’ve been doing on the side recently has just reached a milestone.
Some time in the next year we are going to move out of the city centre.  I want to live as far away from the concrete jungle, rumbling juggernauts and grumpy people as I can, but… it would be nice not to be too far away from friends, activities and shopping….

There are many parameters involved (and in time I will put them all on a map!), but probably the most important one is that our new home must be within a reasonable transit time by public transport of the city centre.  But reasonable is relative… if the new place is great it could be 40 minutes, for “ok” it would have to be less than 30, and “fantastic” might get us to stretch to 50 minutes.  The trips shouldn’t have a frequency (during the day) of more than 20 minutes either, otherwise we’d always be missing/waiting for a train…

So how can I visualise these regions on a map?

1. Write an interface to the MVV (the public transport provider in Munich) web interface.  Unfortunately they don’t have a documented API, but they do have a RESTful interface, and a Java interface already exists (thanks Andreas!).  Between that and some further reverse engineering it wasn’t too difficult.  I’ll be publishing that module soon too. 🙂
2. Localise all the stops in the region, and determine the transit times, and frequencies from the central station (Munich Hauptbahnhof) to each of the stops.  Throw all that data into a database for easy access. (SQLite is your friend).
3. Write a Perl interface to Google Fusion Tables (there are interfaces in Python and PHP already, but I prefer Perl…).
4. Analyse the data, generate polygons around each stop indicating the distance you could walk (assuming 4kmh average walking speed) from that stop after having travelled from the central station, within a time limit (10 minutes, 20 minutes, …)
5. Convert the polygons to KML (that’s an XML variant which google uses for representing geographic data).
6. Upload the polygons to Fusion tables
7. Do some tweaking with the colouring so that the data is visually appealing and understandable.

For my next party trick I’ll be putting proximity to shopping centres, parks on the map, and possibly doing the same representation for transit time with a car.  Any other ideas what would be relevant, and pretty on the map?

The last couple of weeks I’ve been getting accustomed to Google Fusion Tables.

For those who haven’t yet noticed it – Fusion Tables is a new (still in Beta) Google product for data storage: a kind of database.

Some major cool things about Fusion Tables:

1. it’s really accessible (use the usual OAuth2 authentication as for all other Google services to script interaction), or use the web interface to manually edit your tables.
2. If you include geo data in your tables, you can easily (and I mean really easily!) display your data on a map.
3. They’re really clever about geo data… a text address, longitude/latitude, polygons, shapes, … dump it into a “Location” type field, and it will probably be understood.

Net::OAuth2 isn’t actually a new Perl module, but in its previous state it wasn’t really usable for command line applications, so I reworked it a bit.  Both of these modules are very raw/alpha – they are lacking in documentation, error handling, unit tests…. all the things one would expect in productive code… but I want to get them out there now so other people can start using and abusing them.

Here’s a wee screencast of pfusion in action:

And a short instruction to install (I hope it’s complete…):

git clone git://github.com/robin13/Net-OAuth2.git
git clone git://github.com/robin13/Google-Fusion.git
vi ~/.fusion

Edit to look like this, with the id/secret of your application.  If you haven’t got an id/secret yet, you’ll have to register an application for this purpose on the Google API Console and be sure to make it an “installed application” (not “web application”)

---
client_id: 372296649547.apps.googleusercontent.com
client_secret: XDy8Y8xTuwN90F3h7ljCuw4i

# And now run fusion:

perl -I ./Google-Fusion/lib -I ./Net-OAuth2/lib ./Google-Fusion/bin/pfusion

If authentication is successful, you should be able to interact with Fusion Tables to your hearts content, and should find a .fusion.auth file in your home directory with your current tokens (you only have to enter the authentication code the first time).

I hope you have fun with it, and give me some nice pull requests on those repositories!

A typical problem with a home server/NAS used by several people is that someone always turns it on, but who turns it off?  It stays on for days, and the electricity meter whirs happily as the kilowatts get unnecessarily burned…

A better strategy is to let the server shut itself down when it is not being used… intelligently… and this is how I made it happen:

How to determine if the server is not being used

This depends a lot on your setup.  In my case I have a Linux NAS with 6 disks.  One disk (/dev/sda) is the operating system (Linux) disk, and the other 5 are in a (software) raid array serving the shared filesystem which I and my flatmates and I use.  This makes it pretty easy to see if someone is using the system because if none of the 5 raid disks are active, then no one can be using them.  You can see if they are active with hdparm.

A brief introduction to hdparm

hdparm is a really handy linux utility for setting and reading hard disk parameters/status.  To set the spindown time of a disk to 20 minutes:

hdparm -S 240 /dev/sdb

To make your settings persistent, put them into /etc/hdparm.conf

/dev/sdb {
spindown_time = 240
}

To see the current state (active or not) of a disk:

hdparm -C /dev/sdb

Look up the hdparm documentation for all options.

How to shut down the server…

For this we need a daemon which monitors the hard drive status, and initiates a shutdown when the desired… Introducing: Shutdown Daemon (SDD).

This is a script I put together to do exactly what I described above.  For my setup I have used the following configuration:

---
log_level: INFO
log_file: /var/log/sdd.log
sleep_before_run: 60
monitor:
 hdparm:
 loop_sleep: 60
 trigger_time: 3600
 disks:
  - /dev/sdb
  - /dev/sdc
  - /dev/sdd
  - /dev/sde
  - /dev/sdf

This means it waits one minute before starting monitoring, checks the disk status every minute, and will wait for an hour of spun-down disk status before actually shutting down the server (unless one of the disks becomes active again within that time).

Installing SDD

Clone sdd from the github repository:

git clone git@github.com:robin13/sdd.git

Make it:

cd sdd
perl Makefile.PL
make
make test
make install

Copy the init.d script and config, and update the rc.d entries:

cp examples/etc/sdd.conf /etc/sdd.conf
vim /etc/sdd.conf
cp examples/init.d/sdd /etc/init.d/sdd
update-rc.d sdd defaults
/etc/init.d/sdd start

Saving power

In my case, just configuring the spindown on the hard drives reduces the power consumption of the NAS server from 44 to 28Watt.  Now factor in all those times where the server was forgotten and left on for days… the power saving potential is fantastic!

What other monitors could be useful?

Of course spindown time of the disks isn’t the only condition which could be useful for shutting down your server.  Here’s some others I’ve thought of:

• hard drive temperature too high
• No user logged in with ssh (users)
• No locks on samba file shares (smbstatus -L)
• …

If you have any ideas, feel free to fork the project on github and implement a Monitor – I’m looking forward to seeing many more monitors! 🙂