Monthly Archives: April 2011

2011/04/12
2 Comments
ign, mapping

How IGN tile servers work?

In my previous blog article I introduced an IGN – Google Maps integration library. In this post I want to explain how IGN map server interface.

IGN ‘chops’ the world into small square tiles 256 x 256 pixel each. The size is equal to Google map tiles. Image format is JPG (Google uses PNG images).

There are 5 tile servers (as for 2011.04.11) – tsN.iberpix.ign.es, where 0 <= N <= 4. The tile URL is composed in the following manner:
http://ts|tile server index|.iberpix.ign.es/tileserver/n=|map type|;z=|UTM zone|;r=|map scale|;i=|tile x coordinate|;j=|tile y coordinate|.jpg

where:

  • tile server index – as commented before
  • map type – topo map identifiers useful for outdoor activities:
    • 1:1,000,000 – mapa_millon
    • 1:200,000 – mapa_mtn200
    • 1:50,000 – mapa_mtn50
    • 1:25,000 – mapa_mtn25
  • UTM zone – just the number, without hemishere designator and latitude band. Mainland Spain range: 29-31, Canary Islands: 27-28
  • map scale – Map scale pixels per meter multiplied by 1000. Must be a power of 2. The highest value that works (according to my checks) is 2048000 (2,048 m/px), the lowest is 1000 (1 m/px)
  • tile x-y coordinates – cartesian coordinates of a tile (in a propietary IGN coordinate system). More on this later in this article

An example of a tile: http://ts0.iberpix.ign.es/tileserver/n=mapa_millon;z=30;r=256000;i=5;j=68.jpg

How to calculate tile (cartesian) coordinates?

IGN uses UTM (Universal Transverse Mercator) coordinate system indirectly to calculate tile (x, y) references.

Flash introduction to UTM – it “flattens” the earth surface into zones (where we assume that the surface is not spherical but flat). To reference an earth point with an UTM coordinate, you need three numbers: UTM zone (reference to the grid rectangle), x and y. x gives you the distance from the central meridian of each zone (“false easting”) in meters. The zone meridian has a fixed (artificial) value of 500,000 (to avoid negative coordinates). y gives you the distance from earth’s equator (in meters too). As you see, using UTM is useful to get a grasp of distances (if point A’s UTM coordinates are (500,000, 2,000,000) and B’s (500,000, 2,001,000) they are 1000 m = 1 km away).

The IGN tile coordinate is based on the UTM coordinate of the lower left pixel and map scale (in meters per pixel). Lower left pixel of the (0, 0) IGN tile corresponds to (0, 0) UTM point. Since each tile has 256 pixels, it represents the world fragment in a scale and UTM coordinate unit is meter, (1, 1) tile’s lower left pixel corresponds to:

(1 * 256 [px] * scale [m/px], 1 * 256 [px] * scale [m/px]) UTM point.

e.g. for 128 m/px scale, (1, 1) tile’s lower left pixel UTM is (32768, 32768).

generalizing, for given a (i, j) tile and scale, lower left pixel UTM is:

(i * 256 [px] * scale [m/px], j * 256 [px] * scale [m/px])

To calculate tile IGN coordinates (given a tile scale) that cover a point with given UTM coordinates:
(floor(x / (256 [px] * scale [m/px])), floor(y / (256 [px] * scale [m/px])))

Notice that x UTM coordinates can be negative and thus tile coordinates. UTM (-1, 4000000) point (lat-lng: 36.01619ºN, 8.54615ºW) in the 30N zone is legal, although it lies outside of the zone (which spans from 6ºW to 0º). It’s supported by the IGN tile server – it can return tiles covering areas outside of an UTM zone, although with higher projection errors.

2011/04/11
scala

March kata – FizzBuzz

The March kata was FizzBuzz. This is my third kata done in Scala. For the full source code, go to GitHub.

class FizzyBuzzyInt(i: Int) {
  def isFizzy = divisibleBy(3) || contains(3)
  def isNotFizzy = !isFizzy
  def isBuzzy = divisibleBy(5) || contains(5)
  def isNotBuzzy = !isBuzzy

  private def divisibleBy(divisor: Int) = i % divisor == 0
  private def contains(n: Int) = i.toString contains n.toString
}

class FizzBuzz {
  implicit def extendInt(n: Int) = new FizzyBuzzyInt(n)

  def printSequence(out: PrintStream) = {
    (1 to 100).foreach(n => out.println(sayOnNumber(n)))
  }

  private def sayOnNumber(number: Int): String = {
    number match {
      case n if (n isFizzy) && (n isNotBuzzy) => "Fizz"
      case n if (n isBuzzy) && (n isNotFizzy) => "Buzz"
      case n if (n isFizzy) && (n isBuzzy) => "FizzBuzz"
      case _ => number.toString
    }
  }
}

Like for Roman Numerals I used Scala’s pattern matching. I find it extremely powerful, especially if you have a Java/C++ background. I suppose I’m obeying Maslow’s Law of the instrument (“if all you have is a hammer, everything looks like a nail”). Anyway, this is the purpose of a kata – to play with language capabilities on a safe playground.

To use the hammer ever more, I want to implement fizzy and buzzy predicates using Scala’s case classes and combine them with the pattern matching.

2011/04/04
9 Comments
amazon, backup, linux, s3

Configure Amazon S3 backup with Backupninja and duplicity

I’m responsible for launching a new web page for Pedalibre, one of the oldest and biggest Madrid cyclist associations. It uses Drupal as CMS running on Apache httpd and storing everything in an MySQL database. We launched a proof-of-concept version on Amazon EC2 in the cheapest instance (micro, US-east zone). Works smoothly. Now a content work group must migrate the articles from the old site to the new one. Before that, I need a backup system.

My backup plan was the following:

  • create an image (EC AMI). In case of instance crash, I can restore the instance from the AMI and apply backups
  • backup Apache and MySQL configuration (in principle the entire /etc)
  • backup Drupal installation directory (/var/www)
  • backup Drupal database

I checked Google, but I didn’t find any article or a blog post describing how to setup a complete backup using Amazon ecosystem and Amazon Simple Storage Service (S3) in particular. I had to mix various articles and finally reached a solution.

What I used

  • Backupninja – for MySQL database backup and backup orchestration
  • duplicity – filesystem backup (Amazon S3 capable)
  • dt-s3-backup (check also author’s blog post) – shell script performing a backup to an S3 bucket using duplicity
  • s3cmd – command line tool for managing data in Amazon S3; required by dt-s3-backup

Step 1. Install the tools

apt-get install backupninja s3cmd duplicity
download zip from http://github.com/thornomad/dt-s3-backup/zipball/master
extract dt-s3-backup.sh script

Remeber to run everything as super user.

Step 2. Configure Backupninja schedule

Check Backupninja configuration (/etc/backupninja.conf) if it suits your needs. The most important parameter is when – it controls the backup schedule (check the documentation for possible values). I left it with the default value (everyday at 01:00). If you want to get backup report, fill at least reportemail parameter value.

Step 3. Backup MySQL database

Create a Backupninja backup action for MySQL database dump. The easiest way is to start ninjahelper tool and go through create a new backup action/mysql database backup wizard (no tough questions asked :) ). Ninjahelper creates the action configuration in the /etc/backup.d/20.mysql file.

Perform a test:

backupninja --now --debug

Check if there has been no error messages and if the dump has been created in /var/backups/mysql/sqldump.

Step 4. Configure s3cmd

I assume that you are registered in Amazon Web Services platform and you have the Access Key ID and the Secret Access Key.

Type: 

s3cmd --configure

When asked, provice Access Key and Secret Key. At the end of the configuration process, s3cmd will do the configuration test.

Step 5. Create an S3 bucket

In order to store your backup archives on Amazon S3, you have to create a bucket (storage space). You can use AWS Management Console or s3cmd:

s3cmd mb s3://my-backup-bucket

Step 6. Generate a GPG key (if you don’t have one)

Duplicity encrypts your backups with GPG. If you don’t have the GPG key or you want to create a new one for the backup purposes:

gpg --gen-key

and go though the wizard. Check if it has been created and write down the GPG key ID:

gpg --list-keys

pub   2048R/<key ID> 2011-04-03

Step 7. Backup file system and transfer the archive to S3 (MySQL backup included)

Now, edit Damon’s S3 backup script (dt-s3-backup) and fill configuration parameters. The inline documentation is self-explanatory; here I listed only values I had to adapt:

export AWS_ACCESS_KEY_ID="<your AWS Access Key ID>"
export AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>"
export PASSPHRASE="<GPG key passphrase>"
GPG_KEY="<GPG key ID>"
ROOT="/"
DEST="s3+http://my-backup-bucket/"
INCLIST=( "/etc" "/var/www" "/var/backups/mysql" )
EXCLIST=()
STATIC_OPTIONS="--full-if-older-than 7D"
CLEAN_UP_TYPE="remove-older-than"
CLEAN_UP_VARIABLE="14D"
LOGDIR="/var/log/duplicity/"
LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt"
LOG_FILE_OWNER="root:root"
VERBOSITY="-v4"

Test the backup script:

./dt-s3-backup.sh --backup
s3cmd ls s3://my-backup-bucket

You should get the a list similar to this one:

2011-04-04 15:21   1219566   s3://pedalibre-backup/duplicity-full-signatures.20110404T152112Z.sigtar.gpg
2011-04-04 15:21       801   s3://pedalibre-backup/duplicity-full.20110404T152112Z.manifest.gpg
2011-04-04 15:21  17990849   s3://pedalibre-backup/duplicity-full.20110404T152112Z.vol1.difftar.gpg

Check if you can restore from the backup (quite important issue :) ):

mkdir /tmp/restore
./dt-s3-backup.sh --restore /tmp/restore

If everything works fine, orchestrate duplicity S3 backup from Backupninja. In /etc/backup.d, create a new action, e.g. 30.filesystem_to_s3.sh with this content:

<path to dt-s3-backup.sh>/dt-s3-backup.sh --backup

Don’t forget to set the owner and group to root:root and permissions to 700. The numerical prefix must be higher than of the MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (/etc and /var/www), is pushed to the S3 bucket.

Step 8. Test everything together

backupninja --now --debug

and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.

Es todo amigos!

2011/04/01
2 Comments
googlemaps, ign, jasmine, javascript, mapping

IGN – Google Maps integration library

What is IGN?

IGN stands for Instituto Geográfico Nacional (National Institute of Geography), a public institution in Spain. Its functions include maintaining spatial data and to producing topographical maps of the country.

What maps does the IGN offer (that are interesting for outdoor activities)?

  • topo maps in 1:1,000,000, 1:200,000 y 1:50,000 and 1:25,000 scales
  • satellite imagery from Landsat 7 and SPOT 5 satellites
  • PNOA aerial imagery (PNOA – Plan Nacional de Ortofotografía Aérea, National Plan of Aerial Orthophotography)

How can I get the maps?

You can see rendering differences between SigPac and Iberpix on the picture below:

Why another viewer?

My idea was to display IGN map tiles on Google Maps canvas by implementing a custom (Google Maps) map type. In this way I can use Google Maps API to draw waypoints, paths, etc. If you mix it with Garmin Communicator Plugin API, you get a full-blown outdoor trip planning tool like Garmin MapSource, written entirely in JavaScript. Ok, there are maps of Spain for Garmin devices (Garmin’s Topo Spain, free Topo Hispania), but they IGN maps are more vivid and differentiate better between paths, tracks and forest roads.

What is IGN-Google Maps integration library?

Currently, it is not a full-feature viewer. The library offers a custom google.maps.ImageMapType, consisting of a google.maps.Projection and google.maps.ImageMapTypeOptions

I took a different approach than taken e.g. by Wikiloc. I didn’t want to make the IGN tile layer interchangeable with maps offered by Google. It means that for a given zoom or scale, you cannot switch between a Google map and IGN map and get the same view. This is possible in Wikiloc, but at expense of correct visualization and smooth zooming. I wanted just to have an independent IGN map viewer implemented in Javascript (ok, you depend now on Google Maps API that is closed source).

The source code has a comprehensive test suite implemented with Jasmine. Doing BDD on a broader scale and in Javascript was a completely new experience for me. With Jasmin it makes fun – the library is intuitive to use and well documented. As a bonus you get a mocking framework – no extra libraries required.

There are some TODOs in the project (they are described in README.md). On the next days I’ll try to fix them, depending on my spare time.

Where can I get the library?

I just published the very first version on the Github: https://github.com/mgryszko/ign-gmaps. The API is not stable yet and may change in the future.

How is it licenced?

MIT licence. You are free “to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software” = fork it and hack on it.

What are next steps?

I’d like to use it in Abredatos 2011 contest. Abredatos is a 48-hour programming challenge. You are supposed to developed a working application making use of at least one Spanish public datasource. It will be held on May 7th-8th 2011.

My idea is to develop the previously mentioned track & route manager with IGN maps and Garmin integration. If you want to participate, drop me an email/tweet.

Follow

Get every new post delivered to your Inbox.

Join 318 other followers