How to build your own Map Service Part 2 - Auto Updating your database

Minutely Mapnik is the concept of keeping your PostgreSQL database in sync with changes made on the OpenStreetMap server. Changes made by OpenStreetMap user accounts are recorded in "replication files" by the minute and by the hour. The Osmosis java application is used to automate the process of downloading the proper "replication files" that are used to update your database with new changes. The Osm2pgsql application is then used to import those changes into your PostgreSQL database.

Lets start off by setting up our system to automatically import updates at various intervals

Creating the working directories for osmosis. This is where we will be storing all the logs and data from every run.

# mkdir /mnt/changesets

# /mnt/bin/osmosis/bin/osmosis --read-replication-interval-init workingDirectory=/mnt/changesets

This will create 2 files

* configuration.txt - This is settings file for osmosis
* download.lock - This is a lock file to prevent more than a single instance of osmosis

This is my sample configuration.txt file.

baseUrl=http://planet.openstreetmap.org/minute-replicate
intervalLength=60
maxInterval = 3600

We now need to create a state.txt file which has information of the current state of the database. For this to work you need to know the time stamp of the planet.osm file.

Get the timestamp of the planet.osm file.

/mnt/planet/planet-*.osm

<osm version="0.6" generator="OpenStreetMap planet.c" timestamp="2010-08-18T00:11:04Z">

Generating a state.txt file

# wget http://toolserver.org/~mazder/replicate-sequences/?2010-08-18T00:11:04Z -O /mnt/changesets/state.txt

Sample state.txt file:

#Wed Aug 18 00:24:08 UTC 2010
sequenceNumber=441796
txnMaxQueried=85523189
timestamp=2010-08-18T00\:09\:02Z
txnReadyList=
txnMax=85523189
txnActiveList=85517895,85521790,85523083,85523099,85523182

Once you have these three files in place you are ready to import updates from the OSM server.

We are going to setup a cron job that runs every hour to fetch and update your database. I would suggest creating a simple wrapper script instead of directly calling osmosis. Lets call this script update-db.sh

#!/bin/bash
 
if [ -e /tmp/osm_update.lock ]
then
    echo 'Update in progress'
else
    touch /tmp/osm_update.lock
 
    /mnt/bin/osmosis-0.36/bin/osmosis --read-replication-interval workingDirectory=/mnt/changesets/ --simplify-change --write-xml-change /dev/stdout | /mnt/bin/osm2pgsql/osm2pgsql -a -s -C 2048 -S /mnt/bin/osm2pgsql/default.style /dev/stdin
 
    rm /tmp/osm_update.lock
fi

This script will pull in the changes from the Open Street Maps server and push the changes to osm2pgsql without writting the data to the disk

Setting up crontab to run the udpate-db.sh script every hour.

# crontab -e
 
*/10 * * * * /mnt/changesets/update-db.sh

Note: You can also add MAILTO=user@example.com at the top of the file if you want the output to be mailed to you after every run.

Congratulations from now on your database will automatically fetch updates from the Open Street Map databases.