OSM Replication of Data

Over the years access to the OSM data set has got easier with various mirrors each with their own different slant. While working with the whole data set still requires substantial processing power, restricting local working to your immediate area makes things a lot more practical. For my own version of the OSM map server is restricted to the British Isles based on the extracts served up by Geofabrik and uses the .pbf file from the Britsh Isles Page.

What I had not initially spotted was that the replication process is also provided via that link. The .osc.gz files link provides the material necessary to set up a regular sync of the data using approximately daily update files. The link provides a matching state.txt file for the base .pbf extract, and allows a similar update cycle to be set up for the ongoing changes to this slice of the OSM data. The original setup was a little mixed up, but for this new build I've set up a separate disk as '/srv/maps' and am basic all of the data on that base folder, so I have a /planet folder with the base .pdf file along with it's state.txt file, and through which all the updates flow.

# original OSM minutely replication sequence number 1520317
timestamp=2015-08-09T21\:22\:02Z
sequenceNumber=880

To set up the basic replication on this ...

osmosis --read-replication-interval-init workingDirectory=/srv/maps/planet

This builds a configuration.txt, but we need to modified the default baseUrl to match the Geofabrik source. This will then allow subsequent updates to be downloaded and merged with the database.

# The URL of the directory containing change files.
baseUrl=http://download.geofabrik.de/europe/british-isles-updates

# Defines the maximum time interval in seconds to download in a single invocation.
# Setting to 0 disables this feature.
maxInterval = 3600

Once initialized, each call to check for new updates will compare the local state.txt with the current one on the service. The following pair of commands will update changes.osc.gz if necessary, and then process that into the postgresql database. If there are no new updates then changes.osc.gz is just an empty file.

osmosis --read-replication-interval workingDirectory=/srv/maps/planet --simplify-change --write-xml-change changes.osc.gz
osm2pgsql --bbox -14.5,49,3,61 --append --slim changes.osc.gz

Since the Geofabrik updates are only generated daily, a switch to the http://planet.openstreetmap.org/replication/minute/ update source would provide a much faster update service. This is where the first line of the base state.txt file comes in. By switching the sequenceNumber= entry to match the 'minutely replication sequence number' we can then switch back to the 'http://planet.openstreetmap.org/replication/minute/' as a baseUrl in the configuration.txt file. This does have the problem of now working with update files from the whole planet, which are then filtered by the bbox entry in the osm2pgsql, but filtering the data does not add too much delay to the process, and only filtered data is actually then added to the database.

This should be taken in conjunction with Local Tile Server Setup Take 2 which handles adding the necessary software packages and creating an empty database to process the data into. What still needs to be added to this crib sheet is how it find the trim box for different areas, but it's unlikely I will be outside the UK any time son.