Re: [Maposmatic-dev] Database update becoming critical

[Maxime Petazzoni]

* Maxime Petazzoni <address@hidden> [2010-05-26 08:56:07]:

> But this is obviously not a perennial solution, and we should look to
> rapidly move to hourly our minute diffs using Osmosis. I'm currently
> investigation this solution in more detail. If one of you already had
> some information to share on Osmosis other than what's in task #10148
> <https://savannah.nongnu.org/task/?10148>, please do!

Apparently this process has been documented recently (April) on the OSM
wiki at http://wiki.openstreetmap.org/wiki/Minutely_Mapnik

Using Osmosis, we can fetch the exact changes we need to apply to the
database from any timestamp to "now", and append these changes to the
database with osm2pgsql. After a simple two command setup:

  mkdir ~/.osmosis
  osmosis --read-replication-interval-init workingDirectory=~/.osmosis

The update is then as simple as creating the state file from the last
update timestamp, and piping osmosis into osm2pgsql to perform the
update:

  wget http://toolserver.org/~mazder/replicate-sequences/?TIMESTAMP \
    -O ~/.osmosis/state.txt
  osmosis --read-replication-interval workingDirectory=~/.osmosis   \
    --simplify-change --write-xml-change | osm2pgsql -a [our custom args]

With TIMESTAMP being a UTC timestamp in the form YYYY-MM-DDTHH:MM:SSZ. A
very simple hourly crontab could be created from this. We'd have to
check of course that this process has a significant time advantage. I
believe the more frequent we make the update process, the faster it
could run: I don't think the processing time grows linearly with the
update interval (aka it takes way longer than 60 times what it takes to
process a minute diff to process an hourly diff).

- Maxime
-- 
Maxime Petazzoni <http://www.bulix.org>
 ``One by one, the penguins took away my sanity.''
Linux kernel and software developer at MontaVista Software

signature.asc
Description: Digital signature