On Wed, Sep 15, 2010 at 17:10, Maxime Petazzoni
<address@hidden> wrote:
Another solution would be read-only access to a kept up-to-date OSM
database.
Not sure how this would be done nor its impact, just throwing the idea
out there for discussion.
This is just my thinking 'aloud', so to speak.
1. pgsql stores the database in e.g. /var/pgsql/ocitysmap
2. at 00:00, finish any current job if running, then shut down postgres
3. mv /var/pqsql/ocitysmap /var/pgsql/ocityold
4. mv /var/pgsql/ocitynew /var/pgsql/ocitysmap
5. restart pgsql and start processing jobs again
6. meanwhile at low io prio: rsync address@hidden:/somewhere /var/pgsql/ocityold
7. mv /var/pgsql/ocityold /var/pgsql/ocitynew
The other machine 'somehost' can keep the database up to date, just making sure it's in a known good state by the time the public server comes for its rsync.
That or you could have this other server be the initiator of the rsync update, putting a semaphore file on the maposmatic server when it's done; this public server would then upon seeing this file finish any running job, move those directories around, and restart postgres.
Then again, I'm not a postgres wizard, so perhaps postgres' native hot copy, failover, replication (et al) functionality would give similar or less load on the machine and be preferred. It could well be that doing the database upgrade on a 2nd machine and streaming replication to maposmatic as it's running is less of a load than it doing the update itself? For one the other machine would be running all of the queries an update requires, the maposmatic machine would be fielding mostly updates and inserts.
Perhaps some performance can be eked out of map rendering being primarily read only queries.
Another thought is to ask the authors of the update tool if it can delay the actual updating to near the end of the process and write all the updates at once in a single transaction. The database would be doing mostly reads until then and keep more rows around in cache that any rendering job at the time might make use of as well.
For all I know i'm talking absolute bollocks, but if someone better versed in postgres and mapnik comes away inspired, it's been a good use of 5 minutes for me. ;)
Also... have you thought to ask the guys at geofabrik how they keep things up to date? Is it solely throwing hardware at the problem, or did they do something clever with configuring both postgres and the update jobs too.
Regards,
Jeroen.