Re: [Maposmatic-dev] daily update stats

[Jeroen van Rijn]

On Wed, Jan 6, 2010 at 17:47, David Decotigny <address@hidden> wrote:
>
> Hello,

Hi David,

> (05:11:04 PM) d2: From the "update" log above, we can see that the daily
> update takes around 16hr, due to the "normal" requests being rendered
> (05:12:07 PM) d2: it means that we have 8hr "free" of update every day. it
> also means that we have 8hr margin between 2 daily updates. if the daily
> updates grow, we will eventually reach a major problem. this could happen
> very soon...
>
> As of now, almost 500 jobs in the queue. Current rendered job submitted
> yesterday night 11pm... that's around 19h delay.

Question: Is there a comparison between the time it takes to run
osm2pgsql with and without jobs being rendered at the same time?
Likewise, how long does a job take when run at the same time as the
update process, and how long does it take without?

If it takes for example only half the time when the update is run when
there's no rendering activity, it may make sense to schedule the job
to be run from midnight until how-ever-long-it-takes, and to suspend
the rendering queue until it's completed. As long as it's stated
outright that jobs will only be processed during the hours of ... and
... (and saying this is because the map is kept up to date in the
downtime), I can see people accepting this.

As an added benefit the jobs themselves ought to run quicker when the
not contending with the update process for the database. It could be
that pgsql can multiplex the updates and queries so well that the
difference is marginal, and you'll find that osm2pgsql wouldn't run
all that quicker. If on the other hand there's a significant
bottleneck in running this concurrently, there's something to be said
for suspending the render queue.

Alternatively it may be interesting to investigate if the daily update
can be split into parts, to be run at times of lower load; if the
entire update hasn't finished by some deadline, *then* suspend the
rendering queue and run the rest of the update.

Regards,
Jeroen.