[Koha-zebra] Re: Import Speed

[Mike Taylor]

> Date: Thu, 2 Mar 2006 13:29:54 -0800
> From: Joshua Ferraro <address@hidden>
>
>> Why are you doing the search?  So far I can see, it's just a probe
>> to see whether the connection is still alive.  But you don't need
>> to do that: just go ahead and submit the update request, you'll
>> find out soon enough if the connection's dead and you can re-forge
>> it then if x.necessary.
>
> Here's what the connection manager looks like now:
> 
>         if (defined($context->{"Zconn"})) {
>                 $Zconn = $context->{"Zconn"};
>                 return $context->{"Zconn"};
>         } else {
>                 $context->{"Zconn"} = &new_Zconn();
>                 return $context->{"Zconn"};
>                 }
> So ... no search ... if one is defined it just returns it and if
> it's not alive I assume the app will just crash (no fault tolerance
> built into the script).

OK.  Of course, you can reintroduce robustness later by checking for
update-failure and re-forging the connection if the error-code
indicates this course of action.

> And here's the new benchmark for those 5000 records:
> 
> 5000 MARC records imported in 7727.84231996536 seconds

Hmm.  Well, compared with the previous truly astonishing time of 40604
seconds, that's a better than fivefold improvement, which is not a bad
start.  But, still -- more than one second a record, we still have
_plenty_ of scope for improvement here.

How busy is your disk now?

> So it's definitely better without the search, but there is still the
> question of XML ... being able to import raw marc (which would only
> take a few seconds) would be really nice ...

I agree with Seb that the XML is unlikely to be culprit here: the
actual indexing is the only thing I can think of that would show the
pattern you see of taking longer as the database grows.

 _/|_    ___________________________________________________________________
/o ) \/  Mike Taylor  <address@hidden>  http://www.miketaylor.org.uk
)_v__/\  It's hard to retro-fit correctness.