[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Koha-zebra] Re: Import Speed
From: |
Mike Taylor |
Subject: |
[Koha-zebra] Re: Import Speed |
Date: |
Fri, 3 Mar 2006 09:04:48 +0000 (GMT) |
> Date: Thu, 2 Mar 2006 13:29:54 -0800
> From: Joshua Ferraro <address@hidden>
>
>> Why are you doing the search? So far I can see, it's just a probe
>> to see whether the connection is still alive. But you don't need
>> to do that: just go ahead and submit the update request, you'll
>> find out soon enough if the connection's dead and you can re-forge
>> it then if x.necessary.
>
> Here's what the connection manager looks like now:
>
> if (defined($context->{"Zconn"})) {
> $Zconn = $context->{"Zconn"};
> return $context->{"Zconn"};
> } else {
> $context->{"Zconn"} = &new_Zconn();
> return $context->{"Zconn"};
> }
> So ... no search ... if one is defined it just returns it and if
> it's not alive I assume the app will just crash (no fault tolerance
> built into the script).
OK. Of course, you can reintroduce robustness later by checking for
update-failure and re-forging the connection if the error-code
indicates this course of action.
> And here's the new benchmark for those 5000 records:
>
> 5000 MARC records imported in 7727.84231996536 seconds
Hmm. Well, compared with the previous truly astonishing time of 40604
seconds, that's a better than fivefold improvement, which is not a bad
start. But, still -- more than one second a record, we still have
_plenty_ of scope for improvement here.
How busy is your disk now?
> So it's definitely better without the search, but there is still the
> question of XML ... being able to import raw marc (which would only
> take a few seconds) would be really nice ...
I agree with Seb that the XML is unlikely to be culprit here: the
actual indexing is the only thing I can think of that would show the
pattern you see of taking longer as the database grows.
_/|_ ___________________________________________________________________
/o ) \/ Mike Taylor <address@hidden> http://www.miketaylor.org.uk
)_v__/\ It's hard to retro-fit correctness.