[Koha-cvs] koha/misc migration_tools/rebuild_zebra.pl zebr...

[paul poulain]

CVSROOT:        /cvsroot/koha
Module name:    koha
Branch:         
Changes by:     paul poulain <address@hidden>   06/02/09 10:59:34

Modified files:
        misc/migration_tools: rebuild_zebra.pl 
        misc/zebra/unimarc: zebra.cfg 
Added files:
        misc/migration_tools: rebuild_zebra_idx.pl test_cql2rpn.pl 
        misc/zebra/unimarc: collection.abs pqf.properties 

Log message:
        OK, this time it seems to work. The last blocking problem was... a 
space in
        recordId: (bib1,Identifier-standard) just after the comma. Adam agreed 
it was a bug, and it should be solved soon. But now we are aware, we can avoid 
putting the space !
        
        In this commit you have all what is needed to setup a working zebra DB 
in Unimarc :
        * collection.abs is UNIMARC specific and must be rewritten for MARC21, 
in marc21 directory
        * pdf.properties is to be copied unmodified in the marc21 directory 
(can also be put somewhere else)
        * rebuild_zebra.pl is SLOW, but 1 step reindexing tool, using ZOOM
        * rebuild_zebra_idx is FAST, but 2 step reindexing tool, and does not 
use zebra. run it, it will create all biblios XML files in /zebra/biblios 
directory, then zebraidx update biblios in your zebra directory
        * zebra.cfg is the zebra config file ;-)
        * test_cql2rpn.pl is a script that will query the database and show the 
results. Works for me, just change the query at the beginning to get answers 
you expect.
        
        What has to be done :
        * benchmarking : it seems the zebraidx update is faster than lightning 
(400biblios/sec : 10 000biblios in 25seconds), while ZOOM indexing is slow 
(something like 25biblios/second) More benchmarking could be done.
        * completing collection.abs for UNIMARC. I'll take care of it.
        * modifying Biblio.pm to use ZOOM instead of the "zebraidx through 
exec" running actually. I'll take care of it also.
        * modify the search API & tools & screens. I'll let the ball to someone 
else (chris ?) for this. I agree SearchMarc.pm can be dropped and replaced by 
something else (maybe a new-and-clean Search.pm package)

CVSWeb URLs:
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/rebuild_zebra_idx.pl?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/rebuild_zebra.pl.diff?tr1=1.3&tr2=1.4&r1=text&r2=text
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/migration_tools/test_cql2rpn.pl?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/collection.abs?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/pqf.properties?rev=1.1
http://cvs.savannah.gnu.org/viewcvs/koha/koha/misc/zebra/unimarc/zebra.cfg.diff?tr1=1.5&tr2=1.6&r1=text&r2=text

Patches:
Index: koha/misc/migration_tools/rebuild_zebra.pl
diff -u koha/misc/migration_tools/rebuild_zebra.pl:1.3 
koha/misc/migration_tools/rebuild_zebra.pl:1.4
--- koha/misc/migration_tools/rebuild_zebra.pl:1.3      Tue Jan 10 17:03:32 2006
+++ koha/misc/migration_tools/rebuild_zebra.pl  Thu Feb  9 10:59:34 2006
@@ -41,11 +41,11 @@
 }
 
 # first, drop Zebra DB
-eval {
-       my $Zpackage = $Zconn->package();
-       $Zpackage->option(databaseName => 'Koha');
-#      $Zpackage->send("drop");
-};
+# eval {
+#      my $Zpackage = $Zconn->package();
+#      $Zpackage->option(databaseName => 'Koha');
+# #    $Zpackage->send("drop");
+# };
 
 eval {
        my $Zpackage = $Zconn->package();
@@ -56,19 +56,20 @@
 unless (opendir(DIR, "$cgidir")) {
                $cgidir = C4::Context->intranetdir."/";
 } 
-
 my $starttime = gettimeofday;
 my $sth = $dbh->prepare("select biblionumber from biblio");
 $sth->execute;
 my $i=0;
 while ((my $biblionumber) = $sth->fetchrow) {
        my $record = XMLgetbiblio($dbh,$biblionumber);
+#      warn "\n==============\n$record\n==================\n";
        my $Zpackage = $Zconn->package();
        $Zpackage->option(databaseName => 'Koha');
-       $Zpackage->option(action => "recordInsert");
+       $Zpackage->option(action => "specialUpdate");
+#      $Zpackage->option(recordIdNumber => $biblionumber);
        $Zpackage->option(record => $record);
        $Zpackage->send("update");
-       $Zpackage->destroy;
+#      $Zpackage->destroy;
        $i++;
        print '.';
        print "$i\r" unless ($i % 100);
Index: koha/misc/zebra/unimarc/zebra.cfg
diff -u koha/misc/zebra/unimarc/zebra.cfg:1.5 
koha/misc/zebra/unimarc/zebra.cfg:1.6
--- koha/misc/zebra/unimarc/zebra.cfg:1.5       Tue Jan 10 17:01:55 2006
+++ koha/misc/zebra/unimarc/zebra.cfg   Thu Feb  9 10:59:34 2006
@@ -1,18 +1,15 @@
-# Simple Zebra configuration file that defines
-# a database with USMARC records.
-# $Id: zebra.cfg,v 1.5 2006/01/10 17:01:55 tipaul Exp $
-#
 # Where are the config files located?
 profilePath: ${srcdir:-.}:/usr/local/share/idzebra/tab/
 
+
 # Files that describe the attribute sets supported.
 attset: bib1.att
 attset: explain.att
 
-recordId: (bib1,Local-number) 
+recordId: (bib1,Identifier-standard)
 
 # Specify record type
-recordType: grs.marc.unimarc
+recordType: grs.xml
 
 # Lock File Area
 lockDir: lock
@@ -27,7 +24,7 @@
 memMax: 100
 
 perm.anonymous: rw 
-encoding utf8
+encoding utf-8
 
 storeKeys:1
 storeData:1