RE: [Koha-devel] New KOHA3 API -design issue

[Tümer Garip]

Hi Paul,

As I work more on this issue of ZEBRA API more problems arise and thats
why I have written this document.
To have more details in biblio and items tables i also think is not an
issue of CPU or disk space. Both approach is still acceptable. I kept
the barcode as well as you noticed. My idea is that we mainly use ZEBRA
for searching but use SQL to reach a record with unique indexes. This
way we can do some load balancing and let zebra do the heavy work. SQL
is as fast when reaching unique records.

Currently I have a bigger problem that our current zebra design does not
allow merged resultsets. I am working heavily on this to see what can be
done without too much CPU overhead. i.e. Currently I cannot search a
title in a specific library branch.

Regarding koha_attr table.
Well i needed the old field names to exist somewhere so i can use them
in our framework (marc_subfield_structure) designs to map 245$a to
"title" 300$a "publisherid" etc. The old API got these names from the
tables themselves (biblio,biblioitems,items,subject,additionalauthors).
So I created a table with all the old field names in so that all the old
templates and most of the code can work with <!--TMPL VAR
NAME="title"--> notations.
But to create a table just for this was unneccessary so I added extra
functionality to it.
It has another column called "attr". There we can define what is the
corresponding ZEBRA attribute to this virtual_field_name "title",
"publisherid", "barcode" or whatever.
It has data like this:
Koha field              ATTR                            LABEl
SORT            SERVER
itemnumber              @attr 1=1032            Item number
No      itemserver
classification  @attr 1=16                      LC classification
No      biblioserver  
conference_name         @attr 1=3                       Conference name
No      biblioserver 
copyrightdate   @attr 1=30 @attr 4=5    Copyright date           No
biblioserver 

This table now provides me two more functionalities. (I am talking about
the backend workings by the way not front end searching). While we
evolve ZEBRA functionality we are all using different search attributes
for our fields. Such as you are using Identifier-standard (@attr 1=1007)
for you biblionumber I am using Number-db (@attr 1=52). Even at one
stage we all use the same attributes for indexing in time people will
want to add more or change their indexing attributes. This table is
Sysytme admin editable. So within KOHA API I still call
biblionumber=some number and do not use any @attr notation at all. The
code is free of Bib-1 or GILS or KOHA designed attributes.

The second functionality is that i can now create as many virtual fields
as i want on the fly. As you may have noticed I have conference_name
defined. So if I index my zebra with conference_name as long as define
it in my koha_attr it will appear as a searchable field in my search
screens. All these field names do not have to be mapped to
marc_subfield_structures but if you want to use them in your templates
as <!--TMPL VAR NAME="conference_name"--> than you have to map tham to
your MARC tables ofcourse.

One other functionality is that we do not have to hardcode $9 where
authority id is going to reside in our MARC definitions. It has a name
now "authid". You just map this to any record any field and dont bother
whether its $9 $6 or $8 you are using your link from. So flexible
linking is possible.

Similar to MARCfind_marc_from_kohafield we now have
MARCfind_attr_from_kohafield.
And lastly it says whether you sorted on that field or not. If you did
then it gets included in your sort dropdown menus. No hardcoding. You
may decide to sort on a field that you do not think you needed now. When
you do just set the flag sorts to true and your Search page will have it
included.

Whether we use PQF,CQL,CCL in the frontend for searching purposes (which
I belive we should be able to do them all) in the background I needed
something flexible so that developers can go on programming without
having to concern of whats happening with atributes etc.

I hope this explains it better and I await your comments on this.

Cheers 
Tumer


-----Original Message-----
From: Paul POULAIN [mailto:address@hidden 
Sent: Monday, July 17, 2006 6:21 PM
To: Tümer Garip
Cc: address@hidden
Subject: Re: [Koha-devel] New KOHA3 API -design issue


Tümer Garip a écrit :
> Hi all,

Hi Tümer,

> 2- biblio table now has: biblionumber,marc,itemtype,framework
> 3-items table now has : itemnumber,biblionumber,barcode,marc

I'm *strongly against* dropping all the values. Joshua and me have 
spoken of this before, and I really think we MUST kept a minimum of 
datas in biblios & items tables.
Like title, author, and all values that already exists.
It won't cost a lot of disk or CPU, and will be very very useful for at 
least 2 things :
* debuging Koha
* querying the database with pure SQL.

The other question being also to have iso2709 and (or ?) marcxml in the 
datas.
xml is human readable & turned toward future, where iso is more 
efficient in term of CPU & disk usage.

> 6- A new koha_attr table added. This one is for creating virtual 
> fields which we do not have any more. Also helps matching your own 
> MARC indexing rules to KOHA and ZEBRA. (More on this later)

i'm waiting for the "more on this later", because I don't understand 
what you mean here...

> 2- All items (holdings) are single marc records, instead of all items 
> in one marc record for the same bibliographic record. The main idea 
> behind this is flexibility and speed. One full record for an item 
> gives lots of fields to put all that information that you wanted to 
> keep but did not have extra subfields left. All data including 
> circulation data is kept within the marc records, this approach makes 
> circulation updates faster as we do not have to reindex unchanged data

> related to other items again (zebra technicalities!).

sounds correct & good idea to me.

> 3- All MARC framework designs have OPAC:show-hide Intranet:show-hide 
> Editor:show-show collapsed-hide as dropdown selections so not to wory 
> about what we  put in these MARC fields. All handled correctly
> 4- Circulation speed is improved. One marc update instead of 6 
> sepatrate database update calls we used to have.

> The new koha_attr table:
<snip>
> I will be writing more on this issue after the commits but in the mean

> time expect to have your comments on all the issues regarding this 
> upgrade.
I still don't understand what it's done for, sorry...

-- 
Paul POULAIN
Consultant indépendant en logiciels libres
responsable francophone de koha (SIGB libre http://www.koha-fr.org)