Re: Using Emacs Lisp for script writing

[Tim X]

Cecil Westerhof <Cecil@decebal.nl> writes:

> Tim X <timx@nospam.dev.null> writes:
>
> I'll try to keep that in mind. But compiling the regular expression made
> a big difference. Also, I remember someone telling me that lists are not
> very efficient. What should I use instead? Or will I found that out in
> Practical Common Lisp?
>
I'm always very wary of claims such as lists are not efficient. Lists in
lisp tend to be a lot more efficient than lists in other languages.
However, my main reason for being cautious is that it really depends on
what your doing and what the performance expectations are. While I do
try to develop efficient algorithms and I certainly due try to choose
the best abstract data type for the problem at hand, I frequently start
with a list and then possibly change to some other structure once it
becomes clear the list is not going to be efficient or is going to be
less clear code wise compared to using something else, such as s struct,
array or class. Lists are pretty fundamental in lisp - even the code is
a list of lists. Apart from having an efficient list implementation,
lisp also tends to have a lot of useful functions that work on lists. 


>
>> Lisp dialects are somewhat
>> notorious for being easy to learn and very hard to master.
>
> I already experienced a little of it.
>
>
>> For this reason, I've found other
>> resources, such as Practical Common Lisp, CLTL2 and other books really
>> useful. I tend to skim them, find the general area and terms I need and
>> then go back to the hyperspec to get the precise definition/usage. 
>
> Practical Common Lisp I already use. (From there I got the property
> list.) From CLTL2 is said:
>     "The book does not correspond exactly with the ANSI standard: some
>     details are different; some things from the standard are missing;
>     and some things from CLtL2 are not in the final ANSI standard.
>     Programmers should therefore be wary of using it as a reference."
> But I'll add it to my (already very big -maybe I need to sift) arsenal.
>
>
>> I have managed to get myself confused as well by doing this.
>> It is best to concentrate on one until you are quite comfortable and
>> familiar with it and then move on to the next dialect. Apart from
>> reducing the potential for confusion, you can also appreciate the
>> pros/cons of the different dialects better.
>
> With different dialects do you mean different Lisp dialects or different
> CL dialects? The former I think I do not like, the latter is where I
> will aim at. When I write portable code, I can switch to the dialect
> that is best for the situation.

I was referring to the different lisp dialects rather than different CL
dialects. For example, elisp, scheme, guile, rep and cl are all lisp
dialects. They are similar enough that usually you can understand the
code by just reading it, but they have enough differences that you can
easily get confused when switching between them.
>
>
>> (though I still find elisp and emacs the best
>> extensible editor and kitchen sink available!)
>
> I agree. For example I am also using GNUS -just as you I saw-. It is a
> lot of work, but I think/hope that when I have GNUS in my fingers, I can
> easily make it do what I want instead of what the developer thought I
> wanted. ;-)
>
Yep, this is the huge benefit of emacs. Its quite amazing what you can
do with it. I use it pretty much for everything. In fact, I have to as
I'm an emacspeak user. I lost my sight over 12 years ago. Thanks to
emacs and emacspeak, not only have I been able to continue working, I've
actually managed to continue developing a career as a programmer. 

Emacspeak uses a very neat feature of emacs called defadvice. Using
elisp's advice functionality, key functions have been advised to provide
both spoken feedback and auditory icons. Essentially, the advice
mechanism provides a way to modify the behavior of a function without
changing its code. The advice works like a wrapper around the original
function, allowing you to call/run additional bits of elisp before,
after or around the original function. This elisp can do all sorts of
things, such as changing/modifying arguments passed to the function or
values returned by the function or possibly do something quite unrelated
etc. It can be a dangerous feature if not used cautiously and it can
create challenges when debugging, but is a very useful feature. 
>

>> P.S. Another advantage to CL is that if you plan to share/distribute
>> some of what you are doing, you can compile it to native code. This
>> means people don't even have to know you wrote it in lisp. This can help
>> overcome the considerable FUD regarding CL that exists out there. 
>
> That is a good point. But with clisp that does not work as far as I
> know. I tried to install SBCL, but the install process needs Lisp. It
> only mention it to do with SBCL and CMUCL. So that can wait until later.
>
Although I've not done it, looking at the clisp docs, there does appear
to be functionality that will support dumping out the lisp program to
native code. I wouldn't worry about it now, but it probably would be
worth looking at later.


>
>> PPS. Make sure you do put the effort into getting SLIME working.
>
> I already planned that.
>
> Another question. The BBDB and also the example in Practical Common Lisp
> use lists for the database. Is this not inefficient? Would a real
> database not be better. Not that I want to burn me at the moment on
> databases. ;-)

Ive used bbdb for years and have a large .bbdb database file. I have
never encountered any performance issues. Actually, bbdb is possibly a
good example of what I was trying to explain above concerning not
worrying about efficiency and performance too much when developing the
code. 

I would be vary cautious regarding statements such as lists are
inefficient. Such a statement is a huge generalisation and needs to be
considered in context. 

Most argue lists are inefficient mainly because of additional storage
(i.e. each node has a pointer to the next node) and because they don't
support random access (you have to traverse the list to get to the last
element) etc. While this is all true, it is important to balance this
with the fact that lisps have gotten very good at managing lists
efficiently, provide lots of very useful and optimised functions for
operating on lists  and have developed idioms for using lists that avoid
some of the inefficiencies, such as pushing new elements onto the front
of the list and then reversing the list when you are done rather than
traversing the list multiple times to add new elements etc.  Generally
speaking, lists in lisp are a lot more efficient than lists in other
languages. A common mistake/criticism of lisp is that non-lispers think
lists are all there is in lisp and they think lists are inefficient.
This is not necessarily the case and overlooks the fact lisps tend to
support other data types, such as arrays, structs and classes.

Having said that, it is still possible to use lists in lisp
inappropriately and end up with slow or inefficient programs. However,
this isn't a problem with the language as much as with the experience of
the programmer. A good understanding of lisp list structure, how it
shares storage between structures (for example, joining part of one list
with another list usually doesn't create a totally new list with all its
associated storage. More likely, you will modify the 'links' in your
list and will end up with a new linked structure which shares storage
with one of the original lists. Essentially, rather than creating
multiple copies of things, lisps will genrally manipulate the links in a
list to create different representations of the linked atoms that make
up the list. Instead of having multiple copies of the atom, you will
have just one, but it might be pointed to by multiple structures that
represent that atom in different lists. This is why in most lisps you
need to be careful about where you use distructive operations as you
cannot always know what else in the program is sharing some of the
structure. It also explains why lisp is frequently used in a functional
programming style (though this is not the only paradigm CL supports).
However, I digress....

The bbdb uses lists and shows that they can be used in an efficient
manner. While you could use a full blown database for this, you probably
wouldn't get any improvement in performance as you would now have all
the overheads associated with making database connections and managing
database files etc. If on the other hand we were talking about an
application that had to process large amounts of data or had to query
the data in complex ways or support ad hoc queries etc, then maybe a
list would not be the right data type and we might find structures or
classes better. However, in reality, we are talking about a basic
address book. the types of queries are well structured, the records are
fairly well structured and quite small and the system has few
updates/inserts.

At the end of the day, the real measure of whether it is efficient
enough is user experience. If the user is able to add and update records
and query for data in a time that is acceptable without all available
resources being used and essentially, the system is useful and
beneficial, then the program is efficient enough. You could possibly
re-write the whole system to use a different data structure or a
database for storage and maybe you might see a 50% improvement in speed
or reduction in storage requirements, but if that makes no difference to
the user experience, what benefit was it? In addition, adding all the
code to manage a new data structure or manage database communicaitons
wil likely make the code larger and more complex, bringing with it more
bugs and probably more maintenance effort. Increasing responsiveness so
that I get a result in 25 msec rather than 50 msec is unlikely to
impress anyone (or even be noticed).

If you adopt good coding style, changing from one data structure to
another within lisp is usually fairly straight forward. Therefore, my
standard approach is to initially use lists while developing the first
version. If I find that managing the data with lists becomes too coplex
or the efficiency is not good enough, I will change to a new structure.
This usually only involves changing a few low level functions that
manipulate the data at a low level. Doing it this way also usually means
that by the time I've got to this stage, I have much better
understanding of the problem domain and can also make a better decision
as to what is the best structure to use. Something I may not have been
able to do at the very beginning when my knowledge of the problem was
more abstract.

-- 
tcross (at) rapttech dot com dot au