Re: [patch] generator function optimizations

[Paul Pogonyshev]

[please keep me on CC, I'm not subscribed to the list]

> > In my real code (a very large function used as a co-routine)
> > with 60 yields, generator function has over 1000 lambdas.
> > [...]
>
> Hmm... I don't think anyone should really care about the number of
> lambdas, right?  So what is the actual problem (caused by this large
> number of lambdas) that you're trying to improve?  Code size?
> Run time?  Both?  Something else?

Both. This huge function takes a lot of time to compile, produces
lots of bytecode and is fairly slow. Of course it is difficult to compare
with non-generator routine: it is so large and complicated I won't try
to rewrite it just for speed comparisons, not to mention that it is run
by a "master" that can pause at any yield-point to resolve
dependencies with other code.

However, note that there are 60 points where it can yield and thus
lose control. But, there are over 1000 generator states. Meaning
that even between this points it switches many generator states, in
a sense pointlessly, because intermediate states have predetermined
successor (some of them have several, if they are branching points).

Lambda calls are not free and it is certainly faster not to call them
if you don't have to. Basically, this patch eliminates some intermediate
states and makes generator form transformer generate a single lambda
where it would previously create several.

> Have you tried to measure the impact of your patch on the actual problem?

I mentioned 3%. Yes, it is practically negligible, but this is a first
small patch and I'm willing to work on improving generators further.

> > If this patch is accepted, I can work on some other ideas
> > I have.
>
> Indeed, there's probably a lot of opportunity for improvement.
> E.g. cconv.el was designed based on the assumption that there wouldn't
> be many different lambdas within the same scope (i.e. we don't try to
> share the data part of various related closures).

There are also, unfortunately, bugs. I have found and submitted two
already (in the normal bug tracker), for one I attached a fix.

Paul