Are frozen files really that hot?

[Daniel Goldman]

Hi,

The manual says: "Later calls, containing the -R option, are able to 
reload the internal state of m4, from base.m4f, prior to reading any 
other input files. This means instead of starting with a virgin copy of 
m4, input will be read after having effectively recovered the effect of 
a prior run. In our example, the effect is the same as if file base.m4 
has been read anew. However, this effect is achieved a lot faster."

"A lot faster" sounds promising. But also kind of vague. 100% faster? 
200% faster? And of course the frozen file just affects the loading 
phase. If the loading phase is just a fraction of the processing cycle, 
it might not matter much.

Anyway, in the course of doing some development, I figured I would do a 
benchmark on frozen file performance. The task is creating some 350 web 
pages using m4. There are about 450 macros. For the benchmark, I limited 
the test to just the m4 part, customized the shell scripts not to do the 
other parts (sed, grep, etc). On a single user linux e3-1245 v3, with 
crontab turned off, I either 1) make the frozen file first, and use it 
repeatedly to make the 350 outputs, or 2) use the "unfrozen" file each 
time, the so-called "virgin copy". :) I used /usr/bin/time to measure 
the time. Here are the results:

**************************

  freeze=y # use m4 frozen file? 1.65 seconds
  freeze=n # use m4 frozen file? 1.80 seconds
  freeze=y # use m4 frozen file? 1.22 seconds
  freeze=n # use m4 frozen file? 1.98 seconds
  freeze=y # use m4 frozen file? 1.04 seconds
  freeze=n # use m4 frozen file? 1.80 seconds
  freeze=y # use m4 frozen file? 1.27 seconds
  freeze=n # use m4 frozen file? 1.96 seconds
  freeze=y # use m4 frozen file? 1.36 seconds
  freeze=n # use m4 frozen file? 1.88 seconds

{totals[y]} = 6.54
{totals[n]} = 9.42

**********************

That's a 31% improvement. (9.42 - 6.54) / 9.42 = .306

Or one could say the non-frozen test takes 44% longer.

That's better than nothing, but I was hoping for better. I realize this 
is just one test. And sorry not feasible for me to provide the details, 
so a synthetic benchmark would also be helpful. Are there other 
real-world (or synthetic) benchmarks for frozen files, or any cases 
where frozen files are really that helpful for some task?

**********************

Generally, I look for much greater speed up than 31%, especially as the 
overall speed-up (including the non-m4 tasks) would be even lower. For 
example, I recently got into multi-thread programming in C and using 
shell scripts. It was kind of a bother setting up, but the speed up for 
various tasks was dramatic. For the same task making the 350 web pages 
(with all steps included, which take up much more time than the m4 step) 
the single thread time was 15.5 seconds, the multi-thread time (using 
xjobs) was 4.3 seconds. And only a fraction of that time is taken by m4, 
which is already pretty fast.

Also, setting up scripts to use frozen files requires an additional 
step, a small negative (it's not that difficult).

If there were no downside to frozen files, I wouldn't bother. One could 
take them or leave them. I am in favor of giving users choice.

But I imagine there is a fair amount of infrastructure related to frozen 
files, with programming, source code, file formats, systems analysis, 
etc. It would seem to make the software more complex. My guess is that 
frozen files introduce some complications.

If only minimal effort is required related to frozen files (hard for me 
to believe), then I guess there is no problem. Otherwise, I am wondering 
if frozen files are worth it. At this point, based on my benchmarks, I 
don't plan to use frozen files.

I am wondering if maybe frozen files are a failed experiment, or perhaps 
outmoded by improvements in cpu performance. Maybe frozen files were 
useful decades ago. But I would suggest that maybe future effort related 
to frozen files might better be spent improving and fixing other 
somewhat obvious things in m4. I would even suggest maybe frozen files 
might be deprecated at some point. Maybe "less is more" in this case.

If there are other benchmarks showing frozen files really are hot, I 
take all this back. It's always possible I did something wrong. But the 
whole point of frozen files is to speed things up, yet doing a keyword 
search for "benchmark" on this group, this seems to be the first post 
that actually did a benchmark (of any kind). So I think this is worth 
bringing up.

Thanks,
Daniel