Re: Awk Error - Help SOught Urgently

[kahrs]

Rahul Gopal Joshi wrote:

>         I am a student technican in Texas A and M University. I am using Gnu
> Awk (gawk) 2.15, patchlevel 5 for doing text processing. I have to process

gawk 2.15 is a very old version.
Today, Arnold Robbins just announced the availability
of gawk 3.1.4:

 ftp://ftp.gnu.org/gnu/gawk

> about 5000 files, one after another, with each file containing about 200000
> records. WHen I run the awk code, for batches of about 100 -150 files, it
> runs fine., But when I try to run in on the entire list of 5000 files, it
> breaks in between and gives the following error . 

It is possible that this is not a gawk problem.
Having 5000 files you will face completely new kinds of problems
with your shell. Try this one:

 ls * > /dev/null

Looks like an innocent and trivial command, but the command
will fail with an error message on many platforms if you have
thousands of files. Why ? It is the shell which tries to expand
the * and fails because the list is too long. This is not an
academic objection. 3 years ago, I saw someone going mad because
he tried to process 11000 (containg some Genome sequences) this way.