#1710: Major garbage collector runs all the time

[Chicken Trac]

#1710: Major garbage collector runs all the time
--------------------------------+----------------------------
            Reporter:  megane   |       Type:  defect
              Status:  new      |   Priority:  major
           Milestone:  someday  |  Component:  core libraries
             Version:  5.2.0    |   Keywords:  runtime
Estimated difficulty:  easy     |
--------------------------------+----------------------------
 The major garbage collector runs too frequently if you happen to get
 unfortunate
 "correct" combination of
 - heap size,
 - total size of "permanent objects", and
 - you're generating lots of garbage objects.

 (By permanent objects I mean the set of objects that survives a major
 garbage
 collection.)

 In this situation you can effectively get a major GC for every `N`
 allocated
 bytes that survive the 1st generation.

 This `N` is basically just `heap_size - permanent_objects`, i.e. the free
 heap
 space just after a major collection.

 In practice, the `N` may get quite small. In this case the program is
 mostly
 just doing garbage collection.

 One incarnation of this is a program that gradually gets slower and
 slower, and
 then suddenly gets fast again. In this case the sum of permanent objects
 gets
 bigger and bigger until the objects no longer fit in the heap. At this
 point the
 heap is grown and everything gets smooth again.

 One possible workaround is to manually specify a big heap with `-:h` or
 `-:hi`.
 This has the side-effect of
 1. Making finalizers run less often (they are checked during each major
    collection)
 2. Causing memory fragmentation. (Objects are compacted during the major
 GC.)
 3. You have to choose a size. With `-:hi` you can still run into the
 problem if
    you choose too small size.

 One example of affected program:
 https://stackoverflow.com/questions/62826083/chicken-scheme-read-line-
 taking-too-long/63215403
 {{{
 (define *corpus*
   (call-with-input-file "largeish_file.txt"
     (lambda (input-file)
       (let loop ([line (read-line input-file)]
                  [tokens '()])
         (if (eof-object? line)
             tokens
             (loop (read-line input-file)
               (append tokens (string-split line))))))))
 }}}

 This program generates lots of garbage by creating lots of temporary
 lists.
 Using big heap (using `-:h`) makes things a lot faster.

 I've posted a patch here:
 https://lists.nongnu.org/archive/html/chicken-
 hackers/2020-01/msg00002.html
 This works by growing the heap if it's too full after a major collection.

-- 
Ticket URL: <https://bugs.call-cc.org/ticket/1710>
CHICKEN Scheme <https://www.call-cc.org/>
CHICKEN Scheme is a compiler for the Scheme programming language.