Re: fuzzing poke with afl & afl++

[Dan Čermák]

Hi folks,

last week I gave it another try to fuzz poke with afl++, using roughly
the same setup as last time (only using afl-clang-fast instead of
afl-gcc-fast, because afl-gcc-fast wouldn't instrument poke for
reasons…).

The results are relatively meager. Albeit afl found a bunch of crashes,
most of them were variations of
https://sourceware.org/bugzilla/show_bug.cgi?id=25379 (double freeing of
AST variable nodes when there is a syntax error in recursive function).

Another bug that afl found frequently is already known too:
https://sourceware.org/bugzilla/show_bug.cgi?id=25395 (Subpassing
doesn't work when the top node is folded out).

Then there's the memory exhaustion type of bugs, where afl created a
poke file with a ridiculous number of tokens that cause poke to go OOM
and die with a segfault (see for instance the attached
id:000045,sig:06,src:001301,time:61955701+003661,op:splice,rep:8). I'm
not sure if this can be handled more gracefully, but at least poke
needn't segfault.


Now for the "new" bugs (at least I hope that these are new):

- heap-use-after-free on a poke file that is filled with random binary
  garbage
  POC: poke -L id:000044,sig:06,src:004272,time:67719988,op:havoc,rep:16

- segfault in a nested struct/function definition in pkl_ast_type_equal
  in pkl-ast.c:631:
  poke -L id:000068,sig:06,src:000132,time:32646862,op:ext_UO,pos:179

- segfault in pkl_typify1_ps_op_rela in pkl-typify.c:114 for the
  following file:

--8<---------------cut here---------------start------------->8---
deftype Bar=struct{int i;long j;defun j0=void:{}};defvar b=Bar@ 0#B;
b.j0<0; 
--8<---------------cut here---------------end--------------->8---

- Heap buffer overflow in pvm_execute_or_initialize pvm.jitter:2081 for
  the following malformed file:
--8<---------------cut here---------------start------------->8---
deftype Foo=struct{int i;long j;};defvar a= Foo { i =0O0 };
--8<---------------cut here---------------end--------------->8---

- A bunch of assertions in jitter (see for instance
  id:000079,sig:06,src:000158,time:39603823,op:ext_AO,pos:182).
  However, Jose told me that this is caused by poke not having
  implemented printing binary formats yet.


And that's about it, which is not a lot given that I ran this on 6 cores
for 24 hours. This means one of two things: either poke is really quite
robust after the initial fuzzing round around Christmas or the currently
present bugs (especially #25379) skew afl towards finding them over and
over again (this is quite likely, as that bug is pretty generic and can
be triggered by a shear infinite amount of malformed files).


@Jose: could you tell me which of the "new" bugs are actually new? I'll
create Bugzilla entries for those that are not yet known.


Cheers,

Dan

address@hidden (Jose E. Marchesi) writes:

>         Long story short: I've let afl++ fuzz poke's script mode overnight on 
> my
>         laptop (12 fuzzing processes were running, one in deterministic mode 
> and
>         11 in random mode) for about 5 hours until poke filled up my /tmp/ and
>         the fuzzer died (because it was running on the same partition to not
>         murder my SSD). It found 600-something input files that made poke 
> crash,
>         but of these only 164 are distinct crashes (distinct according to afl,
>         at least the failed assertions could be maybe only a handful of
>         bugs). I've put them into the following categories according to the 
> type
>         of crash that occurred:
>         
>         assertion failed: 122
>         double free:      11
>         signal FPE:       2
>         signal SEGV:      19
>         use after free:   8
>         misc:             2 (these just result in an "Aborted" message)
>         
>         The resulting files are attached in the archive crashes.tar.gz.
>     
>     Allright, we definitely want to file bugs for these bugs.  I will look
>     at crashes.tar.gz and do so.
>
> So I did, fixed several bugs, and filed a couple of them that still have
> to be fixed.
>
> Right now we are down to:
> 7 assertion failures
> 0 double frees
> 0 signal FPE
> 0 signal SEGV
> 0 use after free
> 0 misc
>
> Of the reported 265 hangs, after the bug fixes, none of them are
> unjustified.
>
>         I have one suggestion though: could we add a flag to poke (either
>         compile- or run-time), so that it will not create diagnostic files in
>         /tmp/? This is not really useful when running poke a few million times
>         with mostly invalid input.
>     
>     Definitely yes.  Will add it.
>
> Done.  Now poke won't dump anything under /tmp when an ICE is reported,
> if the tool was invoked with --quiet (this is the case when running
> tests.)
>
> So, at this point, you may want to rerun afl.  poke should be much more
> robust now :)
>
> This fuzzing already uncovered 5 bugs.  Definitely worth it.  Thanks for
> doing it Dan!

id:000045,sig:06,src:001301,time:61955701+003661,op:splice,rep:8
Description: Binary data

id:000044,sig:06,src:004272,time:67719988,op:havoc,rep:16
Description: Binary data

id:000068,sig:06,src:000132,time:32646862,op:ext_UO,pos:179
Description: Binary data

id:000079,sig:06,src:000158,time:39603823,op:ext_AO,pos:182
Description: Binary data

signature.asc
Description: PGP signature