[Octave-bug-tracker] [bug #57284] Crash in MEX file when outputs are not used

[Rik]

Follow-up Comment #7, bug #57284 (project octave):

For testing purposes, try compling this file "mexout.c"


#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{

  const mwSize dims[] = {1, 1, 1};

  if (nrhs != 1 || ! mxIsDouble (prhs[0]))
    mexErrMsgTxt ("input must be a single integer");

  int count = mxGetScalar (prhs[0]); 
  
  for (int i = 0; i < count; i++)
    plhs[i] = mxCreateNumericArray(3, dims, mxDOUBLE_CLASS, mxREAL);
}


Then try invoking with


mexout (1)
mexout (2)
mexout (3)
mexout (4)
...


until you get a segfault.

It's not perfect because your machine may have allocated the pointer plhs[] to
a location that has some spare unused memory after it.  Also, I believe that
the Octave interpreter always allocates at least one location to handle the
automatic variable 'ans'.

When I run this in Octave the threshold for segfault seems to be "nlhs + 2". 
For example, 


[a,b] = mexout (3)


works, but this does not.


[a,b] = mexout (4)
free(): invalid next size (fast)
fatal: caught signal Aborted -- stopping myself...
Abort (core dumped)




(file #47915)
    _______________________________________________________

Additional Item Attachment:

File name: mexout.c                       Size:0 KB
    <https://savannah.gnu.org/file/mexout.c?file_id=47915>



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57284>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/