Re: saveing/loading symbol table of annymous functions

[David Bateman]

John W. Eaton wrote:
> On  9-May-2007, David Bateman wrote:
> 
> | I've been working on load/save for inline, function handles anda
> | nonymous function handles in matlabs format. I have inline and function
> | handles working fine, but have difficulties with the format of anonymous
> | function handles. Basically this is a process of reverse engineering as
> | matlab doesn't document the matfile format for anonymous function
> | handles. The format is also excessively complex for what it is.
> 
> Aren't function handles in Matlab just class objects with the fields
> "function", "type", and "file", and anonymous function handles just
> add the additional field "workspace" to that (and set file to '')? 

No function handles have their own class type mxCLASS_FUNCTION, rather
than mxCLASS_OBJECT. Inline functions are objects however, and I used
the object documentation to allow them to be read/written.


> If
> so, then aren't these saved the same as any other class object (which
> I think is documented)?  OTOH, the workspace field seems a little
> opaque and I guess that is where the real trouble is.

Couldn't agree more, and with no documentation, its a real mess
understanding it.

> 
> | The matlab v5 file format for an anonymous function handle is like
> | 
> | mxCLASS_FUNCTION
> |   mxCLASS_STRUCT
> |     matlabroot (mxCLASS_CHAR "/opt/matlab73")
> |     separator (mxCLASS_CHAR "/")
> |     sentinel (mxCLASS_CHAR "@")
> |     function_handle
> |       mxCLASS_STRUCT
> |         function (mxCLASS_CHAR <string with anonymous function handle>)
> |         type (mxCLASS_CHAR "anonymous")
> |         file (mxCLASS_CHAR "")
> |         workspace (mxClassID = 17 [0x11])
> |           MCOS (mxCLASS_UINT32 5x1)
> |           function_handle_workspace (mxCLASS_UINT8 Nx1)
> 
> What anonymous function definition did you use when you discovered
> this structure?

I used three difference definitions that all had the same structure as
above, these being

f = @(x) 2 * x;
save -v6 anon1.mat f
a = 2;
f = @(x) a + x;
save -v6 anon2.mat f
f = @(x) sprand(x,x,0.02);
save -v6 anon3.mat f


> 
> | The function_handle_workspace field is in fact a matfile in itself
> | stored as a UINT8 array which seems to contain a single structure.
> 
> This part is a complete MAT file with header info and all?  Ugh.

No its missing the header of 116 bytes plus an additional 8 bytes for
what is called the subsystem data offset and starts with the endianness,
version number in hex of "0001 494d 0000 0000". One thing I just noticed
is that the 117 to 124 are different in a file containing an anonymous
function handle. The file anon2.mat above has "e003 0000 0000 0000",
whereas a normal file just has 8 spaces (0x20).

Page 1-7 of matfile_format.pdf from the mathworks site states

<quote>
Header Subsystem Data Offset Field
Bytes 117 through 124 of the header contain an offset to
subsystem-specific data in the MAT-file. All zeros or all spaces in this
field indicate that there is no subsystem-specific data stored in the file.
</quote>

Funnily the offset 0x3e0 points to the data block containing
function_handle_workspace which is the embedded matfile....

What I wonder now is if this offset is needed to find the block to
convert. As at the moment to make my life easy my test files all only
have a single saved function handle, I now wonder what will happen to
the file format if it contains multiple variables.. I'll check

In any case the current Octave code ignores bytes 117 to 124 of the
header of matfiles as well.

> 
> | At this point I have no idea what MCOS represents, and I haven't managed to
> | get octave to correctly read the function_handle_workspace variable,
> | which itself decomposes to
> | 
> | function_handle_workspace
> | mxCLASS_STRUCT (1 field MCOS with no name!!!)
> | MCOS (mxCLASS_WORKSPACE)
> | MCOS (mxCLASS_CELL)
> | [1,1] (mxCLASS_UINT8 Nx1)
> | [2,1] (mxCLASS_DOUBLE 0x0)
> | [3,1] (mxCLASS_CELL 2x1)
> | [1,1] (mxCLASS_UINT32 5x1) ## Identical to
> | ## function_handle.workspace.MCOS
> | [2,1] (mxCLASS_STRUCT) ## This has the real workspace in it
> | FileWrapper__ (mxCLASS_UINT8 Nx1)
> | MCOS (mxCLASS_DOUBLE [])
> 
> Ugh^2.

Much what I thought.

> 
> | FileWrapper__ is also an embedded file like function_handle_workspace.
> 
> Is this also a complete MAT file?  If you save it separately and load
> it, what variables does it contain?

It doesn't have the header of 124 bytes either. As point out above it
contains a single variable MCOS that is an empty matrix. What I'm
starting to think is the second workspace might be for persistent or
global variables touched in the function handle, something like

function f0 = f()
  a = 0;
  f0 = @g0;
  function y = g0(x)
    persistent z;
    if (isempty(z))
      z = 1;
    end
    y = z + a + x;
    z = z + 1;
  end
end

though I don't have access to matlab and can't test it at the moment.

> 
> | At this point I could probably get some anonymous function handle load
> | save mat-file functionality running, but wouldn't mind some comments on
> | what people think are the other parts of this structure for a function
> | handle. I attach a loaded matlab anonymous function handle stored as a
> | structure in an octave text file the details of how they were created is
> | 
> | f = @(x) 2 * x;
> | save -v6 anon1.mat f
> | a = 2;
> | g = @(x) a + x;
> | save -v6 anon2.mat g
> 
> Are you asking what information should we save in Octave files?  I
> don't know what to recommend here.

I think I know what to do for Octave file formats know given the above
information. As we don't yet treat sub-functions it is a simpler issue.

What I was looking for was ideas on what the contents of the files I
sent mean, as there are bits I haven't decrypted that make no sense.

In any case something this needlessly complex is really painful to
understand. I wonder if this was a summer project of a student employed
at mathworks...

In any case I've gotten the loading of my test files to work, even
though there are certain parts of the file I don't yet understand.

> 
> In Matlab, I see
> 
>   a = 2;
>   g = @(x) a + x;
>   x = functions (g) => class object with fields as noted above
>   c = x.workspace => cell array with one element
>   c{1} => 1x1 structure array with fields corresponding to variables
>           in the anonymous function that are not in the argument list
> 
> but maybe this is nothing new.

Yeah, I added that in passing to Ffunctions a few days ago and will send
it with the rest of the patch when I'm happy with it.

Regards
David