I figured it out from re-reading the tutorial more carefully. I was using the wrong scaling factors when converting float to FT_Fixed.
matrix.xx = (FT_Fixed)(mat[0] * 64.0);
matrix.xy = (FT_Fixed)(mat[1] * 64.0);
matrix.yx = (FT_Fixed)(mat[2] * 64.0);
matrix.yy = (FT_Fixed)(mat[3] * 64.0);
should be:
matrix.xx = (FT_Fixed)(mat[0] * 0x10000L);
matrix.xy = (FT_Fixed)(mat[1] * 0x10000L);
matrix.yx = (FT_Fixed)(mat[2] * 0x10000L);
matrix.yy = (FT_Fixed)(mat[3] * 0x10000L);