Thanks for your help. I fixed the issue by marking all on-curve points that I moved as touched (and letting IUP do the rest), setting oy = y, and applying the stretch to fy as well. I also had some calculation errors in the height measurement the debug prints used and the algorithm itself.
The next problem is that I relied on an assumption that if the original height of the tilde was about 2 pixels tall, the grid-fitted position would probably be 2 pixels. My method estimates the width of the tilde by looking for the points at the tips of the tilde curves and measuring their distance to the bounding box (see the attached image for a visualization), then adding 1 pixel and scaling the contour vertically, anchored at its lowest point, until it is at least that tall.
Testing with Liberation Sans, the sizes that had flat tildes before still have flat tildes. I can add more than 1 pixel to the measurement to unflatten them, but that causes unnecessarily tall tildes at other sizes. I have confirmed by commenting/uncommenting steps of the hinting process that af_glyph_hints_align_edge_points is the function that snaps the tilde back to flat, so if my understanding is correct, the points at the top and bottom of the tilde are forming edges and are being rounded towards each other, causing the tilde to remain flat. How should I proceed?
I'll also push the code I have after removing the dead code and debug prints that have built up.
n_tilde_measurement_labels.png