So, if my understanding is correct, hb_ot_shape_glyphs_closure will take an input character or characters and tell me all the glyphs that it gets transformed into, as well as the final form. I'm not sure about this interpretation, because the documentation uses the term "Transitive closure", which I'm not familiar with.
As for iterating through auto-hinter styles, do you mean that I should get a list of features and try each one for the 'features' parameter?
Also, I wanted to share my progress in the tilde unflattening. You suggested earlier that I fix the algorithm by removing the segments containing points of the tilde from their edges. This worked remarkably well. In fact, I found that I often got better results with only this change. The problem with the vertical stretch was that while it was able to fix the tilde in many cases, it caused the size to be inconsistent. The tilde would be 2 pixels wide at one ppem, then increase to 3 when downsizing, when this issue wasn't present before (this happens both with and without the segment removal). The segment removal is simpler and should be preferred if it solves the problem adequately. The drawback of segment removal is that it causes the tildes to have blurrier outlines, and it doesn't fix some of the cases that it would if it was combined with the vertical stretch.
The segment removal should be part of the solution, but the question is to what extent the vertical stretch should be part of the solution. To try to answer this, I tested on a bunch of fonts.
To aid testing, different characters use different versions of the algorithm:
õ: vertical stretch, no segment
ñ: no vertical stretch, segment removal
ã: vertical stretch and segment removal
all other tildes: no changes applied
Results:
Liberation Sans Regular: Segment removal delays flattening, but adding vertical stretch delays it even further.
Times New Roman: All approaches had weird behavior at exactly 16 ppem. With only the stretch, the tilde flattened at this size. With both approaches with segment removal, the tilde stretched to 3 pixels tall and looked very out of place. This wasn't an issue with any non-adjusted letters with tildes (indicating it didn't originate from the font). In all cases, the tilde was not flat but very blurry at all sizes where the glyph was legible.
Noto Sans Regular, suggested by Brad: both approaches with segment removal fix the tilde until around 13 ppem. This font has flat tildes at sizes as high as 26 ppem, which occurred with the vertical-stretch-only mode.
Calibri Regular: Didn't have a flat tilde issue to begin with. Both approaches using segment removal had no size anomalies.