I can reproduce this. I suspect that I was doing something that was
causing the GUI window to update and that must have been bringing it
out of App Nap.
Great, glad to hear I'm not the only one.
I don't actually see any performance issues, but that may just be
because my PC is otherwise idle.
Only certain things seem to have issues. read-event specifically does, which is used in many places in my setup. I use evil, and to exit insert mode, I hit ESC which ultimately calls read-event, meaning it takes a second or two to exit. Helm has performance issues too. But yes, if your computer is doing absolutely nothing else, maybe the scheduler is still giving napping apps enough time. For me, I've always got something going on so its pretty unusable.
As a workaround I use:
defaults write org.gnu.Emacs NSAppSleepDisabled -bool YES
But this should be built into the dist.