Hi,
at the current point of time, enabling the toolbar and disabling it
again is not a noop, but rather causes shrinking. That's ugly. On
the other side, Emacs should not increase its frame size beyond the
given initial frame size.
So I think that Emacs should record its startup frame size in _pixels_
(or just record whether it started its geometry calculation with the
toolbar on or off). Turning the toolbar on and off should then cause
the text size to have the largest size fitting inside of the original
startup frame size.
Of course, it would be nicer (and simpler altogether) if Emacs did not
change its frame pixel size at all when turning toolbar (or menubar or
whatever) on and off.