I think the issue here is that even when margins "aren't modified",
olivetti does change them, because it first sets them to 0 before
setting them again to the previous value, as can be seen in
olivetti--set-margins (which is function called for post-command-hook,
window-size-change-functions, and friends):
(defun olivetti--set-margins (&optional window-or-frame)
[...]
(olivetti-reset-window window-or-frame)
[...]
(set-window-margins window-or-frame left-margin right-margin))))
(defun olivetti-reset-window (window)
[...]
(set-window-margins window nil))
I don't know why it starts by calling olivetti-reset-window.
But I think that for Emacs-27, it doesn't matter because it should only
use window-size-change-functions so the extra (set-window-margins window
nil) shouldn't be problematic.