closing emacsclient always focuses another emacs window

[trygve . flathen]

I edit files from my terminal with an alias doing
  emacsclient -c file &
When I later close the window with C-#, focus will always be given to another 
emacs window, even across my virtual desktops.

How can I make focus return to the terminal (or whatever had focus just before) 
when I close the emacsclient window?

Searching for this problem is hard since it is vastly outnumbered by problems 
regarding focus on opening the client instead of closing. I found 
https://bugs.kde.org/show_bug.cgi?id=187718, but that discussion focuses (no 
pun intended) on kde.

-Trygve