[RFC PATCH 0/5] O_IGNORE_CTTY everywhere

[Sergey Bugaev]

Hello!

This is an attempt to use O_IGNORE_CTTY in a lot of places throughout
glibc. But first, what is O_IGNORE_CTTY, how is it different from
O_NOCTTY, and why would we want to use it?

O_IGNORE_CTTY is an open () flag in the Hurd port that makes glibc
access the newly opened file as if it's not the controlling terminal
of the process, even if it actually really is. In particular, reads
from the file while in the background won't generate SIGTTIN, writes
won't generate SIGTTOU. (Doesn't sound very useful yet, does it?)

O_NOCTTY, in contrast, controls whether or not the newly opened file,
if it is a terminal, automatically becomes the ctty of the process. On
the Hurd this is never the case, and O_NOCTTY is defined to 0.

So, why would one want to use O_IGNORE_CTTY when opening random files
that have nothing to do with terminals, let alone cttys? Well, it
turns out you *especially* want to use O_IGNORE_CTTY precisely in case
you know that what you're opening surely is not the current ctty of
the process! This is because unless O_IGNORE_CTTY is set, glibc has to
do an extra RPC (or even two) to check whether the file is our ctty.
Passing O_IGNORE_CTTY is a hint/promise to glibc that the file is not
our ctty, so theres' no need to make the expensive check.

So, O_IGNORE_CTTY is most useful not to alter the user-visible
behavior (treat ctty as if it's not that), but rather to speed up
open () calls *without* altering user-visible behavior.

=======================================

With that in mind, I went through the codebase and added O_IGNORE_CTTY
to __open flags just about everywhere. You only want to *not* pass
O_IGNORE_CTTY when you're not sure what you're opening, such as when
implementing fopen () or a similar wrapper for open (). But when
you're opening, say, a shared library, or a /dev/shm/ file, or a
locale archive -- you know you're not dealing with ttys, and so using
O_IGNORE_CTTY is appropriate and beneficial.

In order to not add tons of ifdefs, I've added a single

#ifndef O_IGNORE_CTTY
#  define O_IGNORE_CTTY 0
#endif

to include/fcntl.h (following Samuel's advice). This of course makes
O_IGNORE_CTTY usable (and no-op) on Linux (and other ports) without
exposing it in the installed headers.

=======================================

While looking through __open calls, I've noticed a few that
(suprisingly!) did not use O_CLOEXEC. I've tried to fix them all. Even
if the rest of this patchset is rejected, please evaluate that one
commit separately.

I had to convert misc/daemon.c to a more GNU-ish style, since my
changes wouldn't easily fit on 79/80 columns otherwise. I've split off
the actual conversion into a separate patch to make review easier; but
that patch is still going to unreadable; nothing I can do about that,
sorry.

I have built glibc on x86_64-linux and run the testsuite; there are no
new test failures. I can't run the full testsuite on i686-gnu, but
some manual testing confirms that the many superfluous term_getctty ()
RPC calls are gone, but the ctty functionality is still there, e.g. I
can fopen ("/dev/tty") and write to it and get the expected behavior.

This is different from my usual patches in that it mostly touches non-
Hurd-specific code.

Sergey

Sergey Bugaev (5):
  misc: Convert daemon () to GNU coding style
  Use O_CLOEXEC in more places
  hurd: Make dl-sysdep's open () cope with O_IGNORE_CTTY
  include/fcntl.h: Define O_IGNORE_CTTY
  Use O_IGNORE_CTTY where appropriate

 catgets/open_catalog.c        |  4 +-
 csu/check_fds.c               |  6 +--
 elf/dl-load.c                 |  2 +-
 elf/dl-misc.c                 |  2 +-
 elf/dl-profile.c              |  3 +-
 gmon/gmon.c                   |  7 +--
 iconv/gconv_cache.c           |  3 +-
 include/fcntl.h               | 16 +++++++
 locale/loadarchive.c          |  7 +--
 locale/loadlocale.c           |  4 +-
 login/openpty.c               |  2 +-
 login/utmp_file.c             |  7 +--
 misc/daemon.c                 | 88 +++++++++++++++++++----------------
 nss/nss_db/db-open.c          |  3 +-
 rt/shm_open.c                 |  2 +-
 shadow/lckpwdf.c              |  2 +-
 sysdeps/mach/hurd/dl-sysdep.c |  4 +-
 sysdeps/mach/hurd/opendir.c   |  2 +-
 sysdeps/pthread/sem_open.c    |  9 ++--
 sysdeps/unix/bsd/getpt.c      |  4 +-
 20 files changed, 106 insertions(+), 71 deletions(-)

-- 
2.39.2