getting EBADF on MSVC

[Bruno Haible]

The newer runtime libraries of MSVC no longer return error codes from
functions like printf(), close(), dup2(), _get_osfhandle(), when you pass
an invalid format string or invalid file descriptor.[1][2]

No, they invoke a so-called "invalid parameter handler" which exists once per
process (i.e. it's a piece of global state).[3]

When no such handler is installed, that is, by default:
  - If the user is logged in on the GUI: a dialog pops up. When acknowledged,
    the program crashes.
  - If the user is not logged in on the GUI (e.g. logged in through ssh),
    the program crashes directly; the dialog is queued up for later.

To make things even more confusing for the developer, Cygwin 1.5.x restarts
crashed programs 6 times. So you have one program that crashes, and it produces
its output 6 times, and you have to acknowledge 6 dialogs.[4][5]

The diagnostics of such a crash through an invalid parameter handler are
miserable, of course. And POSIX requires these functions to return error
codes, not to crash. So the workaround for gnulib must be to install an
invalid parameter handler that turns the notification into a error code
return.

By the way, this is the third notification mechanism:

  - There are signals and signal handlers, specified by ISO C 99.
    Impossible to leave such a handler portably.

  - There are Win32 exception handlers, cf. SetUnhandledExceptionFilter [6],
    __try/__except [7], RaiseException [8][9]. These handlers are made
    for being left. And they are multithread-safe.

  - The  "invalid parameter handler" is yet another mechanism. You can
    leave it through longjmp, but that is not multithread-safe.

[1] http://msdn.microsoft.com/en-us/library/xkh07fe2.aspx
[2] http://www.mail-archive.com/address@hidden/msg22153.html
[3] http://msdn.microsoft.com/en-us/library/a9yf33zb%28v=vs.80%29.aspx
[4] http://www.cygwin.com/ml/cygwin-xfree/2007-05/msg00071.html
[5] http://cygwin.com/cygwin-ug-net/using-cygwinenv.html
[6] http://msdn.microsoft.com/en-us/library/ms680634%28v=vs.85%29.aspx
[7] http://msdn.microsoft.com/en-us/library/s58ftw19.aspx
[8] http://msdn.microsoft.com/en-us/library/het71c37.aspx
[9] http://msdn.microsoft.com/en-us/library/ms680552%28v=vs.85%29.aspx


This patch allows to install an invalid parameter handler for specific pieces
of code.


2011-09-18  Bruno Haible  <address@hidden>

        New module 'msvc-inval'.
        * lib/msvc-inval.h: New file.
        * lib/msvc-inval.c: New file.
        * m4/msvc-inval.m4: New file.
        * modules/msvc-inval: New file.

=============================== lib/msvc-inval.h ===============================
/* Invalid parameter handler for MSVC runtime libraries.
   Copyright (C) 2011 Free Software Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 2, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License along
   with this program; if not, write to the Free Software Foundation,
   Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */

#ifndef _MSVC_INVAL_H
#define _MSVC_INVAL_H

/* With MSVC runtime libraries with the "invalid parameter handler" concept,
   functions like fprintf(), dup2(), or close() crash when the caller passes
   an invalid argument.  But POSIX wants error codes (such as EINVAL or EBADF)
   instead.
   This file defines macros that turn such an invalid parameter notification
   into a non-local exit.  An error code can then be produced at the target
   of this exit.  You can thus write code like

     TRY_MSVC_INVAL
       {
         <Code that can trigger an invalid parameter notification
          but does not do 'return', 'break', nor 'goto'.>
       }
     CATCH_MSVC_INVAL
       {
         <Code that handles an invalid parameter notification
          but does not do 'return', 'break', nor 'goto'.>
       }
     DONE_MSVC_INVAL
 */

#if HAVE_MSVC_INVALID_PARAMETER_HANDLER
/* A native Windows platform with the "invalid parameter handler" concept.  */

/* Get _invalid_parameter_handler type and _set_invalid_parameter_handler
   declaration.  */
#include <stdlib.h>

# if defined _MSC_VER
/* A compiler that supports __try/__except, as described in the page
   "try-except statement" on microsoft.com
   <http://msdn.microsoft.com/en-us/library/s58ftw19.aspx>.
   With __try/__except, we can use the multithread-safe exception handling.  */

/* Gnulib can define its own status codes, as described in the page
   "Raising Software Exceptions" on microsoft.com
   <http://msdn.microsoft.com/en-us/library/het71c37.aspx>.
   Our status codes are composed of
     - 0xE0000000, mandatory for all user-defined status codes,
     - 0x474E550, a API identifier ("GNU"),
     - 0, 1, 2, ..., used to distinguish different status codes from the
       same API.  */
#  define STATUS_GNULIB_INVALID_PARAMETER (0xE0000000 + 0x474E550 + 0)

#  ifdef __cplusplus
extern "C" {
#  endif

/* Ensure that the invalid parameter handler in installed that raises a
   software exception with code STATUS_GNULIB_INVALID_PARAMETER.
   Because we assume no other part of the program installs a different
   invalid parameter handler, this solution is multithread-safe.  */
extern void gl_msvc_inval_ensure_handler (void);

#  ifdef __cplusplus
}
#  endif

#  define TRY_MSVC_INVAL \
     gl_msvc_inval_ensure_handler ();                                          \
     __try
#  define CATCH_MSVC_INVAL \
     __except (GetExceptionCode () == STATUS_GNULIB_INVALID_PARAMETER          \
               ? EXCEPTION_EXECUTE_HANDLER                                     \
               : EXCEPTION_CONTINUE_SEARCH)
#  define DONE_MSVC_INVAL

# else
/* Any compiler.
   We can only use setjmp/longjmp.
   Unfortunately, this is *not* multithread-safe.  */

#  include <setjmp.h>

#  ifdef __cplusplus
extern "C" {
#  endif

/* The restart that will resume execution at the code between
   CATCH_MSVC_INVAL and DONE_MSVC_INVAL.  It is enabled only between
   TRY_MSVC_INVAL and CATCH_MSVC_INVAL.  */
extern jmp_buf gl_msvc_inval_restart;

/* The invalid parameter handler that unwinds the stack up to the
   gl_msvc_inval_restart.  It is enabled only between TRY_MSVC_INVAL
   and CATCH_MSVC_INVAL.  */
extern void cdecl gl_msvc_invalid_parameter_handler (const wchar_t *expression,
                                                     const wchar_t *function,
                                                     const wchar_t *file,
                                                     unsigned int line,
                                                     uintptr_t dummy);

#  ifdef __cplusplus
}
#  endif

#  define TRY_MSVC_INVAL \
     {                                                                         \
       _invalid_parameter_handler orig_handler;                                \
       /* First, initialize gl_msvc_inval_restart.  */                         \
       if (setjmp (gl_msvc_inval_restart) == 0)                                \
         {                                                                     \
           /* Then, enable gl_msvc_invalid_parameter_handler.  */              \
           orig_handler =                                                      \
             _set_invalid_parameter_handler (gl_msvc_invalid_parameter_handler);
#  define CATCH_MSVC_INVAL \
           /* Execution completed.                                             \
              Disable gl_msvc_invalid_parameter_handler.  */                   \
           _set_invalid_parameter_handler (orig_handler);                      \
         }                                                                     \
       else                                                                    \
         {                                                                     \
           /* Execution triggered an invalid parameter notification.           \
              Disable gl_msvc_invalid_parameter_handler.  */                   \
           _set_invalid_parameter_handler (orig_handler);
#  define DONE_MSVC_INVAL \
         }                                                                     \
     }

# endif

#else

# define TRY_MSVC_INVAL if (1)
# define CATCH_MSVC_INVAL else
# define DONE_MSVC_INVAL

#endif

#endif /* _MSVC_INVAL_H */
=============================== lib/msvc-inval.c ===============================
/* Invalid parameter handler for MSVC runtime libraries.
   Copyright (C) 2011 Free Software Foundation, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 2, or (at your option)
   any later version.

   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License along
   with this program; if not, write to the Free Software Foundation,
   Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.  */

#include <config.h>

/* Specification.  */
#include "msvc-inval.h"

#if HAVE_MSVC_INVALID_PARAMETER_HANDLER

# ifdef STATUS_GNULIB_INVALID_PARAMETER

/* Get declarations of the Win32 API functions.  */
#  define WIN32_LEAN_AND_MEAN
#  include <windows.h>

static void cdecl
gl_msvc_invalid_parameter_handler (const wchar_t *expression,
                                   const wchar_t *function,
                                   const wchar_t *file,
                                   unsigned int line,
                                   uintptr_t dummy)
{
  RaiseException (STATUS_GNULIB_INVALID_PARAMETER, 0, 0, NULL);
}

static int gl_msvc_inval_initialized /* = 0 */;

void
gl_msvc_inval_ensure_handler (void)
{
  if (gl_msvc_inval_initialized == 0)
    {
      _set_invalid_parameter_handler (gl_msvc_invalid_parameter_handler);
      gl_msvc_inval_initialized = 1;
    }
}

# else

jmp_buf gl_msvc_inval_restart;

void cdecl
gl_msvc_invalid_parameter_handler (const wchar_t *expression,
                                   const wchar_t *function,
                                   const wchar_t *file,
                                   unsigned int line,
                                   uintptr_t dummy)
{
  longjmp (gl_msvc_inval_restart, 1);
}

# endif

#endif
=============================== m4/msvc-inval.m4 ===============================
# msvc-inval.m4 serial 1
dnl Copyright (C) 2011 Free Software Foundation, Inc.
dnl This file is free software; the Free Software Foundation
dnl gives unlimited permission to copy and/or distribute it,
dnl with or without modifications, as long as this notice is preserved.

AC_DEFUN([gl_MSVC_INVAL],
[
  AC_CHECK_FUNCS_ONCE([_set_invalid_parameter_handler])
  if test $ac_cv_func__set_invalid_parameter_handler = yes; then
    HAVE_MSVC_INVALID_PARAMETER_HANDLER=1
    AC_DEFINE([HAVE_MSVC_INVALID_PARAMETER_HANDLER], [1],
      [Define to 1 on MSVC platforms that have the "invalid parameter handler"
       concept.])
  else
    HAVE_MSVC_INVALID_PARAMETER_HANDLER=0
  fi
  AC_SUBST([HAVE_MSVC_INVALID_PARAMETER_HANDLER])
])
============================== modules/msvc-inval ==============================
Description:
invalid parameter handler for MSVC runtime libraries

Files:
lib/msvc-inval.h
lib/msvc-inval.c
m4/msvc-inval.m4

Depends-on:

configure.ac:
gl_MSVC_INVAL
if test $HAVE_MSVC_INVALID_PARAMETER_HANDLER = 1; then
  AC_LIBOBJ([msvc-inval])
fi

Makefile.am:

Include:
"msvc-inval.h"

License:
LGPLv2+

Maintainer:
Bruno Haible
================================================================================
-- 
In memoriam Bernhard Bästlein <http://en.wikipedia.org/wiki/Bernhard_Bästlein>