[Qemu-devel] [V3 PATCH 03/13] linux-user: Properly Handle semun Structure In Cross-Endian Situations

[Tom Musta]

The semun union used in the semctl system call contains both an int (val) and
pointers.  In cross-endian situations on 64 bit targets, the value passed to
semctl is an 8 byte (abi_long) value and thus does not have the 4-byte val
field in the correct location.  In order to rectify this, the other half
of the union must be accessed.  This is achieved in code by performing
a byte swap on the entire 8 byte union, followed by a 4-byte swap of the
first half.

Also, eliminate an extraneous (dead) line of code that sets target_su.val in
the IPC_SET/IPC_GET case.

Signed-off-by: Tom Musta <address@hidden>
---
V2: Reworked per comments from Peter.

 linux-user/syscall.c |   13 +++++++++++--
 1 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index 229c482..bee1f4e 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -2647,9 +2647,18 @@ static inline abi_long do_semctl(int semid, int semnum, 
int cmd,
     switch( cmd ) {
        case GETVAL:
        case SETVAL:
-            arg.val = tswap32(target_su.val);
+            /* In 64 bit cross-endian situations, we will erroneously pick up
+             * the wrong half of the union for the "val" element.  To rectify
+             * this, the entire 8-byte structure is byteswapped, followed by
+            * a swap of the 4 byte val field. In other cases, the data is
+            * already in proper host byte order. */
+           if (sizeof(target_su.val) != (sizeof(target_su.buf))) {
+               target_su.buf = tswapal(target_su.buf);
+               arg.val = tswap32(target_su.val);
+           } else {
+               arg.val = target_su.val;
+           }
             ret = get_errno(semctl(semid, semnum, cmd, arg));
-            target_su.val = tswap32(arg.val);
             break;
        case GETALL:
        case SETALL:
-- 
1.7.1