The correct result should be symmetric, because rosser() is symmetric. I implemented myself the orthogonal transformation to upper Hessenberg form by calling the LAPACK dgehrd (general) and dsytrd (symmetric) functions and it seems that the result matches what I get with dgehrd. Therefore, I suspect that the problem lies with hess() not recognizing (using) the fact that the argument is symmetric.
I do not know how symmetric matrices are handled in Octave. Is it checked every time? Is there a mark on the matrix for efficiency? It could also be the case that rosser() does not mark the matrix as symmetric and so hess() uses the wrong algorithm.