Well, this is actually a feature:
,----[ C-h v LaTeX-fill-break-at-separators RET ]
| LaTeX-fill-break-at-separators is a variable defined in ‘latex.el’.
| Its value is (\\\( \\\) \\\[ \\\])
|
| Documentation:
| List of separators before or after which respectively a line
| break will be inserted if they do not fit into one line.
`----
,----[ (info "(auctex)Filling") ]
| -- User Option: LaTeX-fill-break-at-separators
| List of separators before or after which respectively linebreaks
| will be inserted if they do not fit into one line. The separators
| can be curly braces, brackets, switches for inline math ('$', '\(',
| '\)') and switches for display math ('\[', '\]'). Such formatting
| can be useful to make macros and math more visible or to prevent
| overfull lines in the LaTeX source in case a package for displaying
| formatted TeX output inside the Emacs buffer, like preview-latex,
| is used.
`----
The "if they do not fit into one line" part is a bit unclear. But I
think it means that a newline is inserted before \( if the formula is
longer than what's still fitting into the current line, and a newline is
inserted after \) when the remainder of the paragraph doesn't fit on the
current line anymore. Well, that second case seems dubious to me. I
don't understand why one would like to have that.
Bye,
Tassilo