it has several xhtml capable markup options, most useful of which is Markdown and itex2MML for the math. it has a latex export and is being used for several large collaborative math sites like http://ncatlab.org/nlab/show/HomePage.
I don't know enough about the architecture of org-mode or Instiki to know how easy or difficult this would be, but I would like to be able to edit an orgmode file with my usual latex environments and links and export it to an instiki, and vice versa. Essentially you could use the instiki as a web-viewable and editable orgmode file. I suspect that this is probably almost doable now and would appreciate some insights from the list.