I'm a bit late to the party here but hope I can contribute something.
About two years ago I made the switch to using Lilypond exclusively as I was getting tired of exporting pdfs with basic music notation and overlaying graphics in Illustrator or some such. This was a real pain when doing parts or making even the smallest of changes. Since then I've used lilypond for a lot of pieces, all of which have some idiosyncratic notational devices. There's very little I haven't been able to implement successfully in lilypond (really just 1 thing that still evades me... customised barlines aligning with first beat of a measure...).
I thought it might be interesting for those wondering what's possible in lilypond to see some examples from the field. I've put together a page collating those things I've done in the past year or so. Many of these notational devices seem to be fairly standardized nowadays; or at least the symbols appear consistently, even if the interpretation of their meaning can vary a lot.
It would be great to develop a contemporary notation library for lilypond making these notations readily available to any user, I'm not sure what that would involve but I know it could be a major selling point for lilypond in the contemporary music world.
For completeness sake here's index of what you see in the linked PDF (naturally eveything you see here is generated using lilypond alone):
1) Split-stem chords/clusters
2) Stemmed glissando
3) Bezier glissando w/arrowhead
4) Variable width bezier glissando
5) Vibrato with variable/random period and slope
6) Interruptive polyphony
7) Lachenmann pressed bow
8) Billone beat notation
9) Pencil line emulation (after Charlie Sdraulig)
10) Sciarrino style jet-whistle
11) Woodwind fingering staff
12) Carin Levine style flute multiphonics
13) Klavierstuck X proof-of-concept
14) Sciarrino tremolo (with bezier hairpins)
15) Stockhausen cluster-glissando
16) Notation from a work of my own for violin
Hope this might be illuminating for others; lilypond is great for this kind of stuff.
best wishes,
piaras hoban