Try this:
\override Clef.font-size = #-3
Adjust size to taste.
Well worth spending the time to read and digest the material in the Internals section of the documentation. Look up Clef in the 'All layout objects' section, and then look at the bottom to see all the interfaces Clef implements, and you will find everything you need. By the way, the clef-interface supports the property full-size-change which is also useful.
Andrew