Re: AJAX-powered search auto-completion for the docs

[Reinhold Kainhofer]

Am Freitag, 2. April 2010 02:58:38 schrieben Sie:
> Ok, I've run out of excuses to avoid tackling issue 815:
> http://code.google.com/p/lilypond/issues/detail?id=815
> 
> See it in action here:
> http://reinhold.kainhofer.com/~lilypond/ajax/user/lilypond/index.html
> 
> Looks pretty cool.  Unfortunately, there's some problems.
> 
> 1.  It uses PHP, which I know nothing about.  I thought this was
> named after a drug, but 2 minutes on wikipedia suggests that I was
> thinking of PCP.  30 seconds of skimming the wikipedia page about
> PHP suggests that it's similar to, or based on, perl.  It also
> reminded me that PHP appears to have an obscene number of security
> flaws -- I vaguely recall seeing "PHP security patches of the
> week" for several months on LWN.

Wow, I thought any CS student would definitely have heard of the most-used 
scripting language on the web. Most CMS sytems (e.g. Joomla, drupal, typo3), 
most database frontends (phpmyadmin, phppgadmin, etc.), and the majority of 
all web applications is written in PHP, so its security cannot be that bad in 
general...

> 2.  lilypond.org is hosted on a shared server, I don't know how
> much CPU power this auto-completion would require, and I don't
> know how much CPU power it would be polite to use.

I don't think the CPU power will be the problem. It might rather be the disk 
access, since each request reads in the file containing the index data.

> 3.  The patch was created almost a year ago, and texi2html has
> undergone a lot of changes since then.  It might still apply
> cleanly, or it might not.

No it will not, since the whole documentation file layout has changed. I might 
try to update it to current master, though.

> With those factors in mind, we have a few options:
> a)  Reject the patch.  We say "yeah, that looks cool, but sorry,
> we can't fit it into lilypond.org and/or nobody wants to work on
> the integration / fix the remaining issues / etc".

Once you have started using that patch, you won't accept that as a viable 
option any more ;-) That search box is even better than the TOC on the left...

> b)  Accept the patch, but disable it in a normal build.  We
> integrate it with the rest of our build system and docs, but leave
> it as an alternate configuration option.  People building the docs
> locally,

No, that won't work. You need to have the docs installed on a web server. Due 
to security issues, a web browser's JS impelementation cannot access the index 
file on the harddrive. Only a server-side script can do that.

> c)  Accept the patch, disable it on a normal build, enable it on
> lilynet.net (which IIRC has tons of resources), and then create a
> doc.lilypond.org which redirects to the AJAX-enabled docs on
> lilynet.net (or any other server that people want to volunteer).
> That way, we don't disrupt the main site, but normal users can
> still play around with the searching stuff.

We can use that to check out whether there are any resource problems created 
by the ajax.

> d)  Accept the patch, make it default, host it on lilypond.org.
> Unless there's a substantial amount of interest from skilled PHP
> and web security people, I do not like this option.

About the security: 
I have not done a full security check. 

The #1 security problem in general is SQL injection attacks. Since there is no 
SQL involved, we don't have that problem.


However, the current version of the patch has one problem, which needs to be 
addressed: It currently allows practically any file on disk to be searched. To 
fix this, we need to ensure that $filename (created from the "lang" and 
"manual" query strings) is below the current dir. I.e. if 
manual="../../../../etc/passwd\0", we need to disallow this. Apparently the 
string is properly escaped anyway, so it will only be possible to access files 
named *.$lang.idx (i.e. giving \0 or so will not result in a truncated file 
name, but the .$lang.idx will always be appended).

The script does not return the verbatim contents of that file, either, but one 
can still get most of the contents.

As I said in my original submission, this is just a proof of concept. If there 
is interest, one needs to tighten these things, of course.

Cheers,
Reinhold

-- 
------------------------------------------------------------------
Reinhold Kainhofer, address@hidden, http://reinhold.kainhofer.com/
 * Financial & Actuarial Math., Vienna Univ. of Technology, Austria
 * http://www.fam.tuwien.ac.at/, DVR: 0005886
 * LilyPond, Music typesetting, http://www.lilypond.org