[Savannah-cvs] [Compromise2010] (edit) recap and timeline

[Beuc]

??changed:
-Current info at:
-
- * http://savannah.beuc.net/
- * http://www.fsf.org/blogs/sysadmin/savannah-and-www.gnu.org-downtime
- * http://savannah.gnu.org/news/
-
-TODO: recap and reorganize
We (GNU Savannah and the FSF) value transparency, so here are the details on 
this page.

Recap: there's been a SQL SELECT injection leading to a leak of unsalted MD5 
account passwords, some of them discovered through online passwords recovery 
services, leading in turn to project membership and admin access, used for 
vandalism on the 'www' project that backs www.gnu.org.
www.gnu.org normally serves static pages, but PHP was found enabled, so 
crackers installed a reverse shell there as well.
Root access is separately maintained and no evidence of root access was found.
No other exploitable SQL injection has been found so far, thanks to the SQL 
parametrization work done back in 2007.
To be safe, we've reinstalled the system and restored the data from a trusted 
backup, November 23th circa 12:00 GMT.

Counter-measures:

 * Crack analysis before re-enabling any service
 * SQL injection fix and code audit before re-enabling the web front-end
 * Removed all passwords (users and system) and sessions
 * Use crypt's SHA-512 for passwords, and phpass's entropy code for salt
 * Enforced password strength (through passwdqc)
 * Added logs analysis reporting tool that keeps us informed of SQL injection 
attacks
 * Upgraded friend website gna.org to our version of Savane

In progress:

 * Auditing changes between the 23th and the 27th to see what was committed (no 
code commits found so far)

Thanks to FSF sysadmins Bernie Innocenti and Ward Vandewege, Savannah Hackers 
Sylvain Beucler, Brian Gough, Michael J. Flickinger, Jim Meyering; Openwall 
hacker Solar Designer; Savannah users for their support.


Timeline:

 * 2010/11/24 21:30 UTC: SQL SELECT injection attack originated from Tbilisi, 
Georgia, access to user encrypted passwords
 * 2010/11/24 21:27 UTC: one Savannah admin password cracked, account 
compromised
 * 2010/11/26 16:02 UTC: cracker gained membership to the www project
 * 2010/11/26 23:51 UTC: cracker tested commit to the www CVS repository
 * 2010/11/27 00:51 UTC: cracker defaced www.gnu.org
 * 2010/11/27 01:35 UTC: cracker committed a reverse shell using unexpectedly 
enabled PHP support
 * 2010/11/27 01:36 UTC: notification of the intrusion
 * 2010/11/27 01:37 UTC: website restored
 * 2010/11/27 04:42 UTC: emergency fix to Savane code (unknowing that an admin 
account was still compromised)
 * 2010/11/27 19:05 UTC: new cracker activity on www.gnu.org - we shutdown the 
machines
 * 2010/11/27 21:35 UTC: reinstalled www.gnu.org
 * 2010/11/29 15:23 UTC: reinstalled Savannah machines to be safe
 * 2010/11/29 21:30 UTC: access to the base host restored, extracting 
incremental backup from the 23th
 * 2010/11/29 23:30 UTC: finished diagnosing original attack
 * 2010/11/30 12:30 UTC: data transfers in progress
 * 2010/11/30 13:30 UTC: read-only access to source repositories
 * 2010/11/30 14:30 UTC: write access to source repositories
 * 2010/11/30 16:30 UTC: data transfers finished
 * 2010/11/30 18:00 UTC: access to downloads and GNU Arch
 * 2010/11/30 21:00 UTC: audited code and found no other SQL injection
 * 2010/11/30 22:30 UTC: found trace of earlier attack on Nov 23th 04:00
 * 2010/11/30 22:45 UTC: stopped write access
 * 2010/11/30 23:45 UTC: found trace of earlier read-only SQL injections as 
back as January, but none with actual account cracking
 * 2010/12/01 00:55 UTC: after fishing through logs, it appears that there was 
no other account cracking
 * 2010/12/01 11:00 UTC: restored write access
 * 2010/12/02 08:02 UTC: web front-end improved and re-enabled


--
forwarded from http://savannah.gnu.org/maintenance/address@hidden/maintenance