More Hacking

Some people have requested more information about the hacking of my box. Alert: nerd content ahead!

I wish I could say that I knew right away that my box was hacked, but I didn’t. I generally don’t have access to my box from work, and certainly don’t have all that much free time to monitor it in any case. The first hints involved degrading performance of my various sites. First things were just slow, then things (Tomcat, etc) would just crash. Eventually I closed my eyes, held my breath, and remotely rebooted my box. Everything came back up, but the problems seem to persist.

Previous to this, I had recieved numerous emails from cron telling me that certain tasks (cleanup, etc) were failing with errors. Being lazy, I ignored these errors for a while, hoping they would go away (duh!). After enough time went by, and they didn’t go away, I investigated more and discovered my box had been hacked.

After poking around the /etc/cron.daily/ directory, I noticed that the logrotate script had been modified. I opened it up, and sure enough someone had added a line telling cron to send an email (to the hacker) with the contents of a particular file. I opened up the file (.sniffer) and was horrified to see the keystroke input of every call to su, passwd, and mysql. Everytime I changed a password or logged-in to something, my keystrokes were being captured and emailed to the hacker.

Needless to say I removed the mailing line from logrotate and changed all my passwords. However, this file is still being written to, by some process I can’t seem to identify. So, my box is still hacked, and likely only a format and re-install will fix it. I also have no idea how my box was hacked. I have heard about an exploit for a http log analyzer I was using, so I’ve disabled it.