I've just spent the last three days trying to get my machine back up. I was away at the Google Ski trip on Thursday/Friday, and I came home Friday night when my machine went zombie... I could ping it, I could connect to it, but nothing would ever respond. Through a combination of complete errors[1], it took my 3 days and many hours to fix the damn thing, but as near as I can tell, my machine got a virus sometime thursday morning, while I was not logged in (I was on a bus to Tahoe).

I know, a Linux virus? Its technically possible, see The ELF Virus Writing HOWTO, but actually getting infected should be nearly impossible (you'd have to run an infected executable, as root no less). This virus modified most of the files in /bin, and then used chattr to set all the ext2 attributes on the files (so they couldn't be deleted). It also managed to attach itself to every process I ran (I think), and a good deal of the one's I just looked at with an infected ls.

Other symptoms, it seemed to fork a process to do this work, so ps would report two copies of everything running. I'm not sure what the other process did, but it was apparently the parent, and didn't exit or wait(2) for its children, so I started getting zombies everywhere. Which I should have noticed more because I had trouble booting the machine, it would hang trying to egrep /etc/conf.modules... At one point INIT completely stopped reaping processes.

I managed to brut force copy and re-install rpms (and play with chattr myself) to get myself mostly back to the point of things running, at least until I can re-install the damn box.

So, that was my wasted weekend. The box was running linux firewall, though it did have wu-ftpd open, in addition to late model opensshd and httpd and named. I didn't think any of those had known exploits, but its possible that the firewall was down (I occassionally disabled it when I needed to open something up temporarily). I'm not all that thrilled to figure it out at this point, this has already caused me enough trouble. No doubt its still got software with the virus in it, or some exploitable stuff still installed. I did notice that my wtmp file for the month of january disappeared...

[1] Errors? Well, I couldn't see what was on the screen, I thought that was due to the machine being dead and the screen blank on, so I hard rebooted it... only to realize I'd plugged the keyboard into the wrong machine, so who knows if I could have unblanked the screen. Because of the zombie state, I figured it was a bad drive, plus I've been getting all sorts of disk errors on this box, so I tossed one of the drives. It probably wasn't bad. Then I thought it was a memory problem (I did get a console message saying NMI received, and the weird INIT behavior...), so I ran around trying to find low profile PC133 memory (after stealing some memory from another server and finding it was too tall), and then still having trouble... albeit not too much. Then I finally noticed some executable file oddities (thanks rpm --verify and md5sum). I also spent a huge amount of time doing an rsync of everything off the box so I could re-install it (since I can only have one set of drives on the 3ware card... I think its a 2 port limitation).