Hacked. A Short Story.

It all started when I was asked (as a freelancer) to configure exim4, so that newsletters would not fall into spam folders. They even sent me a link to a tutorial.

I thought the work would take a couple of hours, including the DNS update, but I was wrong. After logging in as root, I started my favorite screen by running the screen -x command as usual, and watched a curious scene in the favorite for many of you /dev/shm folder. The attacker either did not bother to close the screen session, or was still working in it. That’s where the quest begins:

The first thing I did was looking at the things the attacker was doing:

wget http://ravenul.zzl.org/it/noi/up/8.txt
mv 8.txt list.txt
php lol.php
php lol.php
netstat -an | grep :22
w
rm -rf list.txt
w
rm -rf .x
netstat -an | grep :22

Apparently, he sent out spam and ran some .x file (or was it a directory?). He checked the ssh connection and there was also an archive with the php script lol.php that I, unfortunately, forgot to save.

The output of last and who commands did not show anything special, there were no root sessions during the month, and the owner of the server confirmed this. However…

$ lsof -ni | grep ssh

Showed the established connection with the IP 172.190.125.14, that I killed right away.

I paid attention to /usr/sbin/sshd

$ ls -la /usr/sbin/sshd
-rwxr-xr-x 1 root root 320724 Oct 11 23:29 /usr/sbin/sshd

There was sshd0 next to sshd

$ ls -la /usr/sbin/sshd0
-rwxr-xr-x 1 root root 757356 Jul 31  2010 /usr/sbin/sshd0

File removal took me nowhere:

$ rm -f /usr/sbin/sshd
rm: cannot remove `/usr/sbin/sshd': Operation not permitted

Let’s move on:

$ lsattr /usr/sbin/sshd
-u--ia------------- /usr/sbin/sshd
$ chattr -aui /usr/sbin/sshd
$ rm /usr/sbin/sshd
$ lsattr /usr/bin/* | grep -v -- '-------------------'
-u--ia------------- /usr/bin/ssh
$ chattr -aui /usr/bin/ssh
$ rm /usr/bin/ssh

I reinstall openssh-server and openssh-client. All seems good, there’s no threat anymore, and nothing suspicious was found. I decided to update the system, besides, tzdata was quite old.

I checked /etc/apt/sources.list and /etc/apt/sources.d. All the files were fine, there were no unusual strings, the dates had not been changed for about a year. After apt-get update, I applied all the security updates on Debian Lenny, including new kernel. Now, we need to reboot. I request the KVM just in case (and it was the right thing to do), and waited.

The next day the KVM was provided. I typed reboot and I got dozens of segmentation faults. My hair started to turn gray, hands were shaking. I guess you can imagine the situation. As they say, “Don’t TOUCH it when everything works”, but after detecting the break-in, I had to apply updates and reboot.

To cut the long story short, I pulled myself together and decided to find out what was wrong and logged in as a single user mode. The mount command leads to the segmentation fault after each call, even without any parameters. The file system is readonly and there’s nothing we can do about it. /etc/fstab is fine, df operates as well. Somehow, the date command also leads to the segmentation fault. I run the fsck.ext3 /dev/md0 disk check (raid1). Everything is okay, no abnormalities found. What’s the matter then? That’s when I start to think that I am the one to break the system down after I updated the tzdata package that is connected with time. At this very moment, the DSL connection with my provided is lost. Reboot the modem — the connection returns. Great!

The server owner is angry, as the server has been down for several hours. He decides to write a ticket to the support, while I keep examining the system. The most adequate solution to my mind seems to reboot the system and log in with liveusb, so that the disk would be RW. I begin to debug mount using available techniques. gdb is not installed, but I have ldd. It did not show anything interesting. I also used export LD_DEBUG=all, but it did not find anything special either. The segfault happens after the initialization of all the libraries. That’s when KVM told me that it had been disabled. Got it — the support put their hands on. I stepped away from the laptop and started to think more…

While standing and breathing the fresh air, the following thought comes to my mind: «What if the files that cause the segmentation fault are spoofed?». No sooner said than done. So, there I am waiting for what the client is going to say about the ticket to the support. A few minutes later he sends me the respond:

The partition table has been damaged. It’s impossible to restore it by using rapid methods.
We can engage our system administrators for recovery.
You can also do it yourself. In this case, we recommend using Gpart (http://packages.debian.org/en/sid/gpart).

WTF?! I tell the client that it’s not the case, as fsck has performed the disk check and has not found any violations in the file system. While the client is writing his answer to the support team, the access to KVM returns and I watch the same futile attempts to call mount, hdparm, that are not installed in the system. I also see some fdisk calls.

The latter outputs the following:

$ fdisk -l
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f0571
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       18480   148440568+  fd  Linux raid autodetect
/dev/sda2           18481       19457     7847752+  fd  Linux raid autodetect
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       18480   148440568+  fd  Linux raid autodetect
/dev/sdb2           18481       19457     7847752+  fd  Linux raid autodetect
Disk /dev/md0: 152.0 GB, 152003018752 bytes
2 heads, 4 sectors/track, 37110112 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1: 8036 MB, 8036024320 bytes
2 heads, 4 sectors/track, 1961920 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md1 doesn't contain a valid partition table

Basing on the last Disk /dev/md0 doesn't contain a valid partition table, the support concluded that the problem was in the partition table. Oh, really? fdisk never saw the partition table of the software raid. I send all my thoughts to the client and begin to develop the cunning plan. I can only imagine how the support story would end and how much time it would take if the client accepted their help.

I look at the modification date of the /bin/mount. It’s the time of the last server’s boot. Rebooting again and check the date one more time. It’s the time of the last server’s boot. Strange. This means that something modifies this file during the boot and I should do something about this “something”.

/tmp is readonly. To upload the file to the server, I need a file system with write access. That’s when I remember about /dev/shm. I bring the interface up, assign an IP address, and download deb mount package for lenny. Unpack and run it, and voila! It works! I remount the file system and it's RW now. Things got rolling!

Checking files in /bin/, I see the following:

$ ls -latr /bin
-rwxr-xr-x  1 root root  96408 Nov 15 18:11 vdir
-rwxr-xr-x  1 root root  30896 Nov 15 18:11 pwd
-rwxr-xr-x  1 root root  30712 Nov 15 18:11 ping6
-rwxr-xr-x  1 root root  24252 Nov 15 18:11 nc.traditional
-rwxr-xr-x  1 root root   8612 Nov 15 18:11 mountpoint
-rwxr-xr-x  1 root root  68208 Nov 15 18:11 mount
-rwxr-xr-x  1 root root  32244 Nov 15 18:11 mknod
-rwxr-xr-x  1 root root  39144 Nov 15 18:11 loadkeys
-rwxr-xr-x  1 root root  17244 Nov 15 18:11 kill
-rwxr-xr-x  1 root root   9764 Nov 15 18:11 fgconsole
-rwxr-xr-x  1 root root  26216 Nov 15 18:11 false
-rwxr-xr-x  1 root root   8524 Nov 15 18:11 dmesg
-rwxr-xr-x  1 root root  96408 Nov 15 18:11 dir
-rwxr-xr-x  1 root root  51988 Nov 15 18:11 dd
-rwxr-xr-x  1 root root  59148 Nov 15 18:11 date
-rwxr-xr-x  1 root root  49440 Nov 15 18:11 chgrp
-rwxr-xr-x  1 root root  30956 Nov 15 18:11 cat
-rwxr-xr-x  1 root root  12252 Nov 15 18:11 bzip2recover

The modification date of files changes every 3 minutes and 10 seconds. Start to look though crontabs and find nothing there. lsof does not help me to catch the process that changes files. Printing ps auxww, I see that some cat /sys/class/net/lo/operstate process is hanging there.

I download the package with the kill utility, rename the /bin/cat в /bin/cat_ file and kill the process. Files are no longer being modified. Bingo! Now, I should replace all the modified files with the original ones. So I download the necessary packages and install them via dpkg -i *deb (after checking the creation date of the dpkg file). After all the changes made, I type reboot and watch the KVM window, fingers crossed. The boot is successful, the website works. Next, I scan the infected files that I copied, using clamav, and detect Linux.RST.B-1 FOUND. Who is there to say that there are no viruses under Linux? By the way, the virus is dated to 12/21/2001…

Scanning sshd and ssh does not give anything. Apparently it’s about the modified ssh and sshd. The former one is likely to send the login and password when successfully connected to the server, while the latter one likely allows to connect to the server for everyone with a specific password.

P.S. If there’s anything wrong with the commands, then I apologize; I wrote many of them from memory. I did not feel like configuring exim4 anymore. Also, I did not ask for money. But what would he pay me for? I did not finish the main task anyway :)