Bash Scripting: Why didn't I start this earlier?

Learning bash scripting ended up turning what used to be a 30 minute manual server build process into a perfect lean, mean, server building machine.

I'm a little embarrassed to admit this. I've been building software for fifteen years, and I'm only just now getting into bash scripting.

I had never seen myself as a "DevOps guy" or a "Systems Administrator", so I knew just enough to build good, sturdy, secure servers with little flash and lots of functionality. No flare. No grace. I concentrated on software.

But in the last year or so, I've come to embrace "Unix as an IDE". (Before that, I was a MacVim guy.) The tools are impressive and robust, and having a remotely hosted, SSH-driven development environment is just so convenient and enjoyable. As a result, I've logged nearly a thousand hours on my dev server via iTerm and an SSH connection in the last 6 months.

Despite my growing comfort with the command line, I never really concentrated too much on automating my sysadmin tasks. No premature optimization, I thought. I did everything by hand. All the server builds, the maintenance and upkeep, all done one line at a time. I've even tried Chef and other related tools; I just didn't like them.

But last week I ran into a problem: we had a huge increase in traffic and activity. We needed to scale significantly, and fast. I was used to building one server at a time, at my leisure, all by hand. (I've always felt a zen calm while building servers, so I enjoyed building them by hand.) But that wouldn't cut it now. I needed 8 new app servers in the next few hours.

I bit the bullet and wrote the first real (ie, longer than 10 lines) bash script I've ever written -- to automate my app server builds. While I had the server build procedure committed to memory, it still took me about 90 minutes to write the script and work out all the kinks. (I had a few issues with accidentally calling the script from the wrong path, and made some other similar newbie mistakes.)

But the payoff was the most exciting part. This 90-minute investment of my time ended up transforming what used to be a 30 minute, manual, and error-prone server build process into a perfect lean, mean, server building machine. Now it only takes me 5 minutes to build a server, and they're all perfectly configured out of the box. All due to a crash course in bash scripting.

Part of me wishes it hadn't taken me 15 years to get to this point, but the other part of me recognizes that everyone learns their lessons in a different order. Either way, I'm happy to have finally added bash to my toolbelt.

What do you guys do? How do you manage your server builds and automation? I'd love to see what the full spectrum looks like.