I've been teaching my whole life. I started out teaching music theory at a music school. I tutored math and physics. I've taught courses on JavaScript. I've also taught more technical topics like electromechanical control systems and machine learning.

Over the years I've seen many students succeed, but I've also seen many fail. The failures are more interesting to me, and I've come to develop a theory of why people fail to learn complex skills. If you're about to pick up a new skill, I think an understanding of why people fail can help you avoid failure yourself.

Most Fail Because They Quit

Without a doubt, the most common failure mode is quitting. I have very rarely, if ever, seen a student fail because they weren't "good enough" or "smart enough". Students that stick with it succeed, and students that quit fail.

This alone isn't enough to understand failure. Maybe if there were a handful of common failure modes it would be useful to talk about each one. But it turns out that to ask "why do we fail" is really to ask "why do we quit?" We can also ask the inverse, "why do we succeed?"

Frankly, it really is all about practice. I've never seen a student practice guitar for 1,000 hours and then fail. The same goes for computer science, the same goes for math and physics. Many people dislike Malcolm Gladwell's form of journalistic pseudoscience, but I've seen his "10,000 hours" theory play out time and time again.

Great -- so if you practice, you get better. This still isn't too helpful. If we're concerned about avoiding failure, we need to dig deeper still and talk about why people fail at achieving the practice hours they need. This is where it starts getting interesting.

What I've seen is that there are three primary reasons students quit: confusion, frustration, and boredom. There are others, but those three are the most common.

When both teacher and student solve those three problems success rates shoot way up. Let's talk about each in turn and discuss some ways to get around them.

Confusion

Learning a new complex or valuable skill is difficult. If it were easy, everyone would know it, and it wouldn't be so valuable. Confusion is the first problem that students tend to run into.

Let's say you decide to learn computer programming as an example. The first question you're faced with is "what language should I learn?" Well, there are dozens. How do you choose from those? Maybe you know you want to write Android apps; it's good to have a specific target like this in mind, but even narrowing the scope only helps a little. Do you pick Java or Kotlin? How do you set up your emulator? What are all these tools you need? What is object oriented programming?

It's way worse if you have no idea what type of programming you want to do. How do you choose between Python, Ruby, JavaScript, Erlang, Scala, Go, etc. The landscape is just so vast.

Think about what it takes to build a simple, static webpage, and get it hosted on the internet. Not only do you need to learn HTML, CSS, and possibly JavaScript, you also have to understand what each is for. If you're an experienced programmer reading this, that might sound trivial to you, but imagine someone starting on day 1. Just understanding what HTML, CSS, and JavaScript each do and what their role in this ecosystem is can take hours of work and research.

After that, you also need to understand the difference between a domain name and a host. There's shared hosting and virtual machines. If you want to run your own webserver, you've got to decide between Apache and nginx. Should I use DigitalOcean, AWS, GoDaddy, or one of the thousand other companies out there? Again, the ecosystem is vast.

Many students quit at this point. Stepping into a new world is incredibly daunting. I don't have numbers to back this up, but a good portion of people quit right here, before they even get started.

Frustration

Frustration is a huge problem and is probably responsible for more failures than any other mode. Let's say you've figured out exactly what you want to learn. Now it's time to actually learn it!

But learning new skills is hard. Learning new things is hard. That's what learning is -- it's the process of going from not knowing anything about a skill or subject, to knowing lots about it.

Learning to program is frustrating. Why doesn't this code work? Why does my page crash? How come this doesn't do what I expected? There are also times when a student just can't figure out how to solve a problem. The patterns aren't intuitive to them yet. This causes a great deal of frustration, and often the student will put down the problem and never pick it up again.

I see this a lot with learning to play an instrument as well. Learning guitar is tough. The fingerings are awkward and often painful to a beginner. Transitioning from one chord to another is difficult. These students know what they have to do, but they don't have the muscle memory and articulation to do it yet. This is frustrating. Eventually the student puts down the guitar and never picks it up again. How many of your friends in high school got a guitar for Christmas one year, played for a week and then never touched it again?

I find that students will often make up excuses to justify their quitting to themselves. "My fingers just don't move like that" for guitar, or "my hands are too small" for piano, or "I'm just not a math person" for STEM subjects. These excuses provide a level of cover for the student to protect their egos, when the truth is that they've really just gotten frustrated and quit. If Django Reinhardt, who has only two fingers, can be considered among the best guitar players in history, I assure you that your fingers can move like that.

It's important to understand that frustration is a natural part of the learning process. It gets easier over time if you stick with it. I always tell students that there will be times when they get frustrated and want to quit. Just telling them that helps them get past it -- they now know that everyone gets frustrated and wants to quit at some point.

Boredom

Learning a skill is more about practice than it is about play. Often students will get through the initial confusion and frustration, reach a level where they can play a couple of songs on guitar or throw together a couple of simple programs, and then plateau. The problem here is that continuing to grow this skill through practice becomes boring.

With computer programming, the types of homework problems you get in a 101 course (Fibonacci sequence, bubble sort, etc) are great practice, but frankly quite boring. Similarly, running scales all day on guitar or piano is boring. Students will often revert to play, ie, playing the same song over and over again without stretching their abilities, or else they get too bored with what it takes to reach the next level and quit.

At this stage it's important to be able to balance practice and play. Writing code to design 101-level algorithms is boring on its own; but finding a larger motivating project that requires those algorithms is not. Playing scales and practicing fingerings on its own is boring; but picking a challenging song that uses those scales is not.

This is a bit pseudosciency, but I claim that teachers and students need to figure out ways to trigger the dopamine rewards at this stage. Every hour of practice should be related to an hour of play, otherwise the student will say to themselves "this is not what I expected programming/guitar/physics to be like" and then quit. It's another ego-covering justification. Of course it's not what you expected; you didn't actually know before you started. But if your teacher only ever gives you practice and there's no play or larger goal, that skill will become fatally boring.

How to Succeed in Learning

If you want to succeed at learning (or teaching) a new skill, the ultimate goal is to rack up hours of practice towards the skill. This will sound a bit Malcolm Gladwell-y, but let's say that the three tiers of skill occur at 100, 1,000, and 10,000 hours of practice. Those tiers translate roughly to basic understanding after 100 hours (ie, having a rough understanding of how programming works), proficiency at 1,000 hours (ie, I could get a junior/entry level programming job), and expertise at 10,000 hours (ie, I can get basically any job I want).

To reach those tiers, you need to solve for confusion, frustration, and boredom. When I teach entry-level computer programming, for example, I teach "Building classic video games in Processing". Let's break that approach down.

(Processing is an all-in-one IDE and programming language based on Java; it's geared towards canvas drawing/animation and mouse/keyboard interaction. You don't see it being used "professionally" but it is a fantastic educational tool.)

Confusion: building video games in Processing readily solves the confusion problem. You use one tool as your platform. You write in one language. All the documentation and tutorials are in one place. It doesn't matter that you're not going to become a professional "Processing developer". Processing is simply the platform on which you become a programmer. Become a programmer first, and then figure out where and how to specialize. Don't worry about the huge ecosystem of programming languages, technologies, libraries, and so on, just focus on this one platform that you can install within second and immediately get everything you need to start with.

Boredom: when I teach this topic, I show students how to build Pong in about an hour in Processing, and then tell them to pick 5 other classic Atari arcade games and then go build them. Programming video games is challenging; you'll need to learn about data structures and algorithms, object oriented programming, and so on. It's also fun -- Processing gives you an immediate feedback look. Writing algorithms to solve the Fibonacci sequence is boring; the output is just numbers in a console. Writing video games is rewarding and gets that dopamine flowing. It also makes the process very tangible and visual. Accidentally making a number negative when it should be positive has the clear and distinct effect of moving something in the wrong direction. Overall, recreating these classic arcade games forces you to practice the skills while also contributing to the sense of play and work towards interesting and achievable projects.

Frustration: another reason I teach students to recreate classic video games is because they're narrower in scope than other types of projects students might be interested in. Deciding to build a new social network might be an interesting project, but it's way bigger than the student expects it to be. On the other hand, classic video games tend to be small and achievable. A novice can build Pong in a day. Breakout in two days, asteroids in three. Being able to start and finish many small projects is more rewarding than starting one giant project that has no end in sight. The scope of these games is well-defined, and that keeps frustration in check. It's a lot easier to avoid frustration when you're able to write out a list of 100 discrete tasks that need to be accomplished and check them off one at a time. You also know what the end product is supposed to look like.

Keep in mind that accomplishing 1,000 hours of practice might require 2,000 hours or more of total effort. If you're building your third video game, there's a whole bunch of stuff that you already know how to do -- setting up the canvas, setting drawing modes and colors, things like that. Those tasks might have counted as practice the first or second time you did them, but they no longer count the 10th time around. They're simply reinforcing the muscle memory; they become play.

The same applies to guitar or piano. It's only practice if you're doing something new(ish); it's only practice if it's hard. The first few times you play Wonderwall count as practice, the next 200 times you play it are just that: play.

For this reason it's important to continually level up by choosing more difficult projects as time goes on. Otherwise you'll plateau if you keep doing easy things; you'll rack up a whole bunch of hours that are all play but don't actually contribute to the development of new skills. Every project has a certain amount of boilerplate that goes into it, and as you develop more skill, a larger proportion of your projects will become boilerplate. This is why many students with years of experience may never become experts; if 99% of the things you do are things you already know how to do, you're not actually practicing anything new.

What About "Natural Skill"

What about the claim that some people have natural skills? Some people are naturally better at math, or art, or music. This is a big nature vs nurture debate, and I won't address the question of whether innate skills from birth actually exist. That said, it's clear that some students have a stronger starting point than others (whether that comes from birth or simply early childhood).

I believe that these natural abilities give students a small advantage that apply only at the very beginning of the learning process. Your natural skill in math or logic, or your previous exposure to technology might give you, say, a 20 hour advantage over the next person. Those hours apply only to the earliest stages of learning. If the goal is to reach 1,000 hours of practice, that 20 hour advantage doesn't count for a whole lot.

That slight advantage can be significant, however. I don't think it's significant in terms of the actual skill learned, but it is significant in the context of confusion, boredom, and frustration. Students that start with this advantage tend to get less confused or frustrated, and they're less likely to quit in the crucial first 100 hours. But for me the important point is that students that don't have those natural abilities can still learn the skill to the same degree as those that do; they just need to be more careful to avoid the pitfalls in the early stages.

A Note on Intuition

Ultimately, learning a skill and following it through into expertise is about developing intuition. Intuition is a form of muscle memory, but not just the physical aspect of the skill. Intuition is a sort of muscle memory related to both the practice and theory of the skill.

If you want to develop expertise, therefore, you must develop intuition. To develop intuition, you have to practice both the theory and the execution in lots of different ways. You need to experience, firsthand, how the theory applies itself to lots of varied situations, and how the interface between theory and environment manifests into execution.

Developing intuition once again comes down to practice. But it requires a different type of practice than the practice you need to develop the skill initially. You need to practice taking things you know, or things you think you know, and varying them, using them in different ways and in different places, with different audiences or different environments.

If you want to develop intuition for machine learning algorithms, for instance, you should write the algorithms from scratch lots of different times, in lots of different ways, in different programming languages, using different problem sets, using different constraints, using different parameters.

The way I see it is that every subject or skill is a highly complex multi-dimensional shape. When you solve a problem one way you illuminate one side of it and come to learn a bit of that shape. But if you want to truly understand the shape, to develop an intuition for it, you need to illuminate all the sides. This requires turning it around, redoing the problem, turning it around again, illuminating another side, and so on, until you've ultimately seen all the possible facets of your crazy, complex idea. Eventually you reach a point where you understand the shape of the thing so well that you can intuitively predict what will happen when you shine a certain type of light on it from a certain angle. This requires a thorough understanding of the theory as well as lots of varied experience and exposure; only then can you claim expertise and intuition about a subject.

Final Words of Encouragement

Everything I've experienced in my teaching life leads me to believe that you can learn it, no matter what "it" is. You absolutely can do it.

You'll be confused. That's what learning is. That's ok. Keep reading, keep grinding, keep practicing. You can, and should, limit the scope of the thing you're learning in order to make it less confusing. You can always expand the scope later.

You'll get bored. That's ok. Just make sure you have a bigger, motivating goal in mind. Playing scales is boring. Remind yourself that you have to do the hard parts so that you can get to the fun parts. Make sure that you have a fun part in mind.

You'll get frustrated. That's what practice is for. It's supposed to frustrate you. Practice enough and you won't get frustrated as much. And then pick something harder to practice, so that you get frustrated again. It's ok to get frustrated, it's part of the process. Take a break but make sure to come back later.

If you don't quit, you will learn it.

This takes us back a decade or more to when I was a senior in college. A team of six of us were building a race car both to compete in Formula SAE and to serve as our senior project. My responsibility was the engine computer, electronics, and wiring. It was a small team, though, so we typically worked in pairs or threes on different parts of the car.

One of my teammates was responsible for building and tuning the engine. My engine computer was ready right around the time his engine was built, so we spent a few hours installing everything onto the test stand. We were both delighted that the motor actually turned over, but also surprised at how poorly it ran!

An hour or so of tuning later and we finally have the engine idling nicely. But then it dies inexplicably! We fiddle with the numbers again, but it seemed we had already found the best timings for idling, so that wasn't the problem. But time and time again the engine died. Sometimes it died after a few seconds. Other times we got 30 seconds of measurements out of it. On the rare occasion, it seemed to work perfectly.

My teammate and I went back and tore our components apart and rebuilt them piece by piece, checking everything. We did find some little issues, so that was a good exercise in and of itself, but ultimately we still had the same problem.

After the second day this happened, I set up a few scopes to monitor different parts of the electronics. Each time the engine died, the ECU had a voltage spike and then settled into a safety state, requiring a reboot. The ECU, not the engine, was to blame.

But for days I couldn't find the source of the issue. I focused on interference and power conditioning, figuring that maybe (hand wavy) some magnetic field from the motor was causing a ground loop in the ECU. But shielding the ECU and the cabling didn't help. Using batteries and other conditioned power sources didn't help. I changed every single wire and cable at least twice, checking each. The ground was a literal earth ground: I used the iron engine stand, which itself was embedded in the concrete, in the workshop basement, underground.

I was at wits end. My teammate was understandably frustrated with me. He needed to run the engine a lot, at different loads and throttle positions, in order to design the spark and fuel timing maps. But because of my electronics, his engine wouldn't run reliably for more than a minute or so.

After another failure, and feeling completely lost, I said to him, "I don't know man, the only thing it could be is magic." And right at that very moment a familiar odor drifted in. The magical scent of ozone.

He and I charge into the room next door to find a third teammate happily welding away, working on the car's frame. There was no way the welding rig could interfere with the electronics, right? Not through a couple of walls and 30 feet of earth ground?

We asked him to stop welding and started the engine again. It ran. And it kept running quite happily. We waited a few minutes until we were satisfied, then asked him to weld something. The engine died instantly. It turns out the welding rig could interfere with our electronics through a few walls and 30 feet of earth ground.

To this day I'm still not 100% sure if the interference was RF or a ground current. I think a ground current (some may call it a ground loop) is much more likely, but the fact that our welding rig could dump that much current into ground was pretty astonishing at the time.

Anyway, just goes to show that even when you think you have a handle on your environment, you may not.

SaaSCo is six years old, heavily capitalized, and perpetually "6 months away from profitability". They have a decent product that's marketed well. Steve has overseen the growth of the product and engineering team from 3 to 30. The product lives on a monolith written in something mature and boring, like PHP, Python, or Ruby.

SaaSCo's product has a huge footprint. The important parts are mature and work well. There's a good deal of debt in the codebase. Ever since the front-end shifted to React a few years ago big parts of the legacy codebase are being deprecated.

The engineering team seems antsy. Deadlines are rarely hit. The product is just too big and complex. Changes often have unintended consequences. Somebody's always refactoring something, and internal documentation is essentially worthless.

Steve has heard the microservices success stories, and so have the engineering team. They're eager to try it. Many of them feel that the Python backend is boring, and they're itching to work with new technologies.

Steve decides to start the long and arduous process of converting the monolith to microservices. It'll be a big undertaking, but it'll all be worth it because of the increased velocity we'll have afterwards.

So SaaSCo engineering starts methodically breaking things out into microservices. First they remove image processing and replace it with an IMagick-based microservice. Then comes the Spamassassin service, and then the stats-processor service. This all goes pretty smoothly. Those services were already on the fringes of the monolith, and separated easily. Note that the team is now supporting four service deployments.

Next they try breaking authentication out into a microservice. This is tougher and scarier than planned, but it works out. Encouraged, they start breaking out more core services into microservices.

But then they hit a wall. They quickly discover that the microservices are all talking to each other constantly, and have hidden dependencies they didn't notice during planning. The legacy monolith is still running, albeit in a crippled mode, and has become a sort of service router for the app. Steve decides to pump the brakes on the aggressive microservice-ization in order to stabilize a bit. The solution is to build shared support libraries that help microservices interface with one another. There is also now a microservices style guide that specifies which languages and libraries should be used, in order for the majority of the team to be able to support the majority of the services. This, of course, makes the engineering team upset because a few of them really wanted to use Elixir in one of the services.

Steve now has a lot more problems than the monolith, which, by the way, never fully goes away. Instead of a monolith, Steve now has a dozen different services built by different people using a patchwork of different technologies, libraries, and API conventions.

The development velocity does not actually improve after shifting to microservices. Years go by in this state of microservices purgatory, and then with SaaSCo v4.0 the team proudly announces that they're going back to a monolithic model, except now with a new shiny framework!

I think the moral of the story goes something like this: Steve didn't have a services problem, he had a complexity problem. Moving to microservices does not necessarily solve complexity problems. Instead of focusing on the "DevOps" layer and topology, Steve should have started with an even more abstract view of what the system components are and where the complexities lie. The simple truth is that complex things are hard, and most successful business platforms are complex. And it turns out that teams of engineers are not typically experts at highly-complex system design, and so most will reach for solutions that sound right but don't actually address the root problem.

I'm not claiming that microservices aren't useful or a solid design patterm. I'm just saying that I've seen a lot of CTOs reach for the microservices hammer when they really needed a more subtle tool. In most cases I've observed personally, a much better approach is to reorganize your monolith into a shared-library model. Monoliths by themselves aren't evil. Microservices by themselves aren't good.

I would go even further and offer this rule of thumb: one should not convert a mature monolith to microservices. The move from monolith to microservices, for a sufficiently complex platform, is very expensive and fraught with peril. It's a large jump: microservices is not just one step away from monolith. The more complex a platform is, the more inertia it has, and the tougher it'll be to move that large distance towards a replatform. Moving straight to microservices in this case is like trying to stop a boulder in its path; whereas a subtle refactoring to a shared-library model is more like nudging the boulder so that it doesn't crash into the village below.

By the numbers: In my career I've personally observed maybe 20 examples of CTOs choosing microservices. Of the fresh, green-field platforms that choose microservices, 8 of 10 are successful. However, of the mature platforms that decided to rearchitect to microservices, I've only seen one company actually pull it off to completion. In other words, I've observed a 90% failure rate in large monoliths successfully and completely moving to microservices. The rest have ended up in a state of purgatory, or have rolled back their efforts.

So this is my caution against microservices: if Steve's story feels all-too-familiar to you, please think twice before trying to step in front of the boulder.

I've written and published two books on machine learning, both with tech publishers. I'd like to share some of the more interesting things I learned about the tech publishing industry.

I don't claim to be an expert on the publishing industry. I only have experience with two publishers, neither of my books are best-sellers, and outside of those two books I have no other experience with the publishing industry. So please take this article with a grain of salt; this is simply one author's personal experience in publishing, and my extrapolations may or may not be generalizable to the industry as a whole.

That said, if you're interested in writing books on tech my experiences may be interesting to you.

Publishing is Driven by Marketing

It's probably not surprising that the tech publishing industry is very marketing-driven. Book topics are not chosen by techies, they're chosen by marketers. My impression is that a team of marketers at the publishing company is doing constant market research to determine which topics will be popular in a year's time. Within those topics, they're also figuring out what keywords will be the most sought-out. It's all about predicting demand.

The end product of this market research is a) a book title, b) a table of contents (including page-count per chapter), c) keyword-rich chapter summaries, and d) a timeframe. That is, as an author, you're given a table of contents that you have to write against, and you're required to include specific sub-topics, keywords, and technologies in each chapter in order to check the boxes that marketing believes need to be checked. The table of contents reads like an exercise in SEO. This is required of you regardless of whether those keywords and technologies really make sense for your book, your approach, or your narrative.

Overall, I didn't find this too burdensome. I was forced to include a few technologies that I otherwise would not have included in my last book, and I was forced to write a couple of chapters that I didn't think were necessary. Those chapters did steal some page-count from other topics I personally would have liked to discuss. But I do understand the need for these types of books to be driven by marketing, so I can't get too upset about this.

Be prepared to write against a table of contents that is given to you by a marketer, not a technologist.

That said, you have a lot of creative control over the content itself. As long as you write against the TOC and tick the right boxes, you can craft the content and the narrative however you like.

The Timeline is Insane

Book timelines are decided in advance, and the publisher/your editor will expect a given word-count-per-week from you. They simply work backwards from the publication date and the target page count per chapter and expect chapter submissions at the appropriate date. Different publishers have different approaches to this, but what I found most jarring with my most recent book was that they made no consideration about different chapters having differing levels of complexity or difficulty. It's very easy to crank out 15 pages of intro chapter material; it's a lot more difficult to write 15 pages of technical material concerning neural network topologies. Yet you're given the same amount of time to write and submit each block of text.

On top of the timeline's rigidity, I also found that the timeline was very fast. The publisher is sticking to their marketing schedule, and the book needs to get out by the date they planned on. Part of their market research involves figuring out when competing publishers might be publishing a similar book, and of course each publisher wants a month or two of topic exclusivity.

If you have a full-time job these time constraints make it very difficult to produce high-quality work. Publishers seem to like me because my first drafts are very clean, but that actually became a liability for me; when they saw that my first draft was basically ready-to-publish, they moved up the publication date by a few weeks. This left me with a couple of chapters that I literally was not able to do a thorough second draft of. My work may have met their standard of quality but it did not meet my own, and unfortunately I'm not happy with 100% of the content I published.

Given more time, I also would have gone deeper into the mathematics and algorithms of the more advanced topics, but sadly I was forced to cut corners and use open source libraries in some cases where I would have preferred to write and explain the algorithms from scratch. I simply didn't have the time. And I'm a fast writer.

That said, I found that the target page counts were not at all hard to achieve. Because the TOC is written for you it's pretty easy to write 80% of each chapter as if it were its own essay, and I had no difficulty filling the pages with good information (except for a couple of the "fluff" chapters I wouldn't have personally decided to include; those were tedious at times). So the time constraint is not so much about quantity, but quality.

Most Authors are Flakes, Maybe

One thing I found surpising is that both publishing companies I've worked with had processes in place to deal with "bad authors". Authors can be good or bad across many different dimensions; but reading between the lines, I would guess that many authors are either low-quality writers or have a high probability of abandoning the project. This is very much an extrapolation and little more than guesswork on my part, so take this section with a huge grain of salt.

Both publishers and all of my editors have given me huge praise for the quality of my first drafts. In both cases, the only edits that came back were simple stylistic and formatting edits (eg, not using the correct formatting block for inline code vs inline variable, or not writing references to other chapters correctly). Combining that information with the fact that my publisher wanted to move up the publication date of the latest book leads me to believe that the publishers expect a relatively low-quality work product from their authors and plan for a good deal of internal editing.

I also get the impression that a large percentage of authors are unable to complete their projects. It is a pretty rough timeline, after all. The reason I landed my first book is because the original author abandoned the project and ghosted on the publisher (this is why I have a co-author credit, despite rewriting the book from scratch; the publisher could not get the copyright release from the original author).

I therefore wasn't too surprised when I learned of my second publisher's practices concerning abandoned titles. They had a policy where if a chapter wasn't submitted every 20 days, the book would automatically be flagged as abandoned. Personally I think that policy is a little aggressive, but after seeing what the first publisher had to do to get that book published, I don't blame them. Still -- I went on vacation for a week while writing the 2nd book, informed my editor and all that, and when I returned I found that I was locked out of my author portal because my book had been flagged as abandoned.

I can only assume that there's a large percentage of authors that simply can't keep up with the crazy schedules, or fall behind a little bit and quickly get daunted by how the workload is snowballing, and ultimately abandon their titles.

There's No Money to be Made

I knew from the get-go that I wouldn't be paid a lot for my work, and fortunately I went into both books with no expectations of making money. But if you're reading this article and didn't know that there's no money in publishing, I feel it needs to be said very explicitly: do not do it for the money.

Most books will never recover their advance. Meaning, if you get paid a $2500 advance and have 16% royalties, you should only ever expect to make that $2500. Don't expect to make money on royalties. Unless your book becomes a best-seller (it won't), you won't see much (if any) money beyond your advance. So if you're considering whether or not to write a book, don't even factor in the royalties.

Given the amount of time it takes to write a book, even with all the corners you can cut, you should only expect to make along the lines of $10-12/hr if you're a fast and proficient writer that requires few edits. If you're not a "clean" writer and your publisher requires lots of edits from you, you can expect along the lines of $8/hr for your work.

If you're a technology expert and you want to go into writing full time, you'd find that you can't make a livable wage (in terms of tech expert wages) writing full time for publishers. Even if you could crank out one book a month (which might be possible if you write 60 hours per week), you'd only make roughly $2500 a month, or $30k per year. That's the same as full-time work at minimum wage in NYC.

Now, there's nothing wrong with making $30k per year -- but that's also the insane scenario of landing one book deal a month consistently for a year straight, and writing and editing 12 books in any given year. Furthermore, to write a technical book you are, presumably, a subject matter expert in some technology, and $30k is way below what the market values your skills at.

Long story short: do NOT do it for the money.

I do get some royalty payments from my first book. I think I get something like $30/mo in additional income from it.

Other Things I Didn't Expect

I've learned many other things about the process, but I don't feel I need to write a full section about each. Here are some interesting thoughts though:

Your editors may not understand the subject matter. If you're working on a very technical book, like a book about machine learning, you can't count on your editors to understand the technical subject matter. There are technical editors and language editors. The technical editors of your book will reproduce your code examples and confirm that they are replicable, but they will not vet the quality or sanity of your code or algorithms. Your language editors will simply check that your basic grammar, style, and formatting is correct. Do not count on the editors to make sure your work is correct or high quality. Do not count on the editors checking that the description of your code matches the behavior of your code.

The work doesn't end when you publish. Your publisher will likely ask you to assist with marketing efforts (if not for marketing, then what are they there for??). You will also have to address a steady stream of errata that comes in as readers find mistakes that your editors didn't catch. This adds a bunch of hours of work after publication, which is a little demoralizing.

You have absolutely no idea who your audience will be. Despite writing a full page on "who this book is for", you have no idea who will actually end up reading your book and how the market will react. It's therefore very difficult to target any one audience, so be prepared for lots of people writing in with questions that may or may not be related to your actual book. Also, be prepared to say "no" to lots of people who are looking for help completing their homework assignments or term projects.

Should You Write for a Publisher?

It really depends on your goals. I wrote these books with publishers because I wanted to see what the publishing industry was all about. There was also an aspect of "the bucket list" for me. I just wanted to.

There are, naturally, pros and cons to working with a publisher. Having a TOC written for you can give you some necessary structure if you're at a loss for how to plan your book. While the timeline requirements are pretty intense, it also forces you to get the project done in a timely manner. The advance isn't a lot of money, but it's guaranteed money. There's also the perceived presitge of being a published author and seeing your name on a bookshelf at B&N. It may help your career outside of writing in some ways, and that should not be ignored.

On the other hand, if you're a self-motivated, confident and competent writer and you want to make money doing it, self-publishing is the way to go. Start by building an audience around your blog and mailing list, and grow that audience until you reach a point where 1,000 people have indicated they'd buy a book from you. Then write it. Maybe only 250 will actually buy your book, but if you're selling at $20 (unlike the $60 a publisher would charge), you'll still make double what you'd make doing it with a publisher.

I have to say that I'm glad I wrote these books. Despite the time constraints and the effect on my writing's quality, I'm still proud of the work I did and think it's valuable. That said, I had a good idea of what I was getting into when I started; if I hadn't expected the low pay and tough timelines it might have been a demoralizing shock for me. If you're considering writing a book for a publisher make sure you understand your own personal goals and be sure to balance those against the pros and cons of working for a publisher.

Here's a piece of startup founder advice that's sure to be controversial: don't overinvest in engineering.

I find myself giving this advice more and more often, especially as available capital in the NYC startup market has grown substantially over the last decade or so.

I'm a CTO, an engineer, and a founder, and I love investing in engineering. So I want to be clear: I'm not saying not to invest in engineering. I'm just saying don't over-invest in engineering.

Put another way: you must invest in engineering. But I've seen too many good companies and good founders fail because they've over-invested in engineering.

I'm sure you can come up with dozens of counterexamples and ways this advice can fail you. As long as this article gets you really thinking critically about how much your company in your circumstances should be investing in engineering, I'm happy.

If you build it, they won't come.

I've seen many founders wait until their v1.0/prototype/MVP is launched before starting to hit the pavement and sell. The 1.0 launch is seen as an inflection point: "Time to start selling!". But if you weren't selling beforehand, how do you know what to build? And how do you know how to sell it?

I've also seen many companies build their initial product and engineering team long before they even think of building a sales team. Just as your engineering team is using these early days to learn how to work together and develop the initial product and organizational framework, you should have some semblance of a sales team learning how to do the same.

Many early stage startups will have a CTO and an engineer or two building the prototype, while the CEO is going out and selling solo. The problem here is that it takes time and effort for a sales team to build their materials and relationships, so the company will end up with a full quarter or two where the sales team just hasn't hit their stride enough in order to provide the crucial early product and sales feedback.

At a four-person startup: instead of CTO + 2 engineers and a CEO solo, try CTO + 1 engineer and CEO + 1 salesperson.

Your engineering team, while necessary, simply won't bring you customers, users, or revenue at any scale necessary to build a lasting company. Your product won't magically get customers because of its virtues.

You don't actually need strong technical foundations.

Your early technical foundations may be valuable in some regards, but not nearly as valuable as figuring out a) who your target market actually is, b) learning how to sell to them c) figuring out what pivots are necessary to the product.

Your early technical foundations may even end up being not valuable at all, depending on what you learn in the early days and if you end up piviting your product.

What you need is a strong sales foundation. Yes, even at an early stage. Don't wait for a v1.0 to start getting salespeople in front of customers. Learn what your customers' needs are, learn how to find them, learn how to pitch to them, learn how to close sales.

The engineering team doesn't do that work: the sales and marketing team does. Odds are your early technology is going to be totally unrecognizable in a year or two anyway!

Don't overengineer the technology. You most likely don't need strong technical foundations. You need strong sales and marketing foundations.

You don't need to build a great, award-winning, delightful product.

The best way to burn your entire budget is to try and design the best possible product. A lot of product teams believe that their product needs to deliver a "delightful experience", that it needs to outclass its competitors in every regard, and that no compromises can be made.

If you can afford to build a best-in-class product, that's great. No early-stage startups can afford that, however. Your goal should be much more attainable: your product simply needs to deliver on its promises. That's it. Do not (yet) strive to "exceed expectations". Just strive to meet them.

There was a massive study done on the performance of customer service teams in the book "The Effortless Experience". One of the study's conclusions is that "exceeding expectations" is a) generally unattainable for 90% of companies and b) has no actual impact on customer loyalty, the metric that matters. They found that there is no difference in loyalty between simply meeting expectations
versus exceeding them.

It's true that the above study applies to customer service experience and not product experience, but I'm extrapolating to product just the same.

Build a product that simply delivers on its promises. No more, no less.

A larger engineering team will only do more engineering.

A larger product and engineering team won't necessarily build a better product -- in the eyes of your customers.

They'll build a bigger product, more features, and more technology. They'll build to a higher degree of tolerance. They'll do more engineering. They might build a better product by every internal measure. Those things sound great, but are actually another trap.

Similar to the "exceeding expectations" trap, a larger product and engineering team will only build a better product -- in the eyes of customers -- 10% of the time. Don't invest in those odds.

Most well-staffed engineering teams want to build the software equivalent of a McLaren F1: a beautiful, finely tuned, handcrafted vehicle. Every piece custom built and machined. But if you build a McLaren, it'll cost a million dollars to produce and you'll only sell 100 of them. What you really want to build is a Toyota -- something that meets expectations and reliably delivers on its promises. Just that modest goal is incredibly difficult to achieve, so start there!

Another apropos example is the Juicero juicer. The device was overengineered to the nth degree, and the company almost certainly lost hundreds of dollars on every unit sold.

Because your engineering department is a cost center, you should fund it only to the level that it needs to build a product that meets expectations.

Founders should not run teams in proportion to their ownership.

I think another easy trap to fall into is brought on by the two-founder CEO/CTO model. When two founders split ownership equally it's tempting to feel that each founder should run roughly equivalently sized teams. The company ends up being a 50/50 split of sales/ops and product/engineering, and I've even see this happen at companies with 30+ employees.

Once a company has more than a handful of employees, however, that 50/50 ratio doesn't make much sense. Sales, marketing, operations, and customer support (ie, the "non-tech" side) should begin to outgrow the technical side (product and engineering) relatively early on.

Equal founders want to feel like they have an equal level of responsibility within the company, and often see the number of employees under management as a measure of that responsibility. It isn't.

If you're a founder/CTO, don't forget: you're a founder first. Yes, the bulk of your focus will be on product and engineering, but don't let that be all of your focus. Figure out how you can use your skills to enable the sales, marketing, ops, and support teams. If you're a founder first, you should be doing what the company needs most, not necessarily what you like doing best. Your value and responsibilities don't come from simply managing a headcount, so don't use that as a vanity metric.

Your most important job as a CTO is to figure out how to run a lean team successfully.

Figuring out how to achieve your product's goals with the minimum resources necessary is your biggest responsibility, and your biggest challenge. You should be doing everything in your power to allow your company to invest as much as it possibly can in developing sales, marketing, customer relationships, and customer loyalty: the profit centers of your company.

Get your company to profitability, and then revisit your organizational goals. If, after profitability, you still want growth, then keep investing in sales. If you want polish, then invest a little more in engineering. But until you've reached financial stability, keep your cost centers in check, and make sure not to overinvest in engineering.

I'm well aware the advice above will be controversial to some degree. But I've seen dozens of companies fail because of misallocated team investment. I know many really great product-focused founders who have had to close shop because they didn't invest enough in profit centers, and many of them have told me point-blank that they wished they had invested more in their sales organization. Meanwhile, I know many technical founders who admit they can achieve their product and customer goals with much less. So please, take this opportunity to think critically about how you're allocating your team, and adjust accordingly!

I think mental illness is starting to become a public health crisis. I know only a few people that aren't constantly stressed due to work or school. I've seen many people fall into legitimate depression and only recover after changing their environment, like quitting their job.

We use exercise to improve our bodies. Like exercise, improving your mental health takes time and conscious effort. You need to dedicate time every day for honest self-reflection, self-awareness, meditation, reading and education in the topic, etc.

There are two ways you can lose weight. Accidentally--by getting sick or stressed, for instance--and on purpose, as through concerted diet and exercise. The same can happen with personality changes. It can happen accidentally, through change of environment or mental illness like depression.

But doing it on purpose is more wholesome. Losing weight on purpose is better than losing it accidentally, even though the result is the same, because those people are healthy and the others aren't. Put another way: you shouldn't have to get sick to lose weight.

Everybody should spend some time daily on those mental health items above. It's exercise. Through it you can point your personality and mind in the direction you'd like it to go, and with hard work and patience you'll get there. As your mind gets healthier the less likely you are to become mentally ill. You'll avoid common issues like depression, substance abuse, anxiety and eating disorders.

It's said that half of adults experience depression at least once in their life. I bet it's more like 90%. It's like the flu. Pervasive, transmissible and contagious, with most victims recovering quickly, but a small percentage experiencing fatal cases. Now we have flu shots at every Walgreens.

It's time to do the same for mental health. And until we have more public programs to support mental health, you'll have to take it into your own hands.

It takes ten years to build a successful company.

VCs are good at picking founders. So why do only two in ten make it? Truthfully, probably 70% of VC backed founders are capable of building a successful company on a ten year timeline. But VCs have their own investors, and VCs are expected to give 20% returns per year, at minimum. Their funds also have a timeline of 10 years. Which means that companies that would become successful in the average timeline are below the curve for VC expectations, and end up cut after only five years.

It takes ten years, but you're only given five.

VCs are still successful, of course, because two in ten founders are able to move faster than the timeline. Which means that 5 of 10 founders (the ones that are all likely to build a good company in ten years but aren't the top two) find themselves in unfriendly board meetings where they're challenged to show growth that isn't possible yet while their runway shortens. Those founders end up with unfavorable outcomes, and their potential is wasted because their timeline doesn't match the investment returns. VC portfolios are only 50% efficient.

Bootstrapping isn't any easier. It takes ten years! Now you've got the time, but not the money or resources. Most successful companies have been to the brink of failure. At least VC backed companies have cash to absorb short term losses. Bootstrapped companies may not.

Your choices are not wonderful. You either must outperform 80% of your peers or face an early death at the hands of investors, or you must be able to hunker down through 10 years of financial instability or face an early death because you starved during the seige.

I've seen a lot of great founders get hamstrung by the decision to go with VCs, not realizing how much better than average they'd have to be in order to be allowed to continue on. If you're one of them, if you just got out of a board meeting and are wondering what you did wrong, the answer is probably "nothing". It just takes ten years, and you don't have the time.

[1] There are many ways to define success, but I use "sustainable market significance" as mine.

Common wisdom says that teaching is one of the best ways to develop mastery of a topic. Why is this? It's the grapple - struggling to meet eye to eye with a student and molding your own understanding of the topic to their learning style.

But still: why is this so effective? I'm not an expert in developmental psychology or linguistics, but I propose a hypothesis.

Teaching is an effective learning tool because it forces us to shape and describe a concept with language in many different ways, because of that trial-and-error approach of figuring out a student's learning style.

Language is how we model thought, and using different approaches to model a concept will highlight different facets of the concept. A concept will become intuitive when all facets of the concept are ultimately brought into relief. That's what intuition is, after all: understanding all of the different ways a concept can take form and interact with other concepts.

One analogy is music. You may know how to play a C major chord, but you don't truly understand it until you know all of the voicings and all of the ways it can transit to other chords.

Language shapes concepts, and it's easy to use language to highlight only the parts of a concept that you want to highlight. Biased publications do this all the time. But when you want to learn, or teach, you need to use language to illuminate as much of the concept as possible. That's means saying the same thing in as many different ways as you can.

One way to test this hypothesis is to remove the student - or more accurately, to make yourself the student - and concentrate only on the verbal aspect of explaining concepts.

Take a topic that you have only a rote understanding of, and explain it - out loud - to yourself in as many ways as you can, from as many different angles as you can think of. Add analogies and metaphors, even if they're a bit of a stretch. Make sure to also move both higher and lower in abstraction ("programming languages are really just a grammar for a formal logical system, in that way programming is just like Law" vs "programming languages are just interfaces around hardware-driven bit manipulations").

Your goal, whether learning or teaching, is to shine light into all the nooks and crannies of a complex topic at all levels of abstraction. Imagine you're explaining the topic to an advanced coworker, and then imagine explaining it to a layman. Try this exercise for one hour, as you're showering and getting ready for the day, and you'll find yourself with a deeper intuition around a topic that you once had only a rote understanding of.

In this series I explore learning -- specifically, why it’s difficult and how to get better at it. Sometimes I'll use science, other times I'll call on my decade of experience and observations on teaching a range of subjects -- both formally and informally.

There's no single simple trick or hack. It turns out that learning is hard work. But we all want to be better at it, so we owe ourselves a little bit of hard work to get there.

From the outset, your expectations are a major factor in determining whether your educational venture will be successful or not. This may be surprising to many people. We're not used to the idea that we're actually in control of our ability to succeed in learning.

In my years of teaching -- everything from physics to music to engineering to software to skiing -- I've seen many students fail and many excel. I've seen many different failure modes at different stages of the learning process. Everything here is a generalization. There are always exceptions to the rule.

What I've never seen, however, is a student fail after dedicating 200 hours (for example) to learning something. The students that fail invariably fail in the early stages, before reaching some certain threshold. They fail before they reach their moment of clarity. That moment of clarity where it all “clicks”; we’ll call it the “pass” threshold. A student is very likely to excel after reaching the pass threshold. The failure threshold, on the other hand, is generally when the frustration of learning outweighs the student’s motivation to continue.

I’ve come to this conclusion after observing time and time again that when a student fails it’s because they quit entirely somewhere in the early stages. They drop out. Give up. Leave the classroom and never return. Resign themselves to never trying that subject again. Other times a student performs a "soft quit"--they'll remain physically present in the lessons or the classroom but have mentally checked out. That’s just as bad.

This quit-itude, I've found, stems from a misalignment in expectations. The student gets frustrated that they're not picking up the material as quickly as they hoped. Its almost as if they expect learning to be easy! (Check out the Dunning-Kruger effect.)

But learning is not easy. By definition, learning a new topic is the hardest part of that topic. Learning is the process of going from knowing nothing about a topic to knowing it intuitively. There is no greater mental challenge than learning!

Students that expect learning to be difficult and frustrating tend to succeed not because they're more mentally capable, but rather because the slow-going crawl of learning isn't frustrating to them. It's expected, and they manage their frustration and keep moving forward.

Many students have the mindset that certain things are simply inaccessible to them because they're not smart enough. "I just can't do math" or "I'm hopeless with computers”. But that’s not the case. Barring an actual learning disability, the odds are that you are mentally capable of picking up a subject. It’s just like people claiming they’re tone deaf; many people claim it, but very few people are actually tone deaf.

That mindset — the “I’m just not smart enough” mindset—is poisonous. You’re likely smart enough, you just need to manage your frustration and make sure you don’t quit before you reach your “a-ha!” moment.

I’ve seen it too many times: many students get excited about learning something, and when the first mere 20 or 30 hours of practice prove difficult or frustrating they'll invent an excuse to quit. "My fingers are too weak" when learning guitar or "I don't think I like the industry" when learning music production or "I can't even draw a straight line" when trying to draw (I'm guilty of that one!).

But what if I told you that you could learn a life changing skill, guaranteed, if you put 200 hours of tough, frustrating-at-times effort into it? How would that change your mindset? What if that skill could land you a job at three times the pay? Just imagine how much more willing you'll be to deal with the frustration of learning something new if you knew you’d get there and that the only enemy is frustration? All you have to do is keep pushing forward—even if you feel frustrated and demotivated at times—because you will get the subject to click.

That's what learning's really about: managing frustration. Learning will always be hard. But I believe--and I've seen, and I've done--that anyone is capable of learning anything with the right expectations at the outset. You just need to put in the hours--hours and hours of beautiful frustrating practice.

The right expectations, combined with an unrelenting drive and a good mechanism for dealing with frustration, is the first step in learning even the most difficult subject successfully.

Last week, I mentored a hackathon for beginners and intermediates at The Cooper Union and found myself repeating the same two pieces of advice over and over again. Since programmers shouldn’t ever repeat themselves, I decided to write it down and share with the world. This advice applies if you've been having trouble getting motivated to take the next step or learn new things often.

While I was there, I met a guy who had O'Reillys Python for Big Data open on the desk next to him. He was practicing writing mathematical functions. I had a chat with him and he seemed passionate about learning Python, but really pretty unenthusiastic about the process. It was clear that learning Python was a career goal of his, not a personal goal, and he was going about learning it in a textbook way.

Watching him reminded me of an oft-ignored piece of advice: Learn by doing what you love. Obsession is key when it comes to mastering programming (or anything). While it’s perfectly fine to choose this career path as just that—a career—it also takes a level of dedication and excitement that can’t be generated without a natural spark.

Python for Big Data may be your goal, but that's not how you have to learn Python. There are plenty of ways to develop practical skills. Artists typically learn about art from a young age, crafting personal art projects, drawing what they love drawing, and then later refine those skills to be used towards a career. Writers learn to write by blogging or composing lyrics because they love it—and later figure out they can apply those skills to a profession, like grant writing. Learning quantitative finance is a smart career move, but unless quant really turns you on, why not develop your foundational skills by throwing yourself into something you love?

When in the learning stages, you should only work on projects that are so exciting and interesting to you that you voluntarily stay up until 4am coding. That's the best way to clock the 1,000 hours of experience you'll need before you can become a valued professional while also maintaining your enthusiasm and sanity. Why should you spend those 1,000 hours grinding though quantitative finance or big data (unless that's your thing!)? Learn to be a programmer first. Use projects you love to get there. Once you're comfortable, you can specialize in big data. And I promise, your excitement for your projects will make those 1,000 hours far more educational than if you’re simply walking yourself through the steps.

Another thing to keep in mind—though we’d all like to think of each and every thing we do as deeply meaningful, your passion projects don’t need to be world changing, life-altering attempts. Have fun with it. Embrace the Dinkiness. And through that, you actually will be making quite a difference.

Take on small projects, things that exercise certain skills one at a time, on topics that excite you. Jumping directly into big data is a quantum leap from where you need to be when you’re just starting out. Instead, fill your plate with “dinky” projects: Build a dinky text based RPG. Build a dinky image processing app. Build a dinky dating site.

Dinky is great for a few reasons. First, it takes the pressure off the project; like I said earlier, coding should be fun. Let yourself explore an idea just for you. Sure, it’d be great if the weekend app you’ve created is professionally designed or technically robust, but really, there’s no need for polish. You won't care if anyone uses it. This is just practice for you. Dinky projects are the blog posts and paintings that no one will likely read or see, but that the creator gains value from in personal satisfaction and knowledge. Maybe this dinky little project helped teach you how to work with 3D animation, or efficiently tell a story, or use WebRTC or something else new and exciting.

Plus, there is the off-chance that your dinky project hits it big. Every once in awhile that blog post turns into a novel. That painting gets purchased by a top hotel. Consumers aren't very predictable. If you work on one hundred dinky projects in a year, there's a good possibility that one or two of them will see moderate success, and that's a great morale boost and experience for someone who’s simply trying to build their skills. I've had a few dinky projects accidentally go viral, and each one of those experiences taught me something new. But more importantly, they kept me excited—and routinely up until 4am, working because I couldn’t tear myself away.

You can’t get that experience by following a textbook plan. If the kid who was practicing Python only applied it to big data, chances are he’d either lose steam or flame out completely, and may never feel that personal satisfaction that keeps driving us forward. He also wouldn’t push his limits; exploring the ideas you love with nothing to lose opens the door to new ideas that may never have arrived if simply plunking away on assigned projects. Finally, there’s something powerful about harnessing your creativity and spark to fuel your skill set. But in order to do that best, you have to love what you’re doing and do it for you. If other people think it’s cool, that’s just icing on the cake.

I'm fascinated by Morse code. It teaches us about encoding, language, technology, and our ability to learn to communicate in a revolutionary manner.

Nearly every piece of information around us has been encoded. Even language--human communication--is encoded information. Language started as an abstract concept; we needed to communicate thoughts, ideas, and information from one person to another.

In the very early days, we grunted. Then we developed the spoken word. Very fortunately for us, we have an organ (the vocal cord) that helps us encode language into audio and transmit it out of our throats, through the air, and into a listener's ear. So we did that for a while, but eventually (as it always happens), we figured there could be a different way to encode language such that we could store our thoughts for later. We developed drawings.

Drawing evolved into written language, and we started using paper so that our thoughts were mobile. The written word was a significant advancement in the evolution of human society. Scholars were able to share thoughts with future scholars that they would never meet. We could write plans for tools or machines. We could share ideas without ever needing to know who would read them.

We encode information for lots of different reasons. There are some forms of encoding that help us transmit data. Other forms help us store data. Other forms help us operate on data. And we're always improving the way we encode things.

Take recorded audio, for instance. At one point, someone discovered that you could capture pressure waves on a physical medium so that we could re-play them later. The first record was born. We discovered radio waves, and shortly after that we figured we could encode audio on top of a carrier wave. Thus was born AM radio. AM radio had its limitations in transmission and fidelity, however, so we figured out a better (albeit less intuitive) way to encode audio for radio: FM.

We encoded and recorded audio onto magnetic tape, hard discs, optical discs, and eventually dozens of popular binary formats. We encoded video in much the same fashion: first we used AM, then FM, and eventually landed on digital. Who knows what we'll have in the future. Maybe we'll encode things in quantum states, or in DNA.

This brings me to my fascination with Morse code. Granted, learning Morse code today may be a futile exercise. After all, I don't know a single person who speaks it! But Morse code was the first popular method of electronically encoding human language. What a huge step forward in history that was! What a fundamental breakthrough!

We used Morse code first over telegraph lines and eventually over radio. We used it as civilians and we used it in the military. Morse code was already mature before we could even transmit voice over radio. And even when we started transmitting voice, Morse was more reliable under poor conditions than voice was. The dashes and dots were easier to pick out from the static than spoken words were, so we continued to use Morse for quite some time.

Interestingly, Morse code was originally developed as a sort of written language. The receiving end of the early telegraphs had a roll of paper tape attached, and the dashes and dots would be drawn on the tape, ready for visual human transcription. But after a short while, telegraph operators were able to transcribe Morse simply by ear. In fact, they discovered that listening to Morse code was easier than visually transcribing it.

As you develop proficiency in receiving--or "copying"--Morse, you start hearing the characters rather than the dashes and dots. "Di-dah-dit" just comes across as "R", as passively and intuitively as the letter "R" itself. Eventually you begin to recognize full words, and if you have lots of experience, even common phrases require no active translation.

That's amazing to me. With some practice, we can learn to "natively" speak and translate an entire electronic encoding without active effort. It's even said that Morse operators, particularly during wartime, could identify the actual person they were listening to--because everyone has their own Morse "accent". You'd come to recognize Steve's "voice" without him having to identify himself.

That's why I'm learning Morse code. This simple form of communication, seen my many as antique, encapsulates so much about our world and technology. It shows us how we can use encoding to give information the properties we want it to have (in Morse code's case, ease of transmission, simple electronic components, and high fidelity even when the signal is weak). It teaches us about human language and adaptability (learning to "speak" Morse code). And, long before the internet and the digital age, Morse code revolutionized the way we communicate with one another.

While it may not be practical to learn Morse in this day and age, I feel I must pay homage to such a wonderful and revolutionary system. And who knows--it might come in handy some day. For some reason.

Do any of you speak Morse?

(If anybody is interested in learning, I've been using lcwo.net. There are downloadable programs out there as well, but I like LCWO because it's a webapp.)

(Update Oct 15 2013: I'm proud to say I can now copy Morse code at about 10WPM with 99% accuracy. I love it!)

There are lots of people talking about big data these days. There's a lot of discussion about how to build apps for "web scale", and there's an emphasis on real time apps that collect comprehensive data.

This article is me playing devil's advocate.

I think the big data hype is hurting our ability to think critically about our problems. Obviously this isn't the case everywhere. Many engineers are excellent and brilliant and do the right thing. But many others simply brute-force their problems into submission.

These days it's possible to cheaply collect and analyze billions of data points on commodity resources. On the surface that seems like a good thing, but we should keep in mind that many of the problems we solve today were solvable yesterday too. We went to the moon on a 2MHz processor and 4KB of RAM.

Think about what a feat that was! Think about the problem solving skills that took!

If we could that, why can't we just get by on a million data points instead of a billion? Have we already forgotten statistics? When was the last time you actually asked yourself if you need big data? Unless you actually need to record every single data point -- if that's your core value proposition -- I'm sure you can get by on a random sample.

I'd be willing to bet that any data source that can provide a billion data points will still be statistically significant if you were to sample only 1% or 0.1% of the source.

Big data doesn't need to be so big. Even Google analytics uses sampling for datasets larger than 100,000.

My suggestion to you, then, is to really evaluate your options before you spin up 50 new EC2 instances. If you're a startup and funds are slim, or if your data population is very dense, then brush up on your statistics and figure out if you can get away with sampled or aggregate data. Please don't think that you can avoid doing smarter work just because servers are getting cheap.

Like I said, this is a devil's advocate post. It's meant as food for thought. If you really do need all that data, then ignore this. I won't defend this article to the death. But if you've never thought about this sort of thing, it's time to start. Don't get caught up in the hype of big data, and just do what's best for the project!

I just spent 48 hours without electricity. I'm not complaining; I could have had it much worse (many people in Staten Island did) and very fortunately nobody I know was hurt or lost their home. But my experience without power got me thinking.

I was sitting around in the dark thinking about whether ConEd would give us a pro-rated discount for the 48 hours we went without power. (I don't care if they do or not, I was just curious about the mundane.) Then I realized that 48 hours without power amounts to about $6 on my bill.

Six dollars? That's it? If I could have paid a premium to keep my power on during Hurricane Sandy I gladly would have paid maybe $50 or even $100 just for that $6 worth of electricity. Isn't that amazing? Electricity is so valuable to us and yet so cheap. And also so inaccessible to the undeveloped world.

Of course, I wouldn't be able to pay a sustained $1,500 a month for electricity, but the fact that I'd shell out $100 to keep my power on during a storm made me realize just how valuable it is. And that's not an easy feat. I used to be a hybrid car engineer, and I worked on several renewable energy projects in my grad school years. I always felt I had a grounded, solid understanding of how valuable electricity is.

But actually living without it gave me a new perspective. It made me think about places where power isn't a given, like my grandparents' house in the mountains in Turkey. Power comes and goes there and it's just a way of life. But if that were to happen here it wouldn't be acceptable.

Most people like to think that they're empathetic to poorer nations' way of life. But are we really? I always thought I maintained the appropriate level of gratefulness and empathy, but after just 48 hours with no power I started to go crazy. It was demoralizing, sitting in the dark, cold and helpless and cut off.

And now I have an existential crisis. I used to be a mechanical engineer. I have my master's degree in hybrid car design and I've also done quite a bit of work in renewable energy. And I left that behind to do what? Software engineering. I make websites and write code for the first world. I turned my back on my altruistic goals of helping the environment and designing clean tech because "it wasn't for me" and "I wanted to be my own boss".

Is that fair? Is that a decision I'm morally allowed to make? I have the skills to help other people out but instead I'm running a startup and writing on my blog. Should I feel guilty? Do I have a moral obligation to use my engineering skills to give back to the world in a bigger way? I don't know. I enjoy software more--which is why I switched. I have more experience in software than mechanical engineering, but that may be a moot point: having a couple of degrees and being published in hybrid vehicles engineering may have made me more valuable to the sustainable engineering world than the I am to the web startup world.

And if I'm more valuable to the sustainable world -- if they need me more -- than the startup world, am I supposed to be doing sustainable engineering with my life? Do I get to just recklessly choose which I pursue? Or am I supposed to go the way that needs me more?

Do what you love, or do what you're good at?

I think I'll stick to what I love for now (software and startups), but if anyone wants to buy my company for a few hundred million (it's worth it!) then I promise I'll follow in Elon Musk's footsteps and focus on the rest of the world.