We're putting an end to that. Mayhem Marketplace is now live. It's what Etsy should have been; a seller-first, craft-focused marketplace. Mayhem Marketplace doesn't do ads, transaction percentages, or listing fees. As a seller, you pay $5 or $10 a month to host your storefront/shoppable link-in-bio page and appear in our marketplace app. That's it. You can sell a million dollars on our platform and only ever pay us $100 for the year.

We're also doing the same for adult content and age restricted goods. We're still working on finding credit card processors for that site, but that'll be called Mayhem Red and is now open for seller applications.

That works because smoking has vague, long-term negative effects. Smoking a cigarette today won't kill you. Smoking cigarettes for twenty years might. Her solution was to make the negative effects of smoking immediate, obvious, and painful.

I figured the same approach could be applied to any habit. If you want to write more often, one thing you can do is to make the negative effects of not writing immediate and obvious.

So that's what WritePhobia does: you set writing goals for yourself ("500 words on weekdays"), and for each goal you miss, WritePhobia immediately charges you a dollar and donates it to a writing charity.

So far WritePhobia has a handful of users and it's working great! I plan on building more community features into the app as more people sign up. If you're reading this, and interested in writing more, go sign up for WritePhobia. It's free... if you never miss a goal.

Over the last decade the bulk of my engineering hires have been US-domestic remote employees. I generally hire full-stack developers, but this technique can be adapted easily to front-end devs and, slighly less readily, to backend developers. This technique also works best for hiring mid-level engineers; you will find it more difficult to use for senior talent.

First, Use a Virtual Assistant

You could do this stuff yourself, of course, but you’re a manager and this is not the highest-leverage task you could be doing with your time.

Bring on a VA that’s good at web research and scheduling. Instruct them to look for portfolio websites of developers who have been freelancing for a couple of years.

• Obviously, the portfolio sites they look for should include the relevant technologies. When hiring full-stack devs I tell them to look for people who put some amount of care into their sites. They don’t need to be beautiful, but they need to be “designed”. Bonus points for links to live projects or a GitHub. Bonus points for a technical blog component.
• (When looking for backend devs, I’m ok with just a technical blog and a GitHub.)
• You want them to find freelancers who haven’t been freelancing for too long, but still seem to be successful in their practice.
• Freelancers who have been doing it successfully for more than a year or so are likely to be independent, self-starters, good communicators, and used to working remotely.
• Freelancers who have only been doing it for 1-2 years are more likely to want out. Freelancing is a tough slog and seems more romantic than it is. There’s a sweet spot you’re looking for: they’ve freelanced long enough to develop those important skills, long enough to be sick of the slog, but not so long that they’ve proven they want to stick with freelancing.

Have your assistant find 25 of these per day or so and dump ‘em in a spreadsheet.

Prune, then Phone Screen

Your virtual assistant may need a couple of review and pruning rounds before they start finding the exact right profile for you.

At this point you should have a freelance project queued up and ready to go. Have your VA schedule time to do phone screens with the appropriate candidates.

During the phone screen, make the following things clear:

• You will start with one small isolated (5 hour) freelance project.
• You will have a regular cadence of small and medium freelance projects after that.

If the candidate does well on the phone screen, hire them for the freelance project. Because a single isolated project is a relatively small commitment for you, you should make the interview process a small commitment for them as well. Don’t drag them through the slog of a 3-interview process. Just a 30 minute phone screen is enough. You will be evaluating them constantly over the next three months, so don’t worry about getting it perfect right now.

Go from Project to Retainer

Now you’ve got a contractor on board. I’m not going to give advice here on how to make isolated projects successful (that’s a topic for another issue!), but make sure you get them all the support they need to accomplish your ~5-10 hour freelance project.

There are three possible outcomes: they do poorly, mediocrely, or well. If they do poorly then you simply won’t rehire them for another project, and you haven’t really spent all that much on the small project. If they do well, queue up another project for them.

If they do well on that second project, negotiate a retainer. You will most likely be able to get a reduced hourly rate in exchange for a consistent 10-15 hours per week. If you paid, say, $100/hr for the first two isolated projects, it’s likely you can reduce the hourly to $60/hr during the retainer period. Freelance hourlys are expensive because they need to cover the downtime in between projects. Remember the advice from step one: you’re looking for people who are probably sick of freelancing. Odds are they’ll be thrilled to get some steady work for a 3-month term.

For the next three months or so keep up the retainer work. This retainer period is in lieu of an onerous interview process — and frankly, it’s better than an interview process. You’re working together and learning about each other, but not yet committed to each other. You’re also not wasting their time, and they’re not wasting yours; you’re getting productive work done and they’re getting paid.

If things are going well so far, allow your team to start working with them more closely. Get some cross-polination going. This is essentially a highly productive two-way interview process.

Extend the Full-Time Offer

After a few months of retained work, you’re going to know exactly what you want to do. There are three options: you can keep the freelancer on retainer as-is, let the contract lapse and move on; or extend a full-time offer.

In my experience, if you get to this point the conversion rate to a full-time employee is very high (assuming your company is a good place to work). You’ve been evaluating each other for the last three months, and both parties will just know at this point. Your team has had opportunities to work with the freelancer, and vice-versa. The freelancer has a very good idea of what life will be like at the company; and likewise you have a very good idea of what their work product, communication skills, and temperament will be moving forward.

Anecdata, Expectations, and Why it Works

You may be concerned off the bat about the success rate of that initial freelance project. After all, if the project is an utter failure, or not usable in your product, you’ve just wasted wasted money! And it is true that the first project is the riskiest. In my experience it goes in thirds — out of three freelancers, the breakdown will be:

• One is a total flop. Pay them anyway, but don’t work with them again.
• One is in a grey-area. The work product looked good from the outside, but it wasn’t really to-spec and it had a number of significant issues. Pay them, but keep looking.
• One will be successful all-around. This is the one that you negotiate a retainer with.

This is why you should choose a small and isolated project for the very first one. A 5-hour project will cost around $500-750 (or so). Frankly, spending $1,000 to find a good freelancer is totally worth it (compare to totally absurd recruiter fees). That third person, even if they don’t convert to full-time, can still be a highly productive contributor.

Personally, I’ve never had a case where I wasn’t able to negotiate a reduced hourly in exchange for a retainer. But I suppose that is a risk as well.

Another possible failure point is the retainer period itself. I’ve had a couple of cases of ending the retainer early. I believe that this was largely due to my laziness early on in the process — not wanting to start the process over again, I probably brought a couple of people into the retainer period that I shouldn’t have.

The final failure point is the conversion from retainer to full-time. In my experience, this conversion rate is very high, around 90%.

When you add it all up and calculate the full-funnel conversion, you’ll end up spending around $2,000 to put a handful of freelancers through the process and convert one into a full-time employee. This is way cheaper than a recruiter, and most importantly that money was spent on work-product. It’s not necessarily a “wasted” $2,000 — you still got real work for that money! On average, I’d say the “wastage” is only about $500; that comes from failures in the first small isolated trial project.

I’ll also note that on average, my retention for employees hired in this manner is four years, and their full-time salaries end up below market, since you’re hiring out of Ohio, West Virginia, and so on.

I’ve had a ton of success with this method over the years. It may or may not work for you, but if you’re looking to start hiring US-domestic remote engineers, you may want to give this a try.

So this week’s CTO Worry is: Remote Work Morale. Specifically, CTOs are worried about the downstream effects of forcing people who were previously colocated into a new and isolated lifestyle.

Remote Work isn’t For Everyone

Teams are typically either distributed or colocated. It’s generally difficult to pull off a hybrid. I’ve seen some hybrid models work, of course; typically when there’s an overseas or satellite office involved. But the big fear today is that remote work isn’t for everyone, and yet we’re forcing everyone to do it.

CTOs are seeing productivity and morale slump. This is a bit disheartening to most of the CTOs who previously reported encouraging early results. What CTOs want to know is: how much of this morale slump is related to WFH vs other factors?

These really are “unprecedented times”. But, anecdotally, I can tell you that each and every time I hired a “non-WFH adjusted” employee (that is, someone who never worked remotely before and didn’t know if it’s for them), it took about 18 months for their WFH status to start affecting their morale.

For this reason alone, I have trouble ascribing the morale slump to the abrupt transition to remote work. Or rather, I have trouble ascribing to morale slump to just the transition to remote work. If, say, there were no global pandemic but for some reason all office buildings just stopped working, it would take a lot longer for this forced-WFH to deteriorate morale to where it is today.

So we must instead look at a combination of factors. First and foremost, all of this is going on during what would normally be the midsummer slump. There’s also obviously a huge increase in background stress due to COVID, the political landscape, and the general sense of instability going around. Added to that is the fact that everybody’s social lives disappeared overnight — forcing WFH is one thing, but at least in pre-pandemic days your remote employees could go out at night. There’s also the fact that most people have cancelled their vacation plans and have literally nothing to look forward to anymore.

That last point is perhaps the most poignant. That famous study of Dutch vacationers suggests that planning the vacation is the best part of the vacation; having a vacation to look forward to 6 months from now can really tide one’s mood over. But now that most vacation plans are scrapped people simply have nothing to be happy about anymore.

What the heck do we do?

Nobody likes their boss forcing them to be social. For that reason I’ve never before supported “mandatory fun”-type activities. Until now.

There are probably two important things you can be doing at the moment. The first is to provide fun social interaction. It’s true that hanging out with coworkers is probably not everyones’ definition of “fun”; but it’s better than nothing. And that’s what we have right now: nothing.

I recommend structured, fun, personalized activities. Hire somebody to put together a trivia game on Zoom each week; make the trivia categories focus around an individual. “This week is Joe Trivia, with categories like ‘80s hair bands’ and ‘famous world leaders’. Next week is Jane Trivia, with categories like ‘Hamilton lyrics’ and ‘Seinfeld trivia’”.

Why trivia? Because a) it’s fun, b) it’s structured c) it’s introvert-friendly and d) the ability to personalize each week adds a special something.

Making these activities introvert-friendly is an important point you should not overlook. You probably started doing a weekly Zoom happy hour at the beginning of lockdown; how did that work out? From what I’m hearing, most of those Zoom happy hours have already fallen off. Why? Because they’re not introvert-friendly, yet most of your team are probably introverts.

You don’t have to go with trivia, but pick some other social activity that hits those same bullet points.

Oh, and let trivia cut into work hours. Now is not the time to try and squeeze blood from a stone. Right now your goal is not necessarily to boost morale but rather to preserve as much of it as you can. Do the first couple of trivias from 4-5 on Fridays, and if people really love it you can consider making it after-hours. But if productivity is so low now anyway, why worry about that 2% of the week you’d cut into on a Friday afternoon?

The second thing you should do is give your employees something to look forward to, plan around, and anticipate. Tons of Disney trips and week-long jaunts to Europe have been canceled, but renting a house somewhere beautiful and isolated is still a viable vacation option.

Encourage your people to rent a house or go camping for a week. If you can, offer to have the company pay for one or two nights of the vacation. And try to schedule it as far out as you can — send people away to go see fall foliage in a cabin in Vermont. Help book an early spring Hudson River Valley rental house for an employee who you know loves hiking. Give them something to look forward to. Many might be unwilling to plan that far ahead; that’s why you incentivize them by offering to pay for the first night.

This isn’t easy, and it won’t get easier for a while. Schools reopening next month is a huge variable, and things may get worse then.

The most important thing you can give your team is understanding and support. I know that’s tough to do right now; you’re supposed to look out for the business and you need everyone to be as productive as possible. But everybody’s losing this year, not just you. If you hope to thrive when this is all over, you’ll need to have taken care of your people in the interim.

I Wish I Had Better Advice

This newsletter is called “CTO Worries”, not “CTO Solutions”. I discussed these topics with a group of CTOs last week, and this is the best we could come up with! If nothing else, take solace in the fact that nobody actually knows what they’re doing right now. There’s no guidebook. We simply have to rely on our experience and intuition.

We all seemed to agree on the core points though: for the duration of this pandemic we will likely need to take a more active role than usual in our employees’ personal morale. Typically, CTOs worry about team morale insofar as their relationship to their job goes; but a CTO managing a team of 30 simply can’t delve into individuals’ personal lives to help out. Now, you might have to.

You have to make the time to do these things, and you have to advocate for your employees moreso than you ever have. You’d have no problem shelling out $1000 to send someone to a conference; shell out that same $1000 to pay for a nature retreat instead! That simple act of generosity will simultaneously give your team something to look forward to and also show that you’re serious about supporting your people through this.

Hey, you made it all the way down here! That means  you probably found this content valuable. I won’t force you to give me money, I just ask that you share this with your friends.

Share

If  you want to support me writing this content, however, feel free to give me money! Either by subscribing or giving a gift subscription to a friend:

Subscribe now

Give a gift subscription

And feel free to Tweet your tech leadership worries to @bkanber, and I’ll try to address them in an upcoming newsletter.

The team grows and eventually they hire their first PM. The product manager works with the engineers (and sometimes with sales) to help flesh out requirements and acceptance criteria for individual features and sprints.

This works for a while, but as the CEO and CTO face an ever-expanding scope of responsibilities they find themselves slipping on certain things. The product development process as it stands is prone to errors, the positioning hasn’t been updated in a few years, and deadlines have been slipping.

So the founders start the search for a product executive. They post a job listing along the lines of “product executive for high-growth startup”, and title it “VP Product” or “Product Director”.

One of two things tends to happen at this point:

• They go through dozens of interviews without extending an offer, or
• Or they hire someone that only lasts a few months

Mistakes were made. Let’s discuss the common ones:

What’s the Scorecard?

That’s the first question I ask CTOs looking to hire a product executive. What does the scorecard look like? (If you don’t understand this reference, go read Who: The A Method For Hiring by Geoff Smart and Randy Street.)

It’s surprising just how many CEO/CTO pairs neglect this step. They’re hiring emotionally, looking for a personal connection with the candidate. That’s fine, but not at the cost of ignoring the objective facts.

Before you write the job listing, you need to ask yourself:

• What does success look like in this role?
• What does the average day look like in this role?

In fact, I advise figuring out what the average day looks like and including that in the job listing itself. I give this advice for all job listings, but product executives are such troublesome hires that I think it’s doubly important to lay it all out on the table up-front.

Strategic vs Tactical

I find that product executives fall into two categories, and that in general, an individual strongly prefers one or the other.

Therefore, straight off the bat, CEO/CTOs tend to make an early mistake in expecting that one person can fulfill both the strategic role and the tactical role. This is a unicorn that you should not expect to find.

Things a tactical product executive does:

• Takes measurements, designs A/B tests, looks at data;
• Pays attention to the product development process and tweaks it to improve efficiency or correctness;
• Gets on the phone with actual customers;
• Develops KPIs to track productivity, judge what to build next, and keep the trains running on time.

Things a strategic product executive does:

• Does constant market research to figure out the best positioning of your product;
• Works closely with sales and marketing to execute on positioning;
• Maintains the product and strategic roadmap over a period of years;
• Figures out how to make the product a Purple Cow (see Seth Godin’s Purple Cow)

Again, don’t fall into the trap of thinking you can find someone who excels at both. Even if you find an executive who has proven themselves in both tactics and strategy, you’ll likely soon find that they have a preference for one or the other.

Before you start on this hire it’s important to understand whether you’re looking for a tactical or a strategic product executive. Don’t expect a unicorn.

Not Knowing How To Interview

Sometimes a founding team will get the above points right but still fail at hiring the correct person. This is usually a case of not knowing how to interview an executive when you don’t have the requisite experience in that field. A CTO who has never been a product professional simply might not know the right questions to ask. (Though, again, Who: The A Method For Hiring is a great read.)

The solution to this one is quick and easy. If you don’t know how to interview for a VP Product role, bring on someone who does.

Most likely you have an investor, advisor, or friend who understands both your company’s needs and this VP Product role very well. Ask them for advice, and then bring them in to an interview.

You’re hiring an executive. If there ever was a time to call in favors, this is it. Make sure you hire the right person, and if you need to go so far as to retain a consultant for the interview, then do so.

Clash With The Founders

Sometimes all of the above goes well but the relationship still sours. In many situations, this is related to the inability of the founders to let go of their ownership over product.

Imagine you’re a VP Product just hired by a CEO/CTO cofounder team. They promise you strategic ownership over the product. You, tapping on your decades of expertise, start making changes that you believe will position the product into a growing market. The founders, however, push back and ultimately veto you.

This doesn’t happen just the once, but nearly every time you try and make a strategic change to the product. “But this is what you hired me to do,” you nearly shout. After 6 months of rolling that boulder up the hill, you get frustrated and quit.

This happens a lot. If you’re a founder and you want to hire a strategic product executive, you will have to give up a good deal of your product ownership. If you have faith in the person you hired, you must allow them to run things their own way — at least until they show that your faith is undeserved.

It is very hard for an outsider to step into a startup and take product ownership away from the founding team. The outsider will either have to be bold, or the founders will have to be flexible.

If you think you want to hire a strategic product executive, but know you won’t be able to let go of your ownership, then you can either not hire a strategic product exec, or you can design guardrails for everyone. This really just comes down to the first point of having a good scorecard and clear expectations for what the role should entail.

In this case, the CEO/CTO must decide what parts of product and vision are still owned by them. They must learn how to identify when they’re pushing back because “it’s my baby”. They must learn how to have healthy, honest communication with the new executive they’ve brought into the inner circle. And finally, they must admit and accept that they can no longer run the entire company by themselves.

In Summary

The biggest pitfalls regarding hiring product executives center around a lack of clarity and communication. CTOs and CEOs, because they rarely have professional product management experience, think they can hire someone to “take care of product” at an executive level. It’s surprising how many intelligent people think that a role like this can be simplified that much.

Instead, recognize that this is a significant hire and you should do a lot of work before even writing your job listing. You should go into the process with a full understanding of exactly what this person’s daily responsibilities should look like, what’s in-bounds and what’s out-of-bounds for them, and what success looks like for you. You should have a scorecard and the first couple of major projects lined up. Discuss these with your candidates to make sure that they lie both within your candidate’s skill set and their proclivities.

Founders must also be careful of the emotional attachment they develop towards their product. If you need a strategic product executive, then you also need to let go of some things to make it work.

Hey, you made it all the way down here! That means  you probably found this content valuable. I won’t force you to give me money, I just ask that you share this with your friends.

Share

If  you want to support me writing this content, however, feel free to give me money! Either by subscribing or giving a gift subscription to a friend:

Subscribe now

Give a gift subscription

And feel free to Tweet your tech leadership worries to @bkanber, and I’ll try to address them in an upcoming newsletter.

You don't try,
so you never succeed.

You never succeed,
so you have no confidence.

You have no confidence,
so you're scared of failure.

You're scared of failure,
so you don't try.

You don't try,
so you never succeed.

You never succeed,
so you have no confidence.

You have no confidence,
so you're scared of failure.

Break the cycle.

You're scared of failure,
but you try anyway.

You try and fail,
so you have no confidence.

You have no confidence,
so you're scared of failure.

Break the cycle.

You're scared of failure,
but you try again anyway.

You try again,
and you win. Just a little.

You win,
so you build confidence.

You build confidence,
so you're less scared of failure.

You're still scared,
but you keep trying anyway.

You keep trying anyway,
and one day it works.

One day it works,
and you've broken the cycle.

Conventional Wisdom Says…

As with home security, most intrusions are opportunistic. Every home or car has a window that can be broken. Yet most intruders will simply check the door to see if it’s unlocked. If it’s not, they’ll go see if one of your neighbors was careless. Therefore a CTO’s first step should be to eliminate all opportunistic attack vectors. (UFO VPN’s logs were exposed via an unsecured ElasticSearch instance).

The other piece of convential wisdom says that you will likely not be able to withstand a directed, concerted attack from a skilled attacker. Therefore a CTO’s second step should be to hire a concerted attacker.

Unfortunately, many companies don’t have the budget for this, so the conversation naturally turns to what other concrete steps CTOs can take to secure their environments. We like security in layers and security in-depth, so don’t look at the following ideas as options to choose from but rather a whole set of tactics and strategies you can implement as they become appropriate for your organization.

OWASP Top 10

Most CTOs agree that the OWASP framework is… shoddy. At the same time, most also implement some sort of OWASP review into their code cycles. Why the apparent hypocrasy?

In an ideal world, all of your engineers would also be security experts. But this isn’t an ideal world, and they’re not. The OWASP Top 10 ends up serving not as a tight security framework, but rather as an awareness campaign. By adding a security or OWASP portion to code review, CTOs hope to attune their engineers to a basic understanding of security concerns. The OWASP Top 10 is an easily-digestible checklist that engineers can glance at before hitting commit; and the hope is that a code reviewer — with OWASP in the back of their mind — will be looking for other potential flags to raise outside of the OWASP framework.

If you’re not doing this already, this is a cheap and easy way to a) incorporate security review into code review, hopefully catching the truly low-hanging fruit and b) force engineers to think about the security implications of their code.

Penetration Testing

If you have a little bit of budget you should hire an external penetration testing firm. Conventional wisdom says you should do this at least once per year, preferably twice per year. Take the pentesting findings seriously. Generally, a pentest report will contain lots of items that you consider to be irrelevant or false positives. Don’t ignore them out of hand! Often, a tiny little vulnerability can be leveraged into a larger one, later on. Just because a pentester wasn’t able to convert a vulnerability into an exploit today doesn’t mean someone else won’t be able to 6 months from now as your codebase and architecture changes.

I also strongly recommend buying a copy of Burp Suite for your team and having a portion of your engineers learn it. You won’t be able to do as good a job internally as a professional firm can do, but even naively running Burp Suite against your app once per month can pay dividends. And similarly to the OWASP approach, making an internal pentest part of your regularly scheduled audits will only serve to attune your team more closely to security concerns.

Bug Bounty Programs

The problem with pentest firms is that pentests will typically only scratch the surface. The pentesting firm is paid to generate a pretty report; they’re not actually paid to discover vulnerabilities for you.

If you can’t afford a full-blown grey-hat hacker to run you through your paces, but you still have some budget left after hiring your pentesting firm, then you should consider running a bug bounty program through hackerone or similar.

Unlike pentesters, bounty hunters are paid per exploit. And they will find exploits. In my opinion this is the single most effective thing you can do with your budget.

You are typically allowed to define the parameters of the bug bounty program; ie “look only for authorization exploits” or “ignore our static-hosted sites”. This is a good way to make sure you don’t blow through a budget in the first week. You must be prepared to pay; expect several hundred to a couple of thousand dollars per exploit. (An ounce of prevention is worth a pound of cure.)

A fun exercise here is to get your team to tighten up security as much as they possibly can, until they’re convinced that the app has no holes. Then open up the bug bounty program and watch as your team realizes they’ve reinvented Swiss cheese.

White-Hat and Grey-Hat Researchers

Everything above really only aims to secure your application or software. But just as most undirected attacks are opportunistic, most directed attacks involve some social engineering. This is why conventional wisdom says you won’t be able to withstand a concerted hacker.

The only way to get a handle on this is to preemptively hire a concerted hacker. If you’ve run a bug bounty program already, then a white-hat hacker may not be able to find too many (or any) software exploits. But a grey-hat researcher who also uses social engineering almost certainly will find a way in.

This is where things get really messy and difficult to control. Think about all the ways “trusted users” could (accidentally or nefariously) exfiltrate data from your app. Every admin panel is a risk, and every person with access to an admin panel is a risk.

I still see companies using production exports as seed data for local development. This is both a terrible idea and also a common vector for data leaks.

The important questions you have to ask yourself here are: who has access and how, how do we protect data even when “trusted users” are involved, how do we set up layers of security to mitigate or minimize social engineering, and how can we best monitor for “funnies”?

Because every organization is different, the best thing to do here is to hire a skilled professional to put you through your paces.

Monitoring, Routines, and Incident Response

A healthy organization doesn’t just do these things once a year or once per quarter. They maintain a state of constant vigilance. They install intrusion detection and network monitoring software, use tools like Amazon Macie to detect accidental PII leaks, and most importantly respond to incidents in a predictable and timely manner.

I find about half of startup CTOs have no incident response policy in place whatsoever. This is also a bad idea. Many companies are in an in-betweeny state: they do have an on-call rotation for engineers, they have some monitoring and firewalls, but have no actual policies in place for what to do when they find something. So the on-call engineer calls their boss, and their boss makes an ad-hoc judgement call. Most people are not crisis experts, so it is tough for me to advise allowing people to make ad-hoc judgement calls while they’re under duress. This is how security incidents get swept under the rug.

A better approach is to have scheduled audits and a well-defined incident response policy. When an engineer finds something funny, they should be able to look at their checklist or flowchart and be able to respond to the incident with confidence. When their boss is called in, they should be able to look at the flowchart and the risk categorization of the servers/systems/data that was breached, and know exactly under what conditions customers and authorities need to be notified, and how.

Most CTOs wait until their first security incident to figure this stuff out. That happened to me nearly a decade ago, and it’s not a comfortable position to be in.

This all feels like paperwork and most techies hate paperwork, so unfortunately most ignore it.

Fortunately, the various compliance frameworks make it easy to wrap all this stuff up into a single procedure. GDPR, ISO-27001, SOC 2, and the NIST framework all require that you have risk classifications as well as incident response procedures. So in this case you can drop two birds with one stone: pursuing compliance will also prepare you for these eventualities. Even as a company with little or no budget, you can still strive to hit all the GDPR/ISO/NIST checkboxes even if you don’t pay for a certification at the end of the day.

There are two major, non-technical advantages to pursuing these frameworks. First, they force you to maintain a dataroom with good paperwork. It’s time-consuming to set up, but easy to maintain once it’s in place. Second, they force you to continually audit and record your procedures.

Fortunately for us, our dataroom has only one security incident filing in it. But it does have dozens of audit reports, attestations from engineers about training they’ve received, checklists, procedures, form letters, and anything else we could possibly need if facing either a security incident or an audit from a regulatory agency. The peace of mind that this offers a CTO simply can’t be measured in dollars.

In Summary

All CTOs must practice defense-in-depth. It’s too easy to focus only on software and code security, however, so you must also take a close look at your people and processes.

As always, if you’re not an expert in something, hire an expert. Get an external pentest once per year, pay for internal Burp Suite training, start a bug bounty program, hire a greyhat.

If you can’t afford to hire an expert (and even if you can), strive to make security cultural. Train your engineers on security practices quarterly. Make security review a part of code review. Train your non-engineers about PII, data exfiltration, and social engineering.

Finally, hope for the best but plan for the worst: make sure you have clear incident response procedures, make sure you have an email template informing customers of a data breach prepared, make sure your data room is up-to-date, make sure you know how to contact the appropriate authorities based on your jurisdiction.

And most of all, make sure to keep your front door locked even though you have that huge window in your living room.

Hey, you made it all the way down here! That means you probably found this content valuable. I won’t force you to give me money, I just ask that you share this with your friends.

Share

If you want to support me writing this content, however, feel free to give me money! Either by subscribing or giving a gift subscription to a friend:

Subscribe now

Give a gift subscription

And feel free to Tweet your tech leadership worries to @bkanber, and I’ll try to address them in an upcoming newsletter.

Most of the time, tech leaders face common and relatable problems. We face growing pains as companies and have to figure out how to balance our teams, provide career paths for employees, and so on.

Today though, in mid-July 2020, we’re facing some pretty uncommon problems. I’m hearing lots of new concerns from CTOs over the last four months: employee health and safety, economic collapse, shifting to work-from-home, figuring out how best to support Black Lives Matter, and whether (or how) to attempt a slow shift back to office work.

My plan for this newsletter was to go over old conversations, because over the years I’ve found that there are a finite number of core CTO concerns. But that type of advice isn’t as valuable today. The next few newsletters will therefore consist of more timely topics, and this week’s will be COVID-19.

Here’s what I’m hearing CTOs worry about most.

Are you keeping your offices?

What are your (company’s / team’s) overall plans regarding office vs work-from-home moving forward?

This question was more popular back in May and June, as many companies were adjusting to remote work and trying to figure out if it was viable long-term.

I’m seeing about 60% of companies and teams keeping their office situation “as-is for the foreseeable future”, meaning that they’re going to keep the office but continue to WFH until it is safe or until some other factor changes. In general, these respondents are larger companies with a lease on a single large office space.

I’m also seeing most companies with multiple offices trying to figure out how to combine/reduce/eliminate office space. Here I’m seeing all sorts of permutations; keep the NYC but close the midwest, keep the west coast but close NYC, eliminate the overseas office and have them be WFH, etc.

Of course, a number are going fully remote now. The companies that I see doing this already had a good remote culture in place while maintaining a colocated office. An interesting thing to look out for is how sales and marketing teams will adjust to remote work; the companies I see going full-remote already have a solid product/engineering remote culture, but colocated sales/marketing. This isn’t necessarily a “CTO worry”, but we will need to figure out how to keep extroverted members of the team (regardless of department) fulfilled through quarantine and remote work.

In short, everyone’s just doing what seems best for them. I do worry that some companies are underestimating the effort it takes to develop a strong remote culture. Everyone pulling together to tough-out a few months of WFH is different from shifting your organization to a permanently distributed one, so watch out for landmines here.

Leave a comment

We have people desperate to come back to the office.

Are there any policies that make sense?

This question circulated around in mid-June as NYC had flattened the curve and kept it flat for a bit. THE RESPONSES TO THIS QUESTION ARE THEREFORE OUTDATED.

I DO NOT RECOMMEND YOU ATTEMPT A TRANSITION BACK TO THE OFFICE. Cases in the USA are surging as of this writing and I strongly recommend everyone continue strict social distancing and work-from-home policies.

Now that the disclaimer’s over: last month, I discussed with several CTOs what a return back to office work might look like. Is it possible to do it responsibly? If so, how would it work?

There was unanimous agreement on the following points:

• Return to office work would be opt-in, and ideally limited to the lowest-risk population (how a company could enforce this was beyond us)
• Office occupancy would be kept very low, 25% or less
• Good office ventilation (open windows) required
• The company would need to provide PPE and at very least, cleaning equipment and training, or at best, daily professional cleaning

However, even then, we still couldn’t agree that it was a good idea or safe to return to the office. We brainstormed for a while and came up with some more bullet points to play with:

• Only people who can walk to the office can use it (ie no commuting allowed)
• Co-working in Central Park with PPE and social distancing
• Only one person in the office per day

And even then, we still couldn’t agree that any of this would be a good idea for the company to officially support. So, even when the curve was pretty flat and we were feeling confident, we couldn’t come up with a great way to return to office work quite yet. At best, we were able to come up with a few safety precautions. Again, I do not advise anyone to attempt to return to the office unnecessarily. It is all of our responsibility to flatten the curve.

Leave a comment

We have a team member who’s always been a bit abrasive in written communication.

They’re good in-person but now that we’re remote, it’s becoming a problem. How can we address this?

There were a few variations of this question recently, but it boils down to the basic issue of an employee who was “marginally toxic” pre-COVID becoming more toxic now that communication is largely written and asynchronous.

The obvious reaction to this is “you should not retain a toxic employee”. Since this isn’t a 1-on-1 advice session, only you can know your circumstances. Perhaps it is not possible to let that employee go.

I find that whenever there’s a discussion of a toxic employee, the first question that must get resolved is one of self-awareness. How aware is the employee that their own behavior is seen as toxic? Have you (the manager) ever addressed it with them directly? (A surprising number of managers only dance around these issues, but it’s important to have the hard conversation.)

One idea that floated around this conversation was a code of conduct or a looser “styleguide” for Slack/GitHub/Jira tasks. I’ve not seen anyone implement this yet, so the jury’s out for now.

One of these conversations landed on a good point: if you decide to retain an employee that you know is toxic, it’s now entirely your responsibility to a) figure out why they’re toxic in the first place and fix it and b) protect the rest of your team from that toxicity at all costs.

This really only applies in cases where someone is just marginally toxic, and perhaps is just reacting to recent circumstances rather than displaying chronic behavior, and is also otherwise a very valuable team member.

In one case, the employee in question was one of the early team members, had always been pleasant to work with, and developed some of the core systems early on. However, this person had become more abrasive in written communication over the last year (still pleasant in-person), and it reached a breaking point during the early weeks of COVID lockdown (with no more redemptive in-person meetings).

We discovered that the employee had long shown signs of unhappiness in their work. After contributing and developing core systems, that person’s role shifted to maintenance and support, and had become increasingly frustrated with seeing his work be managed by sales managers and customer demands. Additionally, this person was now in the customer-support loop, and spent most of their time just adding reactive patches, doing maintenance, and fulfilling feature requests.

This is fortunately one of the cases of employee toxicity that may be resolvable. A recommended course of action was to immediately eliminate that employee’s customer support duties (as that seemed to be a large point-source of stress for the employee), and then to figure out if there are any greenfield projects they could take on in Q3.

The moral of this story: some employees are slowly and quietly driven to the brink by the circumstances around them. It’s easy for a manager to accidentally lead someone into a situation like this. If you see a typically-reliable employee start to become more difficult, you had better take a step back and try to figure out if there is a root cause nearby.

Leave a comment

How are you adjusting your roadmaps for Q3 and Q4?

Everyone’s Q1/Q2 roadmaps got messed up — and now CTOs are wondering how other companies are adjusting to this. They’re asking questions like: how far out are you roadmapping? How are you adjusting for remote/WFH? Are you incorporating plans for office reopening?

I’m hearing most CTOs assume that their teams (if not their companies) will be WFH through the rest of 2020 at least.

I’m also hearing some CTOs making downward adjustments to WFH productivity modeling. One CTO said “I’m assuming 4 productive hours per day per employee”. The general consensus is that productivity is a little lower at the moment, but we’re all hoping it’s a temporary condition caused by undue stress, fear, and unexpected life changes.

Overall, it seems like it’s not a bad idea to plan for a less productive Q3, and use the next quarter to focus on how we can support our employees through this transition so that they can approach normal productivity levels again in Q4.

Another general consensus seems to be that nobody is long-term roadmapping at the moment. Most are looking only a quarter ahead, and trying to bullet-point out two-quarters ahead, but all are braced for goalposts to move rapidly and unexpectedly.

Companies who typically do big, annual roadmapping exercises in Dec/Jan are still planning on attempting it. Perhaps we’ll stabilize enough in Q3/Q4 that a 2021 roadmap is possible to plan.

But for now, don’t sweat it. Do your best to plan Q3 around helping employees thrive remotely, and take it sprint by sprint for now.

Thanks for reading this July 12th edition of What CTOs Worry About. Please subscribe and take part in this week’s subscribers’-only discussion.

Have your own CTO or tech leadership worry? Tweet @bkanber #ctoworry and I’ll try to fit you in to the next newsletter!

And don’t forget to tell your friends!

I've been teaching my whole life. I started out teaching music theory at a music school. I tutored math and physics. I've taught courses on JavaScript. I've also taught more technical topics like electromechanical control systems and machine learning.

Over the years I've seen many students succeed, but I've also seen many fail. The failures are more interesting to me, and I've come to develop a theory of why people fail to learn complex skills. If you're about to pick up a new skill, I think an understanding of why people fail can help you avoid failure yourself.

Most Fail Because They Quit

Without a doubt, the most common failure mode is quitting. I have very rarely, if ever, seen a student fail because they weren't "good enough" or "smart enough". Students that stick with it succeed, and students that quit fail.

This alone isn't enough to understand failure. Maybe if there were a handful of common failure modes it would be useful to talk about each one. But it turns out that to ask "why do we fail" is really to ask "why do we quit?" We can also ask the inverse, "why do we succeed?"

Frankly, it really is all about practice. I've never seen a student practice guitar for 1,000 hours and then fail. The same goes for computer science, the same goes for math and physics. Many people dislike Malcolm Gladwell's form of journalistic pseudoscience, but I've seen his "10,000 hours" theory play out time and time again.

Great -- so if you practice, you get better. This still isn't too helpful. If we're concerned about avoiding failure, we need to dig deeper still and talk about why people fail at achieving the practice hours they need. This is where it starts getting interesting.

What I've seen is that there are three primary reasons students quit: confusion, frustration, and boredom. There are others, but those three are the most common.

When both teacher and student solve those three problems success rates shoot way up. Let's talk about each in turn and discuss some ways to get around them.

Confusion

Learning a new complex or valuable skill is difficult. If it were easy, everyone would know it, and it wouldn't be so valuable. Confusion is the first problem that students tend to run into.

Let's say you decide to learn computer programming as an example. The first question you're faced with is "what language should I learn?" Well, there are dozens. How do you choose from those? Maybe you know you want to write Android apps; it's good to have a specific target like this in mind, but even narrowing the scope only helps a little. Do you pick Java or Kotlin? How do you set up your emulator? What are all these tools you need? What is object oriented programming?

It's way worse if you have no idea what type of programming you want to do. How do you choose between Python, Ruby, JavaScript, Erlang, Scala, Go, etc. The landscape is just so vast.

Think about what it takes to build a simple, static webpage, and get it hosted on the internet. Not only do you need to learn HTML, CSS, and possibly JavaScript, you also have to understand what each is for. If you're an experienced programmer reading this, that might sound trivial to you, but imagine someone starting on day 1. Just understanding what HTML, CSS, and JavaScript each do and what their role in this ecosystem is can take hours of work and research.

After that, you also need to understand the difference between a domain name and a host. There's shared hosting and virtual machines. If you want to run your own webserver, you've got to decide between Apache and nginx. Should I use DigitalOcean, AWS, GoDaddy, or one of the thousand other companies out there? Again, the ecosystem is vast.

Many students quit at this point. Stepping into a new world is incredibly daunting. I don't have numbers to back this up, but a good portion of people quit right here, before they even get started.

Frustration

Frustration is a huge problem and is probably responsible for more failures than any other mode. Let's say you've figured out exactly what you want to learn. Now it's time to actually learn it!

But learning new skills is hard. Learning new things is hard. That's what learning is -- it's the process of going from not knowing anything about a skill or subject, to knowing lots about it.

Learning to program is frustrating. Why doesn't this code work? Why does my page crash? How come this doesn't do what I expected? There are also times when a student just can't figure out how to solve a problem. The patterns aren't intuitive to them yet. This causes a great deal of frustration, and often the student will put down the problem and never pick it up again.

I see this a lot with learning to play an instrument as well. Learning guitar is tough. The fingerings are awkward and often painful to a beginner. Transitioning from one chord to another is difficult. These students know what they have to do, but they don't have the muscle memory and articulation to do it yet. This is frustrating. Eventually the student puts down the guitar and never picks it up again. How many of your friends in high school got a guitar for Christmas one year, played for a week and then never touched it again?

I find that students will often make up excuses to justify their quitting to themselves. "My fingers just don't move like that" for guitar, or "my hands are too small" for piano, or "I'm just not a math person" for STEM subjects. These excuses provide a level of cover for the student to protect their egos, when the truth is that they've really just gotten frustrated and quit. If Django Reinhardt, who has only two fingers, can be considered among the best guitar players in history, I assure you that your fingers can move like that.

It's important to understand that frustration is a natural part of the learning process. It gets easier over time if you stick with it. I always tell students that there will be times when they get frustrated and want to quit. Just telling them that helps them get past it -- they now know that everyone gets frustrated and wants to quit at some point.

Boredom

Learning a skill is more about practice than it is about play. Often students will get through the initial confusion and frustration, reach a level where they can play a couple of songs on guitar or throw together a couple of simple programs, and then plateau. The problem here is that continuing to grow this skill through practice becomes boring.

With computer programming, the types of homework problems you get in a 101 course (Fibonacci sequence, bubble sort, etc) are great practice, but frankly quite boring. Similarly, running scales all day on guitar or piano is boring. Students will often revert to play, ie, playing the same song over and over again without stretching their abilities, or else they get too bored with what it takes to reach the next level and quit.

At this stage it's important to be able to balance practice and play. Writing code to design 101-level algorithms is boring on its own; but finding a larger motivating project that requires those algorithms is not. Playing scales and practicing fingerings on its own is boring; but picking a challenging song that uses those scales is not.

This is a bit pseudosciency, but I claim that teachers and students need to figure out ways to trigger the dopamine rewards at this stage. Every hour of practice should be related to an hour of play, otherwise the student will say to themselves "this is not what I expected programming/guitar/physics to be like" and then quit. It's another ego-covering justification. Of course it's not what you expected; you didn't actually know before you started. But if your teacher only ever gives you practice and there's no play or larger goal, that skill will become fatally boring.

How to Succeed in Learning

If you want to succeed at learning (or teaching) a new skill, the ultimate goal is to rack up hours of practice towards the skill. This will sound a bit Malcolm Gladwell-y, but let's say that the three tiers of skill occur at 100, 1,000, and 10,000 hours of practice. Those tiers translate roughly to basic understanding after 100 hours (ie, having a rough understanding of how programming works), proficiency at 1,000 hours (ie, I could get a junior/entry level programming job), and expertise at 10,000 hours (ie, I can get basically any job I want).

To reach those tiers, you need to solve for confusion, frustration, and boredom. When I teach entry-level computer programming, for example, I teach "Building classic video games in Processing". Let's break that approach down.

(Processing is an all-in-one IDE and programming language based on Java; it's geared towards canvas drawing/animation and mouse/keyboard interaction. You don't see it being used "professionally" but it is a fantastic educational tool.)

Confusion: building video games in Processing readily solves the confusion problem. You use one tool as your platform. You write in one language. All the documentation and tutorials are in one place. It doesn't matter that you're not going to become a professional "Processing developer". Processing is simply the platform on which you become a programmer. Become a programmer first, and then figure out where and how to specialize. Don't worry about the huge ecosystem of programming languages, technologies, libraries, and so on, just focus on this one platform that you can install within second and immediately get everything you need to start with.

Boredom: when I teach this topic, I show students how to build Pong in about an hour in Processing, and then tell them to pick 5 other classic Atari arcade games and then go build them. Programming video games is challenging; you'll need to learn about data structures and algorithms, object oriented programming, and so on. It's also fun -- Processing gives you an immediate feedback look. Writing algorithms to solve the Fibonacci sequence is boring; the output is just numbers in a console. Writing video games is rewarding and gets that dopamine flowing. It also makes the process very tangible and visual. Accidentally making a number negative when it should be positive has the clear and distinct effect of moving something in the wrong direction. Overall, recreating these classic arcade games forces you to practice the skills while also contributing to the sense of play and work towards interesting and achievable projects.

Frustration: another reason I teach students to recreate classic video games is because they're narrower in scope than other types of projects students might be interested in. Deciding to build a new social network might be an interesting project, but it's way bigger than the student expects it to be. On the other hand, classic video games tend to be small and achievable. A novice can build Pong in a day. Breakout in two days, asteroids in three. Being able to start and finish many small projects is more rewarding than starting one giant project that has no end in sight. The scope of these games is well-defined, and that keeps frustration in check. It's a lot easier to avoid frustration when you're able to write out a list of 100 discrete tasks that need to be accomplished and check them off one at a time. You also know what the end product is supposed to look like.

Keep in mind that accomplishing 1,000 hours of practice might require 2,000 hours or more of total effort. If you're building your third video game, there's a whole bunch of stuff that you already know how to do -- setting up the canvas, setting drawing modes and colors, things like that. Those tasks might have counted as practice the first or second time you did them, but they no longer count the 10th time around. They're simply reinforcing the muscle memory; they become play.

The same applies to guitar or piano. It's only practice if you're doing something new(ish); it's only practice if it's hard. The first few times you play Wonderwall count as practice, the next 200 times you play it are just that: play.

For this reason it's important to continually level up by choosing more difficult projects as time goes on. Otherwise you'll plateau if you keep doing easy things; you'll rack up a whole bunch of hours that are all play but don't actually contribute to the development of new skills. Every project has a certain amount of boilerplate that goes into it, and as you develop more skill, a larger proportion of your projects will become boilerplate. This is why many students with years of experience may never become experts; if 99% of the things you do are things you already know how to do, you're not actually practicing anything new.

What About "Natural Skill"

What about the claim that some people have natural skills? Some people are naturally better at math, or art, or music. This is a big nature vs nurture debate, and I won't address the question of whether innate skills from birth actually exist. That said, it's clear that some students have a stronger starting point than others (whether that comes from birth or simply early childhood).

I believe that these natural abilities give students a small advantage that apply only at the very beginning of the learning process. Your natural skill in math or logic, or your previous exposure to technology might give you, say, a 20 hour advantage over the next person. Those hours apply only to the earliest stages of learning. If the goal is to reach 1,000 hours of practice, that 20 hour advantage doesn't count for a whole lot.

That slight advantage can be significant, however. I don't think it's significant in terms of the actual skill learned, but it is significant in the context of confusion, boredom, and frustration. Students that start with this advantage tend to get less confused or frustrated, and they're less likely to quit in the crucial first 100 hours. But for me the important point is that students that don't have those natural abilities can still learn the skill to the same degree as those that do; they just need to be more careful to avoid the pitfalls in the early stages.

A Note on Intuition

Ultimately, learning a skill and following it through into expertise is about developing intuition. Intuition is a form of muscle memory, but not just the physical aspect of the skill. Intuition is a sort of muscle memory related to both the practice and theory of the skill.

If you want to develop expertise, therefore, you must develop intuition. To develop intuition, you have to practice both the theory and the execution in lots of different ways. You need to experience, firsthand, how the theory applies itself to lots of varied situations, and how the interface between theory and environment manifests into execution.

Developing intuition once again comes down to practice. But it requires a different type of practice than the practice you need to develop the skill initially. You need to practice taking things you know, or things you think you know, and varying them, using them in different ways and in different places, with different audiences or different environments.

If you want to develop intuition for machine learning algorithms, for instance, you should write the algorithms from scratch lots of different times, in lots of different ways, in different programming languages, using different problem sets, using different constraints, using different parameters.

The way I see it is that every subject or skill is a highly complex multi-dimensional shape. When you solve a problem one way you illuminate one side of it and come to learn a bit of that shape. But if you want to truly understand the shape, to develop an intuition for it, you need to illuminate all the sides. This requires turning it around, redoing the problem, turning it around again, illuminating another side, and so on, until you've ultimately seen all the possible facets of your crazy, complex idea. Eventually you reach a point where you understand the shape of the thing so well that you can intuitively predict what will happen when you shine a certain type of light on it from a certain angle. This requires a thorough understanding of the theory as well as lots of varied experience and exposure; only then can you claim expertise and intuition about a subject.

Final Words of Encouragement

Everything I've experienced in my teaching life leads me to believe that you can learn it, no matter what "it" is. You absolutely can do it.

You'll be confused. That's what learning is. That's ok. Keep reading, keep grinding, keep practicing. You can, and should, limit the scope of the thing you're learning in order to make it less confusing. You can always expand the scope later.

You'll get bored. That's ok. Just make sure you have a bigger, motivating goal in mind. Playing scales is boring. Remind yourself that you have to do the hard parts so that you can get to the fun parts. Make sure that you have a fun part in mind.

You'll get frustrated. That's what practice is for. It's supposed to frustrate you. Practice enough and you won't get frustrated as much. And then pick something harder to practice, so that you get frustrated again. It's ok to get frustrated, it's part of the process. Take a break but make sure to come back later.

If you don't quit, you will learn it.

This takes us back a decade or more to when I was a senior in college. A team of six of us were building a race car both to compete in Formula SAE and to serve as our senior project. My responsibility was the engine computer, electronics, and wiring. It was a small team, though, so we typically worked in pairs or threes on different parts of the car.

One of my teammates was responsible for building and tuning the engine. My engine computer was ready right around the time his engine was built, so we spent a few hours installing everything onto the test stand. We were both delighted that the motor actually turned over, but also surprised at how poorly it ran!

An hour or so of tuning later and we finally have the engine idling nicely. But then it dies inexplicably! We fiddle with the numbers again, but it seemed we had already found the best timings for idling, so that wasn't the problem. But time and time again the engine died. Sometimes it died after a few seconds. Other times we got 30 seconds of measurements out of it. On the rare occasion, it seemed to work perfectly.

My teammate and I went back and tore our components apart and rebuilt them piece by piece, checking everything. We did find some little issues, so that was a good exercise in and of itself, but ultimately we still had the same problem.

After the second day this happened, I set up a few scopes to monitor different parts of the electronics. Each time the engine died, the ECU had a voltage spike and then settled into a safety state, requiring a reboot. The ECU, not the engine, was to blame.

But for days I couldn't find the source of the issue. I focused on interference and power conditioning, figuring that maybe (hand wavy) some magnetic field from the motor was causing a ground loop in the ECU. But shielding the ECU and the cabling didn't help. Using batteries and other conditioned power sources didn't help. I changed every single wire and cable at least twice, checking each. The ground was a literal earth ground: I used the iron engine stand, which itself was embedded in the concrete, in the workshop basement, underground.

I was at wits end. My teammate was understandably frustrated with me. He needed to run the engine a lot, at different loads and throttle positions, in order to design the spark and fuel timing maps. But because of my electronics, his engine wouldn't run reliably for more than a minute or so.

After another failure, and feeling completely lost, I said to him, "I don't know man, the only thing it could be is magic." And right at that very moment a familiar odor drifted in. The magical scent of ozone.

He and I charge into the room next door to find a third teammate happily welding away, working on the car's frame. There was no way the welding rig could interfere with the electronics, right? Not through a couple of walls and 30 feet of earth ground?

We asked him to stop welding and started the engine again. It ran. And it kept running quite happily. We waited a few minutes until we were satisfied, then asked him to weld something. The engine died instantly. It turns out the welding rig could interfere with our electronics through a few walls and 30 feet of earth ground.

To this day I'm still not 100% sure if the interference was RF or a ground current. I think a ground current (some may call it a ground loop) is much more likely, but the fact that our welding rig could dump that much current into ground was pretty astonishing at the time.

Anyway, just goes to show that even when you think you have a handle on your environment, you may not.

SaaSCo is six years old, heavily capitalized, and perpetually "6 months away from profitability". They have a decent product that's marketed well. Steve has overseen the growth of the product and engineering team from 3 to 30. The product lives on a monolith written in something mature and boring, like PHP, Python, or Ruby.

SaaSCo's product has a huge footprint. The important parts are mature and work well. There's a good deal of debt in the codebase. Ever since the front-end shifted to React a few years ago big parts of the legacy codebase are being deprecated.

The engineering team seems antsy. Deadlines are rarely hit. The product is just too big and complex. Changes often have unintended consequences. Somebody's always refactoring something, and internal documentation is essentially worthless.

Steve has heard the microservices success stories, and so have the engineering team. They're eager to try it. Many of them feel that the Python backend is boring, and they're itching to work with new technologies.

Steve decides to start the long and arduous process of converting the monolith to microservices. It'll be a big undertaking, but it'll all be worth it because of the increased velocity we'll have afterwards.

So SaaSCo engineering starts methodically breaking things out into microservices. First they remove image processing and replace it with an IMagick-based microservice. Then comes the Spamassassin service, and then the stats-processor service. This all goes pretty smoothly. Those services were already on the fringes of the monolith, and separated easily. Note that the team is now supporting four service deployments.

Next they try breaking authentication out into a microservice. This is tougher and scarier than planned, but it works out. Encouraged, they start breaking out more core services into microservices.

But then they hit a wall. They quickly discover that the microservices are all talking to each other constantly, and have hidden dependencies they didn't notice during planning. The legacy monolith is still running, albeit in a crippled mode, and has become a sort of service router for the app. Steve decides to pump the brakes on the aggressive microservice-ization in order to stabilize a bit. The solution is to build shared support libraries that help microservices interface with one another. There is also now a microservices style guide that specifies which languages and libraries should be used, in order for the majority of the team to be able to support the majority of the services. This, of course, makes the engineering team upset because a few of them really wanted to use Elixir in one of the services.

Steve now has a lot more problems than the monolith, which, by the way, never fully goes away. Instead of a monolith, Steve now has a dozen different services built by different people using a patchwork of different technologies, libraries, and API conventions.

The development velocity does not actually improve after shifting to microservices. Years go by in this state of microservices purgatory, and then with SaaSCo v4.0 the team proudly announces that they're going back to a monolithic model, except now with a new shiny framework!

I think the moral of the story goes something like this: Steve didn't have a services problem, he had a complexity problem. Moving to microservices does not necessarily solve complexity problems. Instead of focusing on the "DevOps" layer and topology, Steve should have started with an even more abstract view of what the system components are and where the complexities lie. The simple truth is that complex things are hard, and most successful business platforms are complex. And it turns out that teams of engineers are not typically experts at highly-complex system design, and so most will reach for solutions that sound right but don't actually address the root problem.

I'm not claiming that microservices aren't useful or a solid design patterm. I'm just saying that I've seen a lot of CTOs reach for the microservices hammer when they really needed a more subtle tool. In most cases I've observed personally, a much better approach is to reorganize your monolith into a shared-library model. Monoliths by themselves aren't evil. Microservices by themselves aren't good.

I would go even further and offer this rule of thumb: one should not convert a mature monolith to microservices. The move from monolith to microservices, for a sufficiently complex platform, is very expensive and fraught with peril. It's a large jump: microservices is not just one step away from monolith. The more complex a platform is, the more inertia it has, and the tougher it'll be to move that large distance towards a replatform. Moving straight to microservices in this case is like trying to stop a boulder in its path; whereas a subtle refactoring to a shared-library model is more like nudging the boulder so that it doesn't crash into the village below.

By the numbers: In my career I've personally observed maybe 20 examples of CTOs choosing microservices. Of the fresh, green-field platforms that choose microservices, 8 of 10 are successful. However, of the mature platforms that decided to rearchitect to microservices, I've only seen one company actually pull it off to completion. In other words, I've observed a 90% failure rate in large monoliths successfully and completely moving to microservices. The rest have ended up in a state of purgatory, or have rolled back their efforts.

So this is my caution against microservices: if Steve's story feels all-too-familiar to you, please think twice before trying to step in front of the boulder.

Lots of considerations factor in to the debate, but those are out-of-scope for this article. For now, let's assume you've decided to buy. What's the best way to proceed?

I claim that the most common pattern is actually an anti-pattern. It goes like this:

Manager: I've researched 10 options for the best Widget, and found that WidgetizerX is the best tool. I'll buy it now. Please implement it into the product.

Engineering Team: [Implements WidgetizerX into the product]

On the upside, this process is fast and does achieve the product goal. Unfortunately, it introduces tremendous technical debt. The implementation introduces a tight coupling between a third party service and your product.

If WidgetizerX goes out of business, or a better option appears in the future, or the decision to bring the functionality in-house is eventually made, the engineering team will either have to clear the technical debt first (by decoupling the implementation) or re-finance the debt (by way of a drop-in replacement) potentially at less favorable terms.

Here's the way it should go:

Manager: I've researched 10 options for the best Widget and decided on WidgetizerX. My next two choices were Windowmuxer and Panelista. I am most interested in features X, Y, and Z.

Engineering Team: [Researches APIs for all three, designs a lightweight abstraction layer for the functionality, implements a WidgetizerX driver]

Yes, this does slightly increase the cost of implementation. One could argue that the decision to buy was made in order to achieve the lowest possible implementation cost, and so spending the extra time to design an abstraction layer is stupid. My rebuttal is that a little bit of extra capital can eliminate a long-term interest-accruing debt.

Besides, it's comparatively cheap to do this. If an in-house implementation would take 3 months, and a drop-in implementation would only take a day, it's not that big a deal to spend two days on a driver- or adapter- based implementation. You've still saved 3 months in either case.

The adapter layer doesn't need to be complex; it just needs to support the common functionality of the different backends. If we're talking third party APIs, this layer might be as simple as an interface that defines authorization methods and the hooks and events that your application will tie into. Then you write an adapter around WidgetizerX and implement that.

My favorite software design principle is that of appropriate abstractions – "put things where they belong". When you buy a technology, it simply doesn't belong in a place where your product or business closely relies on it. It belongs a degree of abstraction away – behind a driver or adapter layer or something similar. Not implementing it that way will therefore accrue debt or cause other problems down the line.

I find that using this approach is worthwhile most of the time – that is, in most real-life cases, probably 80% of 'buy' technologies need to be replaced over the long term. Functionality moves in-house, or a different vendor is chosen, or the need to support multiple vendors arises. Sometimes you never need to use your adapter layer – maybe you stay on AWS S3 for file storage forever – but most of the time it pays off. Sometimes you get the adapter layer wrong and that introduces a different kind of debt, but teams get better at this with practice.

So if you do decide to buy, the best advice I can give you is to buy like you'd build: with research, a design step, and an implementation at the appropriate degree of abstraction.

I've written and published two books on machine learning, both with tech publishers. I'd like to share some of the more interesting things I learned about the tech publishing industry.

I don't claim to be an expert on the publishing industry. I only have experience with two publishers, neither of my books are best-sellers, and outside of those two books I have no other experience with the publishing industry. So please take this article with a grain of salt; this is simply one author's personal experience in publishing, and my extrapolations may or may not be generalizable to the industry as a whole.

That said, if you're interested in writing books on tech my experiences may be interesting to you.

Publishing is Driven by Marketing

It's probably not surprising that the tech publishing industry is very marketing-driven. Book topics are not chosen by techies, they're chosen by marketers. My impression is that a team of marketers at the publishing company is doing constant market research to determine which topics will be popular in a year's time. Within those topics, they're also figuring out what keywords will be the most sought-out. It's all about predicting demand.

The end product of this market research is a) a book title, b) a table of contents (including page-count per chapter), c) keyword-rich chapter summaries, and d) a timeframe. That is, as an author, you're given a table of contents that you have to write against, and you're required to include specific sub-topics, keywords, and technologies in each chapter in order to check the boxes that marketing believes need to be checked. The table of contents reads like an exercise in SEO. This is required of you regardless of whether those keywords and technologies really make sense for your book, your approach, or your narrative.

Overall, I didn't find this too burdensome. I was forced to include a few technologies that I otherwise would not have included in my last book, and I was forced to write a couple of chapters that I didn't think were necessary. Those chapters did steal some page-count from other topics I personally would have liked to discuss. But I do understand the need for these types of books to be driven by marketing, so I can't get too upset about this.

Be prepared to write against a table of contents that is given to you by a marketer, not a technologist.

That said, you have a lot of creative control over the content itself. As long as you write against the TOC and tick the right boxes, you can craft the content and the narrative however you like.

The Timeline is Insane

Book timelines are decided in advance, and the publisher/your editor will expect a given word-count-per-week from you. They simply work backwards from the publication date and the target page count per chapter and expect chapter submissions at the appropriate date. Different publishers have different approaches to this, but what I found most jarring with my most recent book was that they made no consideration about different chapters having differing levels of complexity or difficulty. It's very easy to crank out 15 pages of intro chapter material; it's a lot more difficult to write 15 pages of technical material concerning neural network topologies. Yet you're given the same amount of time to write and submit each block of text.

On top of the timeline's rigidity, I also found that the timeline was very fast. The publisher is sticking to their marketing schedule, and the book needs to get out by the date they planned on. Part of their market research involves figuring out when competing publishers might be publishing a similar book, and of course each publisher wants a month or two of topic exclusivity.

If you have a full-time job these time constraints make it very difficult to produce high-quality work. Publishers seem to like me because my first drafts are very clean, but that actually became a liability for me; when they saw that my first draft was basically ready-to-publish, they moved up the publication date by a few weeks. This left me with a couple of chapters that I literally was not able to do a thorough second draft of. My work may have met their standard of quality but it did not meet my own, and unfortunately I'm not happy with 100% of the content I published.

Given more time, I also would have gone deeper into the mathematics and algorithms of the more advanced topics, but sadly I was forced to cut corners and use open source libraries in some cases where I would have preferred to write and explain the algorithms from scratch. I simply didn't have the time. And I'm a fast writer.

That said, I found that the target page counts were not at all hard to achieve. Because the TOC is written for you it's pretty easy to write 80% of each chapter as if it were its own essay, and I had no difficulty filling the pages with good information (except for a couple of the "fluff" chapters I wouldn't have personally decided to include; those were tedious at times). So the time constraint is not so much about quantity, but quality.

Most Authors are Flakes, Maybe

One thing I found surpising is that both publishing companies I've worked with had processes in place to deal with "bad authors". Authors can be good or bad across many different dimensions; but reading between the lines, I would guess that many authors are either low-quality writers or have a high probability of abandoning the project. This is very much an extrapolation and little more than guesswork on my part, so take this section with a huge grain of salt.

Both publishers and all of my editors have given me huge praise for the quality of my first drafts. In both cases, the only edits that came back were simple stylistic and formatting edits (eg, not using the correct formatting block for inline code vs inline variable, or not writing references to other chapters correctly). Combining that information with the fact that my publisher wanted to move up the publication date of the latest book leads me to believe that the publishers expect a relatively low-quality work product from their authors and plan for a good deal of internal editing.

I also get the impression that a large percentage of authors are unable to complete their projects. It is a pretty rough timeline, after all. The reason I landed my first book is because the original author abandoned the project and ghosted on the publisher (this is why I have a co-author credit, despite rewriting the book from scratch; the publisher could not get the copyright release from the original author).

I therefore wasn't too surprised when I learned of my second publisher's practices concerning abandoned titles. They had a policy where if a chapter wasn't submitted every 20 days, the book would automatically be flagged as abandoned. Personally I think that policy is a little aggressive, but after seeing what the first publisher had to do to get that book published, I don't blame them. Still -- I went on vacation for a week while writing the 2nd book, informed my editor and all that, and when I returned I found that I was locked out of my author portal because my book had been flagged as abandoned.

I can only assume that there's a large percentage of authors that simply can't keep up with the crazy schedules, or fall behind a little bit and quickly get daunted by how the workload is snowballing, and ultimately abandon their titles.

There's No Money to be Made

I knew from the get-go that I wouldn't be paid a lot for my work, and fortunately I went into both books with no expectations of making money. But if you're reading this article and didn't know that there's no money in publishing, I feel it needs to be said very explicitly: do not do it for the money.

Most books will never recover their advance. Meaning, if you get paid a $2500 advance and have 16% royalties, you should only ever expect to make that $2500. Don't expect to make money on royalties. Unless your book becomes a best-seller (it won't), you won't see much (if any) money beyond your advance. So if you're considering whether or not to write a book, don't even factor in the royalties.

Given the amount of time it takes to write a book, even with all the corners you can cut, you should only expect to make along the lines of $10-12/hr if you're a fast and proficient writer that requires few edits. If you're not a "clean" writer and your publisher requires lots of edits from you, you can expect along the lines of $8/hr for your work.

If you're a technology expert and you want to go into writing full time, you'd find that you can't make a livable wage (in terms of tech expert wages) writing full time for publishers. Even if you could crank out one book a month (which might be possible if you write 60 hours per week), you'd only make roughly $2500 a month, or $30k per year. That's the same as full-time work at minimum wage in NYC.

Now, there's nothing wrong with making $30k per year -- but that's also the insane scenario of landing one book deal a month consistently for a year straight, and writing and editing 12 books in any given year. Furthermore, to write a technical book you are, presumably, a subject matter expert in some technology, and $30k is way below what the market values your skills at.

Long story short: do NOT do it for the money.

I do get some royalty payments from my first book. I think I get something like $30/mo in additional income from it.

Other Things I Didn't Expect

I've learned many other things about the process, but I don't feel I need to write a full section about each. Here are some interesting thoughts though:

Your editors may not understand the subject matter. If you're working on a very technical book, like a book about machine learning, you can't count on your editors to understand the technical subject matter. There are technical editors and language editors. The technical editors of your book will reproduce your code examples and confirm that they are replicable, but they will not vet the quality or sanity of your code or algorithms. Your language editors will simply check that your basic grammar, style, and formatting is correct. Do not count on the editors to make sure your work is correct or high quality. Do not count on the editors checking that the description of your code matches the behavior of your code.

The work doesn't end when you publish. Your publisher will likely ask you to assist with marketing efforts (if not for marketing, then what are they there for??). You will also have to address a steady stream of errata that comes in as readers find mistakes that your editors didn't catch. This adds a bunch of hours of work after publication, which is a little demoralizing.

You have absolutely no idea who your audience will be. Despite writing a full page on "who this book is for", you have no idea who will actually end up reading your book and how the market will react. It's therefore very difficult to target any one audience, so be prepared for lots of people writing in with questions that may or may not be related to your actual book. Also, be prepared to say "no" to lots of people who are looking for help completing their homework assignments or term projects.

Should You Write for a Publisher?

It really depends on your goals. I wrote these books with publishers because I wanted to see what the publishing industry was all about. There was also an aspect of "the bucket list" for me. I just wanted to.

There are, naturally, pros and cons to working with a publisher. Having a TOC written for you can give you some necessary structure if you're at a loss for how to plan your book. While the timeline requirements are pretty intense, it also forces you to get the project done in a timely manner. The advance isn't a lot of money, but it's guaranteed money. There's also the perceived presitge of being a published author and seeing your name on a bookshelf at B&N. It may help your career outside of writing in some ways, and that should not be ignored.

On the other hand, if you're a self-motivated, confident and competent writer and you want to make money doing it, self-publishing is the way to go. Start by building an audience around your blog and mailing list, and grow that audience until you reach a point where 1,000 people have indicated they'd buy a book from you. Then write it. Maybe only 250 will actually buy your book, but if you're selling at $20 (unlike the $60 a publisher would charge), you'll still make double what you'd make doing it with a publisher.

I have to say that I'm glad I wrote these books. Despite the time constraints and the effect on my writing's quality, I'm still proud of the work I did and think it's valuable. That said, I had a good idea of what I was getting into when I started; if I hadn't expected the low pay and tough timelines it might have been a demoralizing shock for me. If you're considering writing a book for a publisher make sure you understand your own personal goals and be sure to balance those against the pros and cons of working for a publisher.