Developing Shared Code with Principles

One of the most high-leverage work in a technical organization is building shared libraries or frameworks. A common library, a piece of code that can be used as is, or a framework, a system that codifies certain decisions and allows further work to be built on top, has the opportunity to benefit many people at once. Not only that, they also institutionalize shared knowledge, put knowledge that’s in people’s head in code for future employees. And of course, there are other benefits such as possibly open-sourcing such work, which comes with its set of benefits to hiring and on boarding.

Of course, there are risks with such a venture. The biggest risk is of course the value risk; that the work goes unused by other teams. That is bad enough, but sometimes it gets worse. Sometimes the adoption comes, but the work instead of enabling teams, hinders them. It gets in the way of actual work, the benefits of standardization is overshadowed by the pains of integration and customization.

So how do we make sure our shared work, be them libraries or frameworks, achieves its goal? In my experience, there are 3 principles that separate successful shared projects from failed ones. Two of them are about how to build the project, and a third one about how to get it in hands of others.

Start with a Real Project

Start with the origin. The ideal way to come up with a shared tool is to extract it from an actual project. This seems straightforward enough. The biggest benefit of this approach is the framework has an immediate customer, so its creators are incentivized to solve actual problems, for actual people. In practice, the most important value comes from the “developer ergonomics”, or essentially the usability of the project is automatically improved. This is basically the cornerstone of the agile movement anyway.

Take Rails, which has been extracted out of Basecamp, a project management software. Or Django, again which was extracted from a content-management system built for a local news site. These two frameworks have different cultures, but they attack the same problem; how do you make sure your developers are more productive?

I realize this way of working is not always easy, or even possible For example, it’s easier to imagine a front-end library like Bootstrap being extracted from Twitter’s internal UI, than, say a library used for internal communications platform. But it is possible.

Keep It Small

The second way to ensure success in a shared product is to keep the surface area as small as possible. In other words, the shared project should try to provide value as soon as possible. In practice, what this means is that the framework should probably undershoot its feature set.

This might sound counter-intuitive. If we are aiming for high adoption, should we not try to cover all the possible use-cases? Shouldn’t we try to solve as many problems as possible, to make sure people find some use in our work?

The problems with trying to solve too many problems at once are plenty. First of all, as modern software development methodologies have discovered, the problem discovery is really a continuous process. Instead of trying to predict what the problems would be and trying to solve them, we should try to deliver value as soon as possible, and then iterate further.

The more subtle problem with a large feature-set is that in my experience, especially more tenured teams see it as more of a liability than leverage. They realize a big investment, especially in a new project will likely result in more work down the line.

A small word of caution here. Especially in the front-end world, this advice to keep the feature set small is taken to an unfortunate extreme. No project should need a library to add left padding to a bunch of strings. The line between a small project that does one thing and does it really well and a comically tiny project is a fine one. A good rule of thumb is to make sure the project should provide some immediate value, and be meaningful by itself. That is, one should be able to do something “production” level with your project and only that.

Take a look at first version of Rails, which is essentially a bunch of ActiveRecord classes that uses Ruby’s dynamism to build an Object Relational Mapper (ORM). All the other features that most Rails developers take for granted came many years later. Similarly, React (in addition to being extracted from an internal Facebook project), barely had many of the features it has now; support for many of the common HTML tags, ES6 classes came later.

Evangelize Constantly

And lastly, but maybe most importantly, teams should actively evangelize their projects to be adopted within a company. This might be initially uncomfortable. Many technical teams have a negative opinion of any sort of marketing. They believe that other teams should be able to evaluate their work on its merits, and everything else is either throwaway or disingenuous.

This is a short-sighted way of looking at it. Developers of shared tools should consider evangelism as not only adverting but also forming a two-way communication channel with their customers.

Take marketing. Many of your potential customers inside your company might have heard of your project, but unless they know the project is well supported and actively maintained, they will probably not consider using it. When you market your project via emails, presentations and such, you are not only letting people know of your project, but sending an active signal that it’s active, maintained project. Many times, just having a face attached to the project that is known inside the company is the difference between real adoption and leverage as opposed to a repository full of insights bit-rotting away.

Moreover, evangelism is also about forming that feedback loop with your customers. When you actively work with your prospective customers, you are getting immediate feedback about the pain points they are having. You see what some of the under-invested parts of your project might have low-hanging fruits for future wins.

Again, a bit of warning here is in order. Evangelism doesn’t come naturally to most developer teams. Moreover, with evangelism sometimes comes sacrifices. It is not unheard of to make one or two small one-off work for a big internal customer, to get some initial adoption. This might feel impure, but sometimes might be necessary. The key here is to keep the scope of the one-off demands as small as possible, and really do it for customers who would be game-changers.

Building shared libraries or frameworks is extremely fulfilling; seeing your code be adopted by others, making their lives easier is why many people get into software development in the first place. And it is an amazing way to create high leverage work in a technical organization. Ability to positively impact the work of tens, hundreds, or even thousands of fellow developers is something most executives would be excited about.

I believe with these guidelines in mind, you would be in a much better place to both deliver value for your company, and have some fun doing it.

Planning for Agile

One of the main tenets of agile methodology is working software trumps extensive documentation. You get something to work, and then iterate based on the quick feedback. It sounds great in theory, and in my experience, works reasonably well in practice. All software estimates are wrong, so agile is also wrong, but it produces software and does it without inflicting too much damage on those who build it.

But how do you square this way of working with a long term vision? If an organization is aligned towards a vision, there has to be a roadmap that people follow. And a roadmap, by definition, is a long term plan. It guides what needs to be done months, and sometimes years in to the future.

These two ideas seem contradictory, and they can be confusing for especially inexperienced software engineers to wrap their head around, like it was for me years ago. But after spending several years in companies big and small, I found a way to reconcile the seemingly contradictory ways of thinking. For me, two ideas bridge this gap: a) Planning is for planning b) “Agile is a state of mind”. Let me explain.

In his seminal book High Output Management, the famed late Intel CEO talks about how Intel creates a five-year roadmap, every year. This seems insane on the surface; every year a bunch of high powered executives come in and spend many hours creating a five-year plan, only to do it next year, seemingly wasting 4 years of planning! Couldn’t they just do a one year plan?

Grove points out the output of the planning process is not really the plan itself, but the mental transformation of the people involved in the process and the organizational effects. The physical, literal exercise of having people sit around a table, and discuss the future establishes a shared vocabulary, and provides a starting point and a framework for all the future ad-hoc decisions that will need to be made.

In other words, once the planning process is finished, it is the people that is reformed, and the plan, on paper, itself is just a small residue in the crucible. That transformation is provides two main things; first is a sane default for all the future decisions and the second is a lingering sense of what needs to be done to keep the momentum. In my experience, the sane default aspect is more important. The key here is not that plan is a fallback for next decisions be followed blindly but it’s a shared framework, a common place to start the conversation to initiate a discussion. This is what Eisenhower meant when he said that “No battle was ever won according to plan, but no battle was ever won without one”. It’s not what happens that matters, it’s that something happens.

This leads me to my next point, namely “the agile mindset (man)”. In most software projects, especially those in consumer field, what matters is the cadence of development. And the hardest part of that momentum is always overcoming the inertia of doing nothing. Ironically, most of the time, this inactivity manifests itself as planning. We need plans, for ourselves, surely, but we also need the antidote.

Let me give an example. One of the projects I worked on involved building a new transport layer that extend all the way from the mobile client almost to the storage layer on the backend of a major enterprise. Almost literally, there was a moving part on every single part of the stack, each owned by different teams on different schedules, sometimes different timezones, different set of hopes and dreams.

The number questions with a project like that is essentially infinite. Some things security and privacy are non-negotiable, but the tail end of requirements have no end in sight. What is the monitoring story? What about error handling? How do we handle rollbacks, exceptions? Typing? Code generation? Compression, and performance? Where do you even start?

This is the point where agile mindset comes in handy. The idea behind agile is not that documentation is not useful (it is definitely useful, which I’ve learned the hard way) but it comes after the working software. The trick lies in being able to identify what really matters and what that initial state of working software looks like. In my experience, it’s always better to err on the side of simpler. Anything more than just a bit, sometimes literally, needs justification that’s simply not worth it.

So here’s what we did: we defined the security and privacy guarantees we need to provide, and only those. Nothing else. And then started building out something where the client can talk to the server, and server can respond back. It was extremely uneventful, when I tapped a button on my phone, and the random string I typed on my laptop appeared back on it. But it worked, and rest of it just followed. We found a way to handle the error handling, and handled the performance bottlenecks as they came along, and some brave souls handled code generation and today, it all works.

This is not to say the process was simple. The art of saying “not today” when people come knocking with their pet feature ideas, either from up or down the management chart, and sounding similarly credible when saying “but tomorrow” is a delicate skill. It requires credibility, resolve, and yes, sometimes a thick skin.

The selling point of “Agile” to the management has been that it provides value instantly and is more amenable to a dynamic, fast changing marketplace. And those are all true, but such verbiage can throw off those working in the trenches as MBA-speak.

For me, the main guiding principle of “agile”, with an intentional lowercase-A, has been the idea of taking into account how humans who build the software work. This isn’t surprising, considering “Agile Manifesto” was penned by actual software developers in the field. The open embrace of the messiness of doing anything that involves flesh-and-bones people is what makes the process more bearable than other forms of building software.

There are known knowns, there are known unknowns, and yes, there are unknown unknowns. There are temper tantrums, there are executive demands that come from nowhere on the 11th hour, there are teams that forgot they are involved, and there are those that casually ignore everything until the last moment.

The best we can do is aligning everyone on the same goals as best as we can, make sure people feel involved in the decisions that affect them, form the personal and organizational connections they will surely need, and have some sense of what success looks like. Rest will follow.