Software development is tough, but software delivery is tougher. Development is just the tip of the iceberg; delivery involves coordination between people across testing, QA, UI/UX, product management, project management, database administration, architecture, business analysis, infrastructure, and of course — the end users of the software you’re building. Figuring out how to orchestrate software delivery across these roles in a manner that maximizes your team’s ROI is what DevOps is all about. Maximizing ROI means reducing costs while increasing value. This can be achieved through:
- Simplifying the processes that bring your products to your users. Sometimes this means simplifying your products.
- Automating as much as possible. Human intervention is risk; risk is expensive.
- Learning through experimentation and fostering a culture that embraces failure as a cost of progress.
To me, DevOps is more of a philosophy than a prescriptive application of tools and processes. Regardless of your ecosystem or domain, devops boils down to continuous risk reduction in all areas of your delivery pipeline.
I’d say that a mature DevOps pipeline can be characterized by the following:
- Your developers can make changes and push them to production with confidence and “on demand” — within a few hours of making them — thanks to automated builds, tests, and deployments.
- It’s easy to roll changes out of production if needed, ideally through some form of feature flagging.
- The time your team spends on production support is minimal.
- Your builds, tests, deployments, monitoring, and infrastructure are all automated.
- Your QA team is focused primarily on exploratory testing as opposed to repetitive testing. They are finding opportunities for automated tests more so than they are hunting for regressions. Unit tests should protect you against regressions.
- Your team does not feel like a fire department — “fires” are infrequent or nonexistent.
- Your team has the luxury of baking unit tests into estimates at sprint planning.
- Production sees frequent, small changes instead of massive, infrequent ones.
- Feedback from users is captured early and often based on these frequent changes yielding shorter feedback loops.
- Your technical debt is measured and continuously improved.
- Your services are outfitted with centralized logging and exception handling.
- As you move from one environment to another along your deployment pipeline, the only things that change are configuration, tests, and scale. If you’re taking advantage of staging slots in production, you should take advantage of the staging slots in every other environment. The procedures should not change between environments. The goal is to eliminate production surprises by never doing anything for the first time in production.
- Deployments to environments are idempotent and reversible.
DevOps isn’t just about building a pipeline though. Building a culture to support that pipeline — and one that sticks — is just as important. I’d say a mature DevOps culture can be characterized by the following (big thanks to Armen Rostamian for encouraging more emphasis on the humanistic aspects of DevOps):
- Your entire team is aware of exactly what it takes for a code change to find its way from a developer’s box to production.
- Your team is constantly learning through experimentation across technical and nontechnical areas.
- High-bandwidth, low-politic feedback loops are available to facilitate rapid iteration, collaboration and knowledge-sharing between the various groups within your team.
- Criticism and doubt are embraced. Persuading and transitioning are huge hurdles, but ensuring that this value persists means that doubts must be surfaced and addressed before they manifest into regressions. Fostering a culture of oppenness and transparency means that the value you’re offering can stand up to (or evolve in response to) critical inquiry — as it should.
- Blame culture is eradicated. You’re delivering as a team — and that team by nature is a hedge against the risks that come with being human. Post-mortems should be blameless and should yield corrections and enhancements, not fear. Individual criticisms can be offered privately while praise is offered publicly to incentivize an open, collaborative dynamic instead of one characterized by competition, the withholding of information, and a fear of experimentation.
- Generalism is encouraged as a defense against regression. We need to ensure continuous improvements and defend against regressions. Part of this defense involves controlling the perception of the changes you’re championing. Effecting perception requires an active effort at all levels of a team to involve participants so that they are tuned into the evolution you’re facilitating. This means that the specialists, influencers, and leaders within your team should become aware of the bigger picture and how it’s improving over time — ideally through data that demonstrates cost reduction. This collective perception will hedge against the risk of someone introducing regressions when you’re no longer there. To encourage this, I like to frame involvement as “skillset enhancement” that improves each individual’s overall value to both their team and the industry at large.
The goal of DevOps is to have a highly aware team delivering software through a highly automated mechanism that carries your code to production with confidence. DevOps is all about making that happen in a way that works well within the constraints of your existing product, team, and technical ecosystem.