Accelerate - Building and Scaling High Performing Technology Organizations
Continuous delivery Architecture Product and process Lean management and monitoring Cultural
accomplishing a task in a single line of code that no one else can understand is less desirable than writing a few lines of code that are easily understood and maintained.
A successful measure of performance should have two key characteristics. First, it should focus on a global outcome to ensure teams aren’t pitted against each other. The classic example is rewarding developers for throughput and operations for stability: this is a key contributor to the “wall of confusion” in which development throws poor quality code over the wall to operations, and operations puts in place painful change management processes as a way to inhibit change.
Second, our measure should focus on outcomes not output: it shouldn’t reward people for putting in large amounts of busywork that doesn’t actually help achieve organizational goals.
We measured product delivery lead time as the time it takes to go from code committed to code successfully running in production
Analysis over several years shows that high-performing organizations were consistently twice as likely to exceed these goals as low performers. This demonstrates that your organization’s software delivery capability can in fact provide a competitive advantage to your business.
Continuous delivery is a set of capabilities that enable us to get changes of all kinds—features, configuration changes, bug fixes, experiments—into production or into the hands of users safely, quickly, and sustainably.
In continuous delivery, we invest in building a culture supported by tools and people where we can detect any issues quickly, so that they can be fixed straight away when they are cheap to detect and resolve.
By splitting work up into much smaller chunks that deliver measurable business outcomes quickly for a small part of our target market, we get essential feedback on the work we are doing so that we can course correct.
Computers perform repetitive tasks; people solve problems.
By giving developers the tools to detect problems when they occur, the time and resources to invest in their development, and the authority to fix problems straight away, we create an environment where developers accept responsibility for global outcomes such as quality and stability.
Teams that can choose their own tools based on what is best for the users of those tools
Unplanned work and rework are useful proxies for quality because they represent a failure to build quality into our products.
Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods.
To enable this, we must also ensure delivery teams are cross-functional, with all the skills necessary to design, develop, test,
To measure productivity, we calculated the following metric from our data: number of deploys per day per developer.
Our research shows that building security into software development not only improves delivery performance but also improves security quality. Organizations with high delivery performance spend significantly less time remediating security issues.
Tags: #security
We found that when teams “shift left” on information security— that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process—this positively impacts their ability to practice continuous delivery. This, in turn, positively impacts delivery performance.
Tags: #product #programming
Limiting work in progress (WIP), and using these limits to drive process improvement and increase throughput
Using data from application performance and infrastructure monitoring tools to make business decisions on a daily basis
We found that approval only for high-risk changes was not correlated with software delivery performance. Teams that reported no approval process or used peer review achieved higher software delivery performance. Finally, teams that required approval by an external body achieved lower performance.
In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate.
Our recommendation based on these results is to use a lightweight change approval process based on peer review, such as pair programming or intrateam code review, combined with a deployment pipeline to detect and reject bad changes.
- The extent to which teams slice up products and features into small batches that can be completed in less than a week and released frequently, including the use of MVPs (minimum viable products).
- Whether teams have a good understanding of the flow of work from the business all the way through to customers, and whether they have visibility into this flow, including the status of products and features.
- Whether organizations actively and regularly seek customer feedback and incorporate this feedback into the design of their products.
- Whether development teams have the authority to create and change specifications as part of the development process without requiring approval.
Our analysis showed that the ability of teams to try out new ideas and create and update specifications during the development process, without requiring the approval of people outside the team, is an important factor in predicting organizational performance as measured in terms of profitability, productivity, and market share.
The fear and anxiety that engineers and technical staff feel when they push code into production can tell us a lot about a team’s software delivery performance. We call this deployment pain, and it is important to measure because it highlights the friction and disconnect that exist between the activities used to develop and test software and the work done to maintain and keep software operational.
Before implementing the technical practices and discipline of continuous delivery on the Bing team, engineers reported work/life balance satisfaction scores of just 38%. After implementing these technical practices, the scores jumped to 75%
How Painful Are Your Deployments? If you want to know how your team is doing, just ask your team how painful deployments are and what specific things are causing that pain.
In order to reduce deployment pain, we should: Build systems that are designed to be deployed easily into multiple environments, can detect and tolerate failures in their environments, and can have various components of the system updated independently
Ensure that the state of production systems can be reproduced (with the exception of production data) in an automated fashion from information in version control Build intelligence into the application and the platform so that the deployment process can be as simple as possible
- Work overload: job demands exceed human limits. 2. Lack of control: inability to influence decisions that affect your job. 3. Insufficient rewards: insufficient financial, institutional, or social rewards. 4. Breakdown of community: unsupportive workplace environment. 5. Absence of fairness: lack of fairness in decision-making processes. 6. Value conflicts: mismatch in organizational values and the individual’s values.
Strong feelings of burnout are found in organizations with a pathological, power-oriented culture. Managers are ultimately responsible for fostering a supportive and respectful work environment, and they can do so by creating a blame-free environment, striving to learn from failures, and communicating a shared sense of purpose.
Managers and leaders should ask their teams how painful their deployments are and fix the things that hurt the most.
Many large technology companies run disaster recovery testing exercises, or “game days,” in which outages are simulated or actually created according to a pre-prepared plan, and teams must work together to maintain or restore service levels.
SURVEYS ALLOW YOU TO COLLECT AND ANALYZE DATA QUICKLY
Trunk-based development has been shown to be a predictor of high performance in software development and delivery. It is characterized by fewer than three active branches in a code repository; branches and forks having very short lifetimes (e.g., less than a day) before being merged into master; and application teams rarely or never having “code lock” periods when no one can check in code or do pull requests due to merging conflicts, code freezes, or stabilization phases.
Integrating security into the design and testing phases of the software development process is key to driving IT performance.
Tags: #security
Implement continuous delivery (CD). CD is a development practice where software is in a deployable state throughout its lifecycle, and the team prioritizes keeping the software in a deployable state over working on new features.
Our research has found that whether organizations actively and regularly seek customer feedback and incorporate this feedback into the design of their products is important to software delivery performance.
Teams should have a good understanding of and visibility into the flow of work from the business all the way through to customers, including the status of products and features.
Team experimentation is the ability of developers to try out new ideas and create and update specifications during the development process, without requiring approval from outside of the team, which allows them to innovate quickly and create value.
Our research shows that a lightweight change approval process based on peer review (pair programming or intrateam code review) produces superior IT performance than using external change approval boards
Use data from application and infrastructure monitoring tools to take action and make business decisions.
The use of work-in-process limits to manage the flow of work is well known in the Lean community. When used effectively, this drives process improvement, increases throughput, and makes constraints visible in the system.