“You can’t manage what you don’t measure.”
In today’s world of technology, every company that runs their business on software is a technology company. During my career as a developer, I’ve seen many bad practices in the software delivery process that were dismissed with the statement, “Well, we’re not a software company, were an X company”, where X would be whatever sector of business the company was doing business in. Deployments for these companies were always slow and painful manual processes and as such we ended up performing infrequent releases because, well, they were slow and painful. In today’s market, gaining a competitive edge over the rest of the market means being able to rapidly respond to the market and delivering value to the business through software.
We know that software development and delivery is an increasingly difficult process as the complexity of systems increases, and that managing and improving any process or system requires insights into that system. Measurements and data are key to creating an effective software value stream.
When measuring devOps values, there are two approaches:
- Survey data
- System data
Neither system data alone nor survey data alone can measure the effectiveness of a modern software delivery pipeline. It is a complementary approach where the survey data provides leading indicators to system data, or provides insights that system data might not be able to capture yet (or at all). Because of this, it is critical for organizations to understand what they can and cannot measure with each approach, and what steps they need to take to gain visibility into their software delivery value streams.
There are several reasons why both system and survey data should be used to measure the value streams that define your software-delivery processes.
Survey Data
Survey-based metrics generally refer to data about systems and people (culture) that comes from responses. When collected correctly, survey data can provide accurate insights into systems, processes, and culture. Surveys are also particularly good at capturing holistic overviews of systems, because the answers that respondents provide synthesize data related to automation, processes, and culture. They are great for getting a top-down view of your overall software delivery process.
System Data
System-based metrics generally refer to data that comes from the various systems of record that make up an end-to-end software delivery value stream. For many development groups, they look for that data where it’s captured from a particular system of record, such as an agile tool like Team Foundation Server for version control, or Jira for bug tracking. However, with many of these agile tools, while they are good at providing some very specific bottom-up metrics, it’s not enough to provide the kind of visibility and reports that can be tied back to the goals of the company. For example, to measure time to market for a customer request, you may need data from a customer tracking system, the requirements system, the agile tool, and the deployment tool chain. You would then need to be able to correlate the value of implementing that feature for the customer relative to the expense of developing it to understand if it was worth doing.
Data from systems can provide very granular data, allowing you to report on subsystems and components. As system data matures in a company, eventually system-level data can to provide a relatively full view of your system, but this requires full instrumentation, plus correlation across measures and maturity in reporting and visualization techniques so that teams can understand system state. It’s generally fairly easy to get a specific data-point out of a sub-system, but harder to correlate that data point to the overall health of the process.
Using Both
There are still some measures that are important to software delivery, such as cultural measures, that survey-based measures will pick up and system-based metrics may miss. In addition, having both types of metrics provides opportunities for cross referencing your data: if your survey measures provide data that is drastically different from the data coming from your systems, this can highlight gaps in the system.
Survey data provides an alternate view of your system, allowing you to identify problems or errors when there are two opposing views. Do not automatically discount your survey measures when this happens: there can often be cases where changes in configurations or system behavior alter the way that system data is collected, while survey measures remain true and it is only the delta in these two measures that calls attention to changes in the underlying system.
Get Baselines
The important thing is to get started with a baseline collection immediately, and for many companies that will mean collecting survey data while efforts to capture and correlate system data are in flight. In the absence of complete system measurements, comprehensive surveys can provide a holistic view of your system relatively quickly (i.e., within several weeks).
Getting end-to-end system data can be a long journey as you first need to deploy a measurement solution across systems, and then make sure that cross-system integration is in place so that the data can be properly correlated.
While it is important to start as early as possible to get the benefits of system data, deploying survey data provides an almost immediate value and source of baseline information. This is valuable both for base-lining current and future survey data, and for comparing survey with system data once they are in place and your instrumentation process matures.
Maturity of Data Collection
Over time, many of the metrics that you first collected through the survey that represented synthesized data for your process can be replaced by the system instrumentation and collection points within your tool chain and automated processes.
Summary
Data collection is a balancing act between survey data and system data. Survey data is a great way to get started and provides leading indicators to system data.
As system data collection and telemetry matures in a company, it can provide a relatively full view of your system, but this requires full instrumentation, plus correlations across different system measures so that teams can understand each system’s holistic state.
The important thing is to get started with a base-lining and start collecting the data now and for many companies that will mean collecting survey data while efforts to capture and correlate system data are in flight.
