Know your project teams: alpha / beta performance analysis

Moderately sophisticated security investors are familiar with the alpha and beta risk ratios. The beta coeficient is supposed to gauge the volatility of a specific security when compared (over the long run) to a benchmark (most often a market indicator such as the S&P 500). If the market as a whole has a beta coeficient of 1, fractional betas indicate securities that are less risky (but also return less over the long run) whereas a beta coefficient over 1 indicates a higher risk / reward ratio. The fundamental thing to understand is that beta is measuring the so called 'market' risk. A certain set of characteristics determines this volatility. Two identical competitors in the same market will have the same beta coefficient because their business volatility should be the same (they are subjected to events in the same way). Alpha on the other hand measures deviations that can be attributed to managers skills (or other aspects).

How does this help knowing your team? While the comparison isn't perfect, and measurements might be hard to accomplish, it might be useful to think about analyzing teams by trying to divide performance variability in its alpha and beta components. Why are some teams overperforming whereas others and underperforming? If one could establish a reasonable benchmark for performance, either across a large organization or across an industry, then the team's beta would be the ratio between cost to performance. Projects under a high flying startup (or a skunkworks type of inner organization) able to attract the brightest talent should perform better than the ones in an established organization. From that perspective, the high performing teams would have a higher than 1 beta, whereas the established ones would have a lower than 1 beta. If one is able to adequately determine the beta coefficient for one's environment, then the deviation from that is each individual's team alpha. A positive alpha over the long run will be correlated with well performing managers / well run projects. A negative alpha - with poor managers.

So how does this help? In assessing teams, instead of looking for absolutes - e.g. project was under / over budget, one can look at the teams' beta and alpha. The latter are much more actionable. Consistently negative project results might point to an organizational problem rather than to team performance. Teams might actually be overperforming and still come short of unrealistic expectations. Shrewd executives would sometimes use setting unrealistic goals as a motivation technique. Thus, when looking at team performance retroactively, it would be more accurate to look at their alpha / beta coefficients instead of how they measure against often arbitrarily set project metrics.

Project decision making with Agile

Some mythologies (like SCRUM Agile) favor a certain flatness and decentralization of the project structure. Project managers become scrum masters and their mission goes from controlling all aspects of the project to removing team impediments. This is an example where the tool affects or prescribes how the craftsman uses it. The problem is that the framework also assumes that other team members can step up to compensate; that the teams can be self organizing and that they can really see the stakeholders needs. Projects are often a series of choices made with imperfect information. When you decentralize or democratize decision making, you can either improve its outcome (more brains are better than one, direct product owner involvement brings the team closer to the client's real needs), but alternatively, it can make it worse. How can the latter happen? Most project participants are specialized in one way or another and by necessity will have less information about at least some areas. Developers for example are more inclined to vote on using capacity to clear out technical debt, often at the detriment of new essential, value-add functionality. A democratic, flatten decision making process will sometimes skew the project in favor of technical gold plating. In another circumstance, the product owner might not see the 'big picture', and spend too much time in low level features; this will have an opportunity cost, since at some point time or other resources (budget, organizational support) will run out. This can happen in every project, whether agile or not. The point is that SCRUM agile tends to magnify these issues in an imperfect, unbalanced team.

The question might arise - would you rather have an unbalanced team sharing decision making, and in the process creating inefficiencies, or would you rather have a flawed project manager? Alternatively, would you rather have a good project manager or a perfect self organizing team?  I think a lot of the push towards democratizing project decision making was a reaction against concentrating (most of the) project decision making in the hand of one individual.

My point of view is that teams make mistakes just as individuals do. Given specialization, it is almost a given that decentralized decision making will not have the same potential for optimization as centralized decision making. In other words, a good project manager will always trump a good Scrum team.

At the other end of the spectrum, an unbalanced scrum team will probably be just as bad as a bad project manager. A bad project manager might be a more convenient scape goat and it might make it easier to identify places for change / improvement. A bad team will lead to the same level of underperformance, but the solution will not be as straightforward. After all, the road to hell is paved with good intentions, and each project individual contributor would be equally guilty (or equally not guilty) of the project's woes.

But shouldn't teams, aided by the right agile ceremonies (such as post mortem sprint reviews), be able to identify problems and correct them? From my experience, no. The team view point / collective bias will permeate those ceremonies, obscuring issues that are outside that view point or that are taboo to discuss. Not often would sprint reviews point out the tendency of technical project participants to reserve too much time for dealing with technical debt. Most often than not, the issue that are brought up are short sighted in nature. Retrospectives are very good at optimizing things such as communication and collaboration, but not necessarily collective decision making. Improving the latter usually relies on project participants self censoring or, conversely, picking up more responsibility when needed (kanban style). If the teams didn't start with this capital, retrospectives and continuous improvement efforts is not going to help.

I can also venture another opinion. I think that a good project manager is easier to find than a good Scrum team. Teams are often assembled by constraints that are not correlated with optimization. More often than not teams are assembled from stakeholders who have availability, not necessarily the people who have complementary skill sets or who have worked well together in the past or who have the best understanding of the domain at stake. While the likelihood of this occurring is the same under classic vs. agile frameworks, the deficiencies of an unbalanced team are magnified in agile. On the flip side, a good project manager will help minimize those deficiencies. But wait, shouldn't the scrum master remove impediments and make sure that the team is running smoothly? Yes, except that removing impediments when the team structure is the impediment is close to impossible. A scrum master is not empowered to control that aspect of the project in the same way a conventional project manager is. Furthermore, a scrum master might not feel like it's her job to do so, not without upsetting team equilibrium, and in the process reverting to a staggered project hierarchy.

The fadeaway project management methodology

The best project management framework or methodology is the one that can fade away easily in the background. In other words, if one is doing the right things, it does not get in the way and it doesn't add new work. The latter part applies both to the practitioners as well as to the standard setters / bearers inside an organization. Enforcing a process should not be a thing. Educating practitioners on the process is the standard setter's mission; if it needs to be enforced, then something is going against the grain.

But are project participants doing the right thing if no one is looking? If there are no PMO enforcers to look at how you are applying the methodology? Is 'what makes sense' congruous with 'what the methodology prescribes'? In a nutshell, it does not matter. If project participants cannot rationalize what they are doing with the conceptual framework of the methodology, then the methodology (or its application) is too heavy handed and it get in the way.

I found that more often than not when people have to work at implementing a process or maintaining its repeatability across an organization, the spirit of the methodology often gets lost. The best run projects are those that don't have to think about methodology, but where the methodology becomes second nature, embedded into everything from individual contributor's updates to team communications and collaboration. It becomes second nature, it feels natural to do things in a certain way. It also becomes natural to speak in certain circumstances and it becomes part of everybody's function to act in a way that enacts the abstractions of the methodology.

The myth of managing projects by the numbers

In my years of managing projects in one fashion or another, I've employed multiple methodologies or frameworks (PMI's PMBok, Agile, CMMI, RUP). None of them claim to be overly prescriptive, instead the stated mission is to structure best practices in a way that make them repeatable. The desire is to reduce project risk and variability through providing guidance for both depth and breadth in dealing with potential 'gotcha's'. They are all taking lessons learned by multiple practitioners in multiple industries and distilling them in an essential (if sterile) form that is waiting to be rehydrated with project specifics.

The problem arises when practitioners put too much faith in the methodology. I call it the silver bullet syndrome. The more structured the methodology, the more entrenched the feeling of security given by it. If I planned for all the practice areas, I'm guaranteed success. Of, metaphorically speaking, if I paint the mandated colors in the mandated areas, I will come up with a masterpiece.

In fact, good management is a more of an art rather than science. If requires continually honing your craft through intelligently listening to feedback. Feedback can come in various shapes and forms: it can be learning from mistakes. Some monumental failures offer the best opportunity for valuable lessons. It could be from really understanding variables that are not obvious immediately - such as corporate culture or the 'real' balance of power in an organization. It would be learning that influencing (and knowing how to do it) is a lot more valuable than dictum authority. It could be taking a humbling experience and turning it into actionable insights for the future. Furthermore, the craft is enhanced through incessantly tinkering with the levels. Project managers have finite availability; figuring out where to place your attention at any given time is methodology or framework independent.

So what does this mean for structured methodologies? It means that a tools is only as good as the skills of its operator. Methodologies are like tools. They are useful for certain tasks, but they are not irreplaceable. You might find it useful to have a paint brush and some outlines to paint a landscape, but the same can be accomplished painting with a painter's knife (or a different brush). More importantly, given the same set of tools, there will be a wide range of individual, non repeatable craft that practitioners will bring to the table. This craft, this individual 'alpha' of the project participants (as well as how the skills of various participants mesh together) is just as important if not more important than the methodology. The craft of the project leader becomes even more essential. As the person whose decisions and steering affect (positively or negatively) the widest range of project aspects. Hiding behind the false sense of security that the methodology offers is even more pernicious if done by a project leader.

Methodologies also provide a framework for collaboration, a common language of sorts. Having all the project practitioners understand the same ceremonies removes friction. If everybody understands interaction etiquette in the same way, it removes a project impediment and allows the team to concentrate on value added tasks instead. But again, this is not the silver bullet. Collaboration frameworks reduce friction, but they won't compensate for poor understanding of the client's needs or for incapacity of seeing the project risks (or where to look for them). But they can give everybody that false sense of security. A tool can only get you so far; the rest is superior craft that can only come from experience, from every project participant, but especially from the project leader.

CMMI compliance - pay lip service better than not doing it at all?

Here's a scenario that I see way to often around the beltway: an organization is mandated to attain a certain CMMI maturity level - usually level 3, as part of a contractual requirement - usually involving goverment work. Because it is a contractual requirement, that organization usually makes it happen. They go through the pain of implementing a SEPG (software engineering process group), they install CMMI czars and institute the software process assessments and all that. On the project side, PMs are required to allocate resources to tailor their processes from the organizational standards and to maintain the paper trails proving compliance. Way too often this degenerates into a state where nobody really believes in the real business value of all this work, and everybody is paying lip service to it because it is 'required'.
Even in this scenario, I think that the organization and project themselves are better off than in a case where no attention is paid to standardizing and improving system engineering processes. CMMI compliance, even if done as a 'requirement', forces an organization and the project leaders to take a step back and evaluate, which is the very spirit of improving. On the project leader side, one might not like all the overhead, but by the very nature of being forced to minimize it, one becomes educated and efficient. I won't comment on the absolute business value of CMMI compliance - it is besides the point in this scenario since it is a contractual requirement. I do feel that projects can get something out of it, even if they don't like the overhead.

Defects vs. enhancements

Is it a defect or an enhancement? The answer to this question makes a big difference to the software team. The developers are very much involved with the product that they are building and often times they take offense if a requirement defect or an enhancement (new functionality request) are mislabeled as defects, i.e., a problem with design or implementation.
The problem is, none of this matters to the business stakeholder. A good Agile team shouldn't even care about the difference between a bug, requirements defect or enhancement, as long as it is work that needs to be done. Clinging to categories is often a symptom of team division and dissension.

The business value of fixing defects

Less defects is always a good thing. Right? Not so fast. Fixing a defect has both benefits (more obvious) and costs (often easy to ignore). In this posting, I'll concentrate on some of the costs and why they can sometimes outweigh the benefits. This analysis is performed from the perspective that everything we do in a software project has to come in line with adding value to our stakeholders.
First, let's discuss the opportunity costs. Allocating resources to fixing a defect takes away from doing other things - such as fixing more important defects or implementing new features. The way to prevent misallocation of precious resources in an agile setting is to treat defects as any other quantity of work - create stories for them and, based on a prioritization exercise that takes into account feature stories, work them into the backlog. In a non agile setting, misallocation of resources due to high opportunity cost is trickier. Since prioritization is usually fractured between requirements and defects, defects usually get prioritized among themselves and get fixed within the allotted time. This usually leads to inefficiencies - since very seldomly the predetermined time for 'fixing bugs' is just right. Many times, teams will over fix some parts of the code and under fix others. There is also a temporal horizon bug fixing problem (I'll deal with that more in depth in a separate entry) - in brief, more recently discovered bugs tend to be fresher in people's minds, and thus, receive more attenting than older bugs.
Second, fixing bugs is risky. Touching a piece of code runs the risk of introducing more bugs, in other words, doing more harm than good. This risk should be managed by the project manager just as she would manage any other project risks. This is particularly heinous since time and attention for retesting are often in short supply.
To conclude, I'm not advocating against fixing defects. The point I'm making is that fixing defects should be treated as any other quantity of resource consuming endeavor, and as such, it should be balanced against other resource demands from the perspective of getting the most bang for the buck for the project stakeholders.