Know your project teams: alpha / beta performance analysis

Moderately sophisticated security investors are familiar with the alpha and beta risk ratios. The beta coeficient is supposed to gauge the volatility of a specific security when compared (over the long run) to a benchmark (most often a market indicator such as the S&P 500). If the market as a whole has a beta coeficient of 1, fractional betas indicate securities that are less risky (but also return less over the long run) whereas a beta coefficient over 1 indicates a higher risk / reward ratio. The fundamental thing to understand is that beta is measuring the so called 'market' risk. A certain set of characteristics determines this volatility. Two identical competitors in the same market will have the same beta coefficient because their business volatility should be the same (they are subjected to events in the same way). Alpha on the other hand measures deviations that can be attributed to managers skills (or other aspects).

How does this help knowing your team? While the comparison isn't perfect, and measurements might be hard to accomplish, it might be useful to think about analyzing teams by trying to divide performance variability in its alpha and beta components. Why are some teams overperforming whereas others and underperforming? If one could establish a reasonable benchmark for performance, either across a large organization or across an industry, then the team's beta would be the ratio between cost to performance. Projects under a high flying startup (or a skunkworks type of inner organization) able to attract the brightest talent should perform better than the ones in an established organization. From that perspective, the high performing teams would have a higher than 1 beta, whereas the established ones would have a lower than 1 beta. If one is able to adequately determine the beta coefficient for one's environment, then the deviation from that is each individual's team alpha. A positive alpha over the long run will be correlated with well performing managers / well run projects. A negative alpha - with poor managers.

So how does this help? In assessing teams, instead of looking for absolutes - e.g. project was under / over budget, one can look at the teams' beta and alpha. The latter are much more actionable. Consistently negative project results might point to an organizational problem rather than to team performance. Teams might actually be overperforming and still come short of unrealistic expectations. Shrewd executives would sometimes use setting unrealistic goals as a motivation technique. Thus, when looking at team performance retroactively, it would be more accurate to look at their alpha / beta coefficients instead of how they measure against often arbitrarily set project metrics.

Project decision making with Agile

Some mythologies (like SCRUM Agile) favor a certain flatness and decentralization of the project structure. Project managers become scrum masters and their mission goes from controlling all aspects of the project to removing team impediments. This is an example where the tool affects or prescribes how the craftsman uses it. The problem is that the framework also assumes that other team members can step up to compensate; that the teams can be self organizing and that they can really see the stakeholders needs. Projects are often a series of choices made with imperfect information. When you decentralize or democratize decision making, you can either improve its outcome (more brains are better than one, direct product owner involvement brings the team closer to the client's real needs), but alternatively, it can make it worse. How can the latter happen? Most project participants are specialized in one way or another and by necessity will have less information about at least some areas. Developers for example are more inclined to vote on using capacity to clear out technical debt, often at the detriment of new essential, value-add functionality. A democratic, flatten decision making process will sometimes skew the project in favor of technical gold plating. In another circumstance, the product owner might not see the 'big picture', and spend too much time in low level features; this will have an opportunity cost, since at some point time or other resources (budget, organizational support) will run out. This can happen in every project, whether agile or not. The point is that SCRUM agile tends to magnify these issues in an imperfect, unbalanced team.

The question might arise - would you rather have an unbalanced team sharing decision making, and in the process creating inefficiencies, or would you rather have a flawed project manager? Alternatively, would you rather have a good project manager or a perfect self organizing team?  I think a lot of the push towards democratizing project decision making was a reaction against concentrating (most of the) project decision making in the hand of one individual.

My point of view is that teams make mistakes just as individuals do. Given specialization, it is almost a given that decentralized decision making will not have the same potential for optimization as centralized decision making. In other words, a good project manager will always trump a good Scrum team.

At the other end of the spectrum, an unbalanced scrum team will probably be just as bad as a bad project manager. A bad project manager might be a more convenient scape goat and it might make it easier to identify places for change / improvement. A bad team will lead to the same level of underperformance, but the solution will not be as straightforward. After all, the road to hell is paved with good intentions, and each project individual contributor would be equally guilty (or equally not guilty) of the project's woes.

But shouldn't teams, aided by the right agile ceremonies (such as post mortem sprint reviews), be able to identify problems and correct them? From my experience, no. The team view point / collective bias will permeate those ceremonies, obscuring issues that are outside that view point or that are taboo to discuss. Not often would sprint reviews point out the tendency of technical project participants to reserve too much time for dealing with technical debt. Most often than not, the issue that are brought up are short sighted in nature. Retrospectives are very good at optimizing things such as communication and collaboration, but not necessarily collective decision making. Improving the latter usually relies on project participants self censoring or, conversely, picking up more responsibility when needed (kanban style). If the teams didn't start with this capital, retrospectives and continuous improvement efforts is not going to help.

I can also venture another opinion. I think that a good project manager is easier to find than a good Scrum team. Teams are often assembled by constraints that are not correlated with optimization. More often than not teams are assembled from stakeholders who have availability, not necessarily the people who have complementary skill sets or who have worked well together in the past or who have the best understanding of the domain at stake. While the likelihood of this occurring is the same under classic vs. agile frameworks, the deficiencies of an unbalanced team are magnified in agile. On the flip side, a good project manager will help minimize those deficiencies. But wait, shouldn't the scrum master remove impediments and make sure that the team is running smoothly? Yes, except that removing impediments when the team structure is the impediment is close to impossible. A scrum master is not empowered to control that aspect of the project in the same way a conventional project manager is. Furthermore, a scrum master might not feel like it's her job to do so, not without upsetting team equilibrium, and in the process reverting to a staggered project hierarchy.

The fadeaway project management methodology

The best project management framework or methodology is the one that can fade away easily in the background. In other words, if one is doing the right things, it does not get in the way and it doesn't add new work. The latter part applies both to the practitioners as well as to the standard setters / bearers inside an organization. Enforcing a process should not be a thing. Educating practitioners on the process is the standard setter's mission; if it needs to be enforced, then something is going against the grain.

But are project participants doing the right thing if no one is looking? If there are no PMO enforcers to look at how you are applying the methodology? Is 'what makes sense' congruous with 'what the methodology prescribes'? In a nutshell, it does not matter. If project participants cannot rationalize what they are doing with the conceptual framework of the methodology, then the methodology (or its application) is too heavy handed and it get in the way.

I found that more often than not when people have to work at implementing a process or maintaining its repeatability across an organization, the spirit of the methodology often gets lost. The best run projects are those that don't have to think about methodology, but where the methodology becomes second nature, embedded into everything from individual contributor's updates to team communications and collaboration. It becomes second nature, it feels natural to do things in a certain way. It also becomes natural to speak in certain circumstances and it becomes part of everybody's function to act in a way that enacts the abstractions of the methodology.

The myth of managing projects by the numbers

In my years of managing projects in one fashion or another, I've employed multiple methodologies or frameworks (PMI's PMBok, Agile, CMMI, RUP). None of them claim to be overly prescriptive, instead the stated mission is to structure best practices in a way that make them repeatable. The desire is to reduce project risk and variability through providing guidance for both depth and breadth in dealing with potential 'gotcha's'. They are all taking lessons learned by multiple practitioners in multiple industries and distilling them in an essential (if sterile) form that is waiting to be rehydrated with project specifics.

The problem arises when practitioners put too much faith in the methodology. I call it the silver bullet syndrome. The more structured the methodology, the more entrenched the feeling of security given by it. If I planned for all the practice areas, I'm guaranteed success. Of, metaphorically speaking, if I paint the mandated colors in the mandated areas, I will come up with a masterpiece.

In fact, good management is a more of an art rather than science. If requires continually honing your craft through intelligently listening to feedback. Feedback can come in various shapes and forms: it can be learning from mistakes. Some monumental failures offer the best opportunity for valuable lessons. It could be from really understanding variables that are not obvious immediately - such as corporate culture or the 'real' balance of power in an organization. It would be learning that influencing (and knowing how to do it) is a lot more valuable than dictum authority. It could be taking a humbling experience and turning it into actionable insights for the future. Furthermore, the craft is enhanced through incessantly tinkering with the levels. Project managers have finite availability; figuring out where to place your attention at any given time is methodology or framework independent.

So what does this mean for structured methodologies? It means that a tools is only as good as the skills of its operator. Methodologies are like tools. They are useful for certain tasks, but they are not irreplaceable. You might find it useful to have a paint brush and some outlines to paint a landscape, but the same can be accomplished painting with a painter's knife (or a different brush). More importantly, given the same set of tools, there will be a wide range of individual, non repeatable craft that practitioners will bring to the table. This craft, this individual 'alpha' of the project participants (as well as how the skills of various participants mesh together) is just as important if not more important than the methodology. The craft of the project leader becomes even more essential. As the person whose decisions and steering affect (positively or negatively) the widest range of project aspects. Hiding behind the false sense of security that the methodology offers is even more pernicious if done by a project leader.

Methodologies also provide a framework for collaboration, a common language of sorts. Having all the project practitioners understand the same ceremonies removes friction. If everybody understands interaction etiquette in the same way, it removes a project impediment and allows the team to concentrate on value added tasks instead. But again, this is not the silver bullet. Collaboration frameworks reduce friction, but they won't compensate for poor understanding of the client's needs or for incapacity of seeing the project risks (or where to look for them). But they can give everybody that false sense of security. A tool can only get you so far; the rest is superior craft that can only come from experience, from every project participant, but especially from the project leader.

CMMI compliance - pay lip service better than not doing it at all?

Here's a scenario that I see way to often around the beltway: an organization is mandated to attain a certain CMMI maturity level - usually level 3, as part of a contractual requirement - usually involving goverment work. Because it is a contractual requirement, that organization usually makes it happen. They go through the pain of implementing a SEPG (software engineering process group), they install CMMI czars and institute the software process assessments and all that. On the project side, PMs are required to allocate resources to tailor their processes from the organizational standards and to maintain the paper trails proving compliance. Way too often this degenerates into a state where nobody really believes in the real business value of all this work, and everybody is paying lip service to it because it is 'required'.
Even in this scenario, I think that the organization and project themselves are better off than in a case where no attention is paid to standardizing and improving system engineering processes. CMMI compliance, even if done as a 'requirement', forces an organization and the project leaders to take a step back and evaluate, which is the very spirit of improving. On the project leader side, one might not like all the overhead, but by the very nature of being forced to minimize it, one becomes educated and efficient. I won't comment on the absolute business value of CMMI compliance - it is besides the point in this scenario since it is a contractual requirement. I do feel that projects can get something out of it, even if they don't like the overhead.

Defects vs. enhancements

Is it a defect or an enhancement? The answer to this question makes a big difference to the software team. The developers are very much involved with the product that they are building and often times they take offense if a requirement defect or an enhancement (new functionality request) are mislabeled as defects, i.e., a problem with design or implementation.
The problem is, none of this matters to the business stakeholder. A good Agile team shouldn't even care about the difference between a bug, requirements defect or enhancement, as long as it is work that needs to be done. Clinging to categories is often a symptom of team division and dissension.

The business value of fixing defects

Less defects is always a good thing. Right? Not so fast. Fixing a defect has both benefits (more obvious) and costs (often easy to ignore). In this posting, I'll concentrate on some of the costs and why they can sometimes outweigh the benefits. This analysis is performed from the perspective that everything we do in a software project has to come in line with adding value to our stakeholders.
First, let's discuss the opportunity costs. Allocating resources to fixing a defect takes away from doing other things - such as fixing more important defects or implementing new features. The way to prevent misallocation of precious resources in an agile setting is to treat defects as any other quantity of work - create stories for them and, based on a prioritization exercise that takes into account feature stories, work them into the backlog. In a non agile setting, misallocation of resources due to high opportunity cost is trickier. Since prioritization is usually fractured between requirements and defects, defects usually get prioritized among themselves and get fixed within the allotted time. This usually leads to inefficiencies - since very seldomly the predetermined time for 'fixing bugs' is just right. Many times, teams will over fix some parts of the code and under fix others. There is also a temporal horizon bug fixing problem (I'll deal with that more in depth in a separate entry) - in brief, more recently discovered bugs tend to be fresher in people's minds, and thus, receive more attenting than older bugs.
Second, fixing bugs is risky. Touching a piece of code runs the risk of introducing more bugs, in other words, doing more harm than good. This risk should be managed by the project manager just as she would manage any other project risks. This is particularly heinous since time and attention for retesting are often in short supply.
To conclude, I'm not advocating against fixing defects. The point I'm making is that fixing defects should be treated as any other quantity of resource consuming endeavor, and as such, it should be balanced against other resource demands from the perspective of getting the most bang for the buck for the project stakeholders.

Small teams: PM or tech lead?

So you're trying to assemble your development team for a software project. You are limited in the number of overhead resources you can afford. Which is better - looking for a strong tech PM type (e.g. someone that can control the project variables - scope, budget, product quality, etc.) or looking for an architect / master coder / lead by example team lead type? Unfortunately, most often than not, the two skill sets don't come in the same package. A lot of small teams (3 - 6 developers) are often forced into having one or the other. The tradeoff may be forced not only by budget constraints but also by trying to avoid a top heavy small team.
The answer - as anyone working with consultants is already predicting - is that it depends. A higher complexity project - using new and unproven technology, building something from scratch will lean the balance in favor of the team lead type. Immovable milestones and a bunch of developers with gold plating tendencies will lean the balance in favor of the PM type.
Regardless of what you decided need most, the mistake to avoid is blaming the project woes (if any) on the absence of the person / type you did not choose. A project, like anything else in life, is subjects to constraints and imperfect situations. Once you made a decision on who will lead your project down in the trenches, live with it.
For example, assume you do not have a team lead to bring the team to consensus once in a while. Well - losing some efficiency by allowing too much bikering about some of the technical decisions is just an opportunity cost of better project control (assuming your tech PM is doing a good job).

Ideas or instantiations?

It is striking how the dichotomy between framework based approaches to managing projects vs. agile methodologies resembles the metaphysical discussion around Plato's theory of forms.
In Plato's opinion, forms / ideas are more real than their instantiations that we see in everyday life. The idea of a human being is more real than all the people you see around you every day.
Take this concept and apply it to framework based approaches such as CMMI. CMMI, we're being told, is a model, a framework for process improvement.
The point I'm trying to make is not that CMMI is more 'real' than any of its implementations / applications. It is that CMMI is a syntetic construct, just as an idea (in the Platonic sense). The problem is that, just like Plato's ideas, CMMI only sounds good when it is not applied. Once applied, it loses its universality, it's just faint shadows on the background of a cave. Thus, the framework itself is never to blame for inefficiencies or unnecessary processes. The framework is never the efficient cause of failure. The only blamable part is the implementation.
Agile on the other hand is a collection of disparate practices (what makes sense) collected during numerous project life cycles. The beauty of it is that, because these processes are already 'implementation', they are the cause of failure or, more often, project success.

Self organizing teams - quo vadis?

I have a team of 5 developers and 2 functional analysts and we use mostly an agile approach to developing software. Coming from a non-agile environment, I had to get used to the concept of letting the team make decisions, including those involving self righting. I instantly recognized some of the benefits. First and foremost - the dream of any project manager, I didn't have to tell everybody what to do all the time. One of the pitfalls of managing project is trying to do too much and by doing so, curtailing the natural lines of communication between project participants. Developers did not expect me to tell them how to communicate with each other or with the functional analysts. If we hit a bump in the road or communication was not efficient, we would discuss it at the iteration post mortem meetings and right it for next time.
However, lately I have been nagged by a side effect. There have been several instances lately where the team took decisions that should have been brought up to my attention. An example is making a decision to refactor code instead of picking up new functionality off of the product backlog. This put the overall schedule and our commitments to the client organization and their board of directors at risk.
The moral is that self organizing teams is a great thing to do if it can be kept under control. Multidimensional communication should still include the PM and the role of the PM as the gatekeeper on scope, schedule and cost should still be recognized. There is a delicate balance to be striken between micromanaging that strangles team initiative and an absolute 'hands off' approach where the project manager is rendered powerless by the voice of the crowd.

Recognizing business value in software development

I think that it is more important than ever for the development team to recognize the business value of the application they develop. The traditional approach had the requirements handed down from the 'business owners'. It was almost sacrilegious for the developers or even analysts to suggest improvements or discuss the ultimate business purpose with the business owners. Think separation of duties, don't do anything you're not 'qualified' of doing.
The problem with this approach (for the development team), is that it is the development team is the place where value is created. Not allowing or encouraging collaboration between business owners and the development team is similar to blindfolding a master painter and ask him to paint a portrait from memory after a cursory look at the model. The development team needs to understand not only what but also why something is built. This ensures project success and prevents finger pointing situations where everybody is right in their own ways.

Project and SDLC methodologies and Agile development

This entry is a quick comment on how 'official', PMBOK-style project management is so far removed from software development using Agile methodology.
All these monolithic methodologies / frameworks - PMBOK, CMMI, etc. claim that you can tailor them to the realities and specificities of your project. This is true. But at what cost? I had an engagement where I had the luxury of spending 3 months tailoring PMBOK and CMMI frameworks before I engaged on a grueling, 15 months software development cycle. But most engagements cannot afford that. Most clients (unless they are in the government space) expect you to come to battle with your arrows already sharpened; they won't pay for you to get your act together as you go. Hence the usual demise of PMBOK and CMMI.
That's where Agile (or Agile-style processes) come into place. In Agile, everything happens as you go (or as you see a need), including creating processes. You might not get it right the first time around, but you eventually get there. The beauty of it is that you only build the processes that you need, when you need them. Team getting bogged down because you don't have a defect management process? It might be time to sit down and create one. Need a more complex process where you include a step for triage and approval of what gets fixed with priority? Better add it in the process. After a while, you're going to get to a state of equilibrium. There - you have the process and not only that, but unlike many monolithic methodology implementations, you will also have buy-in from the process stakeholders.