The answer to both of these questions is yes.  Technical stories represent efforts necessary to improve the infrastructure of a system.  While they do not add value to the system through the addition of new or refined functionality,  they do directly add value to the system in other ways.

It is our responsibility to identify the value directly added by each technical story, and to educate the client on this value.  It may not be as quantifiable as the value of new functionality, but it does exist. Better system stability, lower vulnerability to security or other malicious attacks, enhanced scalability and more robust disaster recovery all intrinsically add value to a system.  Value can also come in the form of more efficient future development, and the associated reduction in cost or schedule.

In the article, Jack observes:

But what about the type of story asked by one of the members “As a development team member, I want the existing unit tests to run under CruiseControl, so that I know if anything breaks”. Where does this belong?

Well this is a perfectly written story but the story really has no ultimate value for the end user at least not directly.

I disagree with Jack that such a story does not have an ultimate value to the end user, or that it’s value is at most indirect.  Automated unit tests allow the development team to deliver new features to the end user much faster.  Every feature added to a system has a value, therefore the lack of any given feature has an opportunity cost.  The direct benefit to the end user is the reduction of that opportunity cost.

If I am unable to articulate the direct value of a technical story to a client, I have no business doing that story let alone asking the client to pay for it.  Effort without value is waste.

We include these infrastructure upgrade tasks in our project MMF workflow.  Cards are written for them, and these cards are added to the project backlog.  Once educated to their value, our clients are then empowered to make priority decisions between technical infrastructure tasks and new features.  We must trust that the customer knows their business the best.  They will know if the opportunity cost of a delay in the implementation of feature X is of higher value than the future cost and time savings of a conversion from Scriptaculous to JQuery, and will set the appropriate priorities.

Related posts:

1. Agile Development for Tiny Teams
2. Eliminating Waste from your Agile Process
3. Our New Development Process

Our transition to a lean development process was the first exposure most of our clients had had to any type of agile environment.  Understandably many of them felt out of their element and truthfully we felt the same way.  What really made the transition successful for us and our clients was the trust that came from the good relationships we’ve maintained.  Being very transparent with our clients went far in cementing that trust.  We let them know that this was a major change for us too, that we were (and still are) learning and refining our process.  Since our clients are very much part of the project team, I openly solicited input from them on how we could improve the process.

Kanban Boards

We employ separate MMF Kanban boards for each client’s project.  Each board has the following queues through which the MMFs flow:

1. Backlog – This is an unordered list of all MMFs that have been conceived. This queue has no limit. It serves as a dumping ground for ideas as simply existing in this queue does not imply any commitment that an MMF will ever be implemented. MMFs in the Backlog may or may not have an estimate.
2. Pending / Scheduled – This queue is prioritized with a limit of 3. Barring priority changes, MMFs in this queue are committed to development. A planning meeting with the client is triggered when this queue reaches 1 item. During this planning meeting the next two MMFs to be developed are identified and added to the queue in order. The client is welcome to change priorities at any time, reordering the queue or swapping MMFs from the Backlog. All MMFs in this queue will have an estimate.
3. Work In Process (WIP) – To address our goal of reducing WIP, this queue has a limit of 1. It is while an MMF is in this queue that the nested, story card based workflow is employed.
4. Acceptance Testing / Show & Tell – Again, to reduce WIP, this queue has a limit of 1. Once the MMF in this queue is ready, a deploy of the software is triggered.

Estimation

Each MMF is given a “shirt size” of small, medium, large or extra large.  These estimates are assigned at any time while the MMF is in the backlog queue.  Sometimes estimates are assigned when an MMF is added to the backlog.  Other times an MMF does not receive its estimate until we are about to pull it into the pending queue.  There is no scientific method for determining the shirt size of a given MMF. It’s a simple process of assigning a “gut feel” based on the description of the MMF.

Even though there is no formal process to calculating them, some of our clients have found them quite valuable in helping determine priority.  We’ve had clients make the decision that the combined value of 3 small MMFs outweighed the value of one large MMF, and pulled them into the pending queue ahead of it.

Once MMFs have been pulled into the pending queue, the estimate serves as the primary tool by which we can answer the client’s question of “How long until I have this?”

Estimates are not carved in stone.  As we work, we understand the scope of an MMF better and uncover knowledge upon which we are able to make a better estimate.  Sometimes development of a higher priority MMF will impact the estimate of an MMF farther back in the workflow.  The more accurate our estimates, the more valuable and useful they are.  We update our estimates as necessary for the overall good of the system.

A change of more than one size, such as small to large or XL to medium, does trigger a review as to why we are making such a significant change to our original estimate.  Even though the process of setting an estimate is one of an educated guess, we do want to continue to improve our ability to make accurate guesses.

Data Tracking, Trailing Indicators and Their Application

There is a significant amount of data that exists around just this part of our development process – enough to keep a data mining, business intelligence geek busy 40+ hours a week.  The truth is, however, that if I were to spend my time doing analysis of our process – there wouldn’t be enough flow through the process to generate data worth analysis.  We are a tiny team of 2, after all.  I applied YAGNI, and we are only tracking the couple of data points that are critical to our management of the system.

For every MMF, we track the date it is pulled into the WIP queue (start date) and the date it is finished in the acceptance test queue (complete date).  From those two dates, we are able to calculate and record the MMF’s cycle time.  We do not record any additional data points for MMFs.

We log MMF cycle times by complete date and estimated size.  I maintain population means by estimate size, rounded to a full day.  I’m then able to use these averages to compute the answer to the client’s question of “How long?” by applying them to all MMFs ahead of the one in question.  MMFs in the WIP or acceptance testing queues  are included by determining days left as the difference between start date + average days and today.

Flexibility to Interface with our Client’s Business

Each client and each project will naturally be a little bit different than the last.  We have adapted this workflow a little bit to better fit with our client’s business process.  For one client, we expanded the limit for the pending queue to 5.  This allows for a longer interval between the triggered planning meetings, which fits better with his schedule.  For another client we actually maintain the backlog as a collection of different lists of MMFs – one for each functional area of the project.

Kaizen

Our goal is to work towards the most efficient and effective process for developing software for our clients.  Just as I solicited input from our clients for improvements, I’m very interested to hear your feedback on this workflow.

Related posts:

1. Agile Development for Tiny Teams
2. Our New Development Process

We determined that our new process needed to be:

• Simple – Being a tiny team, the impact of time used managing the process rather than used generating billable work product is magnified.
• Able to handle concurrent projects – We needed to be able to support ongoing development needs of current clients while continuing to develop new clients and projects.
• Scalable – We will add subcontractors to our team as needs arise. Our new process must work regardless of the team size. It also needed to be easy to teach to new team members.
• Flexible – In addition to client work, Josh and I are partners in Prfessor.com which includes a significant amount of development efforts. The process needed to be flexible enough to manage these obligations too.
• Waste Reducing – We needed to address the four top areas of waste that I identified in our original process (see Eliminating Waste from Your Agile Process)
  • Work In Process (more than the bare minimum necessary)
  • Rework
  • Over-engineering the code
  • Repetitive tasks

From Jon’s presentation I realized that we really had two separate workflows, one nested inside the other. We never formalized that distinction, but when we began to view things in terms of MMFs (Minimally Marketable Features) the nested process jumped right out. Understanding this nesting was critical to the success of our new process.

MMF Workflow

Our MMF workflow is a customer facing part of our process. It relies upon regular input from the client and is the primary interface for our clients into our development process. Clients are welcome to look further into our process and see the details of their project. We have found that some clients prefer to let us completely handle the details. Our new process provides the ability for them to be involved in the process without being overwhelmed by the technology and implementation details.

MMFs flow through a 4 queue process:

The MMF workflow is the part of our business process that needs to interface with our client’s business processes. We found that providing the flexibility for our process to adapt to some level to fit the client has been very valuable. For example, for one of our best clients we maintain multiple Backlog queues – one for each aspect of their application. This organization of MMFs has proven valuable to them in support of accurately setting priorities. Another client was more comfortable with a larger Pending queue as it provides an increased interval between triggered planning meetings. We try, however, to keep this queue limit as small as possible in order to avoid thrash in the queue due to changing priorities.

Story Card Workflow

Nested inside our MMF workflow is our Story Card workflow. This is where the actual coding efforts are managed. The MMF workflow addresses behavior while the Story Card workflow addresses implementation. The MMF workflow is the customer facing portion of our process. The Story Card workflow is primarily for internal consumption. The first thing that happens when an MMF is pulled into the WIP queue is for story cards to be written defining implementation details. .

With our unique but complimentary skill sets, our original workflow was implemented through Kanban queues through which story cards would flow:

Estimation

Clients have consistently presented us with two common questions – “How long until I have this?” and “How much will it cost?”. Obviously the answer to both questions involve some level of estimation. We have formalized the process of estimation towards the primary goal of answering those two client questions in a consistent manner. MMFs and Story Cards are estimated using different methods.

MMFs are given a “shirt size” of small, medium, large or extra large. There is no scientific method for determining the shirt size of a given MMF. It’s a simple process of assigning a “gut feel” based on the description of the MMF. As further understanding of the scope and meaning of an MMF is gained, it is acceptable to change the assigned estimate. Continuous improvement applies to the estimates assigned too. These shirt sized estimates are used along with team performance data collected through the process to answer the question of “How Long?”

Story Cards are assigned a point value. We use the binary sequence of 0,1, 2, 4, 8, 16 as valid point assignments. We’ve determined that if a card should be assigned more than 16 points, it should be further broken down into smaller cards. To date, we have found that 8 points tends to be the largest value commonly used. Point assignment happens as the story cards are written, prioritized and added to the Backlog. We have transitioned from billing our clients by the hour to billing them by the point. These point estimates therefore easily compute to provide the answer to “How expensive?”.

Concurrent Projects

One of the most simple parts of our new process is the management of multiple concurrent projects.  We have set agreements with our clients, committing to a specified number of average points per week of development velocity.  With the rhythm of the development process, the actual weekly velocity for a given project varies.  We track our performance on each project to ensure that on average we are meeting our commitments.  Our clients have embraced this process as it provides them with both steady progress on their projects as well as a predictable monthly invoice from us.

For example, we have calculated that Josh and I can produce an average development velocity of 35 points per week.  Currently we have committed 10 points per week to one client and 15 points a week to another client, leaving us with 10 points per week we can employ on either project as necessary or on internal projects such as Prfessor.

Kaizen

Even though this process is serving us well today, we continue to improve the process.  Planning and writing this series of posts has triggered a number of ideas for ways we could further refine our process.  I know that external review of a process can many times uncover issues that those involved in the day to day operations miss.  I’d really enjoy hearing your feedback on our process, or about a process you’ve implemented to solve similar challenges.

Related posts:

1. Technical Stories in Agile Software Development
2. Agile Development for Tiny Teams
3. Workflow – Minimally Marketable Features
4. Eliminating Waste from your Agile Process

Just as in a car factory, you do not want to build 1000 cars in parallel, similarly in software you do not want to create 1000 features in parallel. As you start to build a car, the waste builds as you add more and more parts, but the waste disappears when the car is completed and ready for shipment. The uncompleted cars are a liability on the organization, and building cars in parallel multiplies the amount of waste. Similarly, once your start designing a feature, and start investing person-hours, the waste builds in direct proportion. Finally, when the new feature is actually in a “customer-ready” shape, that waste disappears. The started-but-uncompleted features are a liability. Features designed but not coded is a waste of time. Code that is written but not tested is a real liability because there is now a hidden amount of work that must be done to correct problems in the coding. Completed features in the code, but can not be installed are still a waste. Building a bunch of features in parallel multiplies the waste.

One of our goals with this process was to be able to handle concurrent projects. That dictates that we may have multiple features in development at the same time. A single feature per project is the bare minimum necessary to meet our production level on each project. Parallel development on multiple features for a single project also brings with it the unnecessary overhead of managing different source code branches. This unnecessary work is waste which can easily be eliminated.

Rework is another area of significant waste in a system, be it building cars or software. We determined that there were two possible root causes of the majority of our rework. Rework was generated out of either doing something wrong (quality control issue) or doing something too early (process issue).

Quality control issues broke down into two classifications – the code we wrote didn’t meet the specification or the specifications didn’t accurately represent the business need. The combination of our move to an agile development process which involves the customer much more frequently has helped address this issue. We are also making fundamental technology changes to an environment which includes tools to support BDD. Combined, these changes will go far in reducing the waste of quality control based rework.

The best example of rework due to doing something too early involves doing in-depth design for a feature long before we were ready to work on it. We would spend time laying out all the technical details of how we intended to implement a feature. Any change to the architecture or the definition of the feature by the development of prior features left us with no option but to discard part or all of our prior work. Generous application of the principle of Deferred Commitment has effectively eliminated this waste in our process.

I have traditionally wanted to over-engineer my code. I have invested a lot time building code with the mindset that someday someone might want this. A few times I’ve been right. Most of the time, however, that effort has been 100% waste. Applying the YAGNI principle has yielded a significant reduction in waste. The change hasn’t been easy for me, but the benefits have outweighed my discomfort.

A personal pet peeve is performing repetitive tasks. Perhaps that’s the fundamental reason I’m in software development. In all areas of my life I find myself asking “How could I automate this?” every time I have to do something more than once. Many of the repetitive tasks we perform in the process of software development could easily be automated and are therefore waste. The reason we developed and continue to refine our Sandstone environment is to eliminate waste. Appropriate use of quality development tools has also helped us to automate tasks and therefore eliminate waste in our process. We’ve also looked at our process itself for repetitive tasks that can be automated. Calculation of the key metrics used to monitor our process was the most common repetitive task we found. Some time invested building some simple spreadsheets has yielded consistency and efficiency in that process.

This is far from a comprehensive list of everything that could be considered waste in the software development process. These were the low hanging fruit we identified in our process. I’m always interested to hear what others have identified as waste and how they’ve worked to eliminate it from their process.

Related posts:

1. Technical Stories in Agile Software Development
2. Agile Development for Tiny Teams
3. Our New Development Process
4. Design Last Design

We believe that software should be high quality. It should be efficient, tested and intuitively designed. When it comes to features, enough is enough. We believe you should build what you need, and nothing else. We also believe that the process of developing the product is as important as the product itself.

Our skill sets are complimentary – I design the architecture that makes the application run, while Josh manages the design of the experience and interfaces.

Dave makes it work, Josh makes it usable and visually beautiful.

In the beginning, we used a traditional waterfall approach. We took on some fixed price projects, and they ended about as well as you would expect. At one point, we realized that scope-creep had lowered our rate to $1.26 an hour. No, that wasn’t with tips either.

Clearly, that’s not a business model to be proud of. So, we changed to trading hours for dollars. We kept our pipeline full of billable hours, and that’s where revenue came from. We booked clients back to back, which kept us busy, but repeat customers did not always get the attention they deserved. At the same time, we weren’t happy either.

Our Original Workflow

1. We spec’d the project to determine what it had to do;
2. We jointly architected a plan of attack;
3. I would code the business logic as fast as possible;
4. I would “toss it over the wall” to Josh to slap a UI on top of it;
5. We would get feedback from the customer;
6. We would make changes and complete the contract;

We wasted a lot of time re-working things, which caused deadlines to slip. By the time Josh got to work on the interfaces, I had to refresh my memory on how I had implemented things. Alas, more wasted time. It became evident that something had to change.

There was another problem. Our goal was to complete the project and get paid for it. However, in most cases the end of our process was just the beginning for our clients. The contract ended, but the project did not. Since we had to move on, our clients, with changes and cash in hand, were left on the sidelines.

Changing our Process

Our process had to change. Josh and I researched agile development processes, and were fixated on SCRUM. We even took a phone call with Jeff Sutherland, the creator of SCRUM, to discuss it. We gave it a fair shake, it just wasn’t a comfortable fit for us in the end.

Jon Stahl, the resident Agile expert here in Cleveland, gave us a preview of his Kanban presentation. Through my background in industrial management, I was quite familiar with Kanban in a manufacturing environment. I had never considered it as a software development process. After hearing Jon’s talk, and talking with him a bit afterward, I was solidly convinced that Kanban was our ticket to a leaner development process.

In a few future posts, I’ll describe our new development process as it’s currently implemented. Even though it’s working very well, we are always making improvements.

Related posts:

1. Technical Stories in Agile Software Development
2. Our New Development Process
3. Eliminating Waste from your Agile Process
4. Workflow – Minimally Marketable Features

3rd grade example: Continually Divide by 2.

100 / 2 = 50
50 / 2 = 25
25 / 2 = 12.5
12.5 / 2 = 6.25

As we continue to evaluate the equation to infinity the answer will approach zero, but never get there.

Grown up example: (The equation isn’t important, the phenomenon is)

As a smart-ass kid in high school, I didn’t really understand why this would be important in real life.  In reality, this is one of the driving principals of my daily work flow.

Time is our most precious resource, and a perfect web application is the ideal output.  Just like everything else in this world, perfection is asymptotic.   We can always improve, and we will (hopefully) keep getting closer to perfection, but never actually get there.

As developers and designers, our perfectionistic tendencies get in our way.  If we shed ourselves of the myth of perfection, and instead focus on a series of iterative improvements, our application will improve faster and more reliably than if we had focused on building it in one perfect attempt.

Pay attention to what tasks are taking up your time.  We should always be working on the things which result in the biggest step toward our goal.  We have a lifetime to knock out the little things.

No related posts.