A few months ago an article was published in Computer Weekly where corporate IT Lawyer Alistair Maughan, argues the case that Agile will not work in Government projects.  I didn’t see this article when it was first published, but it was brought to my attention when a colleague noticed that one of the comments arguing against the article, referenced a presentation I gave.

Many thanks to Francis Fish for referencing my presentation where I discussed introducing Scrum on a UK government project and used Agile methods to rescue a failing waterfall project, again within a UK government programme of work.  I’m really glad my message got across despite the ropey presentation, although in my defence it was my very first attempt at public speaking.

I’ve been working on UK government projects for over 10 years. During my initial 5 years on government accounts I was part of Valtech teams using RUP and aspects of XP, the later 5 years I was fortunate enough to implement Scrum and Kanban on my projects.  With my Agile UK government experience I feel I am justified in challenging many of the misconceptions mentioned in the article.

Firstly, mismanaged projects will go spectacularly wrong independent of the project management framework.  Projects fail for many reasons including ill-conceived requirements, poorly skilled developers, inept management as well as inappropriate SDLC.

So to address some of the points in this article:

“But Agile simply won’t work in the real world of government ICT.“  This is not true, Valtech have implemented a number of successful projects using an Agile SDLC on different government accounts.  Often Agile methods have been used to “rescue” otherwise failing projects but increasingly the merits of running a project using an Agile approach from the outset have been accepted.

“We need a Richard Dawkins to bust the myth of the Agile gospel.”  We need to bust the myths of Agile, that Alastair has heard and believed.  Agile does all of the same value adding activities found in a waterfall project, just at an appropriate time, for example User Acceptance Testing when the real requirement pops out during development rather than right at the very end.  This approach increases the effectiveness of risk management enabling us to make better choices over the life of the project ensuring on-time delivery if (and only if) this is the most critical measure of success.

“Under Agile projects, you pay a given amount of money for a set amount of effort. But you can’t guarantee a specified outcome for a specific price.” You can guarantee an outcome by controlling the richness of the implementation.  A key differentiator of Agile methods when compared with more traditional approaches is the focus on outcomes in preference to outputs.

“Departmental budgets are managed very tightly” Agile enables much more detailed and accurate planning at the appropriate time, using prioritisation and identification of the richness of the solution that is acceptable to the stakeholders. With the added benefit of being able to more accurately account for the overspend as soon as it happens and justify the reduction in richness when appropriate.

“Agile can’t give you a clear specification of outputs up-front.” Neither can Waterfall, the chances of change is too high while the procurement proceeds, this says more about government procurement than Agile.  A well executed Agile project will provide sufficient specification at the right time to satisfy stakeholders.  It is not an issue of Agile projects rejecting documentation (or specification), rather that, when done well, they defer detailed specification to reduce exposure to late breaking changes or lessons learnt resulting in rework to the specifications.

“As if that isn’t problem enough, Agile offers insufficient means of remedy if things go wrong.” It’s a contractual responsibility not the SDLC to offer remedy.  Further, an Agile project should provide evidence of progress in a far more tangible manner. Valtech (and many others) have been working hard to develop contracts that deliver the flexibility in scope that customers deserve while acknowledging the suppliers responsibility for the management of technical risks and the intrinsic qualities of the product.

“You can have an ICT project with a watertight contract, clear deliverables, openly and legally procured, with a fixed price and appropriate remedies if you don’t get what you want.” Waterfalls might deliver what the customer wanted at the time of specification but rarely what the customer needs; as has been often quoted, 40% of features delivered by waterfall projects are never used   (Gartner). The act of insisting on a full specification of everything to deliver will actually make projects bigger. This is a natural consequence of the penalties traditionally applied when new scope is added late. By contrast an Agile project will be accepting changes in scope while encouraging challenge to “gold plated” solutions preferring “good enough” (but no less!).

Please find links to my presentation slides on implementing Agile on Government projects.

http://www.slideshare.net/valtechuk/agile-adoption-in-a-waterfall-environment

http://www.slideshare.net/valtechuk/agile-project-rescue-in-a-waterfall-environment

I feel enthused to blog about my current project, which is the first project I’ve ran where we have a dedicated UX expert on the team. For those of you who are not aware of the term UX, it stands for User eXperience. The UX expert will work closely with the customer and business analysts, defining user journeys throughout the site/system, producing the wireframes that feed into the designs and ultimately the UI development.

Consciously or even sub consciously we all appreciate a website with a good UX and can get really frustrated by poor UX.

While this is the first time I have had a specific UX expert in my delivery team, I have been aware of other Valtech projects delivering UX, over the last couple of years, particularly in the the Agency space for our large retail clients.

Now I wasn’t exactly sceptical, but didn’t fully appreciate just what the UX role entailed and what it could bring to the project. I will be honest and admit to being slightly derogatory about what a UX consultant could offer, and I may have possibly labelled it as ‘colouring in’. My bad

My ignorance was however, strongly influenced by the types of projects I had previously delivered. A lot of my previous experience has been delivering large government systems, where the kudos was always focused on the back-end technologies and not the front-end. Also a good user experience was never a non functional requirement.

The UI design and UX for these large government CMS systems is on the whole quite poor, overly complex and unintuitive. In some instances we may have worked with graphic designers, but most of the time the UI was being designed the customers Business Analysts, where only basic principles of a quickly outdated style guide are to be adhered to. On more than one instance, I was running a dev team which inherited reams of up front, waterfall, external design documents. These had been created using Visio and MSPaint, which led to the designing of near impossible components and cross-field validation. Chaos would generally ensue in System test and UAT when the final screens didn’t look exactly like the signed off Visio and MS Paint designs.

On my current project, the UX expert has made an instant impact. Working with the lead BA, it was obvious that the business owner was engaged and bought in to this way of working. High quality interactive wireframes, are produced in Photoshop with key animation sequences overlaid in PowerPoint. Great care is taken understanding different personas and minimum marketable features. The customer gets near instant feedback of how they want the site to be used.

I sort of expected the benefits that the UX expert would bring to the analysis and design, what I didn’t predict was how important the UX expert has become to the developers. Collaboration between the UX and devs was really visible from the outset and was recognised as one of the most significant positives at our recent retrospective. Key to this success was the way the UX expert researches the technology options, ensuring the design can be achieved using CSS or identifying the necessary JQuery plugin. Non standard design choices are always reviewed with the development team prior to sprit planning to ensure no near impossible designs are agreed with the customer.

Our Agile processes enable this collaborative way of working. We are all collocated provide our updates at a single, large but efficient daily stand up meeting. The devs and UX expert are seeing each others updates, progress and blockers. The BA / UX team are working one or two sprints ahead of the developers, so no design artefacts are going stale in the backlog. We have also ensured that a UX peer review is part of the definition of done policy for the front end developers.

This way of working culminated in a really positive first demo, with great feedback from the business owner and sales director.

Being an aspiring software craftsman goes way beyond just saying it. I’ll quote my own definition of software craftsmanship from my previous post.

Software craftsmanship is a long journey to mastery. It’s a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development.

Software craftsmanship is all about attitude. The attitude of raising the bar of professional software development starting with our own skills and professionalism.

The responsibility shift

Not long ago, I was speaking to a developer and he was complaining about his company, saying that they didn’t have a good career plan, that he did not have any training, that he was not given the opportunity to learn new technologies and, of course, that he was not paid enough. Apparently, from his perspective, his employer was responsible for his career.

Imagine that we need a doctor. Would we pay a doctor to learn while he cut us open or give us a diagnosis? Would we pay an engineer to learn while he draws the plan for our new house? Would we go to a concert and pay the musician to learn how to play the guitar during the show? What about a chef in a restaurant?

So why is it the employer’s obligation to pay for training courses and pay us to learn new technologies and tools while we are working on a project? Should the employers be responsible for what we learn and what we don’t learn.

Software development is not a 9 to 5 profession. To be a professional software developer, we need to take our own time and money to keep learning and improving. As professionals, we should be paid for what we know, our ability to learn fast and for the quality of the work we do. We own our careers and are responsible for them. Working for a customer/employer that helps us with our career in terms of training, books, conferences, seminars, etc, is great but should be considered a bonus.

… but how can we learn and keep ourselves up-to-date?

Different developers have different preferences but here is a list of ways I find useful.

Literature

Books, many books. Having your own library is essential. Books give you a good overview of a specific technology or subject quickly. No, it does not mean you will be proficient and next day you will be a specialist. What a book will give you is an understanding of what a technology or subject is about. It will be up to you then to decide if you want to practice what you’ve learned and become proficient. If you don’t have the habit, try reading 3 to 4 technical books per year. Once you get the habit, try one per month. The most common excuse is that you don’t have time. The cool thing about books is that you can read them during periods of  “dead” time, like on the tube, bus, in your dentist’s waiting room, on the bed before going to sleep, toilet, etc.

Blogs are now one of my favourite types of reading. They tend to fit more the software craftsmanship model since they are much more personal and in general, related to personal findings, opinions, successes and failures. Reading blogs from more experienced professionals and subject matter experts is a good, quick and free way for apprentices and journeymen to learn from multiple master craftsmen at the same time. But don’t think that just experienced professionals should write blogs. Every software developer should write his or her own blogs, sharing their experiences and findings, helping to create a great community of professionals.

Technical websites are also good in order to keep yourself up-to-date with what’s going in the market. New technologies, techniques, etc.

Practice, practice, practice

Pet projects are for me, by far, the best way to learn and study. A pet project is a real project but without the boring bits. There are no deadlines, does not need to make money, you control the requirements and most importantly, you use the technologies and methodologies you want, whenever you want, wherever you want. You are the boss. Pet projects give something for you to focus on and help you to understand why and how you can use certain technologies. It gives you the experience you need to apply what you’ve learned to real projects. Pet projects are meant to be fun.

Contributing to open source projects can also be a great thing. There are thousands of them out there. Find a project that is related to what you want to learn or know more about and download the source code. Start running and reading the tests, if any. Inspect and debug the code. If you want to contribute, start small. Add some documentation and write some tests. Then, check the list of features to be implemented and pick a simple one and give it a go. You can also propose and implement a new and small one to start with.

Pair-programming can be a great experience. Many developers are afraid to try or think they will feel uncomfortable. That’s what I thought as well before I tried. Today I really enjoy it. Pair-programming gives you an opportunity to discuss your ideas, to learn new tricks and techniques from your pair and the resulting code is much better. Pairing with someone from your team is good but pairing with someone that you barely know can be a very interesting experience.

If learning a new language or new technique like TDD / BDD or trying different approaches to OOP or functional programming, try a code kata. Code kata is a small exercise that in general can be solved in a few minutes or in a few hours. Code katas were created to help developers to focus on a problem while improving their skills. You can find a good source of code katas at codingkata.org and codekata.pragprog.com

Community and Events

Be part of your local community. Software craftsmanship is all about a community of professionals, learning and sharing with each other and elevating the level of professionalism and maturity of our industry. Join your nearest user groups and participate in their events. User groups are the best way for making contacts, sharing ideas and learning things. Many user groups promote free talks, coding activities and social events. A great aspect of being part of a community is the feeling that you are not alone. There are many people out there having the same problems you’ve got and many others that will happily share their solution to it.

Raising the bar

The main changes proposed by the software craftsmanship movement are related to the developers attitude. In summary, it’s about being proud about your work, the code you produce and the software you create. It’s about constantly trying to improve your skills and learn new ones.

An aspiring software craftsman is intolerant of bad code will constantly apply the Boy Scout Rule.

If you find that the good code you wrote one year ago is still good enough today, it means you didn’t learn anything during this period and this is unacceptable.

Productive partnerships

The attitude of a software craftsman goes beyond good code. The relationship between a software craftsman and his or her customer (or employer) is of a productive partnership and not employer / employee. As a productive partnership, it’s our role to constantly question the requirements and propose improvements and new features. It’s our job to warn our customers about the problems we see. Due to the knowledge of the code and technology, developers are well positioned to help their customers improve their business.

However, sometimes some customers or employers don’t want or don’t see the advantages of this partnership and will treat developers as production line workers that should just do what they are told. Coding monkeys. In cases like that, any aspiring software craftsman should move on and find another job. Staying in a place where your skills are not appreciated is career suicide.

Raising the bar of our industry is in our own interest. The better we become in writing code and delivering valuable software, the better our lives will be, professionally and financially.

PS: If  you are in London or nearby, join the London Software Craftsmanship Community.

Is software engineering the best approach for developing software? Does it apply for the majority of the software projects or just a very few of them?

Software Engineering was an answer for the perceived “software crisis“, back in 1968, in the First NATO Software Engineering Conference and it was created to solve the problems of extremely large NATO and U.S. Department of Defence projects. In the majority of these projects, the hardware was still being designed and with no hardware to test, there was plenty of time to investigate requirements and write the software specifications. Hardware controlled by the software was generally worth billions of dollars like in the case of space shuttle and the Safeguard Ballistic Missile Defence System. People’s lives and national security were also at stake.

The IEEE Computer Society‘s Software Engineering Body of Knowledge defines “software engineering” as:

Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.

Software Engineering can be very effective when developing safety critical systems like the one for the space shuttle as described on “They Write The Right Stuff” – Fast Company article, from 1996:

The last three versions of the program – each 420,000 lines long – had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.

Of course that this looks really impressive but there is more to it:

Money is not the critical constraint: The group’s $35 million per year budget is a trivial slice of the NASA pie, but on a dollars-per-line basis, it makes the group among the nation’s most expensive software organizations.

Another extract from this same article when they were discussing the process:

And the culture is equally intolerant of creativity, the individual coding flourishes and styles that are the signature of the all-night software world. “People ask, doesn’t this process stifle creativity? You have to do exactly what the manual says, and you’ve got someone looking over your shoulder,” says Ted Keller (senior technical manager). “The answer is, yes, the process does stifle creativity.”

Many of the NATO and US Department of Defence projects took years, some over a decade to complete. Many had hundreds of people involved and almost half of the time was spend in requirements and design specifications, with uncountable reviews and approval cycles. Of course, due to their reliability and quality, many are still in use today.

Software Engineering for the Masses

More and more hardware became cheap and business of all sizes needed software in order to survive and be competitive. The difference was that a the big majority of these businesses couldn’t afford to pay $35 million dollars per year and neither wait for too many years to start benefiting from their software. Also, many of those software projects were not manipulating expensive hardware or dealing with life-threatening situations. Edward Yourdon, in his book Rise and Resurrection of the American Programmer, wrote:

I’m going to deliver a system to you in six months that will have 5,000 bugs in it – and you’re going to be very happy!

It was clear that software engineering processes had to be adapted in order to satisfy a more impatient and lower budget legion of businesses.

The “Good Enough Software” Era

Over the decades, many new software engineering processes and methodologies were created in order to make software development faster and cheaper. The most adopted were the ones based on iterative and incremental development that evolved into agile software development.

Agile software development and “good enough software” were an amazing improvement in bringing costs down, mitigating risks with quicker feedback and much faster time to market.

Regardless the methodology or process used, software projects are still failing. Some fail because they go over-budget, others because they are not delivered on time, others fail to satisfy the requirements and business goals, et al.

The main problem is that for decades, software development was seen as a production line. That’s the software engineering perspective of software development. Processes and methodologies are generally much more focused on making this production line more productive, creating different management and organisational styles than actually trying to make the employees more capable. Developers are treated as mere brain-damaged programmers and are at the bottom of the food chain.

Good enough is not always good enough

Using agile, lean or any methodology is not enough. It is all about the people involved in the project and their willingness to succeed and be proud of what they produce. If developers are seen as the least important and cheapest members of a software project, the maximum that this team is going to produce is mediocre software, regardless of methodology.

A software project would have much better chances to succeed with “good people with a bad process” than “mediocre people with a good process”. Good people would always find a way to improve the process, be productive and produce something that they are proud of. Mediocre people accepts whatever is established, even when it is not good enough, and will produce just what they were asked for.

In a software project, if a company wants a good software, they will need good and empowered software developers. Instead of 10 mediocre people with a good process, it would be better to have 3 or 4 good people and empower them to deliver the project.

A process, imposed by managers and people that have no clue how to write software, will just guarantee that mediocre software is predictably delivered (if lucky). On the other hand, a team of self-organised and great developers would have a better shot at creating a more efficient way to produce great software, constantly trying to improve they way they work.

A process should never be more important than the people. Managers should facilitate the work of great software developers and not tell them what to do. It should always be harder to replace a good software developer than a manager since they are the ones that know the system inside-out. Managing a few well motivated, well paid and good professionals is always easier than managing many mediocre people.

Software development is a creative and highly skilled profession that takes years to master. While software development is treated like a production line, projects will continue to fail.

Software Craftsmanship: The New Imperative – ISBN 0-201-73386-2, 2002
http://en.wikipedia.org/wiki/Software_engineering
http://en.wikipedia.org/wiki/Software_crisis
IEEE Standard Computer Dictionary, ISBN 1-55937-079-3, IEEE 1990
“They Write The Right Stuff”, Fast Company, http://www.fastcompany.com/magazine/06/writestuff.html
Safeguard Program: http://en.wikipedia.org/wiki/Safeguard_Program
Stephenson, W. E. “An analysis of the resources used in the SAFEGUARD system software development”
Edward Yourdon  – http://yourdon.com/about/
http://en.wikipedia.org/wiki/Iterative_and_incremental_development
http://en.wikipedia.org/wiki/Agile_software_development

Our team is currently working with a client on a medium sized, medium complexity Java application which has quite low test coverage. We are introducing characterisation tests to snapshot functionality. These will give us the confidence to refactor away technical debt and extend the application without regression. One of the problems we are experiencing is the concurrent nature of the application. I have have worked on applications in the past which supported very high concurrency without issue but this application is different. I have not fully thought through why this application does differ but there are some obvious points:

• This application spawns threads in Java code a lot. In previous applications we have always avoided this complexity by utilising somebody else’s thread pool code.
• I am used to stateless service classes which operate on domain objects. The stateless service classes obviously have no concurrency issues and the domain objects can be protected using synchronisation blocks. This application seems to have a lot more stateful objects that interact (this is anecdotal, I am have not analysed the code specifically for this attribute).

One of the first refactorings we are looking at is to remove all the Thread.sleep calls from test classes. The CI server reports significant number of test failures which turn out to be false positives. In a significant number of cases the use of Thread.sleep is to blame. I have seen two slightly different uses of Thread.sleep in the test code.

1. The test spawns a thread which is calling some method of the class under test whilst the main test thread interacts with the class under test in some other way. The test thread calls Thread.sleep to ensure that the second thread has time to complete its processing before the test verifies the post conditions.
2. The class under test contains some internal thread spawning code. The test thread again needs to execute a Thread.sleep to remove the chances of a race condition before firing the asserts.

Both these approaches suffer from the same problems.

• The Thread.sleep might be long enough to allow the second thread to complete processing on one machine (e.g. the developers high spec workstation) but it is not long enough to allow the thread to complete its processing on a heavily loaded, differently configured, usually more resource constrained CI server. Under certain load situations the test fails. It works in others. The use of Thread.sleep has made the test non-deterministic.
• Often the response to the above problem is to make the sleep longer. Yesterday I saw a very simple test which took over thirteen seconds to execute. Most of that test duration was sleeps. Refactoring to remove the sleeps resulted in a test that executed in 0.4 seconds. Still a slowish test but a vast improvement. The last application I worked on had 70% coverage with 2200 tests. If each one had taken thirteen seconds to execute then a test run would have taken almost eight hours. In reality that suite took just over a minute on my workstation to complete. You can legitimately ask a developer to run a test suite which takes one minute before every checkin and repeat that execution on the CI server after checkin. The same is not true of a test suite that takes eight hours. You are probably severely impacting the teams velocity and working practices if the build before checkin takes eight minutes. There are very few excuses for tests with arbitrary delays built into them.

To resolve both issues we introduce a count down latch.

Where the test spawns a thread, the latch is decremented inside the spawned thread and where the test code had a sleep a latch.await(timeout) is used. We always specify a timeout to prevent a test that hangs in some odd situation. The timeout can be very generous, e.g. ten seconds where before a one second sleep was used. The latch will only wait until the work is done in the other thread and the race condition has passed. On your high spec workstation it might well not wait at all. On the overloaded CI server it will take longer, but only as long as it needs. A truly massive delay is probably not a great idea as there is a point where you want the test to fail to indicate there is a serious resource issue somewhere.

Where the class under test spawns a thread (an anti-pattern I suspect) then we amend the code so it creates a latch which it then returns to callers. The only user of this latch is the test code. Intrusive as it is, it is often the only way to safely test the code without more significant refactoring.

There are some larger issues here. Is the code fundamentally wrong in its use of threading? Should it be recoded to use a more consistent and simple concurrency model and rely more on third party thread pool support?

At risk of straying from my comfort zone of simple, pragmatic, software delivery, deep down I have never been very happy about the implications of complicated multi-threaded code and automated testing. You can write a class augmented with a simple and straightforward test class which verifies the classes operation and illustrates its use. You can apply coverage tools such as Emma and Cobutura which can give a measure of the amount of code under test and even the amount of complexity that is not being tested. I am not convinced it is always possible to write simple tests that ‘prove’ that a class works as expected when multiple threads are involved (note I say always and simple).

I do not know of any tools that can give you an assurance that you code will always work no matter what threads are involved. Perhaps a paradigm shift such as that introduced by languages such as Scala and Erlang will remove this issue?

There is some good advice available regarding testing concurrent code and I am sure lots of very clever people have spent lots of time thinking this through but its certainly not straight in my head yet.

Quality has always been a “hot” subject in software engineering, and several good development practices used in Agile development, unit testing for example, have been created to improve the quality of the softwares developed.

Programme Managers, Project Managers and Assurance Quality Managers haven’t just been looking for techniques to improve quality, they’ve also been looking for ways to monitor the quality of the projects and the application of these techniques.

This has lead to the development of numerous tools to check the unit testing coverage, the style of the code and its compliance to well accepted rules. Plus, of course, dashboards that gather all that information and present it in a user friendly way.

One of the most commonly used metrics is the code coverage of the unit tests. We often see in Agile teams rules like, “the line coverage should at least 80%”, or “the branch coverage should be at least 70%”. But…herein lies the hidden trap.The code coverage is not enough to verify the quality of the tests. Recently, I saw some unit tests that were exercising the code, but did not have real assertions. The test would only check that the data received was not nil. Something like:


/**
* Example of a useless test.
* As Mashooq Badar explained in a previous entry
* of this blog, tests should have explicit names.
*/
public void
testThatTheCodeRunsWithoutCheckingTheReturnedValue() {
// Prepare Context
// create mock object for instance using Mockito
AnInterfaceToMock mockObject =
mock(AnInterfaceToMock.class);
// Create instance of class under test
// and inject mock objects
ClassUnderTest anInstance = new ClassUnderTest();
anInstance.setInterfaceToMock(mockObject);
// Run code to test
List result = anInstance.operationUnderTest();
// Assertions
assertNotNull(result);
}

This test is almost useless, as it does not check the list of data it received. We do not know whether or not the ‘operationUnderTest’ performed its job properly. The only thing this test shows is that the ‘operationUnderTest’ did not throw up an exception. A change in the code of the class under test will probably not affect this test. However, the quality dashboard will present a healthy project with a high level of code coverage and instil a false confidence in the managers’ minds. There is no tool that can perform that extra check, for now.

This is one of the reasons why code reviews have to be performed, published and monitored as well.

In my previous post on High Level Project Estimates, I talked about 3 points estimates to help provide an up-front estime of effort for a project. Aside from the effort estimate you also need to consider other aspects that the team will spend time on. The following are some aspects I consider with the kind of percentages (above the effort estimate) I have experienced. Note: these percentages are by no means standard or the norm they are essentially a rough guess based on the projects I have worked .

• Planning Meetings @ 10%
• Demos @ 2%
• Code Reviews @ 5%
• Retrospectives @ 5%
• Dark Mater @15%
• Bugs @15%
• New Technical Stories and Spikes @ 25%
• Change Requests (shouldn’t average more than 15%)
• Holidays and Sick Leave @ 5%

You will also need to consider Management Overheads as well as Business Analysts (if included in the team), QA/Functional Testing, Support and Maintenance (if required).

What is the best way to review a backlog? How do you ensure that it is “complete”? How do you ensure that the prioritisation reflects the business vision and goals?

When first faced with a backlog, you are often overwhelmed by the long list of userstories. The most important step is to set a context for these userstories. Are these userstories organised in a hierarchy of “epics”? This hierarchy will help set a context. But first we need to understand what these epics mean at the highest level. Do they represent a user’s high-level goals or are they merely there as a container for some loosely related stories?

When reviewing a backlog for completion it is vitally important that the stories are defined in a context. The context can take different forms depending on the nature of the application. For example if an application has a clear high-level flow that the user journeys along then the epics may be defined as activities in this flow and the userstories can be grouped under each epic representing the functionality required for this activity. This article by Jeff Patton presents such an approach. However, your application my exhibit a more random usage scenario. In this case epics representing high-level user goals may represent the best context for the stories. You can also provide references to other artefacts such as user journeys/wireframes to further enrich the context. This article by Scott Sehlhorst is an interesting discussion of setting a context for user stories.

This grouping of userstories by a context also helps to manage their prioritisation. You can individually prioritise stories within each epic and then also prioritise the epics. Note that just because one epic has a higher priority does not mean that all its child userstories are of a higher priority. You may discover that only the first few userstories can provide enough functionality that further work on that epic is of a lower priority then working on another epic.

Thanks to Rachel Davies for sharing her approach to constructing a diagram of effects.

Rachel proposes that the diagram of effects can trigger a team to discuss how a variety of issues relate and goes on to highlight advice from Bas Vodde and Craig Larman in their first book “Scaling Lean and Agile Development”, the First Law of Diagramming is “The primary value in diagrams is in the discussion while diagramming—we model to have a conversation.”

I have been using a slightly different approach in retrospectives recently. With a similar intent to Rachel I hope to trigger conversations that result in a team sharing there concerns and issues and coming to a shared view as to how these relate and where would be a good point for intervention.

I’ve used this approach a couple of times recently when running retrospectives that cover a few months i.e. not iteration retrospectives where we choose to commit a few hours to investigating issues in some detail.

My favoured approach is borrowed from Eliyahu Goldratt’s Theory of Constraints, it is one of the 5 TOC thinking process and is know as a current reality tree. A current reality tree is a tree of undesirable aspects of your current situation, these are connected to indicate cause and effect.

My approach to constructing this diagram is as follows:

Identify undesirable effects
Invite the whole team to stand around a table thinking of undesirable / unwanted traits of their existing approach (process, tools, methods etc.). Each item should be written on an index card and placed on the table in no particular order. Team members can try to avoid duplicats but if one pops up it is not a problem.

Begin to identify cause effect relationships
Invite the team to select two related cards. These should be stuck on the board and joining with an arrow pointing from cause to effect.

Continue to grow the diagram
The whole team should collaborate to add the cards to the diagram. When I first tried this I wondered how I would cope if groups of cards were not related. I’ve not seen this actually happen yet but should it then I guess we allow for the growth of separate diagrams until we discover a relationship.

Review relationships
In CRT literature there are a set of steps for assessing the existence and relationships between UDEs. My approach is to invite team members to consider where items are missing. Often where many causes relate to a single effect there is a missing intermediate effect.

Since I tend to run this approach in a time boxed session I an looking for the point at which returns begin to diminish. As the group settle on a basic shape and begin to look for intervention points I move to support this.

A good candidate point to intervene at would be contributing to a number of effects or participating in a feedback loop. By focusing on one effect we can select actions that will contribute to reducing this effect and by association those related effects.

Of course, an added benefit of this approach is that we can use the cause and effect diagram to review the effect of our interventions; did we get it right, if not was there something we missed in the analysis?

I hope that this is useful. Do share if you’ve used this approach or something similar yourself.

I walked past two teams doing their daily SCRUM standup today. Both teams claim to be agile. I didn’t join in (even as a chicken) but just observed for a minute or so.

The first team was sitting down in a breakout area. Their body language spoke volumes. There was not one single participant maintaining eye contact with anybody else. Two people were playing on their phones. One developer had his head in his hands. Most had bored expressions. The team leader who is also the SCRUM master was the only person who spoke for the entire time I watched.

The second team was stood in a space near their desks. They were gathered round a task board which appeared to be up to date and the focus of several of the individual’s updates. One person spoke at a time. Almost everybody appeared to be paying attention to whomever was speaking. Most updates were short and concise. A couple rambled on.

Other than both teams calling their meeting a SCRUM I could see no similarities.

As our agile adoption has spread beyond the original teams I suppose it is inevitable that as the experience gets spread a little thinner that people will simply label their existing activities with agile sounding names. Often we have no clear remit in those teams to supply a mentor and to try offer advice would result in rebuttal as team leaders guard their territory. Does this matter? Is there a risk that these teams who are not practicing agile correctly will diminish and discredit agile in the eyes of our programme managers? This is sounding a bit like an excuse for an Agile Inquisition going round checking that no team is using Agile’s name in vain. This cannot be a good thing either.