Agile software development
Gyula Attila Borsós
Hi, I'm Gyula Attila Borsós, someone who loves conversing about Agile!
My full time job is being a Scrum Master, and I'm also working as an Agile Coach, delivering workshops for various companies.
I wrote this handbook as one of the tools for my workshops, a guideline for my new colleagues, a collection of my thoughts.
This is not a book, please don't read it that way. This is my interpretation of agility.
It's rather a handout that I've found worth sharing, hoping that people might get something valuable from it.
Although I'm sharing it here for free, the material itself is copyrighted.
Please ask me for permission before using my work in any form.
Feel free to contact me on linkedin: https://linkedin.com/in/gyulaborsos
History, classic models
What makes SW development special?
- Mandatory, but often difficult cooperation of technical and non-technical people
- Project goals that are hard to determine
- Unexpected challenges
- Difficult estimation, hard to foresee
- Difficult measurement of status
- Rapidly changing technologies, few established rules
- Processes that cannot happen in parallel
- Users and customers aren’t aware of technical difficulties (often neither are the developers)
- Users and customers have a hard time translating their needs to requirements
- Users and customers often change their minds, as they cannot foresee how their requirement will work when implemented
History of software development processes
Sequential model, not yet iterative. The steps strictly follow each other and the time between each step is huge.
Pretty old, it’s used since the 70s, and is still used by third of the software development companies.
Keywords: requirements, planning, deadlines, budget, documentation
Defines the steps before and after coding:
- Collecting requirements
- High level analysis (models, schemas, rules)
- The customer does not always know the valid requirements before seeing the software in action
- Problems that typically emerge during development should be foreseen at designing
- Very hard to introduce changing requirements or resolve design problems in later phases
- The software’s very first version gets ready and is released only at the end of the flow
Half-done, semi-working application versions, sketches, UI designs, these can all be considered as prototypes. Of course the best one is semi-working.
Before the final product, we can give something concrete to the customer, something they can touch and play with. Feedback can be gathered and, even from a technical point of view, the prototype can act as a proof of concept.
Not yet iterative, and not really a method, just an approach.
Creating features as small increments with each increment implemented during a small waterfall process. Finally introduces iterations.
Risk-driven iterative. Iterations typically take months or even years.
It’s iterative since the first step with all its benefits. Even though at first glance the only difference to Scrum is the length of the iteration, the purpose, motivation and the paradigm between them are vastly different. The main goal of the spiral model is to evaluate at the end of each cycle whether the project is going to succeed or not. If not then it should be cancelled. The “fail early” principle is kinda introduced.
Garage development or overcomplicated practices with unclear roles and responsibilities, often unclear priorities. If there is no framework to keep yourself to, then people work on developing the process instead of developing the software.
- Iteration is effective, but the bigger the timeframe is, the harder to react
- We have to realize that the customer should not be expected to specify its detailed needs early
- It’s worth involving the customer during development
- Showing the customer what he will get works and has many benefits
- It’s hard to plan too much ahead, software development is a profession where it’s not always possible
- If we are developing the software in small pieces, we have a product that can be on the market already, and gets more valuable with each addition
The modern approach
Manufacturing method of Toyota (not related to software development) that had a huge impact on what we call Agile today.
- Reduces waste
- Continuous development/improvement
- Team oriented
- Values quality
- Values improvement of human resources
- Promotes revealing the problems in order to solve them
- States that the satisfied customer is most important
Reducing waste means the reduction of anything not contributing to customer value. Unfinished work and unused features, software bugs are clear examples, but unnecessary processes and task switching also cause waste, non-development activities of developers likewise. You can’t really have zero waste, and some waste should NOT be eliminated, but the approach is to keep an eye on these, and reduce what you can.
Amplify learning by having short iterations from which you can learn quickly. Involve the customers and learn directly from them instead of collecting requirements. The mantra is: Build, Measure, Learn.
Decide as late as possible. Postponing decisions to latter phases often reduces ambiguity of the choices, having more clear information on their impact. Working based on facts over assumptions reduces chance of taking the wrong path. Making unnecessary decisions takes time and creates waste. While it seems like lack of planning, it’s far from that. The principle is rather a habit to always ask questions like: “Do we need to make this decision now?”, “Is it worth making the decision after learning a bit more?”.
Deliver as fast as possible. Learn by doing, get your hands dirty. Fail as early as possible to minimize waste and learn from it early. Put the software to production early to learn from the circumstances. Involve the customer as early as possible to learn from their experience.
Deliver the most valuable features.
Empower the team. They should have all the skills to make proper decisions on their level and in their profession. Being able to make their own decision improves their engagement and increases their responsibility.
Build integrity in and focus on the overall experience. Minimize flawed functionalities, bugs, and technical debt. Integration should always be tested and kept.
See the whole. By empowering the team, let them act and manage them as a “whole team”. Collective ownership should get to the level of collective code ownership. Have the same expectations for the whole software. Bring development and operations closer together.
Short statement consisting of a few sentences and a collection of principles, not a specific working method. It was made in 2001, by 17 “legendary” lead developers who got together to discuss lightweight development methods.
Not a framework, it won’t give a workflow. Not a toolset and doesn’t tell HOW to achieve what’s stated in it. Still whatever we will do, we will point back here. Understanding these principles has a key importance and the base for understanding Scrum. As the saying goes nowadays, “Stop doing Scrum, start being Agile!”
“Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.”
- Individuals and interactions over processes and tools
- Pair programming
- Interaction with the customer
- Lightweight processes
- Working software over comprehensive documentation
- The working software is the base of all communication with the customer
- Don’t just show documents, don’t just show images of screens, don’t just talk about it, but
- Give them opportunity to try it and use it regularly
- Get feedback, as early as possible
- Get feedback from the end-users
- Documentation should not be against changing requirements
- Customer collaboration over contract negotiation
- Requirements cannot be FULLY collected at the beginning
- Customers, stakeholders, end-users should be involved
- Responding to change over following a plan
- Changing requirements and software development often goes hand in hand
- Don’t fight it, accept it and adapt to it, provide a solution for it
- Change does not mean change in the requirements. Respond to technical changes, team changes, environmental changes.
- Observe yourself, measure, learn from to achieve
- Continuous improvement
More specific and easier to digest than the manifesto.
- Customer satisfaction by early and continuous delivery of valuable software
- Welcome changing requirements, even in late development
- Working software is delivered frequently (weeks* rather than months)
- Close, daily cooperation between business people and developers
- Projects are built around motivated individuals, who should be trusted
- Face-to-face conversation is the best form of communication (co-location)
- Working software is the principal measurement of progress
- Sustainable development, able to maintain a constant pace
- Continuous attention to technical excellence and good design
- Simplicity - the art of maximizing the amount of work not done - is essential
- Best architectures, requirements, and designs emerge from self-organizing teams
- Regularly, the team reflects on how to become more effective, and adjusts accordingly
Elaborating on the principles
Having the customer involved since the first second and keeping them involved continuously. Delivering regularly what the customer deems as the most valuable. Welcoming changing requirements, seems these as the evolution of the project and a natural part of software development. Being prepared to changes and reacting to them even in the later phases.
The software should be working through the whole time during the development, from the very first second. Including unfinished features. The working software should be working in the production environment. To support that, encourage working in production-like environments.
Delivering features one-by-one, fast, as soon as they are ready, and by this giving competitive advantage for the customer.
Software not working has no value. As soon as it becomes not working, even during development, we have to fix with the highest priority. (Note: this does not mean jumping on all small impact bugs when we find them). This means that, if due to a developer mistake, the software cannot be built, because the compilation fails, then at that moment the software’s value is zero, so the most important task is to bring it back to working. In practice it also means that realizing the mistake made in the last few minutes is easy and fixing that is often trivial because of its freshness.
Development team and servant leadership
Give them the tools and environment they need for their work? and they will be able to use their stills at max. Hire motivated people with great skill sets. Trust them and leave them to do their job.
Don’t rely on micromanaging the team, let them self-organize. They will arrange the work in the optimal way. Let them realize and fix their mistakes in retrospect.
Keep work-life in balance. Don’t ask them to work overtime, it will lead to decreased productivity and morale.
Let them decide the pace they are comfortable working so they’ll make less mistakes and will be able to deliver quality over quantity.
Have all of them in line with the current needs and the foreseeable plans.
Efficiency and simplicity
Avoid over-engineering and working ahead, these are waste
Have something ready by the time it’s needed, not before
Improve efficiency of the team by retrospectives
Any feature, even the small ones that are not valuable for the clients are potential waste
Lightweight documentation and requirements, favoring daily communication over “being written down”
Co-location and face to face communication are keys
Avoid over-engineering and working ahead, these are waste. Have something ready by the time it’s needed, not before. Improve efficiency of the team by retrospectives.
Have people co-located to improve their efficiency. Favor communication over documentation. Communication should be face to face, in person.
Continuous, multi-level feedback loop
To achieve agility
The manifesto and the principles are great! Moreover, they are beyond great, we refer them all the time even when doing Scrum. But HOW to achieve these are not really stated in the principles.
The points are pretty vague and ambiguous, the way to achieve them is not trivial at all. We need a framework which works according to these principles, that’s when Scrum enters which, in itself, is still not enough. Toolsets like XP and techniques such as Continuous Integration or Continuous Delivery are needed.
Keep in mind that it’s a paradigm shift which is always hard. Also, viewing the principles only, it’s quite hard to realize WHY these would work at all.
Scrum is an agile software development framework.
Let’s take a closer look! What’s a framework? There are many kinds of frameworks, software architecture frameworks, web application frameworks, etc. The term always means that it gives a frame, usually default flows or processes and we have to build our own operations INTO the framework.
Scrum is just like that, it defines processes, workflows, artifacts, gives a frame to the daily work. That’s it. People often think of a downloadable “something” based on the framework term, but Scrum is just a bunch of documents.
Agile, as we will see, by correct usage, helps us reaching what the manifesto asks for.
Last but not least, software development. While agile principles (which are still software development principles) CAN be used by other crafts and professions, Scrum is specific enough to be a strictly software development framework.
It’s important to understand that Scrum isn’t a standard. It wasn’t created by an organization or institution, “just” a bunch of very talented people who were lead developers themselves. It’s also an evolving concept, artifacts that are key nowadays weren’t even included in the original paper. Also there are several interpretations but fortunately the core Scrum is pretty clear.
Besides defining the roles, it defines the steps of the iterative development process. General enough to be usable in all kinds of domains and cases, but specific enough to give answers.
The focus is the software development process, improvement, delivering value, and relationship with the customer. Tasks prior and after software development (eg. business plan or operations) are mostly out of scope for Scrum and it won’t give answers to these. Also excluded are the methods of fitting it into an organization, eg. aligning it with managers, managing software support, etc.
Scrum is empirical, meaning that it states that knowledge comes from experience, and
making decisions should be based on what is known. It’s an iterative, incremental approach
to optimize predictability and control risk.
Three pillars holding this empiricism:
Transparency that states that the process must be visible for responsibles, who should have a common understanding of what they are seeing.
Inspection, asking for frequent inspection to detect undesirable variances.
Adaptation, telling that adjustment must be made as soon as possible to minimize further deviation.
Scrum values the following:
- Committing yourself to the team and Sprint Goal
- Willing to change
- even if that means accepting that you are wrong
- if your opinion is not in line with team’s opinion
- Focusing on keeping the customer happy
- Being focused on the sprint and its goal
- Telling everyone everything about all your work
- Letting stakeholders observe
- Helping others to learn what you are good at
- Not judging the things that others aren’t good at
Quick introduction to the roles
Scrum defines the following three roles in details:
- Product Owner
- Development team
- Scrum Master
The Product Owner or PO represents and is the voice of the customer and the stakeholders.
He/she ensures that the software (or product increment) being delivered by the team always holds the most valuable features for the customer. Achieves it by involving the customer, learning their needs, and composing so-called User Stories from the perspective of end-users of the customer. These are going to be the development item, which he will prioritize and order in the so-called product backlog. He’s working close with the development team and gives answers to functional questions continuously.
One Scrum team contains one Product Owner, to ensure that the tasks and their priorities are clear for the team.
The development team consists of 3-9 people who are responsible to deliver the User Stories (chosen by the order given by the Product Owner) in the sufficient quality, by the end of each iteration.
Their task is (based on the agreements of what the quality needs are) to analyse, design, implement, test, and document the items. They are cross-functional, so as a team, they should be capable to do all the aforementioned work.
Like the functional decisions are up to the Product Owner, the technical decisions are up to the team. They are empowered to make choices to reach the goals in the most efficient way.
The last role is the Scrum Master who acts as a servant leader and coach within the Scrum Team, making sure improvement, processes and Scrum rules are kept.
The Scrum workflow briefly
Scrum, being iterative, runs in cycles. The development happens in iterations, which Scrum calls sprints. Sprints are 1-4 weeks long, having the length of the sprint kept long term.
We will talk about 2 weeks long iterations, length of the meetings are scaled accordingly.
One iteration consists of the following steps:
At the beginning of the sprint, a sprint planning takes place, which is usually timeboxed for ~4 hours. Based on the priorities set in the product backlog - which is a dynamic list that contains everything that will be or might be needed in the product - the team defines the goal of the upcoming sprint. They pick the items that the team commits to deliver.
It’s a two-way commitment, the team commits to deliver these, while the PO commits NOT to change the priorities and the content during the sprint. This arranged content, this set of items picked from the product backlog are moved to the so-called sprint backlog. The meeting ends with the developers breaking down these items to smaller tasks, and planning the execution of the sprint. The sprint is started.
The development team every day, at the same time, usually in the morning hours get together for the daily standup. They share their achievements and plans with each other, plan the day, arrange and assign the leftover work to do. They take actions to be able to deliver the work they are committed to. Of course this should not be the exclusive place and time for communication, they always have to be aware of how the sprint goes. This is rather a checkpoint, where they stop for a bit, look around and adjust if needed.
The last day of the sprint approaches. If the sprint is successful, the so-called potentially shippable product increment is available. The sprint is successful if all the items are done (according to the quality standards). The sprint closes with two important meetings.
First is the sprint review, where the team shows the stakeholders the product increment. Customer can be involved as well. The PO gets feedback, the past and planned work of the team is clarified.
The second is the retrospective meeting where the Scrum team looks back to the past weeks and evaluates themselves. The goal is to realize rooms for improvement, and take actions according to these, applying these already in the next sprint.
One other meeting happening in-between, during the sprint is the backlog refinement, which is a preparation meeting for the sprint following the current one. Here the upcoming items are cleared up and the team gives estimations.
In these few pages we defined the core of Scrum, of course many more terms and elaborations are needed to fully understand it.
Product Backlog Items (PBI), backlog, sprint
User stories are short user needs, written from the customer's perspective. It must contain the following three: who wants what and why.
As a ….. I want to…. so that….
As a ROLE, I want GOAL, so that BENEFIT
As an administrator of the user management system, I want to have a new overview of the disabled users, so I can easily find them and re-activate them.
As a user I want to achieve something so I have some benefit.
In the “who” part the Product Owner states whose perspective he is talking in.
The “what” part seems simple, as it states what the user wants, right? Nope.
The user in fact wants to gain the benefit written in the “why” part and the “what” is already a solution, an idea to get that benefit.
The User Story seems to work without the why part as well. However, if we skip it then we would blindly develop what the user asks for, without understanding their perspective and their goals. Moreover, we might find a better and more efficient way to achieve that benefit.
Let’s look at the example. The user asks for a completely new screen. But in fact all he needs is a method to find these users easily. If we already have a screen dealing with user management, we might say:
“What if we just add a filtering option to view disabled users to the already existing users overview? The software would remain simple, we don’t want to add yet another screen which serves one single purpose. Moreover, the administrators already know this existing screen. Last but not least, it takes half the time to develop it.”
The collective knowledge the Scrum team has resulted in a better and cheaper solution. It only works if everyone understands the goal the user wants to achieve.
This is one of the reasons why this simple, few lines long description might be more efficient than several pages of specification. The other important factor is that the more detailed a specification is, the less space we have to improve simplicity. The more detailed, the more you are guided in your solution. You can’t give a quick but satisfying solution if you are narrowed down by unnecessary details.
Trust the developers and ask them to put the button on screen, they will put it where it’s the easiest to put, hence the cheapest time-wise. If you ask them to put it to a specific place, it might be a place eg. outside of a UI component, resulting in a huge refactor and without your knowing, in an expensive solution. If the expensive solution doesn’t give more value to the customer than the cheap one then you are making waste.
This is the reason why agile places prefer short descriptions, stripping down initially anything they can.
One important factor is that the User Story should be small enough. The theoretical maximum size is if it fits a whole sprint by the whole team is working on it. Of course it’s far-far from the ideal size, which is a few days worth of effort.
Follow the INVEST approach
Independent - The PBI should be self-contained, in a way that there is no inherent dependency on another PBI.
Negotiable - PBIs are not explicit contracts and should leave space for discussion.
Valuable - A PBI must deliver value to the stakeholders.
Estimable - You must always be able to estimate the size of a PBI.
Small - PBIs should not be so big as to become impossible to plan/task/prioritize with a certain level of certainty.
Testable - The PBI or its related description must provide the necessary information to make test development possible.
Pros of using User Stories compared to detailed specifications:
- Small enough that it’s ~few days worth of work, but already creates value
- It encourages conversation
- It encourages understanding the user point of view
- Everyone can understand it, developers and mortal people as well
- Not a huge specification, so it’s easy to be maintained
- “One time usage”
- Encourages splitting a big feature to small parts
- Encourages the valuation of each small part
- Relatively easy to estimate
- Forces the developers and users to use the same language
Scrum does not specifically ask for User Stories as items, BDD style scenarios also work.
Given [initial context], when [event occurs], then [ensure some outcomes].
Scenario: Refunded items should be returned to stock
Given that a customer previously bought a black sweater from me
And I have three black sweaters in stock
When he returns the black sweater for a refund
Then I should have four black sweaters in stock
Definition of Done
Definition of Done, in short DoD, is a list of criteria all User Stories should satisfy in order to be considered Done by the PO. A simple yet powerful tool to drive the project to our quality needs. As one product has to have the same quality standards all over, one product should have one and only one DoD.
These criteria are so evident for the project that it would be a waste of time to state them in each of the stories. Eg. automated tests, documentation, code reviewing. Anything needed for the item to be considered as truly delivered. Keeping in mind that no external dependencies should get introduced. That would mean that delivering the item is not 100% in the team’s hand.
Both the DoD and the DoR (in the next chapter) are arranged by the Scrum team during negotiation, according to the external needs. One entry can have huge impact, so it’s mandatory to understand what each need results in for the team.
Eg. if we put into the DoD that the items have to be deployed to the production environment, then we demand Continuous Deployment. Which is a powerful tool, but gives a huge and complex overhead initially for the team. If our goal is indeed Continuous Deployment, surely it has to be put in the DoD, but if not, we are just generating waste by having demands that hold no value for the customer.
DoD criteria examples:
- The development provides the value stated in the story.
- Acceptance tests are fulfilled based on the test cases.
- Automated unit tests and integration tests are developed for the feature.
- The item does not break any existing tests and the build in the Continuous Integration.
- The Product Owner accepted the item.
- The item was developed in pair programming.
- A senior developer reviewed all created code.
- The new feature was documented in the product documentation.
- Customer care is informed about the new feature.
- The feature is deployed (in a hidden way) to the staging environment.
- XYZ non-functional requirements
Definition of Ready
Similar to the DoD, but from the other way around. These are the criteria each User Story should satisfy in order for the item to go into a sprint (as an open item). Similarly, a quality standard and important both for the developers and stakeholders to guide the PO regarding their needs.
DoR criteria examples:
- The Product Owner confirmed with the stakeholders/customer that the feature is valuable for them.
- The PO and the developers discussed the item during backlog refinement, having a common understanding regarding the purpose.
- Item is estimated and the estimation is in line with the value the item provides.
- Common agreement regarding HOW the value will be delivered, WHAT will be developed.
- BDD-like Scenarios should be written in the stories.
- For bugfixes, the steps to reproduce them are clear and added.
Bugfixes in this context mean fixing bugs unrelated to current sprint items. Surely if you are just developing something and it’s buggy, it’s gotta be fixed. These are bugs that got into the system some time, just being realized and should be up to fixing depending on the circumstances.
Scrum does not separate bugfixes from regular User Stories and asks to manage them in the same way. Bugfixes have to be prioritized, estimated, planned just like all items. Both the DoR and DoD applies to them.
As a PO and SM we have to take into account that estimating bugs are pretty hard. It sounds silly, but fixing a bug has two phases: finding what causes the bug, then fixing it. If we already know what causes the bug then it’s easy to estimate how to fix it. But if we aren’t even aware of what causes it, estimating both finding it and fixing it is really hard, especially because you have no idea yet whether it’s a small fix or a huge architectural issue.
A technical debt got into the system during development and gave short term benefits, but is not sufficient on the long term, and sooner or later it will come back and bite us, so we gotta fix it.
There are several ways these can get into the system. It can be a strategic decision by the Scrum team to take a shortcut to meet an important deadline. Can be negligence by a developer, which also means improper processes that cannot catch this negligence. Can be unrelated to the current team, eg. a lack of test coverage on a legacy system. Any of the following:
- Bad design
- Bad architecture
- Unfitting libraries
- Insufficient test coverage
- Too much manual work
- Bad development practices
- Missing development and coding standards
- Lack of general senior knowledge
- Lack of knowledge of a certain functionality
We must deal with the important ones, otherwise they slow the development down. They can get to an “unpredictable tipping point”, after getting to a critical mass and will erupt. Then small changes will cause major uncertainties and bugs in presumably unaffected areas.
As these items do not change any functionality, but change technical things, these usually come internally from within the development team.
- Unmaintainable code (spaghetti code, copy-paste code, magic numbers, dead code)
- Missing development by-products (increase test coverage, documentation, internal drawings, etc.)
- Manual work (automations)
- Lack of knowledge (areas unfamiliar, lack of knowledge regarding technologies, increase common knowledge)
- Slow development (unmaintained architecture, out of date libraries)
People often say, “But my team has to deal with many-many other things.”
The following should be thought over:
- Are these tasks related to the flow of the development?
“My team gets together every Friday and checks all produced code together.”
Code review should be part of the DoD. When the code review happens for supervising purposes, the whole team should not attend.
Code review for learning purposes should be separated.
- Are those tasks truly for the development team to do?
“Every Monday morning the development team sits together with the management and talks about future marketing ideas.”
Time to realize that the developers can spend their time on tasks more suited for themselves eg. development. If marketing needs someone close from development to help, the PO might participate in these meetings.
- If the team’s involvement cannot be avoided, give it a structure.
“Developers often travel to foreign offices, which takes 2-3 days.”
Either add a placeholder story for this activity when it happens, or simply reduce the amount of items the team commits to deliver (velocity).
- Involvement cannot be avoided, but also cannot be planned.
“Developers have to help in second line support. These activities can’t wait for weeks to be planned in, these need action at that very moment.”
It might indicate that the team is interrupt-driven, thus Scrum might not suite them. Still, a timeboxed User Story, or a team-wise agreement in managing these to a certain extent can solve these easily.
Rates of PBIs
You have to see everything in the vein of how much value it gives to the customer. Software developed in the agile way might have less bugs due to quality standards and practices, but it still happens. Fixing these bugs has to be viewed from the value perspective as well.
Fixing bugs affecting all users and eg. causing data loss of course hold huge value. An “annoyance” kind of bugfix affecting a minority of users also holds value, but much much less. Consider comparing the effort needed to fix that bug to a development adding a fancy new feature to a system, from which all customers gain benefit. The latter might be the better choice.
Legacy softwares tend to have hundreds if not thousands of known bugs. Stopping the addition of new features to fix all of these might not give the best value.
Still, bugs and technical debts happen, and should be dealt with to some extent. It’s recommended to gradually work on these, eg. in the following ratio: User Story 80%, Bugfix 10%, Technical Debt 10%
Lifecycle of an item
Responsibilities of the Product Owner
- envisions the future of the product
- collects feedback from stakeholders
- maintains a prioritized list of development items
- evaluates priority of items based on their business value AND estimation
- ensures that the Product Backlog is visible, transparent, and clear to all
- able to show what the Scrum Team will work on next
- works closely with the team during the sprint
- to answer questions which come up during development
- to discuss and make an agreement if the scope of an item slightly changes
- to validate that the delivered functionality is satisfying
- makes decisions on functionality related questions
- gives non-functional requirements
- defines the DoD together with the team
- changes the scope of a “ready” item to lower the estimated effort
- does not make technical decisions
- does not negotiate estimations
- but should collaborate with the team to change the scope of an item to reach a suitable estimation
PO is the only channel where work goes in for the development team. The Product Owner is one person, not a committee. He may represent the desires of a committee in the Product Backlog. Those wanting to change a Product Backlog item’s priority must address the Product Owner!
The organization must respect the PO’s decisions, which should be visible.
Responsibilities of the development team
- self-organizes to achieve the sprint goals
- accountable as a team
- collaborates with the PO regarding delivering the functionality with the minimal effort
- ensures that the development environment of the team supports high quality software development (eg. CI, CD, sandbox)
- communicates with external people (both inside and outside of the company) in order to finish items
- notifies the team and Scrum Master if its work is blocked
- overviews the sprint and the items
- members report their status to the other team members
- does not negotiate with the PO about changing the sprint content
- does not make decisions alone which would affect the team alone
The team has to be able to work independently and administer their own work to achieve their own commitment. It’s often swept under the rug, but it won’t necessarily work with a team full of juniors, or at least the Scrum Master need much bigger control over them.
A team with some experienced, senior developers, that are united with several juniors, will work well. It’s not necessary for each individual to be an expert, but some have to be.
The team, as a black box has to be able to deliver the items according to the DoD. Outside of the team, it’s not their concern how the team achieves it.
It can happen by having everyone as a generalist in the team, but then the lack of true experts in areas will hurt. Other way around, if everyone is a specialist, they will be able to cover all required expertise, but won’t be able to self-organize properly, there will be too many internal dependencies. None are optimal.
The recommendation is to have individuals with T-shaped skillsets, meaning that they are experts in one or two areas, but have a wide range of knowledge to feel comfortable in any area.
The following helps in achieving self-organization of the development team:
- clear boundaries
- certain freedom
- total freedom within the boundaries on HOW to do things
- trust in them
- authority to self-manage themselves
- let them learn by themselves
- accountability and responsibility as a team
- longevity of the team
- competency to do the work
- let them know the WHYs
- give them a compelling mission
Responsibilities of the Scrum Master
- doesn’t assign tasks (dev team)
- doesn’t make decisions regarding technology, implementation (dev team)
- doesn’t make decisions regarding functionality (PO)
- doesn’t command people what to do and how to do
- of course it can be useful to ask for his OPINION
- doesn’t exclusively communicate with other departments
- doesn’t exclusively communicate with external parties
- enforces the rules of Scrum
- ensures everyone acts in their role within the Scrum team
- organizes and facilitates the meetings
- ensures that development items are managed according to the DoD
- distinguishes Scrum rules from company rules and team agreements
- very important for the parties to know if they themselves can change a process or if they have to ask someone else to change it
- authority to change the applied Scrum processes (eg. iteration length)
- authority (together with the PO) to make small changes in the content (should be exceptional)
- ensures the awareness of processes for people outside the Scrum team
- ensures that the purpose of Scrum, it’s workflow, roles and values are clear both inside and outside the Scrum team
- unbiased agent of both the dev team and the PO
- unbiased opinion within the development team
- helps the PO in maintaining the product backlog
- helps in effective backlog management
- coaches the PO to maximize value
- coaches the Development Team in self-organization; cross-functionality and organizational environments that are not agile yet
- removes impediments so the team can focus on the items
- “an issue only becomes an impediment if it can’t be unblocked through the authorizations of the (development) team“
- do NOT solve impediments until they are visible for the team
- often it’s about helping the team to realize how they can solve the impediment themselves
- overviews the sprint and the items, evaluates execution to help the team deliver the whole
- ensures that the team does not fall back, but instead keeps improving by keeping the retrospective action points
- teaches the team and outside the team: agile, process awareness, sharp on missing tech skills, generates new ideas for improvements
- sometimes has to step out of the Scrum Master role and take the lead if it is recognized that the Team got stuck, but in general the Team has to recognize when it is in trouble
- visualize and manage risks (e.g. checklists; like the release template)
The Scrum Master is Servant Leader, Coach, Facilitator, Teacher, Mentor, Manager (of impediments, processes, boundaries, culture), Change Agent.
He’s not a Scribe, Secretary, Boss, he isn’t Scrum police, tool and technology administrator, Sprint Board Owner, Daily Standup Leader.
Pigs and chickens:
By maintenance we mean that the Product Backlog reflects the current priorities and the items are prepared in a level that is reasonable for their priority. This usually means clarifying the most important tasks, estimating them, making them satisfy the DoR. The most important forum of Scrum for this activity is the backlog refinement meeting which often includes giving estimations.
The Product backlog is an essential part of the Scrum process. It should be accessible by all for viewing the priorities. But how items get to the backlog is up to the PO. A badly managed or undermanaged backlog will cause the team to make unreasonable commitments and would deny the stakeholders to review the true state of the project.
It’s very important to have one single backlog, and that one single backlog should represent the priority order. POs often make the mistake that they have several backlogs. While it’s okay to have several physical backlogs for the sake of managing hundreds of items, but it’s key to have a single logical order. You can have a list of bugs, can you also prioritize them among each other, but in the logical single backlog, each bug should stand at its true place of priority.
It’s recommended that the top of the backlog should consist of at least 2-3 sprints worth of items that satisfy the DoR, to ensure the that short term plans are in line with the company goals. Of course a newly found critical bug can jump to the top, but should satisfy the DoR as soon as possible.
The lower we go, the less priority items we find, which should NOT be specified as much as items on the top. That would mean we are potentially creating waste. It’s okay to talk about upcoming features or epics, future company goals, but only in detail that is necessary.
The refinement is found to be an essential part of Scrum nowadays, with teams find it worth spending more and more time on refinement. Interestingly, it wasn’t part of the original core Scrum practice.
Backlog refinement or backlog grooming usually takes about 2-4 hours for a 2 weeks sprint and its goal. It’s an ongoing meeting, held in each iteration. Just like the planning or a retrospective, it should always take place, as in practice there is always room for improvement in the backlog. Although not strictly related to the current sprint succeeding, one of its main goals is to make sure that the very next sprint is prepared.
Early in the meeting, the goal of the next sprint should be discussed, then the foreseeable few next as well, to check if there are dependencies or barriers.
Prioritization happens at the start of the refinement meeting already, as the items are presented to the team in the order of their priority. We start from the top. If the item satisfies the DoR already and no one has any remarks, we move to the next one. If an item does not yet satisfy the DoR, we address the missing points. Estimations can take place. If based on these the priorities change, the PO can immediately make that change in the backlog.
If the top is not good enough then it is the most important role of the meeting to arrange that properly. The team should not be bothered with a future vague item if the next sprint is still unclear. If we are good with the top then we can take time for talking about later items, even future features.
The top items are User Stories, Bugfix items, Technical Debts, so specific todos. These should be clear, estimated, ready to be worked on. Some might even be broken down the tasks. Below there are still same kinds of items, but with rough estimation. Even lower only the goal of the items are talked about. Below these can be upcoming features, epics, rough sketches. This is the graininess or granularity of the backlog, fine-grained on the top, heavy on the bottom.
Note, if the estimation happens separately, the role of the refinement meeting is to answer the WHY and WHAT questions, the HOW questions only happen if they affect the estimation.
During estimation, the team gives an estimation that they all agree on regarding the cost of the item. It is an estimation, not a commitment, but every estimation that is off is a possibility to learn.
Even during the estimation, the items are not specified in extreme details. Aspects not affecting the estimation should not be discussed (we count these are waste). For the estimation it doesn’t matter if that button is yellow or red. It does not matter if it’s on the left or the right side. Only talk about it if it DOES matter, eg. because it will float out of a component and would make it much more difficult. Most of the questions not affecting the estimation can be discussed with PO during development. Remember the “decide as late as possible” principle.
Estimations should be accurate, but not precise. Whether something is 3 or 8 matters. Whether it’s 3.0 or 3.2 does not.
- The whole team will provide a common estimation, based on mutual agreement
- Every developer in the team takes part (the junior, the senior, the new guy, etc.)
- Estimate as “how much would it take for the team” instead of “how much would it take for ME”
- The PO does not participate
- The Scrum Master does not estimate, but moderates the meeting
- The estimation should be worked out until the agreement is reached
- Items on the story level have to be estimated (tasks and features are recommended at certain phases too)
The best and commonly used method for estimation is the so called planning poker. Each member of the team gets a pack of these cards, which represent a Fibonacci-like sequence.
By default the numbers do not have any unit. The units are often translated to hours (in absolute estimation) or an artificial story point unit (in relative). Third alternative is T-shirt sizes eg.: S, M, L, XL, XXL. These different types of units should be used in different phases of the estimation.
In the first round, everyone estimates “blindly”, without being affected by others. Before this round it’s strictly forbidden to tell a number or even imply it. People choose their own cards (only values on the cards, no in-between values).
They put down their card, face down, and when everyone is ready they flip them at once. The team observes the cards. If all agree then they chose the estimation. Otherwise, people giving the lowest and highest estimations should speak up.
We repeat this until the agreement is mutual, with the Scrum Master moderating.
- Each individual is encouraged to think about the item, they must understand it, otherwise they won’t be able to estimate it themselves
- Everyone is involved
- Common estimation, common responsibility
- One talkative guy won’t dominate the others
- One overcomplicating guy won’t mislead them
- Anyone with a good idea that lowers the effort needed can speak up
- The distance between the cards makes sure that we don’t get lost in the details
- remember, it’s not important whether it’s 10 or 12 hours.
There are two main methods, absolute (time-based) and relative (point-based).
With time-based estimations, we define a unit assigned to the pack or cards, eg. hour or day or half-day. If we pick hour as a unit, it simply means that the card with 2 means 2 hours, while the 5 means 5 hours.
- Easy to understand the concept
- We are used to time based estimations
- As our work is measured in hours/days
- Easy to timebox
- Devs won’t foresee any potential problem that may arise and won’t calculate it into the estimation
- How do you measure something that might be 1 hour or 10 hours
- During development, we are focusing on the time being spent and time being left over
- while we should focus on the doneness of the task instead
- Time-based estimation method usually results in time-based sprint planning
In the relative way, the numbers are just ratios. The relationship between 2 and 5 is only that 5 is “more than two times more” than 2 (sometimes even omitting this small meaning). But then what does these mean?
First of all, relative estimation does not only take “time” into account.
The equation often recommended to be used is P= f(E,C,R).
Effort/repetition: It does not take too much brain, but takes many lines of code
Complexity: It won’t yield much code, but needs thorough designing, experimenting, figuring out a complex algorithm, etc.
Risk: uncertainties and ambiguities.
Often reference items are defined early in the process, so there are predefined and commonly agreed items which we have to compare new items to.
- No time-based accountable “promise”
- Items are measured against each other rather than in time
- Paradigm shift: The importance is the doneness of the item, not the time spent on it
- Sprint planning is more commitment-based rather than time-based
- by committing to a set of stories
- Result of the estimation is easier to be discussed due to the 3-factors
- Estimation process is smoother and faster
- Hard to understand at first
- Reference items should be specified
- Hard to timebox
Items should not be prioritized based on Business Value only. Estimations should be taken into account as well. Don’t forget, the goal of Scrum is to deliver the best value possible during a sprint.
By default from two items that hold the same BV, the one with the smaller estimation should be prioritized higher. This is not a process that can be automated, human decision cannot be avoided. Which one is better? Picking small BV items with small estimations or a few big items with high BV? These are not trivial decisions, the Product Owner should make decisions.
Taking estimation into account for prioritizing is essential, but also to achieve simplicity. Estimations are not negotiable, but should be discussed. If for a presumably small task the estimation is very high, the PO should think about discussing it, as it might happen that the team misunderstood something. Or even if not, he might reduce the scope of the item to achieve maximum value with lower effort.
- PO: “Can I make any simplification to reduce the estimation?”
- Dev: “Well, from the 40 hours 20 would go to the deletion of the user. Deleting the user means removing all references to him and ...”
- PO: “Wait, 20 hours for deleting? I don’t need it to be deleted, it’s okay if you only disable it.”
- Dev: “You mean it should not be deleted, just denied from logging in?”
- PO: “Sure! But it should be shown to the other users with some icon that he is a disabled user.”
- Dev: “Okay, let’s re-estimate it. Okay it’s only 20 hours now.”
- PO: “Cool, I’ll move up with higher priority. Can you also notify the disabled user in an email?”
- Dev: “Well, that again introduces complexity, as we don’t have any mailing in the system yet. Let’s take that separately.”
Clearly this reduction would not have happened without the discussion. The team would have spent double the time on developing something that is waste for the PO (and the customer as well).
The sprint will start with the sprint planning meeting. All the previous meetings were just preparations. It usually takes 4 hours to plan a 2 weeks sprint. The main goal is to pick the items for the upcoming two weeks, and plan the execution of the sprint.
Defining the sprint goal
Before jumping into details, it’s worth stating what the goal of the sprint would be, so at the end of the planning we can check whether we arranged the content accordingly.
We determine the velocity and the capacity of the team, then open an empty sprint backlog, which will hold the items for the next two weeks. Based on the priorities, we pull in the items one by one from the product backlog to the sprint backlog.
While it looks like this would only take a couple of minutes and we are done, those practicing Scrum know that in practice it takes time and is often painful.
We have to realize that although we talked a lot about the items during the preparations, their goals are clear, they are estimated, we might even composed use cases and acceptance tests, these all are plans for what we are going to do when we actually pull the item in. When the time comes that the item does go into a sprint, things might get re-evaluated.
Estimation, priority, scope, etc. all can change, and this is the last time these can change significantly. Dependencies might be realized, items getting split. Stakeholders try to push in items in the last minute.
How much work does the team take?
The team should take amount of work they are comfortable being able to deliver in a sprint. The goal is not to maximize the busyness of the people, but to maximize the value they deliver.
The nightmare of the management is that if people define their own capacity and velocity, they will undercommit and won’t take enough work. In practice, the team usually underestimates the work to be done. No need to be afraid they will be without work in the last few days, especially that there are processes to manage slack time as well.
The amount of items pulled in are non-negotiable by the PO, while the priority of the items are non-negotiable by the developers.
In time-based planning, the capacity is the time based capacity of the developers, excluding holidays, etc. The problem with this is that it encourages to stuff the whole time-frame with tasks, and does not really leave place for negotiations and reasoning regarding why to take less. Even in this case, the recommendation is to pull in enough items the team is comfortable with, and don’t stuff the whole timeframe.
With relative estimation, we forecast the velocity of the team based on how many story points they could deliver in the last few sprints. We pull in items one by one, and close to reaching this forecast velocity, the team determines if they can take more work. They do it until they are comfortable.
At the end of the planning, the Scrum team gets to a two-way commitment. The team commits to deliver the items by the end of the sprint, according to the DoD. The PO commits not to change the content of the sprint. Both are commitments which are taken very seriously, but not promises.
The team commits to the delivery as a team. No individual developer is responsible for the delivery of certain items, but the whole team is responsible for the whole.
Proper measurement of velocity is only possible if the team takes commitment and time-based planning is not abused to be a budget. Based on the amount of delivered story points, we can forecast the team’s velocity, moreover, they can measure their own improvement.
Even though story points are artificial units, the PO and the stakeholders can forecast the progress of the project. Note that due the artificial units, one team’s velocity only represents that team. There is no point in comparing the performance of two teams based on this manner.
Finalizing the sprint backlog
The first part of the planning was about WHAT to deliver. In the second part of the planning, developers spend significant time based on the HOW questions as well. They break down the items to tasks and plan the execution of the sprint, potentially even assigning tasks for the first days.
It’s advised to also spend time estimating the tasks in an hourly manner, so during the sprint the team has a good view on their progress.
This phase has a key importance, as the whole sprint depends on a proper execution. Revealing dependencies, defining the right order to take the items, realizing potential parts that can go in parallel, taking risky work ahead, starting longer tasks.
Recommendation for splitting the tasks - SMART:
- Specific enough that everyone can understand what's involved, avoiding overlapping
- Mostly “done or not”
- Task should be achievable in the scope of the story and the task owner should be up to it
- It should contribute to the story. If it’s not needed for the story, it should not be a task of the story.
- Unlike with stories time should be the primary measure for it.
Every morning, in the same time of the day, at the same place the team gets together for the daily stand-up. All the developers participate, with the PO and SM suggested to be there as well. Chickens might join as well, but only observe. The meeting starts at the time, waiting for people who are late is not accepted.
Why standing? This also indicates that it should be a quick meeting, no time to get comfortable. It should be very short. For an 8 people team, it should not exceed 15 minutes. It’s recommended to observe and use the physical or virtual sprint backlog during the conversation, to make sure everyone understands which item people are talking about and to check what work is left to be done.
It’s quite easy to measure the maturity of a Scrum team by observing the meeting.
Who reports to whom about what?
The development team members report to each other, not for the Product Owner or the Scrum Master. The main goal is to share information regarding the state of the items and take actions regarding them. Technical discussions should be minimal, these should be taken outside the meeting.
This daily meeting should not be an exclusive place for sharing information and taking actions, but having a daily fixed point when everyone stops for a bit and looks around helps a lot. Tasks depending on each other, people being stuck, misc. blocking factors are often realized at this meeting.
In the original paper, each team member should answer three questions.
- What did I do yesterday that helped the Development Team meet the Sprint Goal?
- Eg. “I’ve finished the implementation of X item! It can be moved to QA!”
- What will I do today to help the Development Team meet the Sprint Goal?
- Eg. “I’ll pick up X task for QA, then help Joseph in the implementation of Y, because he’s a bit stuck.”
- Do I see any impediment that prevents me or the Development Team from meeting the Sprint Goal?
- Eg. “Alice indicated that X story should be picked up for testing, however, I can only do it when someone did the code review, where I need Bob’s help!”
- Or “Guys, I think the task Cecil is working on is getting more and more risky and pretty overdue. After the daily a few of us shall sit together regarding that technical difficulty he mentioned to help him in moving further.”
A new approach is instead of focusing on “who did what”, rather check the items themselves.
Traits of a good daily are:
- The SM barely has to speak up
- People are talking to each other, instead of the SM/PO
- People are focusing on the state of the item, instead of the time being spent on it
- People are switching tasks
- People are volunteering to hop in to help others or to take items to the next phase
- People bring up blocking problems
- And give possible solutions for them
- Honesty regarding the state of the sprint
- The meeting is pretty short without too much moderation
The burn-down charts help in forecasting the progress based on the time spent and the time left in the items.
At the end of the sprint the demo/review happens and lasts for usually 1-2 hours for a two weeks sprint.
The development team presents the done (and only the done) items to the stakeholders, even the customer can participate. Shows and presents the working software, not just description of the items. The key is to show it as live as possible.
The team does not demo for the PO, the PO should have reviewed these items already, as he’s actively involved during development. Feedback gathering is an essential part of the meeting, so is talking about the foreseeable future of the project.
Although an informal event, it’s okay to have a “ceremony” feeling for it. The two extremes can be a common mistake, neither no one attending and being interested, nor having too much people at the review are healthy.
Improvement and development of the team itself is key in any Lean/Agile/Scrum method. Its forum and most important part, the meeting responsible for the improvement is the retrospective meeting. It takes 2-3 hours for a two weeks sprint and the last closing meeting of a sprint.
In short, the following should happen:
- The team looks back at the previous sprint regarding what when good and what went wrong
- Realizes rooms for improvement and agree on action points which ensure the development of the team.
There are so many reason why this meeting is great. As it happens often it’s quite easy to remember for the whole sprint and draw right conclusions. Also as one sprint is quite short, it’s worth experimenting and trying new processes out. In the end we can always revert these changes in two weeks.
This meeting is often embraced by non-Scrum teams as well, as a great tool for self-improvement.
Evaluating the sprint
The best case is of course if the team met the sprint goals and delivered all they committed to. Of course if too much slack time accumulated at the end, it means the team undercommits.
The next best is a failing sprint. Why? Because the worst is a sprint that is forced to be successful, but should have been considered as a failing one. This hides problems and does not let those problems to emerge, holding improvement back.
The team is committed to make a successful sprint, and by this, they often try to deem find shortcuts to consider it done. This is where the Scrum Master's presence is important, who watches the events unbiased and does not let the team sweeping issues under the rug. If he sees that an item does not meet the DoD, he has to intervene. If he sees that the last two sprints succeeded only because two people stayed in till midnight, he has to intervene.
Failing should not be avoided! It’s important to fail, make mistakes and learn from it. In this way of working, where there are many feedback loops, the goal is to fail as early as possible and change accordingly. Never failing means playing safe and not being on the edge of maximum productivity.
The sprint should be viewed as a whole, from the refinement to the review. The true reason for a failing item might not be at the last few days, but hides in the lack of sufficient preparation of it. It can be related to a completely different item taking double the time early in the sprint, which was developed properly, but was vastly underestimated. It’s very important to find the root cause, otherwise, we are fixing non-issues.
The goal of the retrospective is that team itself realizes these problems and figures out the proper actions, and NOT the Scrum Master. Of course he can make suggestions, but it’s preferred to make them realize these by questions. This will make the team admit the problems to themselves and will feel more engaged to fix these. Directly point out to problems often leads to frustration and the team might also will just lay back and wait for the SM to come up with a solution as well.
It should be an honest meeting, without blaming. If the lack of skills of one person holds the team back, it should not be hidden, but the honest discussion should rely on constructive ideas. Maybe that guy only needs a short training or a few pair programming sessions.
What to improve?
The team should collect these ideas before the meeting on a “whiteboard” or a shared online space. It’s worth throwing in all ideas and problems. It’s very common for people to holding back a problem, because they feel it’s only their problem and don’t want to be ashamed. They should do it, as many times it turns out that that very problem makes half of the team go crazy as well, but everyone was waiting for someone else to start the discussion.
When collecting the points, focus on the problem you are trying the solve and let the whole team find the solution suiting all. The solution you gave might get voted down. If you have had raised the problem, others might have felt the same, and worked out a solution together.
Vote on the points if there are too many to make sure you are addressing the most important ones.
Being action oriented
Immature retrospectives tend to focus on the problem without taking actions. We must take actions, even if won’t solve the whole problem, or even if we are not sure whether if it will solve it or not. If you don’t take any action, just talk, nothing will change, and the retrospective will only be a therapy session for the people in the team.
Spending ages whining about manual building of the application taking much time won’t yield results. Taking action and starting to work on it might will.
The whole team should agree together on what actions to take, and commit themselves to follow it up by reviewing whether it worked or not on the next retro meeting. The Scrum Master should facilitate as always.
Questioning helps to get to the bottom of the issues:
Senior Dev: “The code several people put out is of very poor quality and we spend much time at the end of the sprint to rewrite these.”
SM: “Is it expected from everyone in the team to write proper code?”
Junior Dev: “Of course! But as a junior I’m not aware of my code being poor quality. I don’t recognize bad quality myself.”
SM: “Would a training help?”
Senior Dev: “I don’t think so. It’s a long term thingy to learn these, not a 1 hour one. ”
SM: “Okay, what can YOU do to help this? Without always rewriting these bad codes yourself.”
Senior Dev: “Well, I can collect the most important coding standards that even a junior can keep himself to. I’ll collect these, share it and if there are no objections I want everyone to code according to it.”
SM: “Does everyone agree that it?”
Junior Dev: “Yes, but I’m afraid it won’t help much. I’d rather go with something drastic, like pair programming or a detailed code review.”
SM: “These are all great idea. But I suggest you to take one step now instead of too many. Pick one which is “cheap”, see if it works, and if not, we will do further improvements based on other ideas.”
Give them time
Too many drastic changes at once should be avoided. Many independent small changes are okay. Major changes one by one are are okay as well. If we are taking too many big changes at once and see that the process not working, we won’t have any idea which of those major changes were good or bad.
Changes can go as far changing the DoD, eg. introducing code review for quality reasons, demanding automated tests, or doing Continuous Delivery.
“We spend too much time at the end of the sprint with the manual deployment of the application”
ACTION: Let’s automate the deployment
“An item took way too much time because of the spaghetti-code in that area. The worse? That code was written by us, 2 sprints ago!”
ACTION: Coding standards
“The feature we delivered last time is full of bugs, which only happen during customer usage”
ACTION: Using real-life projects during testing
“The delivered functionality could not have been installed to the production environment”
ACTION: Bringing test environment closed to prod-like
ACTION: Create a staging (pre-production) environment
Scrum misc. artifacts
Exceptional events during a sprints
Cancelling a sprint
This is the answer to the question “What are we going to do if we need immediate action from the team?” If it’s really that important, you have the option to cancel the sprint. It might not happen due to external input, it can even happen that during execution, we realize that the sprint goal is invalid. Of course cancellation should happen rarely, but the option is there.
Cancellation often comes from the Product Owner, with the Scrum Master checking whether the cancellation is indeed valid. The SM also has to make everyone in and outside the team, that the impact of a cancelled sprint is quite big.
It’s also a powerful option when protecting the team. If the team team is asked about a presumably important and urgent matter, the team has the tool: “Look, if it’s really that important we can talk with the Product Owner and the Scrum Master to cancel the sprint. Of course everyone around the team, including the management would be interested in why we got to this point, are you going to have a proper answer for that?” It often turns out that it’s not THAT important and could wait a few more days.
It still happens that due to the circumstances, it has to be cancelled. The recommendation is to plan a short, exceptional sprint for the remaining days with the new goal, then jump back to the original rhythm (as the original eg. 2 weeks went by).
Changing the content
It also happens that only one item gets invalidated, meaning that one item’s goal is invalid. In this case, simply remove the item from the sprint. Do NOT put anything else in its place. That might screw up the execution the developers planned. If they have slack time at the end of the sprint, you have tools for that as well, which comes as follows.
Sometimes the team meets the sprint goals - meaning, puts all items to done- much earlier than the end of the sprint. First of all, if it happens often it means they don’t commit to enough work, or play it too safe, or there are major estimation problems and the Scrum Master should coach them to avoid it.
If the team feels that they can take more work till the end of the sprint, then they can take additional work. The priority is still determined by the Product Owner and often comes from the very top of the backlog, but as a rule of thumb, the team often prefers to pick up something small they can still finish, instead of jumping on a 13 SP monster story.
This content change can be only made by the Scrum Master, to make sure that he’s gatekeeping the integrity of the board.
It’s not really content change, but an important question what to do when the team realizes that they won’t be able to reach the sprint goals. In this case the Product Owner should be involved to determine which items are the most important to be finished. A failing sprint is a failing sprint, but it matters what impact that failure has. If it’s a small UI bugfix, it might not affect much, if it’s a story on which future sprints depend on, it surely hurts. Keep yourself to the Agile values, and recognize that it’s still about maximizing the value, even in the case of a failing sprint.
It depends on the project, but sometimes the team has to work with tools (hardware or software) that are yet unknown, not familiar for them, and it’s hard to estimate working with these. This unknown factor can also be a nasty bug which is hard to be reproduced. These all hold some kind of uncertainty, ambiguity, doubt, something that makes them hard to be estimated.
The tool for managing these are called spikes. Spikes are like regular items, but produce information rather than code.
The goal of the spike is to reduce the ambiguity. A spike should give answer to one or few uncertainty questions, and should not be a “filler item”. Also keep in mind that uncertainty is part of software development, it happens all the time. You shouldn’t take spikes for every small unknown factors. Spikes should be the exception, not the rule.
Guidelines for spikes:
- Estimable: No timeboxing! Should be foreseeable and have a clear goal.
- Demonstrable: to the team and at least we should be able to explain it outside the team on the review.
- Acceptable: PO should be able to accept or reject the outcome.
Considered to be the holy grail or the anti-pattern of Agile.
With a timeboxed item, the commitment is not a specific commitment, but rather a commitment to “spend X amount of time on working on it”. It means spending that much and only that much.
It’s already clear that estimating these items in a relative way is problematic. It also should be avoided because it’s a great soil for unprepared item and sprints without concrete commitment.
How done can an item like this be? Did it produce code? Is it tested? What happens when the time gets spent in the middle of QAing?
Timeboxing items goes against many Scrum concept, that’s why Scrum Masters might tell the team not to use timeboxing at all. Rather, pull in spikes that answer the uncertainty, rather than “just spending time on it”.
Regular timeboxing shows unprepared sprints. Still, it’s sometimes used, but should be used carefully as it can introduce extra work and processes.
High level planning
Tasks are the level where developers work. A User Story is decomposed to tasks.
User Stores are the level the Scrum team works on, the PO states his needs in this form. DoD is applicable for stories. One story hold one single value, but usually is part of a bigger picture.
The need for higher level entities is clear.
“As web-portal administrator, I’d like to have a list of users, so that I can search for a specific user.”
“As web-portal administrator, I’d like to disable users, so that I can suspend their disturbing activities.”
“As web-portal administrator, I’d like to generate a new password for a user, so that when he requests a new password, I can email it for him.”
These are all User Stories but clearly belong to a higher unit, which is about the user administration of that web portal. We call these features.
It’s way easier to manage whole features if we manage them in a way that stories working towards that feature are collected in the same entity. Moreover, having the possibility to check all the potential developments for the feature helps all in understanding the big picture. Developers might realize a missing story from the feature. Not to mention that during development, they often have to choose between seemingly equivalent paths. Knowing what comes next or what other aspects they have to keep themselves to in that feature help in choosing the path that leads to the fewer waste.
This new entity helps the Product Owner to manage items on a higher level and organize them more easily. Seeing these all in a whole helps in saying “Yes, we are indeed ready with the user management!”. Valuation of the stories is also easier this way. “Do we want to include user disabling feature initially in our Minimal Viable Product? Can we postpone it to a later phase?”
Outside the team, having this is level is even necessary. People from the management (at the company or even at the customer) think and prioritize on higher levels. They are probably not interested in small details, at least they should not be. For them, the doneness of a story does not matter, but the doneness of a feature does.
The Product Owner if sometimes considered as a “Product CEO”, but usually he gets the high level priorities from the stakeholders or the customer and has to align to them. It often not the Product Owner who decides between features, but someone higher in the management. “We need the user administrator to sell the software, it cannot go without it. But I can sacrifice the companion mobile application feature.”
A level even higher than this is the Epic. With the previous example, the Epic can be “web-portal administration”, consisting of the features “user management”, “content management” and “authorization management”, etc.
Epics also can represent business cases, which consist of features required to achieve that business case.
These are the commonly used entities, but each Scrum software has it’s own higher level ones. eg modules, projects.
Levels of Agile Planning
An elevator pitch, answering the following questions:
Who is going to buy the product? Who is the target customer?
Which customer needs will the product address?
Which product attributes are critical to satisfy the needs selected, and therefore for the success of the product?
How does the product compare against existing products, both from competitors and the same company? What are the product’s unique selling points?
What is the target timeframe and budget to develop and launch the product?
A typical format can be the following:
For (target customer)
Who (statement of the need or opportunity)
The (product name) is a (product category)
That (key benefit, compelling reason to buy)
Unlike (primary competitive alternative)
Our product (statement of primary differentiation)
For the software startup to midsize company,
who needs a simple, intuitive agile project management tool,
Cognosco is a cloud and mobile based offering
that provides scrum facilitation, agile metrics, reporting, and low admin overhead via social log-ins.
Unlike similar services or package software products,
our product provides offline capability with peer to peer screen sharing, and competitive cost.
The vision should not change often. A change in the vision means a change in WHAT we want to achieve, and where we are going. Changing the vision often means you don’t know where you want to go and you might not get anywhere.
Roadmap and release planning
Roadmap should be maintained around every 3 months and should show the plans for a few quarters ahead.. It is not carved in stone and typically does change every in each quarter a bit. As with the Vision, change means change in the plans, which is good if you found a better way to achieve your goal, and bad if you change it often, as it can indicate lack of planning
Release planning is about which features, stories should get to a certain release. Heavily depends on the product and the meaning of “release” for that product. Should focus defining the MVP and GA product, by cutting down less valuable parts of items.
Clarity for each level is important and everyone should be aware of all levels. However, responsibility is key for each level of planning. These are just rules of thumb to get the idea.
- Vision is rather high C-level about the elevator pitch
- Roadmap is for stakeholders, PO with epics and features
- Release is for stakeholders, PO, developers with features and stories
- Sprint is for SM, PO, developers with stories and tasks
- Daily is for developers with stories and tasks
It’s worth putting effort into having the different levels of estimations.
Features should be estimated on T-shirt level to get an idea on their size relative to each other. It implies that the feature should be understood by the developers. As it takes time like any refinement, it should only happen for a couple of features (filling in the roadmap is a good rule of thumb). Understanding the features and being aware of the order helps the developers. Developers notifying stakeholders about dependencies between the features helps the roadmap planning. As with all levels, big features should be splitted, as they are probably an epic.
Stories are suggested to be estimated on story points level, at the refinement meeting, this topic has been discussed already.
Breaking down stories to tasks also help in sprint and daily planning. It’s suggested to do hour based estimation on this level, with each task being smaller than a whole day. It also helps during inspection and retrospectives if the development of the item does not go as expected.
What does Scrum give to us?
- Short iteration
- Regular and quick feedback (for the product, development, etc.)
- Low risk for each sprint (which makes us brave enough to take risks)
- Adapts to changing priorities and requirements
- Provides a rhythm
- Between the appropriate parties
- Structured, short, regular meetings
- Fixed sprint content
- Clear responsibilities
- Flat management
- Collective knowledge
- By the “Definition of Done”
- By pulling the most valuable features
- By the customer being involved
Just a framework
It gives a frame and answers many questions. Tell us that you need to have a DoD, but does not say what it should contain. Tell a lot about the development team but barely talks about software development itself. This is intentional. It has to be general in order to work for the so many different software development cases.
There aren’t two identical teams! Moreover, if one guys leaves a 7 people team, the whole chemistry of the team changes, and by this their work and behaviour as well. Two teams consisting of many different individuals will have different problems, needs, attitudes, which all require different solutions. To goal of Scrum is not give solution to all of these, but to give a process to recognize the problems and find the solutions.
There aren’t two identical companies! For one, a two weeks iteration with Continuous Delivery works, while the other has to release it every three weeks. One company might have to deal with the huge amount of ideas that has to be prioritized, while at the other the PO has to fight for proper involvement of the stakeholders.
There aren’t two identical customers! One might over-exceed the involvement and distract the team, while the other has to be dragged into the sprint review!
There aren’t two identical projects! One might not have a goal yet, but the deadline is already there, the other has pixel perfect UI designs and waits for the team to estimate the deadline.
Clearly, what would work in one case would hurt the other.
You have to integrate your uniqueness into Scrum. Find what works for your team, project, company, customer. Keep in mind that things change over time, and adjust to that.
Integration into the framework
These rules and procedures has to be set up either by the team or outside the team, either for internal or for external work.
“As a team, we can’t let a 60 hours item, which cannot run parallel into the sprint. If the PO pushes an item like that, we have to educate him regarding the problems it might cause, and we should help him in splitting it to smaller ones.”
“We agreed with the PO that instead of pre-drawn UI design, we will arrange the UI together with him, during the sprint. Let’s try it out and if it works well, we can keep this habit.”
“As the PO, I’ve asked the stakeholders that whenever they report a bug, they should attach a video file while reproducing it. It helps during estimation a lot, as we immediately know what part of the software talk about and what are the circumstances. They said that whenever it’s doable, they’ll attach it.”
“We agreed with the development team that everyone should commit and push their changes at the end of the day, so when someone goes on a sick leave, we can continue his work. We made sure these commits do not affect the branch from which we build the production package.”
“The management asked us that if a feature is so overdue that it affects the roadmap, we should inform them. We agreed that we review the progress of each feature at the end of the demo. ”
“The customer wants to try out all the ready and done features as soon as possible. We’ve set up a staging environment, which we will upgrade at the end of every sprint.”
“As the development team, we agree that whenever the Continuous Integration system alerts us regarding a failure, we either fix it immediately, or revert it - to ensure the working software.”
The most important a clear acceptance from the management. If the CEO himself distracts the developers and cancels the sprint all the item, it means that the working culture does not support agile values, or at least a wrong method is chosen.
External factors also affect the if the team can work as a Scrum Team. If the Scrum Master is not empowered to protect the team, if the Product Owner isn’t empowered to make decisions, it will never work properly.
Let the management understand the agile values, how it works, how it creates value and why it’s often more efficient. Send them to a training. There is no best way to convince them, they are individual people with own thoughts.
There are some other recommendations as well.
Co-location is very important. If the team is scattered in place or time, they won’t be able to work as a team, just as a bunch of individual developers.
Avoid over-specialized tasks and over-specialized people. Again, teamwork is nullified if only certain people can do certain tasks.
Getting rid of external dependencies is also recommended. If the team depends on another team to be able to deliver, it might kill being able to plan and to properly finish items. An external QA department makes sprint execution very complicated. Being distracted all the time with support requests makes the team losing focus.
When Scrum might not be the best
The above mentioned aspects can lower the efficiency, or can completely kill Scrum. Taking away a few key aspects might make the whole process collapse. Keeping efficiency high is key!
There ain’t no “good” or “bad” methods. You are able to deliver software using any method. The question is, which one suits the situation better. Also note that there are differences between Agile methods as well (Scrum, KanBan, ScrumBan).
- If the deadline isn’t, only the end result counts (if everything is needed and nothing can be valued over another, then there is no prioritization)
- When bureaucracy can’t be avoided you don’t get the advantage of being lightweight (banking, government)
- Too many external dependencies
- Too much uncertainty (technical or functional)
- The environment does not support Agile values
- Requires specific knowledge in a huge enterprise environment (Scrum of Scrums, LeSS, SAFE)
- Team members are too specific in areas and cannot take tasks from each other
- You need several experienced seniors
- When you are too agile (CD, ScrumBan)
ScrumBut, common mistakes
These are often called ScrumBut as it often starts with “We are using Scrum, but….”
We use Scrum, but we need a whole hour for our daily meeting
We use Scrum, and we are so good already that we do not even need retrospectives anymore
We use Scrum, but we are developing heavy features, so our iterations are 8 weeks long
We use Scrum, but our Scrum Masters are developers as well.
We use Scrum, and we are so good already that we don’t even need SMs.
We use Scrum, but of course if the CEO asks for something, we change the content.
- Using Scrum without understanding Agile
- Thinking that Scrum will solve the problems
- It will reveal and help you realizing the problems.
- But it won’t solve them (eg. people problem, lack of strategy, bad vision)
- Getting lost in the different Scrum interpretation
- Bossy Scrum Master
- Backlog without DoR items
- Skipping customer involvement
- Not using feedback
- Not taking DoD seriously
- Done stories without the feature being usable
- PO who isn’t there
- Proxy PO
- Not involving the stakeholders
- Cost targeted
- Agree on a realistic projected final price
- Shared risk
- Share the benefits of going underbudget
- Share the penalty of going overbudget
- Eg. 100K budget, finished in 80K, pay 90K, so both parties gain 10K
- 100K budget, needs another 15K worth of work, the customer pays 110K
- Capped T&M
- Agreement on upper bounds
- Protects the customer in going overbudget
- Incremental delivery
- Customer decides to continue and pays per iteration
- Customer decides if they gained sufficient value already
Related methods and toolsets
What Scrum misses
Scrum, although huge books are written about it, is just a few pages worth of lightweight framework. The Scrum Guide is just 12 pages long! It gives a rhythm, but we have to fill the framework that it gives to us with specifics. Even though those are usually unique, there are common practices that agile teams do nowadays.
Even though it’s software development framework, it lacks technical talk. This is the place where XP will fill the gap.
Agility works best the whole company embraces it, and being Agile should be a core value for the whole company. But the size of one Scrum is pretty small comparing to what companies need. Even small companies often exceed the 9 developer limit that Scrum suggests for one team.
What are we going to do if there are 100 developers? What if they are 1000? If these developers work on separate product, that’s fine, just separate the teams by products. But what if they are working on the very same software? There are several options to choose from.
Often missed that the core Scrum does contain a solution for scaling, which allows several teams to work together, called Scrum of Scrums, which allows 2-5 Scrum teams to work together.
There is also Large Scale Scrum (LeSS), which allows 2-8 teams, or even more than 8 teams to work together (with LeSS Huge).
Scaled Agile Framework (SAFe) is also an option with 3-level SAFe working with 10 teams, and 4-level with more than 10 teams.
Scrum of Scrums and LeSS are pretty lightweight, but LeSS Huge and SAFe (on both levels) are quite heavyweight, and requires specific knowledge and expertise to be run.
With Scrum of Scrums we put another layer of Scrum over the original Scrum process. After the usual daily, we delegate someone from the team to represent the team on the “Scrum of Scrums” daily, where these delegates meet and discuss topics. These are the RIDAs (Risk, Impediment, Dependency, Assumption) and the questions each answer are the following:
- What RIDAs has your team moved on since we last met?
- What RIDAs will your team move on before we meet again?
- Are there any new RIDAs slowing your team down or getting in their way?
- Are you about to introduce a new RIDA that will get in another team's way?
A LeSS, although introduces some new rules and term, mostly just specifies cases that are missed in the original core Scrum. Defines synchronization points for the multiple teams, states how many team a PO and SM should take, how the sprints run in parallel etc. Talks about why one PO and one backlog should not be exceeded with 8 teams.
Less Huge, so the frameworks allowing you to work with 60+ developers are way more complex. These introduce new terms totally independent from the core Scrum, eg. Area Product Owner, Solution Architect, Release Manager, Enabler, Epic Owner.
KanBan is discussed as an Agile method, not as the Lean manufacturing method.
It allows agile behaviour for teams which are unable to use Scrum (because they are two small, or their work is so interrupt driven that they cannot plan ahead).
Roles, meetings are often the same as with Scrum. But it omits Scrum’s core feature, the iterations with strictly fixed content. There is no iteration at all, just continuous work with changing priorities immediately being reflected in the work. One task card can get on the board at any time with all its pros and cons.
KanBan also has own rules, eg. that items are more on the right (closer to being done) have priority over cards on the left. The top of backlog has to be clear and DoR satisfied always, in every minute, as the top item can get into work potentially at any time. It requires heavier collaboration and control, but allows teams to deliver items the very day it got on the board.
The mixture of the two are ScrumBan. In practice we are planning a sprint like with Scrum, but then work KanBan-style during it. There is an iteration, but priorities can change.
None of these are better than Scrum, they are just different. Eg. picking up an item that is new and delivering it over the others can be great, but also make focus lost. Less planning can be misused and can potentially yield less collective knowledge.
Is this a framework too?
There are different opinions, let’s just take Extreme Programming now as a toolset.
Scrum does not really talk about automation, yet it’s one of the most important trait of modern software development and agility. Scrum teams, by themselves can recognize the value of automation, but why would you take time when it’s clearly beneficial.
What to automate? Anything you can! Anything that you have to do more than once. Anything you do manually and take time. Anything that would help in faster delivery
- Automated tests (unit, integrity, UI)
- Running of automated test
- Build of the application
- Cluster configuration
Team,s especially if several teams work on the same software, often find the integration as the biggest challenge. There is a huge difference between running the code in the developer's own small environment (in a vacuum) and in the production system. Continuous Integration inspects the software as a whole, typically in a production like environment.
It keeps on continuously fetching the changes of the code, and build the application, puts it together, puts it in an environment, installs it, then runs many-many automates tests to change if the integrity is kept. It does it all the time, continuous, so when a mistake is made the feedback happens as early as possible.
Code formatting, usage, warning, compilation or general software design standards set by the team and usually kept in check by a tool. Helps achieving collective code ownership, in understanding each other’s code, lowers the amount of poor quality code.
A new paradigm in writing automated tests for the production code. Instead of writing the production code, then the test, rather
- Repeat the whole process in very small chunks
- Write the test first (which fails due to the lack of production code at this point)
- Write the code that is just enough to fulfill the test
- Refactor the code
- Code will be much simples, as the method forces you to structure your code properly
- The developers writes just enough code that the feature needs
- Automatically full test coverage
- Code quality is kept due to refactoring
- Quite hard to do it initially
- Especially with a yet unknown technology
Simply put two developers, together work on the task. It seems to be double the effort, but has many benefits:
- Keeping each other in focus
- Learning from each other
- Getting over obstacles more quickly
- Better code quality
- Encouraging team-work and collective code ownership
Comparison of Agile and Classic methods
Follows the plan
Only in certain phases
At the very end
Continuous, but low-key
Initial and thorough
Copyright © 2019 scrum-handbook.com Gyula Attila Borsós All Rights Reserved