1. No category

MTAT.03.243
Software Engineering Management
Lecture 11:
SPI & Measurement –
Part B
Dietmar Pfahl
Spring 2015
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
email: dietmar.pfahl@ut.ee
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Background information:
• Recall Exercise 2 from the lecture where you modeled a process using one of the
(equivalent) notations shown in Figure 1 below. Then perform the tasks described on
the next page.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T1 [3 marks]:
• Have a look at the description of ‘Scrum’ published in the Eclipse Process
Framework (EPF) Wiki at http://epf.eclipse.org/wikis/scrum/index.htm.
• Create a process model using the table notation shown in Figure 1.
• Make a note/comment, if you think, information is missing (incomplete) or
inconsistent in the process description provided by the EPF Wiki.
• Note: Marking will consider completeness and correctness of the model as
well as appropriateness of your notes/comments
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 –
Task 1:
Solution
Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T2 [2 marks]:
• In the model you created in Task T1, select at least three activities that are
directly connected via product-flow (i.e., one activity consumes at least one
artifact produced by another activity). Then create a graphical representation
of this subset of the Scrum process model. You can choose one of the
following three notations (and tools):
– Notation from Figure 1
– Spearmint (you find information on how to download and install
Spearmint on the course wiki – Lecture 4)
– Bizagi (you find information on how to download and install Bizagi
Process Modeler on the course wiki – Lecture 4)
• Note: Marking will consider completeness and correctness of the model and
consistency with the model developed in Task T1.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 – Task 2: Solution Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T3 [1 mark]:
• Explain your choice of the notation/tool in Task T2. What was the reason
why you chose it and what are the advantages and disadvantages of the
notation you chose? To get full marks, you need to define and discuss at
least two criteria that guided your decision. For example, you may use a
table notation as shown below:
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 – Task 3: Solution Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Distribution of Marks
Students enrolled: 26
HW1 submitted: 25
Penalty of -1: 1
Penalty of -2: 1
Average marks: 4.8
No name on sheet: >3
Not PDF format: 1
Late submitted: 1
Not submitted: 1
Mean: 4.8
Median: 5.3
Mode: 6
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurable Entities in a SW Process (Model)
Ressourcetool
An entity can represent any of the following:
• Process/Activity: any activity (or set of
activities) related to software
development and/or maintenance (e.g.,
requirements analysis, design, testing) –
these can be defined at different levels of
granularity
• Product/Artifact: any artifact produced or
changed during software development
and/or maintenance (e.g., source code,
software design documents)
• Resources: people, time, money,
hardware or software needed to perform
the processes
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Productin
Ressourcerole
Activity
Productout
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
•
Name:
Avg. Sprint Prod.
????????
•
Entity:
Software
Sprint
•
Attribute:
•
Unit:
•
Range:
Size
[0, ∞)
...
Effort
[0, 1, 2, ..., n]
[0, ∞)
•
[ph] = TM [p] * TD [d] * WHD [h/d]
Productivity = Output / Input
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
[0, ∞)
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
•
Name:
Avg. Sprint Prod.
????????
•
Entity:
are
Sprint
•
Attribute:
•
Unit:
•
Range:
Size
[0, ∞)
????????
Effort
[0, 1, 2, ..., n]
[0, ∞)
•
[ph] = TM [p] * TD [d] * WHD [h/d]
Productivity = Output / Input
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
????????
[0, ∞)
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
•
Name:
Avg. Sprint Prod.
•
Entity:
Dev. Process
•
Attribute:
Efficiency
•
Unit:
•
Range:
Sprint Count
Sprint Output
Sprint count
LOC/person-hour
[0, ∞)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
dimensionless
[0, 1, 2, ..., n]
Sprint Input
Size
person-hour
[0, ∞)
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
•
Name:
Avg. Sprint Prod.
Sprint Count
Sprint Output
Sprint Input
•
Entity:
Dev. Process
Dev. Process
Software
Sprint
•
Attribute:
Efficiency
Sprint Number
Size
Effort
•
Unit:
LOC/person-hour
dimensionless
LOC
person-hour
•
Range:
[0, ∞)
[0, 1, 2, ..., n]
[0, ∞)
[0, ∞)
•
Formula (Model): ASP [loc/ph] = (SUMsprint (SO [loc] / SI [ph])) / SC [-]
– Question: Does this formula make sense?
•
Note: Since a sprint has a fixed time in days (TD) and a fixed number of fulltime team-members
(TM) with a fixed number of work-hours per day (WHD), one could ’Sprint Input’ (SI) consider a
derived measure, calculated as SI [ph] = TM [p] * TD [d] * WHD [h/d]
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Core Elements
GQM has three elements:
• Goals
• Questions (and associated Models)
• Measures
Implicit
models
Q2
Q1
M1
M2
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Goal
M3
Q3
...
Q4
GQM Core Elements: Goals
• GQM goal (or: Measurement
Goals) are derived from
business or improvement
goals
• A GQM goal defines
–which object is
measured,
–for which purpose,
–with respect to which
quality focus (aspect),
–from which viewpoint,
–and in which context
(environment).
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Goal Template
GQM Goal – Example
Analyze
for the purpose of
with respect to (quality aspect)
from the viewpoint of the
in the environment of
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
test process
characterization (understanding)
effectiveness
test team
project X, organization Y.
GQM Question – Examples
Test
Effectiveness,
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
• Goal: Analyze the test process for the
purpose of characterization with respect
to (quality aspect) effectiveness from the
viewpoint of the test team in the
environment of project X, organization Y.
• Question 1: How many failures are
detected during testing?
• Question 2: When are failures detected
(time)?
• Question 3: What types of failures are
detected?
• Question 4: How much testing effort is
spent?
• Question 5: Which test techniques/tools
are applied?
• Etc.
Developing the GQM Hierarchy
Example GQM Hierarchy (incomplete):
Goal1
Q3
M1.1 M3.1
• Question 3: What is the distribution of failures by criticality?
• Model: D = F(x, y) = x[y]/x[all], x = Measure 1.1, y = Measure 3.1,
where D: distribution of # failures per criticality class
• Measure 1.1: Failure count (ST: absolute; U: n/a; S: positive
integer; O: product version 1.0)
–Hypothesis: 120 failures
• Measure 3.1: Failure criticality (ST: nominal; U: n/a; S: {critical =
complete breakdown of system, uncritical = unable to perform
one or more of the functions F1, ..., F6, other}, O: failure report)
–Hypothesis: 5% critical failures, 15% major failures, 80%
minor failures
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Plan
• The models and measures are
defined by answering ”What kind of
information do we need in order to
answer the questions?"
• The GQM-tree is documented in
tabular form
• Each measure is defined by:
–Name, ID
–Scale, unit, etc.
–Hypotheses
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurement Plan – Example
• Table for tracing Measurement Plan entries to GQM Plan, Project Plan
(based on Process Model) and Data Collection Forms
GQM Plan
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Project Plan
Data Collection Forms
The GQM Process
Business Goals
Development Process
Measurement
Data
Analysis
&
Feedback
Initiate Measurment
Project / Set Goals
Measurement
Goals
(Why ?)
Define Measures
GQM Plan
(What ?)
Execute Measurement
(Collect data)
Measurement
Plan
(How ?)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Define Measurement
(and Analysis)
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 3: SW Measurement
Administrative information:
•
This homework has to be done individually. Maximum number of marks: 6
•
Submission deadline is Wednesday, 08-Apr-2015, at 20:00 sharp. If you don’t submit in time,
you receive a penalty as follows:
– Late delivery until Thursday, 08-Apr-2015, at 20:00  2 marks penalty (-33%)
– Late delivery after Thursday, 08-Apr-2015, at 20:00  6 marks penalty (-100%)
•
No exceptions from the penalty-rules will be made!
•
Submit your homework using the ‘Submission’ function provided on the course web-page:
https://courses.cs.ut.ee/2015/SE_Man/spring/Main/Submission
– Make sure to select the correct task, i.e., ‘Homework 3’, when you submit your file.
•
IMPORTANT: Only files in PDF format will be accepted! I won’t look at files that are not PDF.
– Please make sure that you have put your name and student ID somewhere at the top of
the first page of the file you are submitting.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 3: SW Measurement
• Task T1 [1 mark]
– Understand
measurement
definitions
• Task T2 [1.5 marks]
– Develop a measure
• Task T3 [1 mark]
• Task T4 [1.5 marks]
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
• Task T5 [1 mark]
– Subjective Meas. &
Reliability
Tasks 3 and 4 are
about ’Abstraction
Sheets’
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurement Process -> ISO 15939
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
SW Product Quality -> ISO 9126
Software Quality Model
(ISO 9126 – Part 1)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
SW Product Quality -> ISO 9126 (cont’d)
External and Internal Measures
(ISO 9126 – Parts 2 & 3)
Example:
Maturity – internal: actual defect detection/correction during development, test adequacy, ...
Maturity – external: freedom of the SW system from failures caused by defects in the SW
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Proposals for measures in
ISO9126 - Appendix C
(-> reading materials)
ISO 9126 – Example Application
Measures how many defects (faults, bugs)
remain in the SW system that may emerge as
future failure.
Formula: X = (abs(A1 - A2)) / B
with:
A1 is the total number of predicted latent
defects in the system.
A2 is the total number of actually occurring
failures.
B is the system size.
Note: A1 is an internal measure derived from a
reliability growth model.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Internal vs. External Measures
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
ISO 9126 –
Software Product Quality
Quality in Use Measures
(ISO 9126 – Part 4)
Effectiveness relates to how capable is the
software in enabling users to achieve
specific goals with accuracy and
completeness.
Productivity relates to how capable is the
software in enabling users to reach
effective goals with appropriate amounts
of resources.
Safety relates to the software ability to
operate within acceptable level of risk of
harm to people, business, property or the
environment.
Satisfaction relates to the ability of the
software product to satisfy the needs of
users.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
ISO 9126 – Future Developments: ISO 25000
• New series of standards currently under development
– Name: Software Product Quality Requirements and
Evaluation (SQuaRE - ISO 25000)
• This series of standards will replace the current ISO
9126 (and ISO 14598) series of standards
– Note: the new standard will replace the word ”metric” by
“measure”
• Link: http://maisqual.squoring.com/wiki/index.php/ISO/IEC_SQuaRE
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Software Process & Product Quality
Software products
Process
influences
influences
process
quality
depends on
Process measures
(ISO 15504)
influences
internal
quality
attributes
Internal
measures
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Effect of software
product
external
quality
attributes
quality
in use
attributes
depends on
External
measures
Quality in use Context
measures
of use
Software Process Assessment
CMMI / ISO 15504 (SPICE)
Staged
Continuous
ML5
ML4
ML3
ML2
ML 1
PA
Defines 5 maturity levels (MLs); in order to
achieve a maturity level all process areas
associated to this level, plus all process areas
associated with levels below must have a certain
minimal capability.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
PA
PA
A maturity profile is established
based on the capabilities of
individual process areas
CMMI Levels and
Process Areas
(staged)
Level
Process Areas
5 Optimizing
Causal Analysis and Resolution
Organizational Innovation and Deployment
4 Quantitatively
Managed
Quantitative Project Management
Organizational Process Performance
3 Defined
* Integrated Product/Process
Development (IPPD) –
add-on to the Engineering
processes
** Acquisition – add-on to the
Engineering processes
2 Managed
1 Performed
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Organizational Process Focus
Organizational Process Definition
Organizational Training
Risk Management
Integrated Project Management (for IPPD*)
Integrated Teaming*
Integrated Supplier Management**
Decision Analysis and Resolution
Organizational Environment for Integration*
Requirements Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
CMM Assessment Results (continuous)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
KPI (Key Performance Indicator)
A set of quantifiable measures that a company or
industry uses to gauge or compare performance in
terms of meeting their strategic and operational goals.
Similar to GQM – but:
1. Stronger focus on traceability of
Goals to Business Objectives
2. Explicit targets
3. Lack of guidance on how to find
the right measures (no ’Question’
step)
4. Lack of explicit baselining (where
are we today?)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Example:
KPI – A Software Example
Source: An Example of Using Key Performance
Indicators for Software Development Process
Efficiency Evaluation by Ž. Antolić, R&D Center
Ericsson Nikola tesla d.d. (available in reading
materials)
Context:
•
Ericsson R&D, Zagreb
•
Projects on CPP platform
Objective:
•
Improve project performance
Key areas of concern:
•
Schedule and Progress
•
Resources and Cost
•
Product Size and Stability
•
Product Quality
•
Process Performance
•
Technology Effectiveness
•
Customer Satisfaction
Example KPIs:
Cost Adherence:
[1 – (ECost - CCost) / CCost] x 100%
measured monthly
ECost: expected cost (actual + forecast of remaining cost)
CCost: commited cost
Fault Slip Through:
[1 – FT Faults / All Faults] x 100%
FT: Function Test
I&V: Intgration & Validation Test
All = FT + I&V
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
measured monthly
KPI – A Software Example (cont’d)
Project Benchmarking
Projects:
D1, ..., D6
DFU1, DFU2
M3
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
•
•
•
•
•
Homework 2 – Solution Sketch
Exercise
GQM (Goal/Question/Metric) Process
Homework 3
Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Balanced Scorecard – 4-Box Model
• A strategic performance
management tool
• A mixture of financial and nonfinancial measures each compared
to a 'target' value within a single
concise report
• Four strategic dimensions
• Financial
• Customer
• Internal Business
• Learning and Growth
More info in materials on course-web: Cobbold, I. and
Lawrie, G. (2002a). "The Development of the Balanced
Scorecard as a Strategic Management Tool".
Performance Measurement Association 2002
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
See also:
http://www.balancedscorecard.org/
http://www.ap-institute.com/Balanced%20Scorecard.html
Balanced
Scorecard –
Strategy Map
Cause-and-Effect Logic:
•
A Strategy Map highlights that
delivering the right performance
in the one perspective (e.g.
financial success) can only be
achieved by delivering the
objectives in the other
perspectives (e.g. delivering
what customers want).
•
You basically create a map of
interlinked objectives.
See also:
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
http://www.balancedscorecard.org/
http://www.ap-institute.com/Balanced%20Scorecard.html
Next Lecture
• Topic:
– Lean & Flow-based (KANBAN) Principles and
Processes
• For you to do:
– Finish Homework 3 – Deadline: Wednesday, Apr 8, 20:00
– Continue working on Project
– Optional: Read articles posted on course wiki (Lecture 11)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015