/ In“NormalView”scrollleft[C1] to see headings;
For keyboard shortcuts and instructions see Guide DocTpls;
use: links to navigate; use keys ALT+← to go back; (Mac: +←).
PQA Forum[C2]
PQA definition, scope, positioning

Synopsis:

This document is intended to start a discussion on what is meant by Project Quality Assurance (PQA), the scope of the assurance activity (process), and the place PQA has within the hierarchy of portfolio, programme, project, business process modelling, software development, i.e. all processes involved in creating outputs for use in the business operational context.
PQA has morphed out of a number of disciplines. The word “assurance” has very different meanings to different readers. For an accountant it touches on audit which implies very specific rigorous methods of deriving surety. For software development / ISO-9000 people, assurance means ensuring the process of creating the outputswill yield a product that is of quality within a statistical tolerance. For non- finance or engineering folk in roles of management, assurance may mean what is stated in the dictionary, i.e. the assurance (guarantee) that all is well. In addition,“Quality” is a term with many interpretations. Last but not least there is the span of control, being: within a single project, a single programme, or the behaviour across a playing field of hundreds, or even thousands of projects, in terms of statistical variance of outcome.
This document is for anyone involved in the NZ PQA-forum,to facilitate discussion and converge on a clear definition of PQA terminology, scope and positioning.
Version(history on p.15) : / v0.1, draft
Author: / J Wijninckx

Approved by:

Name / Title / Organisation / Date / Signature

Copyright © SmartMatix Ltd. This copyright work is licensed under the Creative Commons Attribution 3.0 New Zealand licence. In essence, you are free to copy, distribute and adapt the work, as long as you attribute the copy righted diagrams contained in it to its rightful ownersand the document as a whole to NZ PQAF. To view a copy of this licence, visit

This document is based on template “TNoteTpl v3.3.x”[C3] ( sourced [C4]).

PQA-Forum_discussion_paper_v0.1.docxPage 1 of 27Saved 14-Aug-16 22:13

PQA Forum - PQA definition, scope, hierarchy

© SmartMatix Ltd, Creative Commons Attribution 3.0

Table of Contents

1Introduction

2Defining PQA

2.1Unravelling the terminology

2.1.1Project, Programme, Portfolio

2.1.2Understanding process

2.1.3Introducing Assurance into the process mix

2.1.4But wait, there is more to assurance…

2.1.5Enter the Quality debate

2.2Definition PQA

3PQA process definition

3.1Introduction

3.2IQA process

3.3Process Quality Assurance process

3.3.1Brief Explanation from a user (staff) perspective

4PQA within the process order

5Postscript

6Document Control

Appendices

Appendix A - About the Author(s)

Appendix B - Meaning searches

Appendix C - CMMI PPQA

Appendix D - Quality Management

For typographic conventions see Document Control; terminology see glossary[1].

1Introduction

The document is structured as follows:

  1. Dissect the meanings of “Assurance”, “Quality”, “Project” yielding a definition of PQA;
  2. Armed with that definition, define the process or processes of PQA, and how to assure that those processes will yield an assurance outcome within expected tolerance (pre-take on the definition );
  3. Defining the place PQA has within the hierarchy of portfolio, programme, project, business process modelling, software development processes, i.e. all processes involved in creating outputs for use in the business operational context.
  4. To support the thinking in this document, in the appendices please find supporting text-fragments and the web-site links as references.
  5. Document control information, and background information on the authors and contributors can be found at the end of this document.

2Defining PQA

2.1Unravelling the terminology

2.1.1Project, Programme, Portfolio

PQA through its name “project” constrains itself to the domain of project and possibly programme and portfolio management. Based on the name the scope of applicability pertains to the work we do in organisations to create new ways of working (people processes) and introduce new software and possibly IT infrastructure, which the business will own to produce a step change in business efficiency or create new capability such as legal, compliance, or products and services.

Programme and project management these days for many are blurred terms meaning to be about one the same; a programme manager is just some kind of bigger project manager.This is far from the truth, managing projects is a distinctly different process (e.g. PRINCE2, PMBOK, SCRUM) from managing programmes, i.e. the process of managing a collection of projects (e.g. MSP). As an umbrella across projects and programs there is a very distinct process of managing an investment portfolio, which is about optimizing the corporate resource pool and funding to maximize benefit and return (e.g. MoP).

In turn, projects manage work specifics, such as the creation of new business processes to be performed by people, the development of software, and in a product company setting, marketing, and possibly some form of hardware engineering. Each of those have their own process areas which guide the workers to create outputs. In addition to that we may need processes to procure e.g. a Request for Information (RFI) or proposal (RFP) process, or a standardize procurement process (Vendor Management) which covers legal aspects such as master contracts etc.

In diagram the relationship of these process areas over time would look something like this:

Figure 1: Process layers relative to time

Note that portfolio management has no start nor an end, programmes have no defined end, other than when an outcome threshold has been reached, but projects do have ends, with marketing and vendor management surviving and extending beyond the project end / span of control.

2.1.2Understanding process

Business processes create repeatable experiences for the organisation’s customers, as is well described by E. Gerber in his books “the E-myth” (the enterprise myth).

Similarly, in any factory setting, “the process” ensures consistency of output. Both in business and production, the process churns out widgets, services etc. In the sixties, Deming etc started to measure the deviation and cost of fixing in case products or services do not meet their required quality or sameness. We humans love sameness, that’s why we have McDonalds, the Hilton, or Honda’s where the experience of the product or service is consistent and as can be expected.

The quality of a product results from the discipline and rigour of the process to produce it, as well as the ability (agility) to react to process deviations causing defects. In industry we measure that deviation as a deviation from the standard (the average). One standard deviation means some 32% of outputs have defects, 3standard deviations means 0.3% defects, 6standard deviations,aka six-sigma[1], means one deviation per 300,000 “productions”[2].

These are the origins of “Quality Assurance” or “Quality Management”, i.e. the management of the process to “assure” that the product or service is of consistent “quality” (quality is also ambiguous, see below).

2.1.3Introducing Assurance into the process mix

The word “assure” here deviates from the English language dictionary meaning. Where assure in the dictionary has a meaning of “guarantee” (as in “I assure you it will be done by tomorrow”) in any production process that guarantee is a guarantee with a margin of error. That margin of error is of importance when considering projects, and it is important when considering what we mean by “assurance”.

Unlike a production process where we service hundreds of thousands of customers in the same way, or produce hundreds of thousands of the same cars differing only in the minutest of detail, a project is defined as a unique undertaking with a definitive start, and a definitive end. As such we have a statistical sample of one only!

We cannot compare the output of project A with Project B, because if we could, project A and B would not be unique. Projects can only be compared in the way they are executed. A project manager manages the process of getting from start to finish, but what each project A, B, or Z is producing, is very different. The uniqueness aspect of project management makes it hard to “assure” the outcome with certainty such as the English dictionary defines it.

The problem of uniqueness / one-ness of projects, and how to ensure / “assure” that when we undertake a project it will deliver within its stated tolerances of time, cost, scope and quality, has been addressed in the 1970’s 80’s and 90’s through the emergence of the discipline of “Quality Management”. Think of works like the CMM and ISO-9000. The CMMemerged in 1991, and came about as a result of 4 years of research into what makes a measurable difference to time and cost, as well as time cost variance in delivering a software project to scope and to quality. The CMM finding: the difference is in the processes and practices performed at the project level. However, the most important (and least understood) key finding of the CMM:only when certain groupings of process areas are applied by the project, does the probability of on time / to cost delivery make a significant step-up in performance.

Figure 2: Applying specific groups of processes reduce time / cost overrun

Similarly to the CMM the ISO (international standards organisation) came up with ISO-9000 / ISO-9001. The ISO 9000 family of standards function by “assuring” / ensuring within statistical variation, that the processes are executed in such a way that products stemming from the process meet the required “quality” (we’ll get back to that). In the 1990’s the criticism of ISO-9000 was that work tuned into paper tigers, as a result of quality managers overstating what the organisation would perform. In ISO9000, if you say you will perform a process then it only gives you the tick if you do so. The way around it is to state that you’ll do as little as possible. The current 2015 version now has a risk based approach to “assurance”, “assurance” as in staying within acceptable risk / tolerance.

When we talk in the context of software development about process areas, think about: Requirements management, Project Planning (PP), Project Monitoring and Control (PMC), Configuration Management(CM), Product and Process Quality Assurance,Service Agreement Management, or at a level 3 capability, Requirements Definition (ensuring you have the right requirements as opposed to the requirements right), Risk Management, Verification, Validation, Technical Solution (design) etc.

When you consider these process areas, PP, PMC, CM are covered by standards such as PRINCE2 or PMBOK. Project management’s management of scope, in software development is achieved through the process area and practices of requirements management. Requirements, Verification, Validation etc are specifics managed under the umbrella of a PRINCE2, PMBOK or SCRUM method, like this:

Figure 3: Process layers relative to time

Here comes the crux:

to “assure” / guarantee the successful delivery of softwareprojects, project management alone, as in PRINCE2, is insufficient to make a measurable difference to time and cost variation. With just PRINCE2, you keep operating at CMMI level 1, i.e., on average (across 100+ projects) you will see a result of delivering twice over time / cost as estimated.

Intermediate conclusion 1

When we talk about assurance in the context of projects, the traditional Quality Management angle is to think of assurance as creating the process guidance to stay within tolerance. Assurance here does not mean “guaranteeing” the outcome. It is about assuring “guaranteeing” that we do the process as described,which if we do, will result in project deliveries within historic time / cost variation.

2.1.4But wait, there is more to assurance…

Due to consistent and persistent project time / cost overrun and delivery of project dubious outcomes, since the mid-2000’s (could be earlier) at the executive and board level there has been a demand for assurance (services) that projects will deliver their stated objectives.

The executive management (rightfully) wishes to know as early as possible what the “risk”, i.e. uncertainty is of projects not delivering, and to gain assurance how these projects will.

In situations where projects are dangerously close to failure often the executive may require assurance that the outputs of the project are “correct”.

Let’s unravel this:

  1. Regarding “risk, i.e. uncertainty is of projects not delivering, and to gain assurance how these projects will.” :
  2. As explained above, the risk of not delivering is reduced by applying those processes and practices which, if applied, will make a measurable difference to the reduction of time / cost variation [CMMI].
  3. To gain assurance that the project will deliver requires an independent assessment to which extent the project processes are setup to be able to assure management that the project can deliver as intended. These are often referred to as Independent Quality Assurance or IQA.
  4. In this context “assessment” could be interpreted as “audit”. Also in this context, the executive often includes the CFO, or projects actually “tree up” straight to the CFO.
    A CFO’s comes from the domain of finance and in the finance domain“assurance” and “audit” have a very specific meaning, involving very precise methods of determining where there is deviation. Hence for a CFO “assurance” may well mean an absolute guarantee that the project when delivering at a significant distance in the future, will deliver as intended. However, as shown in the previous section, there cannot be such an absolute guarantee, there can only be an assessment that due process is applied (i.e. an assurance of process application, which is not the same as an assurance of delivery).
  5. Please note that this is why the big four accountants firms (e.g. PwC) will have clauses in their IQA contracts to show this is not financial assurance or even remotely similar or associate with the rigour of a financial assurance (See Appendix B.6, Wikipedia on Assurance services, page 21).
  1. Regarding “assurance that the outputs of the project will actually be correct” :
  2. This is probably the most contentious and difficult area.
  3. In some recent instances of working on such assignments I [JWx] have come across management seeking an absolute guarantee that the software does what the requirements state. Problem is: mostly the requirements are unclear; at best the requirements are clear and it can be proven that they are met; in the latter case it could then still be that they are the wrong requirements and thus the wrong delivery. This is what the CMMI process areas of RD, REQM, VER and VAL try to minimize (minimize - statistically it cannot be ruled out).
  4. Sometimes the quest for “correct” is narrowed down by the assurance seeking party tomean for example “the software design is correct and extendable”. This is an even more of a contentious and flawed area. As every 201 engineering student can tell you, if we consider 100 designs of the same system, there will be a few bad designs, many good designs, and a few excellent designs. Furthermore “extendable” is a requirement, and how often do we see requirements that express “extendable” as a measurable, testable requirement? To state “it must be extendable” is a non-measurable requirement. How can one test that it can be extended in the way one may think about it at some future point in time?
  5. Often this area is referred to as Technical Quality Assurance.
Intermediate conclusion 2

The analysis in this section 2.1.x implies that when we speak of “Assurance” in the context of Project Quality Assurance, we have at least three types of assurance, and possibly four:

  1. Process Assurance – operating at the organisational level applicable to projects, programmes, etc, which we may wish to label as “proactive assurance”:
  2. through process definition and tailoring of international standards, organisational standards, and project circumstances, set up a process definition for the project, programme, portfolio etc, which assures the delivery of outcomes within reasonable time / cost / scope tolerance
  3. Independent Assurance – an independently of the organisation appointed assessment that:
  4. the project processes are defined and assured, as measured against the organisation’s standards, or industry standards which specify what will make a measurable difference to reduce the variation of time / cost and scope.

or that:

b.2the project defined processes are applied.

But assurance is NOT some function to guarantee the product outputs (that is a quality control function)

2.1.5Enter the Quality debate

An endless number of books contribute to the eternal quality debate. To cut to the chase in an engineering mind, quality means meeting the stated requirements and no more.

If quality to you is a plush couch, then state the requirements which would define your ideal couch (height of the fluffy poles of the plush, how deep you should sink into it, the colour measured by spectrum, the height, tilt, how hard wearing it is, and so on), and someone can make it.

As is often the case we won’t know it till we see it. That is why in software projects the processes aim at ensuring that we have the right requirements in the first place and then test that those requirements are met.

Quality Assurance then is not to be confused with quality control. Here is a nice definition found on the internet[3]:

  • Quality Assurance makes sure you are doing the right things, the right way.
  • Quality Control makes sure the results of what you've done are what you expected.

Quality Assurance these days for many people starts to mean “testing that it doesn’t break”. As we’ve seen above this is a definition that is just not appropriate to the complex domain of software development, contractual management and everything else that is involved. That something “doesn’t break” is one simple requirement:

  • Requirement 1 – the system is deemed accepted when it has zero severity-1 defects, n severity-2 defects, m severity-3 defects, with the severities as defined by the requirements Sev1, Sev2, Sev3.

2.2Definition PQA

Project Quality Assurance concerns itself with:

  1. (Process) Quality Assurance – at the organisational level:
  2. Definesthe process - through process definition and tailoring of international standards, organisational standards, and project circumstances, set up a process definition for the project, programme, portfolio etc, which assures the delivery of outcomes within reasonable time / cost / scope tolerance
  3. Assessesthe application of process - the process is of Quality if the execution (by people) adheres to the requirements which the process sets
  4. Measures and Analyses the process performance data through which assurance within tolerance can be provided
  1. Independent Quality Assurance – an independently of the organisation appointed assessment that:
  2. the project processes are defined and assured, as measured against the organisation’s standards, or industry standards which specify what will make a measurable difference to reduce the variation of time / cost and scope.

or that: