Verification and Examination Management of Complex Systems

As ship systems become more complex, with an increasing number of safety-critical functions, many interconnected subsystems, tight integration to other systems, and a large amount of potential failure modes, several industry parties have identified the need for improved methods for managing the verification and examination efforts of such complex systems. Such needs are even more prominent now that the marine and offshore industries are targeting more activities and operations in the Arctic environment. In this paper, a set of requirements and a method for verification and examination management are proposed for allocating examination efforts to selected subsystems. The method is based on a definition of a verification risk function for a given system topology and given requirements. The marginal verification risks for the subsystems may then be evaluated, so that examination efforts for the subsystem can be allocated. Two cases of requirements and systems are used to demonstrate the proposed method. The method establishes a systematic relationship between the verification loss, the logic system topology, verification method performance, examination stop criterion, the required examination effort, and a proposed sequence of examinations to reach the examination stop criterion.


Introduction
With the accelerated evolution of information and communication technology, the maritime industries have experienced in a short amount of time a significant change from conventional mechanical ships to modern computer-controlled ships.Ship system technology has to some extent developed and been taken into use faster than corresponding verification methods.In (Skjetne and Sørensen, 2004) a consortium of maritime industry partners expressed a need for further research to describe the experienced problems related to increased computer-based integration and software problems.As an answer to this, the need for managing the verification and examination efforts were identified by the in-dustry in the years 2005-2010 by, for instance, DNV, Kongsberg Maritime, Statoil, Farstad, Marine Cybernetics, and Global Maritime during the development of advanced verification and certification methods for complex ship systems (such as DP systems), resulting among others in the DNV GL Recommended Practice (DNV, 2012).An example of a more advanced verification method in the maritime industry is the industry-established Hardware-In-the-Loop (HIL) simulation method (Skjetne and Egeland, 2006), which makes it possible to generate many more detailed and low-level test cases compared to the traditional Failure Mode and Effect Analysis (FMEA) testing of redundant ship systems.The reason for this, as explained by Skjetne and Egeland (2006), is that the HIL test tool makes many more software and hardware functions testable in the target test system, that is, more functions are controllable (can be trigged and manipulated) and observable (the function behaviors can be observed and measured) by this test tool.
Besides control systems, also offshore operations are becoming more complex as new extreme frontiers are challenged.For instance, the reduced sea-ice extent in the Arctic due to global warming in recent years has provided new industrial opportunities.The shipping industry together with the Arctic countries has initiated development and increased use of the Northerns Sea Route for more efficient transportation from Europe to Asia.The offshore industry has shown an increased interest in integrated offshore operations for petroleum activities in Arctic ice-covered waters.Such operations are technically and physically more challenging than conventional open-water operations due to remoteness and general lack of infrastructure, low temperatures, darkness, and the presence of sea-ice and icebergs.Stationkeeping operations by position mooring (PM) or dynamic positioning (DP) are, as an example, challenging since ice forces are much stronger and rapidly varying compared to conventional openwater environmental forces, and the technical control systems have not been developed for the Arctic climate and ice loads.Despite this fact, it is said that the risk of offshore activities in the Arctics should not be higher than in the North Sea.Assuming then that consequences of an accident is higher in an ecological sensitive Arctic area, this means that the probability of an incidents must be reduced by additional technical and operational barriers.It follows from this that improved management of examination and verification methods of the new barriers is needed.
The new situation in computer-based control systems and more operations in extreme environments have raised some concerns, such as: • The new system topologies are more integrated on a ship-wide scale and become more complex.
The physical component topologies are well specified and can be assumed to be known for the verifiers, while software and computer-based communication topologies are not.
• The number of operational modes and combinations of user-allowed settings of the systems are increasing.
• Larger integrated operations in sensitive environments, like in the Arctics, require a better overall assessment of integrated functions, systems, and barriers against operational failure.Vessels should be verified as an integrated part of a larger system.
• There is a need for explicit assessment of the applicable failure modes, that is, to identify potential failure modes and to determine a relevant subset to verify while discriminating other failure modes.
• The potential verification scope of software functionality is very large, where the functionality is not specified in detail, and the inter-dependencies between software functions are unclear.
• There is a need to select the optimal sequence of examinations, to define adequate verification methods and sufficient quantities of examinations, under the assumption and general acceptance that verification cannot require complete coverage of all possible examinations.Hence, selection of the most beneficial examination method for a given requirement is needed.
• The system topologies or requirements structures may influence the need for examination effort, the selection of verification method, and the verification result.
In (Skjetne and Sørensen, 2004) it is stated that many different verification methods now exist for use in an industrial context, where testing by HIL simulation is one example of a verification method.Application of each such method require significant costs, especially if testing is to happen at sea trial for a vessel or within the Arctic environment.This further emphasizes the need for better management of verification.
Based on the above described situation, the industry partners of the project leading to (Skjetne and Sørensen, 2004) expressed a need for further research to obtain a more general understanding of how to describe and estimate verification contributions, and how to optimally put together a verification portfolio for a vessel or a system.In, for instance, the context of ship systems or integrated power systems, different types of representation and visual presentation of complex systems should be studied, and precise definitions of verification benefits should be proposed.Such definitions may then be used to collect empirical information about properties and relevance of existing verification methods and activities, for instance, standards like IEC 61508 (IEC, 2010), ISO 9000 types of assessments, class rules, certification, approval, manufacturing surveys, FMEAs, HIL simulation, software quality assurance techniques, dock trials, sea trials, field trials, and annual trials.
Given that sufficient knowledge related to the benefits of verification methods for various types of components or systems is established, the industry asks for a systematic approach to verification management.A key issue in verification management is to find methods for allocation and sequencing different verification activities (traditional surveys, FMEAs, HIL, and other methods) in different parts of the lifecycle of the ship or system.To achieve this, relationships between system complexities, operational modes, verification volumes, and verification confidence levels must be studied, including the possible need to set limits on how complex systems can be built in order to ensure sustainable development with verification within reasonable use of resources.
In financial audits, the auditors are applying audit risk concepts in their planning of audits.The audit planning and the proposed verification management are in general representing the same type of considerations and parameters to be used for allocating examination resources as proposed in this paper.In the text box below are given brief quotes of the main concepts relating to the financial audit planning and audit risk (Arens et al., 2006;AICPA, 2006).
Audit risk Audit risk (AR) is the risk that the auditor may unknowingly fail to appropriately modify his or her opinion on financial statements that are materially misstated.The model AR = RM M × DR expresses the general relationship of audit risk, the risks associated with the auditor's assessments of risk of material misstatement (RMM) (inherent and control risks), and the detection risk (DR).Risk of material misstatement Risk of material misstatement (RMM) is the product of inherent risk (IR) and control risk (CR).Control risk Control risk (CR) is the risk that a misstatement that could occur in a relevant assertion and that could be material, either individually or when aggregated with other misstatements, will not be prevented or detected on a timely basis by the entity's internal control.Detection risk Detection risk (DR) is the risk that the auditor will not detect a misstatement that exists in a relevant assertion that could be material, either individually or when aggregated with other misstatements.Detection risk is a function of the effectiveness of an audit procedure and of its application by the auditor.Detection risk cannot be reduced to zero because the auditor does not examine 100 percent of an account balance or a class of transactions and because of other factors.
The objectives of this article are to explain the industrial need for examination and verification management of complex systems, and then to propose a novel method for verification and examination management based on industrial needs and some concepts and prac-tices found in financial auditing.
The proposed method will be demonstrated through two case studies on how to manage the quantity and sequence of examination activities on subsystem level in order to reach an acceptable level of verification risk.The first case is one requirement to give a decision if the specified examination should be carried out.The second case illustrates verification management for a redundant system with a common component.

Problem formulation 2.1 The verification management approach
We define verification according to (IEC, 2010) as "confirmation by examination and provision of objective evidence that the requirements have been fulfilled".We consider a verification on a system -subsystemcomponent level, where the system is constructed by a set of subsystems {A, B, C, . ..}, and each subsystem is constructed by a set of components that all must function for the subsystem to function.The overall requirement is typically related to some main mode or system function (e.g.stationkeeping mode for a DP system) or an operation (e.g.Ice Management operation for an Arctic offshore operation).Correspondingly, we assume there is an overall requirement Y representing the overall system function.This is constructed by a set of main requirements {H A , H B , H C , . ..} corresponding to subsystems.Each main requirement H A is again constructed by a set of subrequirements {h A1 , h A2 , h A3 , . ..} on component level.
Verification management (VM) is proposed for costefficient verification of the overall system requirement Y .This involves establishing a system verification loss L, examination methods for each main requirement, an examination stop criterion Z for the overall verification process, and a decision rule of the verification result.Finally, VM must apply the decision rule for concluding the verification result -which is the result of the verification activity.
Examination management (EM) is a part of the verification management process.The examination management is to decide on zero examination effort or select a sequence of examination efforts {x A , x B , x C , . ..} by corresponding methods for the main requirements by means of an estimate of the verification risk Ψ subject to minimization.Based on the background situation and industry needs a method for examination management is proposed, including the following main elements: • A system is represented by an overall requirement Y .This is constructed by a set of main requirements, represented by the vector H = (H A , H B , H C , . ..), through a Boolean structure function Y = Φ(H).
• Each main requirement H j , j = A, B, C, . .., is assumed constructed from a set of subrequirements H j = {h j1 , h j2 , h j3 , . ..} that all must be satisfied for the corresponding main requirement to be satisfied, i.e.
• Each subrequirement h ji and each main requirement H j take a state value (True) if the requirement is 'compliant', and (False) if the requirement is 'noncompliant'.
• Based on previous experience, statistics, or conservative estimates, we assume a priori (before examination; denoted by subscript 0) knowledge of the probability p c 0,Hj = P (H j ) that requirement H j is compliant.Conversely, we assume the probability p nc 0,Hj = P (¬H j ) that H j is noncompliant.• The examination of a main requirement H j is characterised by the subset of 'examined subrequirements' E j (x j ) ⊆ H j and the subset of 'unexamined subrequirements' U j (x j ) ⊆ H j , where x j is some examination effort for H j .
• For the overall verification activity there is a stop criterion Z for stopping further examination (Z is a Boolean expression).
• To the overall requirement Y is associated a potential loss L due to potentially wrong verification result (for instance that a noncompliant requirement is accepted).
Note the distinction between the main requirement H j , which is a scalar Boolean state variable, and the set H j that merely lists the "set of subrequirements" that H j is constructed from.The restriction that all subrequirements in H j must be compliant (true) for H j to be compliant (true) means that we can relate the logical outcome of H j to the "size" of the subset in H j with equivalent outcome.

Illustrating example to clarify the concepts
Consider the situation where a forklift shall either be accepted or rejected by the buyer.The machine shall be accepted if the requirement Y is complied with.Assume that there is given an a priori probability p c 0,Y = P (Y ) that the requirement Y is complied with, and conversely p nc 0,Y = P (¬Y ) that the requirement is not complied with.
The buyer is offered the possibility to carry out a verification and examination effort x, and this examination will clarify for certain if the requirements are complied with or not.The value of the machine is L, and in the case that the buyer accepts the machine, he has to pay the value L of the machine.This means that if the forklift is accepted without examination, the possible loss may be L for the buyer.We will revisit this example in the first case study in order to demonstrate the detailed steps of the verification management method.

Abbreviations and notations
Boolean operators are ∧ for logical AND, ∨ for logical OR, and ¬ for negation.In addition we have the following nomenclature: • A, B, C, . ..: Applied names of subsystems.
• h ji : Subrequirements for the subsystem j = A, B, C, . .., i = 1, 2, . . ., M j , typically representing components or functions.Each subrequirement is assumed statistically mutually independent and takes a state value 'True' if the requirement is compliant, and 'False' if it is noncompliant.
• H j : Main requirements, j = A, B, C, . .., constructed by a series of subrequirements.Each main requirement takes a state value 'True' if the requirement is compliant and 'False' if it is noncompliant.All main requirements are collected in a state vector H = (H A , H B , H C , . ..).
• Y : Overall system requirement constructed by a Boolean structure function of main requirements Y = Φ(H).
• H j : Set of subrequirements for • H c j : The subset of subrequirements, H c j ⊆ H j , that are compliant.
• H nc j : The subset of subrequirements, H nc j ⊆ H j , that are noncompliant.This gives H c j ∪H nc j = H j .• x j : Examination effort by means of an examination method for requirement H j .We collect all efforts into a vector x = (x A , x B , x C , . ..).
• E j (x j ): Examined set of subrequirements for H j as a function of examination effort x j .Note: • e j (x j ): Scalar measure of examined subrequirements in H j .
We collect all examination functions into a vector e(x) = (e A (x A ), e B (x B ), e C (x C ), . ..).
• U j (x j ): Unexamined set of subrequirements for H j as a function of examination effort x j .Note: • u j (x j ): Scalar measure of unexamined subrequirements in H j .We collect all the functions into a vector u • L: The loss which may follow as a consequence of wrong verification decision for requirement Y .
• Ψ(x): Verification risk for the system requirement Y .Initial verification risk is denoted Ψ 0 .
• ∂Ψ ∂xj : Sensitivity of verification risk Ψ with respect to effort x j for the respective verification method.
• Z: Examination stop criterion, assumed to take a logic value (true or false).
Note that scripted notation is used for sets of requirements, e.g.H, E, and U.If we in the text discuss a single main requirement, we typically use j = A and H A (without loss of generality).In most of the paper, this is the case.

Proposed examination and verification management method
Verification or examination management shall recommend if and how much examination effort x one should perform before accepting or rejecting conformance to the given requirement Y .We make the assumptions: • Subsystems, components, and examination of subrequirements are assumed to be statistically mutually independent.
• In a complex system consisting of a number of subsystems (A, B, C, . ..) the examination is assumed to be performed at the subsystem level and the result of the examination can be aggregated to the top system level by means of standard risk and reliability methods.All requirements are possible to be examined.
• If all subrequirements are examined and thus all main requirements verified, then a completely correct verification decision will be made for Y .This assumes that a selected verification method gives perfect certainty of compliance if performed.
• An examination of a requirement shall establish the state of the requirement, being either compliant or non-compliant.
• The state of a requirement is assumed to be unchanged due to an examination effort.This implies that examination must be nondestructive.We note, however, that for some types of examination the testing may influence the state of the equipment to be improved or possibly to become worse (destructive testing).
• In the case that a noncompliant requirement is identified by the examination, it has to be decided if the state of the requirement (or the component) should be fixed, or if the state shall remain noncompliant as identified.
Note that the verification risk will be the same in both cases if the noncompliant requirement is restored or not, since verification risk is only related to the knowledge of the requirement's state for making the correct verification decision.

Requirements and outcome of verification decision
The overall requirement Y may be a single requirement, Y = H A , or a complex Boolean expression containing a number of requirements.This is generally represented by a Boolean structure function Each main requirement H A is constructed by a series of subrequirements to be satisfied, that is, The corresponding set of subrequirements, e.g.H A = {h A1 , h A2 , h A3 }, may originate from standards, class rules, recommended practices, functional specifications, or be agreed between the user of the verification result and the verifier.
The verifier's task is then to decide on an examination scope for the requirements, perform the planned examinations, and thereby provide evidence for accepting or rejecting the requirements.To be able to make a decision regarding whether to accept or reject the requirement, the verifier must choose a sufficient set of examinations.Since the main requirement H A logically needs all respective subrequirements to be compliant, a scope of examinations of the subrequirements is needed.The examination may include verification of the complete set of subrequirements in H A , a subset of H A , or no examination at all.The outcome of such a process may in principle be four different scenarios, as indicated in Figure 4: 1. H A is actually true and verifier accepts requirements: Right decision, green box.
2. H A is actually true but verifier rejects requirements: Wrong decision, yellow box, Type I error.
3. H A is actually false but verifier accepts requirements: Wrong decision, red box, Type II error.
4. H A is actually false and verifier rejects requirements: Right decision, green box.

Verification loss
In this paper the outcome that a wrong verification decision, by accepting a noncompliant requirement, is denoted as a Type II error, which is similar to the terminology used in hypothesis testing (NIST/SEMATECH, 2014).Obviously, the critical requirements are those that in reality are noncompliant and unexamined, that is, those in the set H nc A ∩ U A (x A ). Verification loss L is the worst-case loss that may follow as a result of a wrong verification decision for a Type II error.L is the consequence of the verification decision for a given stakeholder (owner, yard, verifier, etc.) in the case that requirement Y is wrongly accepted in the verification decision.This value should be established by the verifier together with the stakeholders applying the verification result.The loss due to a Type I error is assumed smaller and, thus, not considered in this paper.
The verification loss parameter L may be described by consequence classes related to, for instance: • Fatalities/injuries from accidents.
• Loss of facilities.
• Income losses due to operational unavailability.
• System operational risk; risk without barriers or risk with barriers.
The idea in this context is to assume that the verification loss L can be established upfront the verification work and be given as a prerequisite for the verification and examination management.At the start of the verification, no examination work x has been done and the potential verification loss is L.
For a given loss value L, a conservative definition of overall verification risk Ψ can be proposed as: (1) In this paper only the potential loss of a Type II error is considered, but it is possible to include more error types and loss effects.
In the conservative definition of the verification risk it is assumed that no effects of a priori knowledge, or examination, or other effects (degradation) have been taken into account in the estimate Ψ.Compared to a normal risk equation, this definition assumes by its conservative nature that the a priori probability of Y being noncompliant (false) is equal to 1.

A priori probability of state of requirement
In most cases the verification manager has some presumed and possibly conservative a priori information about the expected (probable) outcome of an examination of H A .The verifier may decide to not perform any examination and only base the verification decision on the a priori probability p c 0,H A = P (H A ). Another general assumption for verification and examination management is that exhaustive examination of a complex system and complex requirements is often not possible.This means that the practical verification results may only be possible based on a partial examination or no examination at all.Another situation is when the requirement H A consists of many subrequirements {h A1 , h A2 , h A3 , . ..},where the verifier must evaluate which subrequirement that shall be given the highest examination priority and which shall be given less effort.
We assume that the set of subrequirements can be divided into a set of compliant requirements H c A and a set of noncompliant requirements H nc A , as illustrated in Figure 5.
for the main requirement H A .The a priori assumed initial state of H A could be given by the estimated probabilities P (H A ) = p c 0,H A and P (¬H A ) = p nc 0,H A .This is used for initiating the verification and examination management process.
There is then a need for describing some a priori knowledge of the initial (or unexamined) state of the requirements.This a priori information is given as a probability that the requirement H A is initially compliant, that is, where the initial estimate could, for instance, be that p c 0,H A = 0.5.Conversely, the a priori information could be given as the probability of H A being noncompliant, that is, We let Φ c a be the algebraic function relating the probabilities of the main requirements H being compliant to the probability of the Boolean function Y = Φ(H) being compliant.Equivalently, we let Φ nc a relate the probabilities of the main requirements H being noncompliant to Y being noncompliant.This is simply obtained by the substitutions where p * 0,H = (p * 0,H A , p * 0,H B , p * 0,H C , . ..).Based on the specified verification loss and a priori given probabilities, we propose to estimate the a priori verification risk as a fraction of the verification loss by This means that initially with no examination (x = 0), the verification risk is equal to the risk of the overall system requirement Y being noncompliant.

Examination of requirements
In a verification and examination activity the verifier will normally examine the specified sets of requirements H A , H B , H C , . . .before concluding the verification decision (accept/reject) for Y .For each main requirement H A , the examined set of subrequirements is expressed by E A (x A ), which grows with increasing examination effort x A .The unexamined part is the set U A (x A ) of subrequirements that have not yet been examined.This set decreases with increasing examination effort x A .It follows that ) and E A (x A ) ∩ U A (x A ) = ∅ as illustrated in Figure 6.At the start of the examination (x A = 0) the examination status of the requirement H A is 'not examined'.Let a scalar characteristics function u A (x A ) describe how the unexamined set U A (x) decreases for increasing examination effort, that is, u A (x A ) ∈ [0, 1] with u A (0) = 1 and u A (x A ) = 0 when the requirement H A has been completely examined -in which case U A (x A ) = ∅.Similarly, we define the scalar characteristics function e A (x A ) to describe how the set E A (x A ) grows with increased examination effort.Without loss of generality, we let e A (x A ) ∈ [0, 1] with e A (0) = 0 and e A (x A ) = 1 when the requirement H A has been completely examined.We choose u A (x A ) and e A (x A ) such that u A (x A ) + e A (x a ) = 1.(or e A (0) = 0) for x A = 0 and examined e A (x A ) = 1 (or u A (x A ) = 0) at x A = x 1 .
For complex systems and complex requirements, exhaustive examinations of all requirements and subrequirements can in practice not be accomplished as the costs of the examinations will typically be too large compared to the possible benefits.To quantity suf-ficient examinations of the requirements in order to make the verification process cost-efficient is the key result to be established by the verification and examination management process.
Figure 8: Examples of functions for unexamined parts of H.Such curves may be relevant and applicable for complex requirements (upper curve could e.g.be HIL, while the lower curve could be FMEA).
The characteristics function u A (x A ) may take different shapes.If, for instance, a single test can verify the status of a single requirement H A , then u(x A ) may take the shape illustrated in Figure 7.If H A is constructed by a series of subrequirements, then u A (x) may take the form of a staircase function linearly stepping down from one towards zero as all subrequirement are tested.However, typically a requirement will consist of a large number of subrequirements, each subrequirement will possibly need several tests, examination will have an initial cost and need preparations, and results will need post-analysis.Thus, u A (x A ) will more generally be characterized by some curve as illustrated in Figure 8.
Correspondingly, we assume that the characteristics function u A (x A ) describing the examination of a requirement H A is a continuous function that monotonically decreases with increasing examination effort x A .
An important assumption made, is that if u A (x A ) = 0 and the verifier has completely examined the set of requirements H A , then the verification decision will always be correct -either accepting or rejecting H A (green boxes outcome in Figure 4).This means that the intended effect of increased examination is to reduce the number of unexamined subrequirements that may lead to a verification loss; see Figure 9.However, since some examinations (efforts) will give larger verification risk reduction than others, it is important in verification and examination management to prioritize the examinations based on cost-benefit considerations.
In order to estimate how the expected verification risk changes with examination effort, we note that the probability of Type II errors becomes Using the characteristics function u A (x A ) as the mea- sure of U A (x A ), we analytically calculate (10) by Recalling the function Φ nc a (•) in ( 8), the proposed measure of the verification risk as a function of examination effort is then where • denotes the element-wise product between the two corresponding vectors, that is As illustrated in Figure 10, this shows that the verification risk initially (with no examination x = 0 such that u(x) = (1, 1, 1, . ..)) takes the value of the a priori estimated risk Ψ 0 .Then the verification risk reduces with increased examination effort according to the examination function u(x).If for some effort x 1 the requirement Y is completeley examined, then u(x 1 ) = (0, 0, 0, . ..) and Ψ(x 1 ) = 0.

Marginal verification risk
In order to determine the effect of a specific examination effort x j , one could elaborate Ψ(x) by calculating ∂Ψ/∂x j , and use this expression for marginal verification risk efficiency with regard to the examination effort x j of requirements in H j , that is Figure 10: The verification risk measure is based on the verification loss L, the a priori estimated state of Y , and the effect u(x) of increasing examination effort x by the given examination methods.
For complex systems or requirements consisting of many subrequirements (for instance related to different subsystems or components {A, B, C . ..}), the expression ∂Ψ(x)/∂x j can be further elaborated in order to find the marginal verification risk (also called the Birnbaums measure (Rausand and Høyland, 2004)) for examination of a given requirement.Verification and examination management is now to determine the sequence of examinations that should be carried out among the main requirements H j before concluding the outcome of verification.The sequence of examinations can be decided by selecting the requirements that achieve largest risk reduction in Ψ(x)/∂x j (steepest decent) for a given examination effort x j .

Stop examination criteria
The verification management model handles verifications that potentially may contain large quantities of effort.In order to limit the examination effort, the model proposes a criterion for stopping the examination and concluding the verification.Two examples of possible stop criteria are illustrated in Figure 11.
The first criterion Z 1 illustrated in Figure 11 is related to the marginal change of the verification risk function.For example, Z 1 ='true' if ∂Ψ(x)/∂x j > −1 for a given method and corresponding effort x j then the examination should be stopped.This means that the number −1 is an example threshold indicating when the marginal verification risk reduction is less than or equal to the marginal examination effort x j .
The other proposed criterion Z 2 is related to the achieved level of verification risk.
For example, Z 2 ='true' if Ψ(x) ≤ 20 000 after an effort x, where the number 20 000 is an example threshold to be selected.We return to the illustrating example of Section 2.2, where a forklift shall either be accepted or rejected by the buyer.The forklift shall be accepted if the single requirement Y is complied with.Assume that there is given a priori probability p c 0,Y = P (Y ) that the requirement Y is complied with, or p nc 0,Y = P (¬Y ) that the requirement is not complied with.
The buyer is offered the possibility to carry out verification and examination at cost x 1 , where this examination will clarify for certain if the requirement is complied with or not; see Figure 3.The value of the forklift is L. In the case that the buyer accepts the forklift, he has to pay the value L. The verification management question is to decide whether the examination and verification shall be carried out at an effort of x 1 ; see Figure 7.
In this case we have Φ nc a (p nc 0,Y ) = p nc 0,Y and Let the examination stop criterion Z ='true' be to stop examination when the examination effort exceeds the verification risk, that is, Z = {x 1 > Ψ(x)}.Inserting the expression (15) for the verification risk gives the stop criterion Assume that a priori it is a 10% probability that the requirement Y is noncompliant, that is, p nc 0,Y = P (¬Y ) = 0.1.Initially, we then get the condition on examination effort (x 1 ≤ 0.1L) for performing the examination.This means that if the cost of performing the examination is less than 10% of the cost L of the machine, then the buyer should decide to perform the examination of Y .
4.2 Case 2. Verification and examination management of a redundant system with one common component We are given a redundant power generating and thruster system as indicated in the block diagram in Figure 12.The requirement to this system may be described by and by de Morgan's theorem we get the negated requirement The probability that the requirement is noncompliant becomes where are the a priori probabilities that respective requirements H A , H B , H C are not initially complied with.Assume that the verification loss is of Type II and this loss is represented by L, and let x A , x B , x C be the respective examination efforts on subsystems A, B, C. Based on the above formula for P (¬Y ) and ( 12) the verification risk now becomes where it is assumed that the examinations of the subsystems A, B, C are statistically independent.The marginal verification risk (Birnbaums measure) with regards to examination efforts can now be described analytically by The stop examination criterion Z is defined as and the verification loss is L = 1000 000 (NOK).Let the a priori assumptions of compliance (c) and noncompliance (nc) of the requirements H A , H B , H C be p c 0,H A = 0.8, p nc 0,H A = 0.2 (26) and assume that the subsystems, requirements, and examinations for A, B, C are statistically independent.
The examination characteristics functions are all assumed expressed as with j = A, B, C, where x j for instance corresponds to days of examination.This gives which is inserted into the verification risk and marginal verification risk equations for the A, B, C subsystems.
The model has been implemented in an Excel spreadsheet, which is used for proposing the examination efforts and the sequence of the efforts.Table 1 shows the sequence of the examinations {x C , x A , x C , x B } selected on the basis of the marginal verification risk for the requirements.The examination is stopped when the verification risk is below 50 000 as {Z = 43000 < 50000} satisfies the given stop examination criterion.At this stage the verifier will make the verification decision either to accept or reject the Y requirement.
When the stop criterion is reached, the result is a proposed examination effort of x A = 1 day on the Asystem, x B = 1 day on the B-system, and x C = 2 days on the C-system, in total 4 days of examinations.

Results and discussion
This paper reports the expressed original industry need for examination and verification management as formulated in 2011.In the RCN research projects "D2V" (RCN project no.210670) and "Arctic DP" (RCN project no.199567) a conceptual examination and verification management (VM/EM) model was proposed in the years 2012-2014 and presented in this paper.
The main properties of the VM/EM model are: • The VM/EM method is based on a holistic topdown approach and based on examination of combinations of subsystems.A top-down recursive method to any level of detail of subsystems is presented.
• Based on possible consequences of Type II errors, the concept of 'verification risk' has been proposed as the key parameter to be reduced by means of examination efforts.Verification risk is to be used as the main parameter for selecting and managing examination efforts.
• The VM/EM method establishes relations between the initial verification loss, the complex system design topology, verification method performance, and the required examination effort to achieve the examination stop criterion.
• Establishment of u-functions for describing relation between the examined parts of a requirement and the examination effort.An assumption made is that different u-functions may have different forms, and that some barriers might be more efficient to examine than the others in order to reduce the overall verification risk with lowest use of examination resources.
• The VM/EM method allows for complex system descriptions using standard Boolean and reliability methods, such as de Morgans theorem, Birnbaums measure, reliability block diagrams (RBD), event tree analysis (ETA), and fault tree analysis (FTA).
• Requirement for an examination stop criterion to clarify when examination should be stopped.
• The Excel implementation model is quite simple and intuitive, although the background analytical equations for a medium size system might be quite large and complex.
The paper provides two simple verification management cases based on an Excel implementation.The cases demonstrate how the verification and examination management process of examination efforts may work in a general and in a practical manner.In future, the steepest decent algorithm to find the sequence of examinations should be implemented in an optimization-based framework.
The model requires that the user must be more explicit on specification of the verification task to be carried out, compared to a traditional planning of examination efforts.These issues are normally taken implicitly into consideration in the approval or verification processes.However, in the proposed verification risk model it is required to be more explicit on topics like: • Potential verification consequence loss L.
• Specification of complex requirements through the Boolean structure function Y = Φ(H).
• Examination stop criterion Z on how accurate and how much cost or effort that should be spent on reaching the examination result and then make the verification decision.
The model allows for scaling up to a large number of components and subsystems and their corresponding requirements.This will also make it possible to select parts of the model that could be modeled in more detail.
Examination and verification management as indicated in this paper is mathematical and will produce the same result every time.However, the initial assumptions of a priori values and u-functions will be based on various types of considerations that could result in different results from time to time.
The model seems to be robust, and the resulting distribution of examination efforts in the different subsystems will probably depend on the component position in the system topology.The design of the system topology strongly influences the efficiency of examination efforts in the different subsystems.
In the approach described in this paper, the negative concept 'verification risk' has been chosen to be the key concept for selecting examination efforts.In the very beginning of this development process, the positive concept of 'verification benefit' was proposed as the key concept.However, at a certain stage the conformity to audit risk concepts motivated the verification risk concept, and the mathematical expressions for verification risk was perceived to be easier to handle in calculations than the expressions for confirmation and benefit.
Complex systems often have a high number of system operational modes that should be verified.Such advanced modeling could be included in future development of the models.
The estimation method demonstrated in this paper is based on a manual selection method of one single step.It is obviously possible to carry out the estimation with more automatic methods and this should be considered in the future when more basic knowledge about the prerequisites discussed above have been elaborated and justified.

Conclusion
The main result of the work reported in this paper is a proposal of a conceptual framework for verification and examination management as requested by the verification industry in 2010-2011.The main requested issue at that time was to manage how much examination and verification effort should be allocated on the various parts of a system and how the sequence of such efforts should be distributed on the subsystem parts.Another important issue was to estimate the effect of different verification and examination methods.The main lesson learned by the work is that it is possible to model and organize the selection of the examination effort for complex systems in the proposed manner with the given examination and verification risk management assumptions.
The VM/EM modeling is a simple and flexible approach that is similar to existing models for expressing reliability and risk.An extended verification risk model may make it possible to model a wide range of system requirements and system topologies.
The conceptual model clarifies and specifies key concepts regarding verification and examination management.The verification risk concept also has similarities to the audit risk method applied in planning of financial auditing.
Complex systems or operations have complex logic relations between subsystems and subfunctions.These relations have effects on the overall system properties (e.g.reliability, availability, restoration time, risk, etc.) The relations also have effects on the required examination efforts that may be needed in order to achieve an overall goal on verification risk at system level.

Figure 1 :
Figure 1: Illustration of the verification process.

Figure 3 :
Figure 3: The buyer of the forklift may examine the forklift before accepting or rejecting it.

Figure 4 :
Figure 4: Possible outcomes of verification decision.

Figure 5 :
Figure 5: Illustration of the set of subrequirementsH A = H c A ∪ H nc Afor the main requirement H A .The a priori assumed initial state of H A could be given by the estimated probabilities P (H A ) = p c 0,H A and P (¬H A ) = p nc 0,H A .This is used for initiating the verification and examination management process.

Figure 6 :
Figure 6: The set H A of subrequirements is divided into examined and unexamined requirements.Initially, all requirements in H A are unexamined such that U A (0) = H A and E A (0) = ∅.When effort x A is increased, the set E A (x A ) will grow and U A (x A ) will decrease.

Figure 7 :
Figure 7: A requirement is unexamined u A (0) = 1 (or e A (0) = 0) for x A = 0 and examined e A (x A ) = 1 (or u A (x A ) = 0) at x A = x 1 .

Figure 9 :
Figure 9: Illustration of the sets of subrequirements H c A , H nc A , and how these are overlapped with E A (x A ) and U A (x A ).The verification risk is estimated by means of the intersection (red part) of the unexamined set U A (x A ) and the set of noncompliant requirements H nc A .

Figure 11 :
Figure 11: Examples of examination stop criteria Z 1and Z 2 , where Z 1 dictates a stop when the marginal value of verification risk (∂Ψ/∂x) reduction is lower than the marginal examination effort ∂x, and Z 2 dictates a stop when the overall verification risk is below the value 20.000.

Figure 12 :
Figure 12: A DP vessel is arranged with 4 trusters and 4 diesel-generators.The A-side (port) and B-side (starboard) consist of switchboard SW BA and switchboard SW BB, each with 2 connected diesel-generators and 2 thrusters, respectively.The switchboards SW BA and SW BB are connected with a bus-tie breaker X, labeled subsystem C.

Figure 13 :
Figure 13: Reliability diagram of complex system (A ∨ B) ∧ C. The a priori assumptions for the compliant parts and the unexamined parts for each subsystem (or subrequirements) are indicated.The verification and examination management objective is to estimate the individual x A , x B , x C examination efforts in order to fulfill the stop criterion (Z).

Table 1 :
Verification and examination management applied to find an effective sequence of examination for the subsystems A, B, C. The table shows the proposed sequence of examination calculated from a steepest decent approach.The red numbers illustrate the information used for selecting examination.The resulting examination effort becomes: (x A = 1, x B = 1, x C = 2).