DRAFT DOCUMENT FOR PUBLIC COMMENT
Draft Failure Definitions and Scoring Criteria
for the Gun Safety Technology Challenge
National Institute of Justice
June 2016
Introduction
This document has been developed for the National Institute of Justice (NIJ) Gun Safety Technology Challenge. It describes the failuredefinition (FD)and scoringcriteria (SC) that will be usedto “score”test events that occurduring the testingof handguns, such as pistols and revolvers,in the Challenge. These firearms or firearms accessories can be understood to useintegrated components that exclusively permit an authorized user or set of users to operate or fire the gun and automatically deactivate it under a set of specific circumstances, reducing the chances of accidental or purposeful use by an unauthorized user. The integrated gun safety technology may include different authentication technologies such asradio frequency identification and fingerprint sensors.
This document follows the testing methodology published in Draft Test Procedures for the Gun Safety Technology Challenge to provide meaningful comparison between similar firearms in order to determine whether the reliability of the tested firearms differs significantly based on performance. Testing and evaluation is designed to prioritize the collection and use of data that can substantiate conclusions about the relative performance of firearms such that firearms with and without advanced gun safety technology that are similar with respect to type, form factor, caliber, and other physical characteristics are tested and evaluated using a common methodology and equivalent ammunition. Testing and evaluation is not designed to provide comparison of test results against absolute performance requirements or safety criteria, but rather to provide a meaningful comparison of test results of one firearm against another, similar firearm, or a firearm with and without a relevant safety accessory. The endresultof thescoringprocess is to provide“scored”data points in order to form a basis to determine whether the addition of a smart gun technology does or does not significantly reduce the reliability of the firearm system compared to existing firearms.
Together,the FD and SC serve as atoolforguidingareliabilityscoring panelthrough thetestevent scoringprocess, with the intent of eliminatingasmuch subjectivityfrom theprocessas possible. Thepurpose ofthe FD is to define degraded handgunperformance or functionalitywhich is considered afailurewhen evidenced bya malfunction. Thepurpose of the SC is to defineaspecificand agreed upon processforscoringtest events which occur during firearmtesting. Thescoringprocess involves thecharacterization ofeachtest event into the proper categorybased upon its impact on thefirearm’s operational performance or functionality,followed byattributingthecause of thetest eventto aparticular cause. The content of thisFDSCfollows U.S. Army guidance, such as Guidelinesfor Developing ReliabilityFailure Definition and Scoring Criteria,to characterize theimpact of malfunctions on the handguns tested in the Challenge,consistentwith the failuredefinitions that havetraditionallybeen appliedto smallarms.
Please direct any feedback on this document by email to .
1Scope
1.1This document describes the failuredefinition (FD)and scoringcriteria (SC) that will be usedto “score” test events that occurduring the testingof handguns, such as pistols and revolvers, in the Challenge.
1.2This document follows the testing methodology published in Draft Test Procedures for the Gun Safety Technology Challengeto provide meaningful comparison between similar firearms in order to determine whether the reliability of the tested firearms differs significantly based on performance.
1.3Testing and evaluation is designed to determine whether the addition of a smart gun technology does or does not significantly reduce the reliability of the firearm system, as compared to existing firearms.
1.4Reliability shall be assessed by a panel of qualified experts familiar with testing and evaluation of firearms using the FDSC in this document.
1.5Inherenthardware failures shall be recorded and appropriately scored according to the FDSC.
1.6Induced failures attributable to the operator, technical documentation, maintenance, training, andapplicable support equipment shall be recorded and appropriately scored according to the FDSC. Induced failures areincluded in theoverall evaluation to capture andconveythe impact of deficiencies whichare rootedin sources other than hardware,but which ultimatelymanifest themselves in the form ofequipment malfunctions,in order to provide a more complete picture of performance to the end user.
1.7Malfunctions of the firearms induced by ammunitionfailure shall be recorded and appropriately scored according to the FD, however further separate assessment of ammunition reliability is outside the scope of this document.
2References
Draft Test Procedures for the Gun Safety Technology Challenge, National Institute of Justice, Washington, DC, April 2016,
Guidelinesfor Developing ReliabilityFailure Definition and Scoring Criteria, 3rd ed., U.S. Army Training and Doctrine Command, Army Capabilities Integration Center, April 2012.
Charles E. Ebeling, An Introduction to Reliability and Maintainability Engineering, 2nd ed., (Long Grove, IL: Waveland Press), 2010.
Siegmund Halpern, The Assurance Sciences: An Introduction to Quality Control
and Reliability (Englewood Cliffs, NJ: Prentice-Hall), 1978.
3Terms and definitions
Reliability
The probability that a device will perform its intended function for a specified period of time under stated conditions.[1]
Maintainability
The probability that a failed component or system will be restored or repaired to a specified condition within a period of time when maintenance is performed in accordance with prescribed procedures.[2]
Malfunction
A test event where the firearm does not perform its intended function properly.
Test event
Any occurrence during testing, whether expected or unexpected, that requires the operator to make an adjustment or take corrective action.
Operator
Individual person using the firearm.
4Failure definition
The failure definition formally defines what constitutes a failure as it relates to testing the firearm’s essential functions.
4.1General considerations
4.1.1Observed test events are divided into three general categories:
—“Failure” events related to reliability;
—“Non-failure” events related to maintainability; and
—Events or failures not related to reliability or maintainability.
4.1.2A failure related to reliability is characterized byamalfunction in which thefirearmfails to perform its intended function properly.
4.1.3Anytest event that involves a malfunctionofthefirearmshallbe scored asa “failure”forthepurpose of evaluating reliability.
4.1.4Thecauseshall be assignedafter thefailurehas beenscoredby referencingtheestablished event chargeabilitycategories.
4.1.5Asfailuresgenerate maintenancedemands, atest event shouldstillbeclassified as afailureeven if “corrective”actionto address the malfunction may bedeferred until testinghas beencompleted.
4.1.6Any observed problem that does notrequireimmediate corrective action and which can be remedied through theexecution ofa routineoperatingprocedure,such as an adjustment,prescribed in thefirearm’sinstructional materials should not be scored as a failure.
4.1.7Any expected test events that can be considered exceptions to exclude from scoring as a failure shall be documented in writing prior to testing.
4.1.8Any test methods that may not conform to, may deviatesignificantlyfrom, ormay be obviously in conflict with the acceptable range of test firearm use shall be identified and documented in writing prior to testing.
4.1.9Inorder for a deficiencyof thefirearm to be included in the evaluation of reliability, the firearmmust first be capable of performingthe functionto which the deficiency pertains.
4.1.10If the firearmis incapable ofperformingaparticularfunction, then amalfunction related to that function cannot have occurred.
4.1.11Performance limitations caused by design deficiencies where no malfunction has occurred shallnotbescored as failures related to reliability,and shouldbe evaluated as a performance limitation.
4.1.12Failureto observethis protocol willresultin thefirearmbeingpenalized twice forthe same test event byimproperlychargingthefirearmwith a reliability-related failurein addition to a performance-typefailureevent.
4.2Failure events related to reliability
4.2.1A test eventmeetingthe definition of a “failure”related to reliability shallbescored in accordancewith the failurecategories in this section.
4.2.2Failures related to reliability are divided into three general categories:
—Essential function failures (EFF);
—Non-essential function failures (NEFF); and
—Dependent malfunctions (DM).
4.2.3Essentialfunctionfailure (EFF)
An EFF is a failure that renders thefirearmincapable ofperformingoneormore essential functions, or which degrades the abilityto properlyperform anessential function to thepoint wheretheresidual utilityofthe function is deemed“operationally”inadequate.
Essentialfunctionfailuresare further characterized into threesubcategoriesbylevel ofcriticalityin accordancewith the followinghistorical small arms failure definitions:
—A ClassI EFFis an immediatelyclearable failure or stoppageof thefirearm adversely impacting one or more essential functions that the operatorcan correctin approximately10 secondsorlessthroughthe execution ofimmediate action procedures, such asremovalandreinstallation ofthe magazine ormanuallyremovingan ammunition round or emptycasing;
—A ClassII EFF is anoperatorclearable failure or stoppageof thefirearmadversely impacting one or more essential functions which requires the operator to take morethan 10 seconds to correctthrough execution of prescribed operatorlevel troubleshooting and correctiveactionprocedures, either with orwithoutthe useof common tools orcleaningkitcomponents; and
—A ClassIII EFF isa severefailure or stoppageof thefirearm that theoperator is unableto correctthrough execution ofprescribed operator level troubleshootingandcorrective action procedures, either because the malfunction requireshigher level maintenancethat is non-deferrable,orthe need fortools, equipment, or parts go beyond common tools orcleaningkitcomponents.
4.2.4Non-essentialfunctionfailure (NEFF)
A NEFF is a failure that renders thefirearmincapable ofperformingoneor morenon-essential functions, but which does notadversely or significantlyimpact the abilityof the firearm to perform anessential function.
ANEFFmay beassigned toagroupofrecurringoperator actions of a particular typethat,while theymayfitin the categoryofroutineoperating procedures[in section 4.x], havebecome so frequent that theconstant interruption becomes asignificant problem.
4.2.5Dependent malfunction (DM)
A DM is afailurecaused by or directlyattributable to another“primary”test event, such as apreceding or prior malfunction of the firearm. To characterize a malfunction as a DM, analysisof the failureshallpositively confirm that it occurred as the direct resultofaspecifiedprimaryevent.
Thescoreultimately assigned to theprimaryeventshallbebased onthe overallimpactorseverityofboth test events on system functionality. If the impact of thedependent malfunction wasgreaterthan that ofthe primaryevent, then thescore assigned to the primaryevent should reflect that level of severity, based on the secondary failure’s functional impact.
4.2.6Reliability-related failureevent chargeabilitycategories
Primaryfailure
Thiscategoryis used to identifythecause offailureeventsthat are classified as DM. The chargeabilityforeachconfirmed DM should be assigned to the primary failure category.
Qualitycontrol
This categoryis used toaddress failure events that are attributable to inadequatequalitycontrol in the equipment manufacturing, production, or assemblyprocess.
Manufacturer furnished hardware
This category is used for malfunctioninghardwaredeveloped or furnished bytheequipment manufacturer and operator-related test events attributable to the hardwaredesign that maylead to malfunctions. This is distinct from a performance limitation described in 4.3.7, in that the hardware is the cause of a malfunction.
Operator
This category is used for failure events induced bytheoperator that are not rootedin hardwaredesign problems, inadequate training, deficiencies or errors in thetechnical documentation, or the result of abuse.
Technical documentation
This category is used for failure events attributable to misleading, incorrect, ornonexistentinformation in technical documentation provided with the firearm. Inadequate instructional materialsmaycause operator ormaintenanceerrors, andin such cases the failure event should becharged to the technical documentation.
Maintenancepersonnel
This category is used for failure eventsattributable to errorscausedbymaintenancepersonnel, such as not adheringto the established repair procedures.
Training
This category is used for failure events that can bedirectlyattributed toinadequacies in trainingdueto omitted orincorrect trainingprocedures, inappropriate training material, or inadequate or insufficient trainingtime.
Manufacturerfurnished support equipment
This category is used for failure events directlyattributable to tools and external test, measurement,and diagnostic equipment furnished by the manufacturer.
Laboratoryfurnishedsupport equipment
This category is used for failure events directlyattributable to tools and external tests, measurements,and diagnostic equipment furnished by the laboratory.
Accident
This category is used only for thosefailureevents resultingfrom accidents which arenot attributed to thedesign of thefirearm. However, accidents dueto inadequate training, inadequate warnings in thetechnical documentation, careless operation, or failureto perform preventative maintenanceshould not be charged as an “accident” but rather tothe appropriaterootcauseof theaccident.
Unknown
This category is used only for thosefailure events forwhich the cause or source cannot be determined.
4.3“Non-failure” events related to maintainability
This includes test events leading to maintenance actions that areperformed for reasonsother than to correct failures or malfunctions related to reliability.
Maintenancetypeactions in this categoryinclude:
—Preventivemaintenance;
—Scheduled maintenance;
—On-conditionmaintenance;
—Cosmetic maintenance;
—Routineoperatingprocedures;
—Malfunctions or maintenance induced bythe ammunition; and
—Performance limitation.
4.3.1Preventivemaintenance
Actionsthat the operator is required to perform which are necessaryto maintainthe system in proper workingconditionin accordancewith procedures definedas preventative maintenanceinthe applicable technical documentation. This does not include correctivetypemaintenancethat is performedto correctfirearmdeficiencies detected duringpreventative maintenance,as such actions arerequired either to remedya“failure”or execute a routineoperatingprocedure.
4.3.2Scheduled maintenance
This includes any periodic preventivemaintenancedefinedin thefirearm’s technical documentation that isto beperformedat prescribed intervalsby dedicated maintenancepersonnelin order to keepthe firearm in proper operational condition, and that areconsideredto be scheduled maintenance. To qualifyas scheduled maintenance,an event must be consistent with the prescribed serviceinterval, such as rounds fired,operatinghours, or calendartime.
This does not include correctivetypemaintenancethat is performedto correctfirearmdeficiencies detected duringscheduled maintenance,as such actions arerequired to either remedya“failure”or execute a routineoperatingprocedure. This categorydoesnot include correctivemaintenancethat is performed duringthe scheduled maintenanceperiod to fixknown deficiencies.
4.3.3On-conditionmaintenance
This includes maintenancenot relatedto a failureexecuted in order to replaceworn parts or componentswhich havemettheir expected service life. Forsuchactions to becharacterized as on-condition maintenance, thepart or component must havebeen functioning atthetimeof its replacement.
A test event in which a part or componentphysicallyfails almost immediately after its expected service life may not be characterized as a “non-failure” depending on a number of factors, including monitoringaccuracy of the part or component by cognizant personnel, shortcomings in the technical documentation, or periodic maintenance schedules.
Parts or components that continueto exhibit gradual degradation after meeting their expected service life that do not experiencesudden or catastrophicphysical failure should notbescored as a reliability-related failure, if they continueto beusedfor an extended period of timeprior to replacement,andconsequentlyshowexcessivewear at the timeof replacement.
Parts or components that experiencesudden or catastrophicphysical failure if they continueto beusedfor an extended period of time after meeting their expected service life should notbescored as a reliability-related failure, but attributed to another cause, such as abuse due to excessive use.
4.3.4Cosmeticmaintenance
This includes anymaintenancenot related to a malfunctionthat is undertaken to correcta cosmetic deficiency or anomaly, such as deburringa rough external surfaceor sharp edgeto prevent personal injury,rather than forthe purposeof restoringlostor degraded functions.
4.3.5Routineoperating procedures
This includes tasks thatcan be performed quicklyand readily, such as those prescribed in firearm instructional materials,which are not consideredimmediate corrective action. This is to preventthesystem from being penalized for needing a minoroperator action thatis considered a normal, routine operating procedure, such as an adjustment. Atask to correct a malfunction does not constitute a routineoperatingprocedure nor shouldanimmediateaction suchas clearing a stoppagebe considered routine.
4.3.6Malfunction or maintenance induced by the ammunition
This includes malfunctions or stoppages caused by the ammunition. These test events could be due to improper physical dimensions of thecartridgewhich preventsit from beingproperlyfed or chambered,failure of the ammunition to fire when adequatelystruck bythe firingpin,or an improper powder chargecausinginadequategas pressure orrecoilforce thatmaylead to an extraction or ejection failure. Should a malfunction be due to excessive carbon build-up in thefirearm, the failure may be attributed to inadequate preventative maintenance.
4.3.7Performancelimitation
Theinabilityof a firearm to meet a specifiedperformance criterion whereno malfunction has occurredshould be considered a performance limitation attributable to a design deficiency,ratherthana failure related to reliability. Adesign deficiencygenerally cannot be eliminated through typical maintenance or repair actions. Instead, some form of equipment redesign or modification willbe required to remedytheproblemthat is causingthe limitation in performance.
An indicatorof adesign deficiencywould be theinabilityof thefirearm to perform aparticular function atanytimewhereno maintenance action can beperformed to correct theproblem. In orderforadesign deficiencyto be scored asa performancelimitation instead of afailure related to reliability,adefinitivenumeric performancecriterionshallexistwith regard to theaffected areaof performance.
4.4Test eventsnotapplicable toreliability or maintainability
This includes test events that arenot related to and haveno impact on the reliabilityormaintainabilityof thefirearm. The assignment of events to this categoryshould in no waybe construed to implythat theyareinvalid or insignificant, as theycan impacton other functional areas of systemevaluation. This categoryincludes:
—Pretest inspection;
—Equipment modifications;
—Test-peculiar events;
—Abuse;
—Unrelated damage;and
—Other test events not applicable to reliability or maintainability.
4.4.1Pretest inspection
This includes test events observed or detected duringpretest inspection orotherdesignated pretest activities. Pretest events arenot scored againstthe reliabilityor maintainabilityof system as theyhaveoccurred outside oftheactual testing“phase,” however all events detected after thepretest period will be scored on theirown merit.
4.4.2Equipment modifications
This includes test events involvingmaintenanceactions associated with the installation or incorporation ofmodified or upgradedfirearm hardware. These events havenoapplicabilityto reliabilityor maintainabilityunless the original hardwareitemwas nonfunctional, orwas malfunctioning prior to being upgraded or replaced as part of themodification. In those cases wherethere is applicabilityto reliabilityor maintainability, the event of removingand replacing a faultyitem will be scoredon its own merit.
4.4.3Test-peculiar events
This includes test events consistingof malfunctions and maintenanceefforts caused byequipment that is notpart ofthe system beingtested, such as tools or instrumentation peculiar to the test, orbypeople who arenot official test participants, such as people other than operators or maintenance personnel. Test-peculiar eventsarenot attributableto thesystem beingtested, andhaveno applicabilityto its reliabilityor maintainability.
4.4.4Abuse
This includes test eventsin which official test participants, such as operators ormaintenancepersonnel,causedamageto thesystem eitherwillfullyor throughgrosscarelessness or negligence. This categoryalso includes anyeventsin which the test personnel directs thedeliberate abuseof thesystem, such asatest excursion to over-stress or exceed the performancelimits ofthe systemfor investigativepurposes, whethercalled for bythe test plan or not.
4.4.5Unrelated damage
This includes test events in which damageis causedbynatural phenomenaor other influences that arebeyond the controlofofficial test participants. Because thesourceof these events is external to the system beingtested, they should not be included in the evaluation of reliability and maintainability.
4.4.6Other test events not applicable to reliability or maintainability
This includes test events which haveno bearingonreliability and maintainability that do not fit into other categories. Examples may include suggested improvements, reports ofinadequate test procedures,recommended improvements to technical manuals,unusable orunacceptablereplacement parts discovered prior to orduringinstallation, test delays,general information regarding the condition ofequipment or components where no failure related to reliability hasoccurred,and suggested human factors improvements.
5Essential functions
Essential functions represent the coreoperationalfunctionsthat the firearmsshallbecapable of performing. Five essential functions areenumerated below and described more in 5.1 through 5.5:
—The operator shall be able to install a full load of rounds into the ammunition magazine and subsequently both insert the magazine into and remove it from the firearm;
—Safety mechanismsshallfunction properly and remainintheselected state until actuated by theoperator;
—The firearm shall feedandproperlychamber eachindividual round/cartridgewithout inducing a stoppage that requires correctiveaction;
—The firearm shall firechamberedroundsby properly strikingthe primerofeach individual cartridgewithsufficient impact to initiatefiring in all firing modes available on the firearm without inducinga stoppage that requires correctiveaction; and