Toward a Scalable Method for Quantifying Aspects ... - Semantic Scholar
Carnegie Mellon University
Research Showcase @ CMU Institute for Software Research
School of Computer Science
1998
Toward a Scalable Method for Quantifying Aspects of Fault Tolerance, Software Assurance, and Computer Security Philip Koopman Carnegie Mellon University
Follow this and additional works at: http://repository.cmu.edu/isr
This Conference Proceeding is brought to you for free and open access by the School of Computer Science at Research Showcase @ CMU. It has been accepted for inclusion in Institute for Software Research by an authorized administrator of Research Showcase @ CMU. For more information, please contact [email protected].
.RRSPDQ&6'$¶
,(((3RVWSURFHHGLQJVRIWKH&RPSXWHU6HFXULW\'HSHQGDELOLW\DQG$VVXUDQFH )URP 1HHGVWR6ROXWLRQV &6'$ 1RYHPEHU :DVKLQJWRQ'&
7RZDUGD6FDODEOH0HWKRGIRU4XDQWLI\LQJ$VSHFWVRI )DXOW7ROHUDQFH6RIWZDUH$VVXUDQFHDQG&RPSXWHU6HFXULW\ 3KLOLS.RRSPDQ 'HSDUWPHQWRI(OHFWULFDODQG&RPSXWHU(QJLQHHULQJ ,QVWLWXWHIRU&RPSOH[(QJLQHHUHG6\VWHPV &DUQHJLH0HOORQ8QLYHUVLW\3LWWVEXUJK3$ NRRSPDQ#FPXHGX $EVWUDFW 4XDQWLWDWLYH DVVHVVPHQW WRROV DUH XUJHQWO\ QHHGHG LQ WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUHDVVXUDQFHDQGFRPSXWHUVHFXULW\$VVHVVPHQWPHWKRGVW\SLFDOO\HPSOR\HGLQ YDULRXV FRPELQDWLRQV DUH IDXOW LQMHFWLRQ IRUPDO YHULILFDWLRQ DQG WHVWLQJ +RZHYHU WKHVHPHWKRGVDUHH[SHQVLYHEHFDXVHWKH\DUHODERULQWHQVLYHZLWKFRVWVVFDOLQJDWOHDVW OLQHDUO\ZLWKWKHQXPEHURIVRIWZDUHPRGXOHVWHVWHG$GGLWLRQDOO\WKH\DUHVXEMHFWWR KXPDQODSVHVDQGRYHUVLJKWVEHFDXVHWKH\UHTXLUHWZRGLIIHUHQWUHSUHVHQWDWLRQVIRUHDFK V\VWHPDQGWKHQEDVHUHVXOWVRQDGLUHFWRUDQLQGLUHFWUHSUHVHQWDWLRQFRPSDULVRQ 7KH%DOOLVWDSURMHFWKDVIRXQGWKDWUREXVWQHVVWHVWLQJIRUPVDQLFKHLQZKLFKVFDODEOH TXDQWLWDWLYH DVVHVVPHQWFDQ EH DFKLHYHGDWORZFRVW 7KLV VFDODELOLW\ VWHPVIURP WZR WHFKQLTXHV DVVRFLDWLQJ VWDWHVHWWLQJ LQIRUPDWLRQ ZLWK WHVW FDVHV EDVHG RQ GDWD W\SHV DQGXVLQJRQHJHQHULFEXWQDUURZEHKDYLRUDOVSHFLILFDWLRQIRUDOOPRGXOHV*LYHQWKDW WKLVDSSURDFKKDVVXFFHHGHGLQFRPSDULQJWKHUREXVWQHVVRIYDULRXVRSHUDWLQJV\VWHPV LWLVQDWXUDOWRDVNLILWFDQEHPDGHPRUHJHQHUDOO\DSSOLFDEOH ,WDSSHDUVWKDW%DOOLVWDOLNHWHVWLQJFDQEHXVHGLQWKHIDXOWWROHUDQFHDUHDWRPHDVXUH WKHJHQHULFUREXVWQHVVRIDYDULHW\RI$3,LPSOHPHQWDWLRQVDQGLQSDUWLFXODUWRLGHQWLI\ UHSURGXFLEOHZD\VWRFUDVKDQGKDQJVRIWZDUH,QVRIWZDUHDVVXUDQFHLWFDQEHXVHGDV DTXDOLW\FKHFNRQH[FHSWLRQKDQGOLQJDQGLQSDUWLFXODUDVDPHDQVWRDXJPHQWEODFN ER[WHVWLQJ$SSO\LQJLWWRFRPSXWHUVHFXULW\DSSHDUVPRUHSUREOHPDWLFEXWPLJKWEH SRVVLEOHLIWKHUHLVDZD\WRRUWKRJRQDOO\GHFRPSRVHYDULRXVDVSHFWVRIVHFXULW\UHOHYDQW V\VWHPVWDWHLQWRDQDORJVRI%DOOLVWDGDWDW\SHV :KLOH %DOOLVWDOLNH WHVWLQJ LV QR VXEVWLWXWH IRU WUDGLWLRQDO PHWKRGV LW FDQ VHUYH WR SURYLGH D XVHIXO TXDOLW\ DVVXUDQFHFKHFN WKDW DXJPHQWV H[LVWLQJ SUDFWLFH DW UHODWLYHO\ ORZFRVW$OWHUQDWHO\LWFDQVHUYHWRTXDQWLI\WKHH[WHQWRISRWHQWLDOSUREOHPVHQDEOLQJ EHWWHULQIRUPHGGHFLVLRQVE\ERWKGHYHORSHUVDQGFXVWRPHUV :KHQ\RXFDQPHDVXUHZKDW\RXDUHVSHDNLQJDERXWDQGH[SUHVV LWLQQXPEHUV\RXNQRZVRPHWKLQJDERXWLW EXWZKHQ\RXFDQQRW PHDVXUH LW ZKHQ \RX FDQ QRW H[SUHVV LW LQ QXPEHUV \RXU NQRZOHGJH LV RI D PHDJHU DQG XQVDWLVIDFWRU\ NLQG LW PD\ EH WKH EHJLQQLQJ RI NQRZOHGJH EXW \RX KDYH VFDUFHO\ LQ \RXU WKRXJKW DGYDQFHGWRWKHVWDJHRIVFLHQFH ²/RUG.HOYLQ
.RRSPDQ&6'$¶
,QWURGXFWLRQ 2QH RI WKH PRUH GLIILFXOW DVSHFWV RI ZRUNLQJ LQ WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUH DVVXUDQFH DQG FRPSXWHU VHFXULW\ LV TXDQWLI\LQJ UHVXOWV 4XDQWLILFDWLRQ LV DEVROXWHO\ HVVHQWLDO DWWDLQDPDWXUHFDSDELOLW\IRUFUHDWLQJV\VWHPVRIDQ\NLQG$QG\HWGHVSLWHVLJQLILFDQWUHVHDUFK HIIRUWV WKH DWWDLQPHQW RI TXDQWLILFDWLRQ LV SURYLQJ HOXVLYH LQ WKH DUHD RI VRIWZDUHLQWHQVLYH V\VWHPV:KLOHWKHUHLVQRGRXEWWKDWSURJUHVVKDVEHHQPDGHLQWKHDUHDRIJHQHUDOVRIWZDUH GHYHORSPHQW PHWKRGRORJ\ WKH IRFXV KDV EHHQ RQ WKH SULPDU\ DWWULEXWHV RI SURGXFWLYLW\ DQG VRIWZDUH FRUUHFWQHVV /HVV DWWHQWLRQ KDV EHHQ SDLG WR ZKDW DUH RIWHQ FRQVLGHUHG VHFRQGDU\ DWWULEXWHVVXFKDVIDXOWWROHUDQFHVRIWZDUHVDIHW\DQGVHFXULW\7KHVHWKUHHDUHDVVHHPWRVKDUH DFRPPRQSUREOHPLQWKDWLWLVGLIILFXOWWRDWWDLQTXDQWLILFDWLRQODUJHO\EHFDXVHRIWKHLQWHUDFWLRQ RIIRXUSUREOHPV )LUVWWKHSURSHUWLHVRIVRIWZDUHLQWHQVLYHV\VWHPVWKDWQHHGWREHPHDVXUHGDUHW\SLFDOO\YHU\ GHSHQGHQW RQ WKH GHWDLOV RI WKH V\VWHP VSHFLILFDWLRQ RU XVDJH ([DPSOHV LQFOXGH IXQFWLRQDO GHIHFWUDWHVZKHWKHUDV\VWHPLV³VDIH´DQGZKHWKHUDV\VWHPLV³VHFXUH´7KLVPHDQVWKDWDQ\ PHDVXUHPHQW WRRO PXVW EH SURYLGHG ZLWK D ZHDOWK RI LQIRUPDWLRQ DERXW WKH V\VWHP DQG LQ JHQHUDOWKLVUHTXLUHVDFFHVVWRDVXEVWDQWLDOSRUWLRQRIWKHV\VWHPVSHFLILFDWLRQ&RPSOLFDWLQJ WKH VLWXDWLRQ LV WKDW WKH VSHFLILFDWLRQ RU LQGHHG HYHQ WKH GHWDLOHG LPSOHPHQWDWLRQ PD\ EH XQDYDLODEOHIRUVRIWZDUHFRPSRQHQWVSXUFKDVHGRIIWKHVKHOI2EWDLQLQJVXFKLQIRUPDWLRQFRVWV WLPH DQG PRQH\ 7R LOOXVWUDWH KRZ GLIILFXOW WKLV PDNHV WKLQJV FRPSDUH WKH GLIILFXOW\ RI GHWHUPLQLQJ ZKHWKHU D FRPSXWHU V\VWHP LV VHFXUH WR GHWHUPLQLQJ LWV VSHHG DW H[HFXWLQJ D SUHSDFNDJHGEHQFKPDUNVXLWH 6HFRQGWKHGHVLUHGPHDVXUHPHQWRXWFRPHLVRIWHQGHVLUHGWREHSHUIHFWLRQ DWWDLQLQJD]HUR GHIHFW UDWH FRPSOHWHO\ VDIH RU QHYHU IDLOV (YHQ WKRXJK VXFK SHUIHFWLRQ LV JHQHUDOO\ XQDWWDLQDEOH LQ SUDFWLFH LW PHDQV WKDW WHVWV DUH IRFXVVHG RQ ILQGLQJ GHYLDWLRQV IURP D VSHFLILFDWLRQRIVRPHVRUW7KLVLVLQVKDUSFRQWUDVWWRPRUHWUDGLWLRQDOFRPSXWHUV\VWHPPHWULFV VXFKDVFRPSXWDWLRQVSHHGRUDFKLHYHGQHWZRUNEDQGZLGWK,WDOVRUDLVHVLVVXHVLIDVLVRIWHQ DOZD\V" WKHFDVHWKHVSHFLILFDWLRQLWVHOILVLPSHUIHFW 7KLUGLWLVFRPPRQIRUWKHVDPHWHVWVWKDWDUHXVHGWRPHDVXUHWKHV\VWHPWREHXVHGGLUHFWO\ WRIL[DQ\VKRUWFRPLQJVIRXQG)RUH[DPSOHDSURMHFWPHDVXULQJEXJUHSRUWVSHUZHHNRIWHVWLQJ ZRXOGQDWXUDOO\IHHGDOOEXJUHSRUWVWRWKHPDLQWHQDQFHJURXSIRUFRUUHFWLRQ7KXVLWLVFRPPRQ IRU WKH PHDVXUHPHQW SURFHVV LWVHOI WR EH FRXSOHG WR FKDQJHV LQ WKH V\VWHP EHLQJ PHDVXUHG :KLOH LW LV FOHDU WKDW WUDGLWLRQDO VSHHG PHDVXUHPHQWV UHVXOW LQ V\VWHPV EHLQJ RSWLPL]HG WR SHUIRUP EHWWHU WKH VLWXDWLRQ LV GLIIHUHQW KHUH LQ WKDW WKHUH LV QR IXQGDPHQWDO TXDQWLW\ WR EH PHDVXUHG VXFKDVDSDUWLFXODUZRUNORDGWRFRPSOHWH 5DWKHUGHYLDWLRQVIURPSHUIHFWLRQDUH EHLQJERWKPHDVXUHGDQGFRUUHFWHGE\WKHVDPHSURFHVV )RXUWKDQGSHUKDSVPRVWLPSRUWDQWIURPDSUDFWLFDOLPSOHPHQWDWLRQSRLQWRIYLHZLVWKDWWKH SURFHVVRIPHDVXULQJ³EDGWKLQJV´VXFKDVEXJVLVRIWHQVHHQDVDGHVWUXFWLYHDFWDVRSSRVHGWR PHDVXULQJ ³JRRG WKLQJV´ VXFK DV SHUIRUPDQFH 7KH RXWFRPH RI D PHWULF YDOXH RI OHVV WKDQ SHUIHFWLRQFDQEHVHHQDVDODEHOLPSO\LQJGHIHFWLYHJRRGVZKHUHDVWKHRXWFRPHRIDSRVLWLYH PHWULFVXFKDVVSHHGLVPHUHO\DPHDVXUHPHQWRIDQLQKHUHQWSURSHUW\ 'HVSLWHWKHVHSUREOHPVLWLVLPSHUDWLYHWKDWZHILQGZD\VWRTXDQWLI\SURSHUWLHVRIV\VWHPV WKDW ZH ZLVK WR XQGHUVWDQG DQG WR LPSURYH RYHU WLPH 2I FRXUVH WKHUH LV DOZD\V WKH ULVN RI LQYHQWLQJQXPEHUVMXVWIRUWKHVDNHRIKDYLQJQXPEHUVVRLWLVDOVRLPSRUWDQWWRILQGPHWULFVWKDW DUH DW OHDVW FRUUHODWHG WR DFWXDO GHVLUDEOH SURSHUWLHV 7KXV LW ZRXOG EH GHVLUDEOH WR KDYH D PHDVXUHPHQW WRRO RU PHWKRGRORJ\ WKDW GRHV QRW UHTXLUH PXFK GHWDLOHG LQIRUPDWLRQ DERXW WKH V\VWHP LWLV PHDVXULQJGRHV QRW DWWHPSW WR PHDVXUH TXDQWLWLHV WKDW GHYLDWH YHU\ VOLJKWO\ IURP SHUIHFWLRQXVHVQRYHOWHVWVIRUHDFKPHDVXUHPHQWRURWKHUZLVHGLVHQWDQJOHVPHDVXUHPHQWIURP GHIHFWFRUUHFWLRQDQGLVVRPHKRZVHHQDVDFRQVWUXFWLYHPHDVXUHPHQWUDWKHUWKDQDGHVWUXFWLYH
.RRSPDQ&6'$¶
DFWLYLW\ 7KHZRUNGLVFXVVHGKHUHLQGRHVQRWVDWLVI\DOOWKHFULWHULDIRUSURYLGLQJKLJKTXDOLW\QXPEHUV WKDWZLOOVWDQGWKHWHVWRIWLPH7KXVZHVWLOOVHHNWKHWUXHQXPEHUVWKDW/RUG.HOYLQH[KRUWVXV WRXVH+RZHYHURQWKHSULQFLSOHWKDWSURJUHVVLQPHDVXUHPHQWLVDJHQHUDOO\ZRUWK\JRDORXU DSSURDFKPLJKWEHFRQVLGHUHGDVWHSLQWKHULJKWGLUHFWLRQ,QSDUWLFXODUWKLVZRUNGHPRQVWUDWHV WKDWLWLVSRVVLEOHLQVRPHFDVHVWRDXWRPDWHWHVWLQJDODUJHDPRXQWRIVRIWZDUHZLWKRXWVLPSO\ WXUQLQJDODUJHWHVWLQJHIIRUWLQWRDVLPLODUO\ODUJHH[HUFLVHLQSURYLGLQJGHWDLOHGLQIRUPDWLRQWR WKHDXWRPDWHGWHVWLQJV\VWHP *HQHUDOH[LVWLQJWHFKQLTXHVIRUVRIWZDUHPHDVXUHPHQW ,WLVRIFRXUVHFXUUHQWO\SRVVLEOHWRPHDVXUHPDQ\SURSHUWLHVRIVRIWZDUH7KHPRVWPDWXUH RIWKHVHDSSURDFKHVZHUHERUQRIWKHVRFDOOHGVRIWZDUHFULVLVWKDWKDVVRPHKRZPDQDJHGWRODVW DWWKLVSRLQWVHYHUDOGHFDGHV7KHVHPHWULFVDWWHPSWWRPHDVXUHSURGXFWLYLW\DQGWKXVHYDOXDWH ZD\VWRLPSURYHWKDWSURGXFWLYLW\7KH\LQFOXGHWKHYHQHUDEOHVRXUFHOLQHVRIFRGH 6/2& WKH ZLGHO\ XVHG F\FORPDWLF FRPSOH[LW\ PHWULFV >@ DQG RWKHU PRUH DGYDQFHG PHWULFV HJ DV GLVFXVVHGLQ>@ :KLOHWKHVHPHWULFVYDULRXVO\XVHPHDVXUHPHQWVRIWKHVRIWZDUHLWVHOIWKH VRIWZDUHLVQRWEHLQJHYDOXDWHGIRULQKHUHQWXWLOLW\,QVWHDGVRIWZDUHLVPHDVXUHGWRGHWHUPLQH FRPSOH[LW\ ZKLFK LV WKHQ XVHG WR HVWLPDWH GHYHORSPHQW DQG PDLQWHQDQFH FRVWV DQG SHUKDSV LQGLUHFWO\WRJHQHUDWHDQHVWLPDWHGGHIHFWUDWH 0RUHUHFHQWO\DZLGHYDULHW\RIPHWULFVDQGPHDVXUHPHQWPHWKRGRORJLHVKDYHEHHQSURSRVHG IRU WKH VRIWZDUH UHOLDELOLW\ DUHD 7KHVH PHWULFV WHQG WR XVH YDU\LQJ FRPELQDWLRQV RI VRIWZDUH FRPSOH[LW\GHYHORSPHQWSKDVHGHIHFWGLVFRYHU\WHVWLQJSKDVHGHIHFWGLVFRYHU\DQGLQVHUYLFH GHIHFWGLVFRYHU\WRPDNHSUHGLFWLRQVDERXWUHVLGXDOGHIHFWUDWHVIRUQHZDQGH[LVWLQJVRIWZDUH V\VWHPV DEURDGGLVFXVVLRQRIVXFKDSSURDFKHVFDQEHIRXQGLQ>@DQG>@ :KLOHVRIWZDUHUHOLDELOLW\PHWULFVDUHGHILQLWHO\DSURGXFWLYHVWHSLQWKHULJKWGLUHFWLRQWKHLU XVHLVQRWZLWKRXWSUREOHPV7HVWLQJEDVHGDSSURDFKHVDPRXQWWRUHTXLULQJDVRIWZDUHWHVWVXLWH GHYHORSPHQWHIIRUWSURSRUWLRQDOWRGHYHORSLQJWKHRULJLQDOVRIWZDUHLQWKHILUVWSODFHDQGWKXV DUHH[SHQVLYH:KLOHLQPRVWFDVHVWHVWLQJRIQRUPDOIXQFWLRQDOLW\LVVHHQDVQHFHVVDU\WKHUHLV DSUREOHPLQWKDWH[FHSWLRQKDQGOLQJFRGHDQGRWKHUSRUWLRQVRIWKHV\VWHPFDQRYHUVKDGRZWKH EDVLF IXQFWLRQDOLW\ LQ WHUPV RI WHVWLQJ FRPSOH[LW\ >@ 7KLV PHDQV WKDW LQ DOO EXW WKH PRVW H[WUHPHVDIHW\FULWLFDOFDVHVOLPLWHGWLPHEXGJHWDQGPDQSRZHURIWHQUHVXOWLQWHVWLQJHIIRUW EHLQJ FRQFHQWUDWHG RQ ³QRUPDO´ IXQFWLRQDOLW\ JLYLQJ VKRUW VKULIW WR GHSHQGDELOLW\ DVSHFWV ,Q RWKHUZRUGVUHOLDELOLW\WHVWLQJFDQIDOOYLFWLPWRWKHSUREOHPVRIFRVWDQGQRWEHLQJVHHQDVD SULPDU\YDOXHDGGHGEXVLQHVVDFWLYLW\EH\RQGDFHUWDLQSRLQW $QDOWHUQDWHDSSURDFKWKDWLVRIWHQFRPELQHGZLWKDWHVWLQJEDVHGDSSURDFKLVRQHEDVHGRQ PRQLWRULQJGHIHFWGLVFRYHU\UDWHV,IRQHDVVXPHVWKDWGHIHFWVDUHGLVFRYHUHGDWORZPDUJLQDO FRVW HJ WKURXJK DQ H[WHQVLYH EHWD WHVWLQJ SURJUDP RU D PDQGDWHG IXQFWLRQDO FRUUHFWQHVV WHVWLQJSURJUDP WKHQVXFKPRQLWRULQJEHFRPHVDUHDVRQDEO\ORZFRVWH[HUFLVH+RZHYHUWKLV DSSURDFK GRHV VXIIHU IURP WKH SRWHQWLDO SUREOHP RI GHOD\LQJ PHDVXUHPHQW IHHGEDFN XQWLO GRZQVWUHDP LQ WKH GHYHORSPHQW F\FOH ZKHQ SUREOHPV DUH SRWHQWLDOO\ PRUH H[SHQVLYH WR IL[ $GGLWLRQDOO\GHIHFWUDWHPRQLWRULQJKDVWKHSUREOHPWKDW SUHVXPDEO\ WKHLPSRUWDQWGHIHFWVDUH IL[HGDOWHULQJWKHV\VWHPXQGHUWHVWRYHUWLPH 5REXVWQHVVWHVWLQJDVDQH[DPSOHRISRWHQWLDOO\JHQHUDODSSURDFK 7KLVSDSHUGHVFULEHVWKHDWWDLQPHQWRIDUHODWLYHO\JHQHULFORZFRVWVFDODEOHSRUWDEOHWHVWLQJ DQG PHDVXUHPHQW PHWKRGRORJ\ IRU UREXVWQHVV ,Q SDUWLFXODU WKH PHWKRGRORJ\ PHDVXUHV WKH UREXVWQHVV RI DQ $SSOLFDWLRQ 3URJUDPPLQJ ,QWHUIDFH $3, ZLWK UHVSHFW WR H[FHSWLRQDO
.RRSPDQ&6'$¶
SDUDPHWHUYDOXHV7KHFXUUHQWUHVXOWVKDYHLPSOLFDWLRQVIRUIDXOWWROHUDQFHVRIWZDUHDVVXUDQFH DQGFRPSXWHUVHFXULW\DQGWKHPHWKRGRORJ\PLJKWEHH[WHQGHGIXUWKHU 5REXVWQHVV ZDV VHOHFWHG IRU PHDVXUHPHQW EHFDXVH V\VWHP FUDVKHV DUH D ZD\ RI OLIH LQ DQ\ UHDOZRUOG V\VWHP QR PDWWHU KRZ FDUHIXOO\ GHVLJQHG 6RIWZDUH LV LQFUHDVLQJO\ EHFRPLQJ WKH VRXUFH RI V\VWHP IDLOXUHV DQG WKH PDMRULW\ RI VRIWZDUH IDLOXUHV LQ SUDFWLFH VHHP WR EH GXH WR SUREOHPV ZLWK UREXVWQHVV >@ 7KLUW\ \HDUV DJR WKH $SROOR PLVVLRQ H[SHULHQFHG WKUHH FRPSXWHU FUDVKHV DQG UHERRWV GXULQJ GHVFHQW WR OXQDU ODQGLQJ FDXVHG E\ H[FHSWLRQDO UDGDU FRQILJXUDWLRQVHWWLQJVWKDWUHVXOWHGLQWKHV\VWHPUXQQLQJRXWRIPHPRU\EXIIHUV> @'HFDGHV ODWHUWKHPDLGHQIOLJKWRIWKH$ULDQHKHDY\OLIWLQJURFNHWZDVORVWGXHWRHYHQWVDULVLQJIURPD IORDWLQJ SRLQWWRLQWHJHU FRQYHUVLRQ H[FHSWLRQ >@ 7RGD\ H[FHSWLRQDO FRQGLWLRQV URXWLQHO\ FDXVHV\VWHPIDLOXUHVLQWHOHFRPPXQLFDWLRQV\VWHPVGHVNWRSFRPSXWHUVDQGHOVHZKHUH*LYHQ WKDW VXFK SUREOHPV KDYH SHUVLVWHG LQ PLVVLRQFULWLFDO VSDFH FRPSXWHUV IRU GHFDGHV LW LV QRW GLIILFXOW WR SUHGLFW WKH\ ZLOO VWLOO EH ZLWK XV VHYHUDO GHFDGHV KHQFH LQ HYHU\GD\ FRPSXWLQJ +RZHYHU WKH SUREOHP WKHQ ZLOO EH WKDW HYHU\GD\ FRPSXWLQJ ZLOO KDYH FRPSOHWHG LWV FKDQJH IURP OX[XU\ WR QHFHVVLW\ PDNLQJ VXFK IDLOXUHV SRWHQWLDOO\ GHYDVWDWLQJ LQVWHDG RI PHUHO\ LQFRQYHQLHQWRUH[SHQVLYH 7KXV WKH %DOOLVWD SURMHFW ZDV FUHDWHG WR PHDVXUH WKH UREXVWQHVV RI V\VWHPV %HFDXVH RSHUDWLQJV\VWHP 26 VRIWZDUHXQGHUOLHVPRVWODUJHV\VWHPVWKHILUVWDSSOLFDWLRQDUHDWKDWZDV FKRVHQ ZDV WR SURGXFH TXDQWLILHG UHVXOWV IRU UREXVWQHVV WHVWLQJ RI IXOOVFDOH RIIWKHVKHOI RSHUDWLQJV\VWHPV$XWRPDWHGWHVWLQJZDVSHUIRUPHGRQILIWHHQ326,;>@RSHUDWLQJV\VWHP YHUVLRQV IURP WHQ GLIIHUHQW YHQGRUV DFURVV D YDULHW\ RI KDUGZDUH SODWIRUPV 0RUH WKDQ RQH PLOOLRQWHVWVZHUHH[HFXWHGLQDOOFRYHULQJXSWRGLVWLQFWIXQFWLRQVDQGV\VWHPFDOOVIRUHDFK 260DQ\RIWKHWHVWVUHVXOWHGLQUREXVWQHVVIDLOXUHVUDQJLQJLQVHYHULW\IURPFRPSOHWHV\VWHP FUDVKHVWRIDOVHLQGLFDWLRQRIVXFFHVVIXORSHUDWLRQLQXQVSHFLILHGFLUFXPVWDQFHV ,Q EULHI WKH %DOOLVWD WHVWLQJ PHWKRGRORJ\ LQYROYHV DXWRPDWLFDOO\ JHQHUDWLQJ VHWV RI H[FHSWLRQDOSDUDPHWHUYDOXHVWREHXVHGLQFDOOLQJVRIWZDUHPRGXOHV7KHUHVXOWVRIWKHVHFDOOV DUHH[DPLQHGWRGHWHUPLQHZKHWKHUWKHPRGXOHGHWHFWHGDQGQRWLILHGWKHFDOOLQJSURJUDPRIDQ HUURU DQG ZKHWKHU WKH WDVN RU HYHQ V\VWHP VXIIHUHG D FUDVK RU KDQJ 7KHUH DUH WZR NH\ DSSURDFKHV WKDW PDNH WKLV DSSURDFK LQH[SHQVLYH DQG VFDODEOH WKH XVH RI DQ REMHFWRULHQWHG WHVWLQJ DSSURDFK EDVHG RQ GDWD W\SHV UDWKHU WKDQ IXQFWLRQDOLW\ DQG WKH IDFW WKDW WHVWLQJ LV SHUIRUPHG IRU D UHODWLYHO\ VLPSOH SURSHUW\ RI UREXVWQHVV UDWKHU WKDQ IRU WKH PRUH FRPSOH[ SURSHUW\RIFRUUHFWQHVV ,Q WKH IROORZLQJ VHFWLRQV ZH VKDOO GLVFXVV SUHYLRXV ZRUN LQ WKLV DUHD WKH %DOOLVWD WHVWLQJ PHWKRGRORJ\ DQG SRWHQWLDO DSSOLFDWLRQV RI WKDW PHWKRGRORJ\ WR WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUHDVVXUDQFHDQGFRPSXWHUVHFXULW\:KLOHSDUWRIWKHVWRU\LVVLPSO\WKHDSSOLFDWLRQRI H[LVWLQJ%DOOLVWDUHVXOWVDQGPHWKRGRORJLHVWRWKHVHDUHDVWKHUHVWRIWKHGLVFXVVLRQZLOOEHPRUH VSHFXODWLYHDQGK\SRWKHVL]HZD\VLQZKLFKWKHDSSURDFKPLJKWEHJHQHUDOL]HGDQGH[WHQGHG
3UHYLRXVZRUN :KLOH WKH %DOOLVWD UREXVWQHVV WHVWLQJ PHWKRG GHVFULEHG LQ WKLV SDSHU LV D IRUP RI VRIWZDUH WHVWLQJLWVKHULWDJHWUDFHVEDFNQRWRQO\WRWKHVRIWZDUHWHVWLQJFRPPXQLW\EXWDOVRWRWKHIDXOW WROHUDQFH FRPPXQLW\ DV D IRUP RI VRIWZDUHEDVHG IDXOW LQMHFWLRQ ,Q VRIWZDUH WHVWLQJ WHUPV %DOOLVWDSHUIRUPVWHVWVIRUUHVSRQVHVWRH[FHSWLRQDOLQSXWFRQGLWLRQV VRPHWLPHVFDOOHG³GLUW\´ WHVWVZKLFKLQYROYHH[FHSWLRQDOVLWXDWLRQVDVRSSRVHGWR³FOHDQ´WHVWVRIFRUUHFWIXQFWLRQDOLW\LQ QRUPDO VLWXDWLRQV 7KH WHVW LGHDV XVHG DUH EDVHG RQ ³EODFN ER[´ RU IXQFWLRQDO WHVWLQJ WHFKQLTXHV>%H]LHU @LQZKLFKRQO\IXQFWLRQDOLW\LVRIFRQFHUQQRWWKHDFWXDOVWUXFWXUHRIWKH VRXUFHFRGH+RZHYHU%DOOLVWDLQLWVFXUUHQWIRUPLVQRWFRQFHUQHGZLWKYDOLGDWLQJIXQFWLRQDOLW\ IRURUGLQDU\RSHUDWLQJFRQGLWLRQVEXWUDWKHUGHWHUPLQLQJ ZKHWKHURUQRWDVRIWZDUHPRGXOHLV
.RRSPDQ&6'$¶
UREXVW ,Q FRPPRQ XVDJH WKH WHUP UREXVWQHVV PLJKW UHIHU WR WKH WLPH EHWZHHQ RSHUDWLQJ V\VWHP FUDVKHVXQGHUVRPHXVDJHSURILOH+RZHYHUWKHDXWKRULWDWLYHGHILQLWLRQRIUREXVWQHVVWKDWZLOO EHXVHGLQWKHFXUUHQWGLVFXVVLRQLV³WKHGHJUHHWRZKLFKDV\VWHPRUFRPSRQHQWFDQIXQFWLRQ FRUUHFWO\ LQ WKH SUHVHQFH RI LQYDOLG LQSXWV RU VWUHVVIXO HQYLURQPHQWDO FRQGLWLRQV´ >@ 7KLV H[SDQGVWKHQRWLRQRIUREXVWQHVVWREHPRUHWKDQFDWDVWURSKLFV\VWHPFUDVKHVDQGHQFRPSDVVHV VLWXDWLRQVLQZKLFKVPDOOUHFRYHUDEOHIDLOXUHVPLJKWDOVRRFFXU%DOOLVWDZRUNFRQFHQWUDWHVRQ WKH SRUWLRQ RI UREXVWQHVV GHDOLQJ ZLWK LQYDOLG LQSXWV :KLOH UREXVWQHVV XQGHU VWUHVVIXO HQYLURQPHQWDOFRQGLWLRQVLVLQGHHGDQLPSRUWDQWLVVXHDGHVLUHWRDWWDLQKLJKO\UHSHDWDEOHUHVXOWV KDVOHGWKH%DOOLVWDSURMHFWWRFRQVLGHURQO\UREXVWQHVVLVVXHVGHDOLQJZLWKDVLQJOHLQYRFDWLRQRI DVRIWZDUHPRGXOHIURPDVLQJOHH[HFXWLRQWKUHDG KRZHYHUDVZLOOEHGLVFXVVHGWKLVLVIDUOHVV OLPLWLQJDQDSSURDFKWKDQRQHPLJKWWKLQN &UDVKPH $QHDUO\DQGZHOONQRZQPHWKRGIRUDXWRPDWLFDOO\WHVWLQJRSHUDWLQJV\VWHPVIRUUREXVWQHVV ZDVWKHGHYHORSPHQWRIWKH&UDVKPHSURJUDP>@&UDVKPHRSHUDWHVE\ZULWLQJUDQGRPL]HGGDWD YDOXHVWRPHPRU\WKHQVSDZQLQJODUJHQXPEHUVRIWDVNVWKDWDWWHPSWHGWRH[HFXWHWKRVHUDQGRP E\WHVDVSURJUDPV:KLOHPDQ\ WDVNVWHUPLQDWHDOPRVWLPPHGLDWHO\GXHWRLOOHJDOLQVWUXFWLRQ H[FHSWLRQVRQRFFDVLRQDVLQJOHWDVNRUDFRQIOXHQFHRIPXOWLSOHWDVNVFDQFDXVHDQRSHUDWLQJ V\VWHP WR IDLO 7KH HIIHFWLYHQHVV RI WKH &UDVKPH DSSURDFK UHOLHV XSRQ VHUHQGLSLW\ LQ RWKHU ZRUGVLIUXQORQJHQRXJKLWPD\HYHQWXDOO\ILQGVRPHZD\WRFUDVKWKHV\VWHP 7KHFUDVKPHDSSURDFKKDVWKHGLVWLQFWDGYDQWDJHWKDWLWLVUHODWLYHO\HDV\WRLPSOHPHQWDQG FDQ EH SRUWHG UHDGLO\ WR D ZLGH YDULHW\ RI V\VWHPV ,W DOVR KDV WKH FRPSHOOLQJ YLUWXH WKDW LW DFWXDOO\ GRHVILQGZD\VWR FUDVKRSHUDWLQJV\VWHPV+RZHYHU DVDWHFKQLTXHLWLVQRWUHDGLO\ DSSOLHGWRRWKHUDSSOLFDWLRQDUHDVDQGGRHVQRWSURGXFHUHSHDWDEOHGHWHUPLQLVWLFUHVXOWVWKDWDUH GHVLUDEOH IRU FRQFLVH EXJ UHSRUWV )LQDOO\ LW LV QRW UHDOO\ GHVLJQHG WR TXDQWLI\ DQG FRPSDUH V\VWHPVVRPXFKDVGLVFRYHUSUREOHPVLQDSDUWLFXODUV\VWHP )DXOWLQMHFWLRQ 2WKHUZRUNLQWKHIDXOWLQMHFWLRQDUHDKDVDOVRWHVWHGDVSHFWVRIUREXVWQHVV7KH)LDWV\VWHP >@XVHVSUREHVSODFHGE\WKHSURJUDPPHUWRDOWHUWKHELQDU\SURFHVVLPDJHLQPHPRU\GXULQJ H[HFXWLRQ 7KH )HUUDUL V\VWHP >@ LV VLPLODU LQ LQWHQW WR ),$7 EXW XVHV VRIWZDUH WUDSV LQ D PDQQHUVLPLODUWRGHEXJJHUEUHDNSRLQWVWRSHUPLWHPXODWLRQRIVSHFLILFV\VWHPOHYHOKDUGZDUH IDXOWV HJGDWDDGGUHVVOLQHVFRQGLWLRQFRGHV 7KH)7$3(V\VWHP>@LQMHFWVIDXOWVLQWRD V\VWHPEHLQJH[HUFLVHGZLWKDUDQGRPZRUNORDGJHQHUDWRUE\XVLQJDSODWIRUPVSHFLILFGHYLFH GULYHUWRLQMHFWWKHIDXOWV:KLOHDOORIWKHVHV\VWHPVKDYHSURGXFHGLQWHUHVWLQJUHVXOWVQRQHZDV LQWHQGHGWRTXDQWLI\UREXVWQHVVRQWKHVFDOHRIDQHQWLUH26$3, 7KH)X]]SURMHFWDWWKH8QLYHUVLW\RI:LVFRQVLQKDVXVHGUDQGRPQRLVH RUIX]] LQMHFWLRQ WR GLVFRYHU UREXVWQHVV SUREOHPV LQ RSHUDWLQJ V\VWHPV 7KDW ZRUN GRFXPHQWHG WKH VRXUFH RI VHYHUDO SUREOHPV > @ DQG WKHQ GLVFRYHUHG WKDW WKH SUREOHPV ZHUH VWLOO SUHVHQW LQ RSHUDWLQJ V\VWHPVVHYHUDO\HDUVODWHU>@7KH)X]]DSSURDFKWHVWHGVSHFLILF26HOHPHQWVDQGLQWHUIDFHV FRPSDUHGWRWKHFRPSOHWHO\UDQGRPDSSURDFKRI&UDVKPH DOWKRXJKLWVWLOOUHOLHGRQUDQGRP GDWDLQMHFWLRQ 7KH %DOOLVWD WHVWLQJ DSSURDFK EXLOGV XSRQ D OHJDF\ RI ZRUN LQ WKH IDXOW LQMHFWLRQ DUHD SHUIRUPHGE\6LHZLRUHNDQG KLVVWXGHQWV DW&DUQHJLH0HOORQ8QLYHUVLW\ 7KLVZRUN KDVORQJ VRXJKWDFRPELQDWLRQRIUHSHDWDELOLW\KLJKOHYHOLPSOHPHQWDWLRQDQGSRUWDELOLW\ TXDQWLILFDWLRQ ZDVQRWDQH[SOLFLWJRDOEXWFDQEHDFFRPSOLVKHGJLYHQWKDWWKHRWKHUJRDOVKDYHEHHQPHW
.RRSPDQ&6'$¶
$ILUVWHIIRUWDW&DUQHJLH0HOORQZDVWKHFUHDWLRQRIDGHVFHQGHQWRIFUDVKPHFDOOHG&08 FUDVKPH,QVWHDGRIH[HFXWLQJUDQGRPGDWD&08FUDVKPHUHVWULFWHGUDQGRPQHVVWRSDUDPHWHU YDOXHVIRUFRQFXUUHQWO\H[HFXWHGV\VWHPFDOOV7KLVKDGWKHHIIHFWRIFRQFHQWUDWLQJWHVWLQJRQWKH 26LQWHUIDFHDQGHOLPLQDWLQJDODUJHQXPEHURIWHVWVWKDWVLPSO\UHVXOWHGLQLOOHJDOLQVWUXFWLRQ H[FHSWLRQV,WVWLOOKRZHYHUUHOLHGRQVHUHQGLSLW\WRGLVFRYHUSUREOHPVDQGZDVQRWUHSHDWDEOH 7KHQH[WJHQHUDWLRQRIZRUNZDVFDOOHGUREXVWQHVVEHQFKPDUNLQJLQZKLFKFDUHIXOO\VHOHFWHG GDWDYDOXHVZHUHVHQWWRVHOHFWHGIXQFWLRQFDOOVZLWKRQHFDOOH[HFXWHGDWDWLPH7KHHIIRUWZDV ODERULQWHQVLYHLQWKDWLWUHTXLUHGGHYHORSLQJDWHVWVXLWHEXWDWWDLQHGUHSHDWDELOLW\$QGSHUKDSV VXUSULVLQJO\HYHQZLWKDVLQJOHFDOOPRGHOLWIRXQGZD\VWRFUDVKRSHUDWLQJV\VWHPVIURPXVHU PRGH LQFOXGLQJD VSDFHTXDOLILHG IDXOW WROHUDQW DHURVSDFH FRPSXWHU>@ (YHQWXDOO\ VHYHUDO RSHUDWLQJV\VWHPVZHUHWHVWHGZLWKDKDQGIXORIFDOOVGHPRQVWUDWLQJSRUWDELOLW\DQGUHYHDOLQJ WKDWUREXVWQHVVIDLOXUHVZHUHSUHYDOHQWDFURVVYDULRXVV\VWHPV>@ 2QFHUHSHDWDELOLW\KDGEHHQDFKLHYHGWKHREYLRXVQH[WVWHSZDVWRJHQHULFL]HWKHWHVWVDQG DWWDLQVFDODELOLW\IRUODUJHU$3,V7KXVZRUNZDVGRQHWRH[WHQGWKHUREXVWQHVVEHQFKPDUNVE\ JHQHULFL]LQJ GLIIHUHQW RSHUDWLRQV ZLWKLQ D IXQFWLRQ DQG DWWHPSWLQJ WR DVVRFLDWH WKHP ZLWK GLIIHUHQWWHVWGDWDW\SHV>@)RUH[DPSOHPDQ\IXQFWLRQVVHUYHWRFUHDWHDGDWDREMHFWZKHWKHU LWEHDILOHDQDOORFDWHGEORFNRIPHPRU\RUDGDWDVWUXFWXUH6LPLODUO\RWKHUIXQFWLRQVPRGLI\ GHOHWHDQGFRS\VXFKREMHFWV7KXVRQHFRXOGWKLQNRIJHQHULFL]LQJWKHIXQFWLRQRIDSDUWLFXODU PRGXOHLQWRRQHRIDSUHVHWFDWHJRU\DQGWKHQGLUHFWWHVWLQJEDVHGRQDFRPELQDWLRQRIJHQHULF IXQFWLRQDQGGDWDW\SH:KLOHWKLVDSSURDFKGLGZRUNDWWHPSWVWRVFDOHLWWRIXOOVL]HV\VWHPV ZHUH XQVXFFHVVIXO EHFDXVH WKH HIIRUW UHTXLUHG WR VSHFLI\ JHQHULF IXQFWLRQDOLW\ VFDOHG OLQHDUO\ ZLWKWKHQXPEHURIIXQFWLRQVEHLQJWHVWHG7KHDSSURDFKUHPDLQVSURPLVLQJEXWLVQRWZLWKRXW IXUWKHULVVXHVWREHUHVROYHG $IWHU DWWHPSWLQJ WR JHQHULFL]H IXQFWLRQV D IUHVK DSSURDFK WR WKH SUREOHP ZDV WDNHQ E\ HOLPLQDWLQJ WKH LQYROYHPHQW RI IXQFWLRQDOLW\ DOWRJHWKHU DQG FUHDWLQJ WHVW FDVHV EDVHG RQ GDWD W\SHLQIRUPDWLRQDORQH7KLVDSSURDFKZDVWKHVRXUFHRIWKH%DOOLVWDWHVWLQJPHWKRGRORJ\>@ ZKLFKDWWDLQHGUHSHDWDELOLW\SRUWDELOLW\DQGVFDODELOLW\DQGLVGHVFULEHGLQPRUHGHWDLOLQODWHU VHFWLRQV $V DQ DGGLWLRQDO EHQHILW WKH DELOLW\ WR SHUIRUP RYHU D PLOOLRQ WHVWV RQ PRUH WKDQ D GR]HQ RSHUDWLQJ V\VWHPV SURYLGHG HQRXJK WHVW GDWD WR FUHDWH D VWUDWHJ\ IRU TXDQWLILFDWLRQ RI UREXVWQHVV 6RIWZDUHWHVWLQJ *LYHQDGHVLUHWRSURYLGHFUDVKPHOLNHWHVWLQJLQDPRUHGHWHUPLQLVWLFUHSHDWDEOHVHWWLQJRQH FDQWXUQWRVRIWZDUHWHVWLQJWRERUURZFRQFHSWV6RIWZDUHWHVWLQJIRUWKHSXUSRVHRIGHWHUPLQLQJ UHOLDELOLW\ LV RIWHQ FDUULHG RXW E\ H[HUFLVLQJ D VRIWZDUH V\VWHP XQGHU UHSUHVHQWDWLYH ZRUNORDG FRQGLWLRQVDQGPHDVXULQJIDLOXUHUDWHV,QDGGLWLRQHPSKDVLVLVRIWHQSODFHGRQFRGHFRYHUDJH DVDZD\RIDVVHVVLQJZKHWKHUDPRGXOHKDVEHHQWKRURXJKO\WHVWHG8QIRUWXQDWHO\WUDGLWLRQDO VRIWZDUH UHOLDELOLW\ WHVWLQJ PD\ QRW XQFRYHU UREXVWQHVV SUREOHPV WKDW RFFXU EHFDXVH RI XQH[SHFWHGLQSXWYDOXHVJHQHUDWHGE\EXJVLQRWKHUPRGXOHVRUEHFDXVHRIDQHQFRXQWHUZLWK DW\SLFDORSHUDWLQJFRQGLWLRQV 6WUXFWXUDO RU ZKLWHER[ WHVWLQJ WHFKQLTXHV DUH XVHIXO IRU DWWDLQLQJ KLJK WHVW FRYHUDJH RI SURJUDPV %XWWKH\W\SLFDOO\IRFXVRQ WKH FRQWUROIORZRI DSURJUDPUDWKHUWKDQKDQGOLQJRI H[FHSWLRQDO GDWD YDOXHV )RU H[DPSOH VWUXFWXUDO WHVWLQJ DVFHUWDLQV ZKHWKHU FRGH GHVLJQHG WR GHWHFWLQYDOLGGDWDLVH[HFXWHGE\DWHVWVXLWHEXWPD\QRWGHWHFWZKHWKHUDFKHFNIRULQYDOLGGDWD LVPLVVLQJIURPWKHSURJUDPDOWRJHWKHU$GGLWLRQDOO\VWUXFWXUDOWHVWLQJW\SLFDOO\UHTXLUHVDFFHVV WRVRXUFHFRGHZKLFKPD\EHXQDYDLODEOHZKHQXVLQJ&276VRIWZDUHFRPSRQHQWV $FRPSOHPHQWDU\DSSURDFKLVEODFNER[WHVWLQJDOVRFDOOHGEHKDYLRUDOWHVWLQJ>@%ODFNER[ WHVWLQJ WHFKQLTXHV DUH GHVLJQHG WR GHPRQVWUDWH FRUUHFW UHVSRQVH WR YDULRXV LQSXW YDOXHV
.RRSPDQ&6'$¶
UHJDUGOHVV RI WKH VRIWZDUH LPSOHPHQWDWLRQ DQG VHHP PRUH DSSURSULDWH IRU UREXVWQHVV WHVWLQJ 7ZRW\SHVRIEODFNER[WHVWLQJDUHSDUWLFXODUO\XVHIXODVVWDUWLQJSRLQWVIRUUREXVWQHVVWHVWLQJ GRPDLQWHVWLQJDQGV\QWD[WHVWLQJ'RPDLQWHVWLQJORFDWHVDQGSUREHVSRLQWVDURXQGH[WUHPDDQG GLVFRQWLQXLWLHVLQWKHLQSXWGRPDLQ6\QWD[WHVWLQJFRQVWUXFWVFKDUDFWHUVWULQJVWKDWDUHGHVLJQHG WRWHVWWKHUREXVWQHVVRIVWULQJOH[LQJDQGSDUVLQJV\VWHPV%RWKW\SHVRIWHVWLQJDUHDPRQJWKH DSSURDFKHVXVHGLQ%DOOLVWD $XWRPDWLFDOO\JHQHUDWLQJVRIWZDUHWHVWVUHTXLUHVWKUHHWKLQJV D0RGXOHXQGHU7HVW 0X7 D PDFKLQHXQGHUVWDQGDEOH VSHFLILFDWLRQ RI FRUUHFW EHKDYLRU DQG DQ DXWRPDWLF ZD\ WR FRPSDUH UHVXOWV RI H[HFXWLQJ WKH 0X7 ZLWK WKH VSHFLILFDWLRQ 8QIRUWXQDWHO\ REWDLQLQJ RU FUHDWLQJ D EHKDYLRUDOVSHFLILFDWLRQIRUD&276RUOHJDF\VRIWZDUHFRPSRQHQWLVRIWHQLPSUDFWLFDOGXHWR XQDYDLODELOLW\RUFRVW )RUWXQDWHO\UREXVWQHVVWHVWLQJQHHGQRWXVHDGHWDLOHGEHKDYLRUDOVSHFLILFDWLRQ,QVWHDGWKH DOPRVW WULYLDO VSHFLILFDWLRQ RI ³GRHVQ¶W FUDVK GRHVQ¶W KDQJ´ VXIILFHV 'HWHUPLQLQJ ZKHWKHU D 0X7PHHWVWKLVVSHFLILFDWLRQLVVWUDLJKWIRUZDUGWKHRSHUDWLQJV\VWHPFDQEHTXHULHGWRVHHLID WHVWSURJUDPWHUPLQDWHVDEQRUPDOO\DQGDZDWFKGRJWLPHUFDQEHXVHGWRGHWHFWKDQJV7KXV UREXVWQHVVWHVWLQJFDQEHSHUIRUPHGRQPRGXOHV WKDWGRQ¶WLQWHQWLRQDOO\FUDVKRUKDQJ LQWKH DEVHQFHRILQGLYLGXDOEHKDYLRUDOVSHFLILFDWLRQV 63(&,),(' %(+$9,25
,1387 63$&(
6+28/' :25.
9$/,' ,13876
5(63216( 63$&( 52%867 23(5$7,21 02'8/( 81'(5 7(67 815(352'8&,%/( )$,/85(
)LJXUH%DOOLVWDKDVDJRDORILGHQWLI\LQJUHSURGXFLEOHIDLOXUHVIRULQYDOLGLQSXWVWRD PRGXOHXQGHUWHVW $Q\H[LVWLQJVSHFLILFDWLRQIRUD0X7PLJKWGHILQHLQSXWVDVIDOOLQJLQWRWKUHHFDWHJRULHV YDOLG LQSXWVLQSXWVZKLFKDUHVSHFLILHGWREHKDQGOHGDVH[FHSWLRQVDQGLQSXWVIRUZKLFKWKHEHKDYLRU LV XQVSHFLILHG )LJXUH %DOOLVWD WHVWLQJ EHFDXVH LW LV QRW FRQFHUQHG ZLWK WKH VSHFLILHG EHKDYLRUFROODSVHVWKHXQVSHFLILHGDQGVSHFLILHGH[FHSWLRQDOLQSXWVLQWRDVLQJOHLQYDOLGLQSXW VSDFH 7KH UREXVWQHVV RI WKH UHVSRQVHV RI WKH 0X7 FDQ EH FKDUDFWHUL]HG DV UREXVW QHLWKHU FUDVKHV QRU KDQJV EXW LV QRW QHFHVVDULO\ FRUUHFW IURP D GHWDLOHG EHKDYLRUDO YLHZ KDYLQJ D UHSURGXFLEOH IDLOXUH D FUDVK RU KDQJ WKDW LV FRQVLVWHQWO\ UHSURGXFHG DQG DQ XQUHSURGXFLEOH IDLOXUH D UREXVWQHVV IDLOXUH WKDW LV QRW UHDGLO\ UHSURGXFLEOH 7KH REMHFWLYH RI %DOOLVWD LV WR LGHQWLI\UHSURGXFLEOHIDLOXUHV ,QVWUXPHQWDWLRQDSSURDFKHV 7KHUH DUH VHYHUDO FRPPHUFLDO SURGXFWV WKDW KHOS LQ GHYHORSLQJ UREXVW FRGH VXFK DV 3XULI\ >@DQG%RXQGVFKHFNHU>@ZKLFKLQVWUXPHQWVRIWZDUHWRGHWHFWH[FHSWLRQDOVLWXDWLRQV7KH\ ZRUN E\ GHWHFWLQJ H[FHSWLRQV WKDW DULVH GXULQJ GHYHORSPHQW WHVWLQJ RU XVDJH RI WKH VRIWZDUH +RZHYHUWKH\DUHQRWDEOHWRILQGUREXVWQHVVIDLOXUHVWKDWPLJKWRFFXULQVLWXDWLRQVZKLFKDUH QRWWHVWHG DQGHYHQZLWKZKDWZRXOGQRUPDOO\EHFDOOHGWHVWFRYHUDJHLWLVXQOLNHO\LQ
.RRSPDQ&6'$¶
SUDFWLFHWKDWHYHU\H[FHSWLRQDOFRQGLWLRQZKLFKZLOOEHHQFRXQWHUHGLQWKHILHOGLVLQFOXGHGLQWKH VRIWZDUHWHVWVXLWH 7KH%DOOLVWDDSSURDFKGLIIHUVIURPDQGFRPSOHPHQWVWKHVHDSSURDFKHVE\ DFWLYHO\ VHHNLQJ RXW UREXVWQHVV IDLOXUHV UDWKHU WKDQ EHLQJ DQ LQVWUXPHQWDWLRQ WRRO LW DFWXDOO\ JHQHUDWHV WHVWV IRU H[FHSWLRQ KDQGOLQJ DELOLW\ DQG IHHGV WKHP GLUHFWO\ LQWR VRIWZDUH PRGXOHV 7KXV%DOOLVWDLVOLNHO\WRILQGUREXVWQHVVIDLOXUHVWKDWZRXOGRWKHUZLVHEHPLVVHGGXULQJQRUPDO VRIWZDUHWHVWLQJHYHQZLWKDYDLODEOHLQVWUXPHQWDWLRQWRROV
7KH%DOOLVWDVFDODEOHUREXVWQHVVWHVWLQJIUDPHZRUN $VRIWZDUHFRPSRQHQWIRURXUSXUSRVHVLVDQ\SLHFHRIVRIWZDUHWKDWFDQEHLQYRNHGDVD SURFHGXUH IXQFWLRQ RU PHWKRG ZLWK D QRQQXOO VHW RI LQSXW SDUDPHWHUV :KLOH WKDW LV QRW D XQLYHUVDO GHILQLWLRQ RI DOO VRIWZDUH LQWHUIDFHV LW LV VXIILFLHQWO\ EURDG WR EH RI LQWHUHVW ,Q WKH %DOOLVWD DSSURDFK UREXVWQHVV WHVWLQJ RI VXFK D VRIWZDUH FRPSRQHQW D 0RGXOH XQGHU 7HVW RU 0X7 FRQVLVWV RI HVWDEOLVKLQJ DQ LQLWLDO V\VWHP VWDWH H[HFXWLQJ D VLQJOH FDOO WR WKH 0X7 GHWHUPLQLQJZKHWKHUDUREXVWQHVVSUREOHPRFFXUUHGDQGWKHQUHVWRULQJV\VWHPVWDWHWRSUHWHVW FRQGLWLRQVLQSUHSDUDWLRQIRUWKHQH[WWHVW$OWKRXJKH[HFXWLQJFRPELQDWLRQVRIFDOOVWRRQHRU PRUH0X7VGXULQJDWHVWFDQEHXVHIXOLQVRPHVLWXDWLRQVZHKDYHIRXQGWKDWHYHQWKLVVLPSOH DSSURDFKRIWHVWLQJDVLQJOHFDOODWDWLPHSURYLGHVDULFKVHWRIWHVWVDQGXQFRYHUVDVLJQLILFDQW QXPEHURIUREXVWQHVVSUREOHPV %DOOLVWDGUDZVXSRQLGHDVIURPWKHDUHDVRIERWKVRIWZDUHWHVWLQJDQGIDXOWLQMHFWLRQ$NH\ LGHDLVWKHXVHRIDQREMHFWRULHQWHGDSSURDFKGULYHQE\SDUDPHWHUOLVWGDWDW\SHLQIRUPDWLRQWR DFKLHYHVFDODELOLW\DQGDXWRPDWHGLQLWLDOL]DWLRQRIV\VWHPVWDWHIRUHDFKWHVWFDVH %DOOLVWDDSSURDFK 7KH %DOOLVWD UREXVWQHVV WHVWLQJ PHWKRGRORJ\ LV EDVHG RQ FRPELQDWLRQDO WHVWV RI YDOLG DQG LQYDOLGSDUDPHWHUYDOXHVIRUV\VWHPFDOOVDQGIXQFWLRQV,QHDFKWHVWFDVHDVLQJOHVRIWZDUH0X7 LV FDOOHG D VLQJOH WLPH WR GHWHUPLQH ZKHWKHU LW LV UREXVW ZKHQ FDOOHG ZLWK D SDUWLFXODU VHW RI SDUDPHWHUYDOXHV7KHVHSDUDPHWHUYDOXHVRUWHVWYDOXHVDUHGUDZQIURPDSRRORIQRUPDODQG H[FHSWLRQDO YDOXHV EDVHG RQ WKH GDWD W\SH RI HDFK DUJXPHQW SDVVHG WR WKH 0X7 $ WHVW FDVH WKHUHIRUHFRQVLVWVRIWKHQDPHRIWKH0X7DQGDWXSOHRIWHVWYDOXHVWKDWDUHSDVVHGDVSDUDPHWHUV LH D WHVW FDVH FRXOG EH GHVFULEHG DV D WXSOH ^0X7BQDPH WHVWBYDOXH WHVWBYDOXH ` FRUUHVSRQGLQJWRDSURFHGXUHFDOORIWKHIRUP 0X7BQDPH WHVWBYDOXHWHVWBYDOXH 7KXV WKHJHQHUDODSSURDFKWR%DOOLVWDWHVWLQJLVWRWHVWWKHUREXVWQHVVRIDVLQJOHFDOOWRD0X7IRUD VLQJOH WXSOH RI WHVW YDOXHV DQG WKHQ UHSHDW WKLV SURFHVV IRU PXOWLSOH WHVW FDVHV WKDW HDFK KDYH GLIIHUHQWFRPELQDWLRQVRIYDOLGDQGLQYDOLGWHVWYDOXHV$GHWDLOHGGLVFXVVLRQIROORZV ,Q%DOOLVWDWHVWVDUHEDVHGRQWKHYDOXHVRISDUDPHWHUVDQGQRWRQWKHEHKDYLRUDOGHWDLOVRIWKH 0X77KHVHWRIWHVWFDVHVXVHGWRWHVWD0X7LVFRPSOHWHO\GHWHUPLQHGE\WKHGDWDW\SHVRIWKH SDUDPHWHUOLVWRIWKH0X7DQGGRHVQRWGHSHQGRQDEHKDYLRUDOVSHFLILFDWLRQ )LJXUHVKRZVWKH%DOOLVWDDSSURDFKWRJHQHUDWLQJWHVWFDVHVIRUD0X7%HIRUHFRQGXFWLQJ WHVWLQJDVHWRIWHVWYDOXHVPXVWEHZULWWHQIRUHDFKGDWDW\SHXVHGLQWKH0X7)RUH[DPSOHLI RQHRUPRUHPRGXOHVWREHWHVWHGUHTXLUHDQLQWHJHUGDWDW\SHDVDQLQSXWSDUDPHWHUWHVWYDOXHV PXVWEHFUHDWHGIRUWHVWLQJLQWHJHUV9DOXHVWRWHVWLQWHJHUVPLJKWLQFOXGHDQG0$;,17 PD[LPXPLQWHJHUYDOXH $GGLWLRQDOO\LIDSRLQWHUGDWDW\SHLVXVHGZLWKLQWKH0X7YDOXHVRI 18//DQG DPRQJ RWKHUV PLJKW EH XVHG $ PRGXOH FDQQRWEHWHVWHG XQWLOWHVW YDOXHVDUH FUHDWHGIRUHDFKRILWVSDUDPHWHUGDWDW\SHV$XWRPDWLFWHVWLQJJHQHUDWHVPRGXOHWHVWFDVHVE\ GUDZLQJIURPSRROVRIGHILQHGWHVWYDOXHV (DFKVHWRIWHVWYDOXHV RQHVHWSHUGDWDW\SH LVLPSOHPHQWHGDVDWHVWLQJREMHFWKDYLQJDSDLU RI FRQVWUXFWRU DQG GHVWUXFWRU IXQFWLRQV IRU HDFK GHILQHG WHVW YDOXH ,QVWDQWLDWLRQ RI D WHVWLQJ
.RRSPDQ&6'$¶
$3, 7(67,1* 2%-(&76
7(67 9$/8(6
ZULWH LQWILOHGHVFRQVWYRLG EXIIHUVL]HBWQE\WHV ),/(
0(025
@ &DWDVWURSKLFIDLOXUHVRFFXUZKHQWKHHQWLUH26EHFRPHVFRUUXSWHGRUWKHPDFKLQHFUDVKHVRU UHERRWV,QRWKHUZRUGVWKLVLVDFRPSOHWHV\VWHPFUDVK7KHVHIDLOXUHVDUHLGHQWLILHGPDQX DOO\ EHFDXVH RI GLIILFXOWLHV HQFRXQWHUHG ZLWK ORVV RI QHZO\ ZULWWHQ FRPPLWWHG ILOH GDWD DFURVVV\VWHPFUDVKHVRQVHYHUDORSHUDWLQJV\VWHPV
.RRSPDQ&6'$¶
5HVWDUW IDLOXUHVRFFXUZKHQDIXQFWLRQ FDOOWRDQ26IXQFWLRQ QHYHUUHWXUQVUHVXOWLQJLQD WDVNWKDWKDVKXQJDQGPXVWEHWHUPLQDWHGXVLQJDFRPPDQGVXFKDVNLOO 7KHVHIDLO XUHVDUHLGHQWLILHGE\DZDWFKGRJWLPHUZKLFKWLPHVRXWDIWHUVHYHUDOVHFRQGVRIZDLWLQJIRUD WHVWFDVHWRFRPSOHWH $ERUW IDLOXUHV WHQG WR EH WKH PRVW SUHYDOHQW DQG UHVXOW LQ DEQRUPDO WHUPLQDWLRQ D FRUH GXPS RIDWDVNFDXVHGE\DVLJQDOJHQHUDWHGZLWKLQDQ26IXQFWLRQ$ERUWIDLOXUHVDUHLGHQ WLILHGE\PRQLWRULQJWKHVWDWXVRIWKHFKLOGSURFHVVH[HFXWLQJWKHWHVWFDVH 6LOHQWIDLOXUHVRFFXUZKHQDQ26UHWXUQVQRLQGLFDWLRQRIHUURURQDQH[FHSWLRQDORSHUDWLRQ ZKLFKFOHDUO\FDQQRWEHSHUIRUPHG IRUH[DPSOHZULWLQJWRDUHDGRQO\ILOH 7KHVHIDLOXUHV DUHQRWGLUHFWO\PHDVXUHGEXWFDQEHLQIHUUHGDVGLVFXVVHGLQ6HFWLRQ +LQGHULQJIDLOXUHVRFFXUZKHQDQLQFRUUHFWHUURUFRGHLVUHWXUQHGIURPD0X7ZKLFKFRXOG PDNHLWPRUHGLIILFXOWWRH[HFXWHDSSURSULDWHHUURUUHFRYHU\+LQGHULQJIDLOXUHVKDYHEHHQ REVHUYHG DV IDLUO\ FRPPRQ IRUPLQJ D VXEVWDQWLDO IUDFWLRQ RI FDVHV ZKLFK UHWXUQHG HUURU FRGHV LQSUHYLRXVZRUN>@EXWDUHQRWIXUWKHUGLVFXVVHGLQWKLVSDSHUGXHWRODFNRIDZD\ WRSHUIRUPDXWRPDWHGLGHQWLILFDWLRQRIWKHVHUREXVWQHVVIDLOXUHV 7KHUHDUHWZRDGGLWLRQDOSRVVLEOHRXWFRPHVRIH[HFXWLQJDWHVWFDVH,WLVSRVVLEOHWKDWDWHVW FDVHUHWXUQVZLWKDQHUURUFRGHWKDWLVDSSURSULDWHIRULQYDOLGSDUDPHWHUVIRUPLQJWKHWHVWFDVH 7KLVLVDFDVHLQZKLFKWKHWHVWFDVHSDVVHVLQRWKHUZRUGVJHQHUDWLQJDQHUURUFRGHLVWKH FRUUHFW UHVSRQVH $GGLWLRQDOO\ LQ VRPH WHVWV WKH 0X7 OHJLWLPDWHO\ UHWXUQV QR HUURU FRGH DQG VXFFHVVIXOO\FRPSOHWHVWKHGHVLUHGRSHUDWLRQ7KLVKDSSHQVZKHQWKHSDUDPHWHUVLQWKHWHVWFDVH KDSSHQ WR EH DOO YDOLG RU ZKHQ LW LV XQUHDVRQDEOH WR H[SHFW WKH 26 WR GHWHFW DQ H[FHSWLRQDO VLWXDWLRQ VXFKDVSRLQWLQJWRDQDGGUHVVSDVWWKHHQGRIDEXIIHUEXWQRVRIDUSDVWDVWRWULJJHUD PHPRU\SURWHFWLRQH[FHSWLRQE\WRXFKLQJDQXQDOORFDWHGYLUWXDOPHPRU\SDJH 2QHRIWKHWUDGHRIIVPDGHWRDWWDLQVFDODELOLW\LQ%DOOLVWDWHVWLQJLVWKDWWKHWHVWKDUQHVVKDVQR ZD\ WR WHOO ZKLFK WHVW FDVHV DUH YDOLG RU LQYDOLG IRU DQ\ SDUWLFXODU 0X7 7KXV VRPH WHVWV UHWXUQLQJ QR HUURU FRGH DUH 6LOHQW IDLOXUHV ZKLOH RWKHUV DUH DFWXDOO\ D VHW RI YDOLG SDUDPHWHU YDOXHV ZKLFK VKRXOG OHJLWLPDWHO\ UHWXUQ ZLWK QR HUURU LQGLFDWLRQ 7KLV KDV WKH HIIHFW RI ³ZDWHULQJ GRZQ´ WKH WHVW FDVHV ZLWK QRQH[FHSWLRQDO WHVWV PDNLQJ WKH UDZ IDLOXUH UDWHV XQGHUHVWLPDWHV:HHVWLPDWHWKDWRIWHVWVDUHRIWKLVQRQH[FHSWLRQDOW\SHIRU26WHVWV EXW HYHQ WKH UDZ IDLOXUH UDWHV DUH VLJQLILFDQW HQRXJK WR PDNH WKH SRLQW WKDW %DOOLVWD WHVWLQJ LV HIIHFWLYHLQILQGLQJUREXVWQHVVIDLOXUHV
5HVXOWV ,QDOOGDWDSRLQWVZHUHFROOHFWHG2SHUDWLQJV\VWHPVZKLFKVXSSRUWHGDOORIWKH VHOHFWHG326,;IXQFWLRQVDQGV\VWHPFDOOVHDFKKDG WRWDOWHVWFDVHVEXWWKRVHVXSSRUWLQJ D VXEVHW RIWKH IXQFWLRQDOLW\WHVWHG KDGIHZHUWHVW FDVHV7KHILQDOQXPEHUZDVGHSHQGHQW RQ ZKLFK IXQFWLRQV ZHUH VXSSRUWHG DQG WKH QXPEHU RI FRPELQDWLRQV RI WHVWV IRU HDFK VXSSRUWHG IXQFWLRQ 0HDVXUHGIDLOXUHUDWHV 7KH FRPSLOHUV DQG OLEUDULHV XVHG WR JHQHUDWH WKH WHVW VXLWH ZHUH WKRVH SURYLGHG E\ WKH 26 YHQGRU,QWKHFDVHRI)UHH%6'1HW%6'/LQX[DQG/\Q[26*18&YHUVLRQZDVXVHG WREXLOGWKHWHVWVXLWH 7KHUH ZHUH ILYH GLIIHUHQW IXQFWLRQV WKDW FRXOG EH PDGH WR FDXVH HQWLUH RSHUDWLQJ V\VWHP FUDVKHVRQVRPH26 HLWKHUDXWRPDWLFUHERRWVRUV\VWHPKDQJV 5HVWDUWIDLOXUHVZHUHUHODWLYHO\ VFDUFH EXWSUHVHQW LQ DOO EXWWZR RSHUDWLQJV\VWHPV $ERUW IDLOXUHV ZHUHFRPPRQ LQGLFDWLQJ WKDW LQ DOO RSHUDWLQJ V\VWHPV LW LV UHODWLYHO\ VWUDLJKWIRUZDUG WR HOLFLW D FRUH GXPS IURP DQ
.RRSPDQ&6'$¶
LQVWUXFWLRQZLWKLQDIXQFWLRQRUV\VWHPFDOO $ERUWIDLOXUHVGRQRWKDYHWRGRZLWKVXEVHTXHQW XVH RI DQ H[FHSWLRQDO YDOXH UHWXUQHG IURP D V\VWHP FDOO WKH\ KDSSHQ LQ UHVSRQVH WR DQ LQVWUXFWLRQZLWKLQWKHYHQGRUSURYLGHGVRIWZDUHLWVHOI $FKHFNZDVPDGHWRHQVXUHWKDW$ERUW IDLOXUHV ZHUH QRW GXH WR FRUUXSWLRQ RI VWDFN YDOXHV DQG VXEVHTXHQW FRUUXSWLRQPLVGLUHFWLRQ RI FDOOLQJSURJUDPRSHUDWLRQ 0X7VWKDWXQGHUZHQWDFDWDVWURSKLFIDLOXUHFRXOGQRWEHFRPSOHWHO\WHVWHGDQGUHVXOWHGLQQR GDWD RQ WKDW 0X7 RWKHU WKDQ WKH SUHVHQFH RI D FDWDVWURSKLF IDLOXUH 6LQFH WKH WHVWLQJ VXLWH LV DXWRPDWHG D V\VWHP FUDVK OHDYHV LW LQ DQ XQUHFRYHUDEOH VWDWH ZLWK UHVSHFW WR WKH IXQFWLRQ LQ TXHVWLRQ )XUWKHU WHVWLQJ D IXQFWLRQ ZKLFK VXIIHUHG D FDWDVWURSKLF WHVW ZRXOG UHTXLUH HLWKHU PDQXDOH[HFXWLRQRILQGLYLGXDOFDVHVRUDGGLQJWULFN\ZDLWLQJDQGV\QFKURQL]DWLRQFRGHLQWRWKH WHVW EHQFKPDUN 0DQXDO H[HFXWLRQ ZDV SHUIRUPHG IRU WKH ,UL[ FDWDVWURSKLF IDLOXUH LQ PXQPDS DQG DOORZHG WKH LGHQWLILFDWLRQ RI WKLV VLQJOH OLQH RI XVHU FRGH ZKLFK FDQ FUDVK WKH HQWLUH26UHTXLULQJDPDQXDOUHERRW PXQPDS PDOORF 0$;,17 1RUPDOL]HGIDLOXUHUDWHV :KLOHLWLVWHPSWLQJWRVLPSO\XVHWKHUDZQXPEHURIWHVWVWKDWIDLODVDFRPSDUDWLYHPHWULF WKLVDSSURDFKLVSUREOHPDWLF0RVW26LPSOHPHQWDWLRQVGRQRWVXSSRUWWKHIXOOVHWRI326,; UHDOWLPHH[WHQVLRQVVRWKHUDZQXPEHURIIDLOXUHVFDQQRWEHXVHGIRUFRPSDULVRQV,QDGGLWLRQ WKHQXPEHURIWHVWVH[HFXWHGSHU0X7LVGHWHUPLQHGE\WKHQXPEHUDQGW\SHVRIWKHDUJXPHQWV 6RDVLQJOH0X7ZLWKDODUJHQXPEHURIWHVWFDVHVFRXOGVLJQLILFDQWO\DIIHFWERWKWKHQXPEHURI IDLOXUHVDQGWKHUDWLRRIIDLOXUHVWRWHVWVH[HFXWHG6LPLODUO\DQ26IXQFWLRQZLWKIHZWHVWFDVHV ZRXOG KDYH PLQLPDO HIIHFW RQ UDZ IDLOXUH UDWHV HYHQ LI LW GHPRQVWUDWHG D ODUJH SHUFHQWDJH RI IDLOXUHV7KXVVRPHVRUWRIQRUPDOL]HGIDLOXUHUDWHPHWULFLVFDOOHGIRUDQGLVUHSRUWHGLQWKHODVW FROXPQRI7DEOH 7DEOH 0HDVXUHGIDLOXUHUDWHVIRUILIWHHQ326,;RSHUDWLQJV\VWHPV
6\VWHP
326,;&DOOV 7HVWHG
&DOOVZLWK &DWDVWURSKLF )DLOXUHV
1XPEHURI 7HVWV
$ERUW )DLOXUHV
5HVWDUW )DLOXUHV
1RUPDOL]HG $ERUW 5HVWDUW5DWH
$,;
)UHH%6'
+38;
+38;
,5,;
,5,;
/LQX[
/\Q[
1HW%6'
'LJLWDO8QL[
'LJLWDO8QL[
41;
41;
6XQ26
6XQ26
.RRSPDQ&6'$¶
:HGHILQHWKHQRUPDOL]HGIDLOXUHUDWHIRUDSDUWLFXODURSHUDWLQJV\VWHPWREH N
∑
F =
fi w i ZLWKDUDQJHRIYDOXHVIURPWRLQFOXVLYHZKHUH ti
i=1
1 QXPEHURIIXQFWLRQVWHVWHG ZLLVDZHLJKWLQJRILPSRUWDQFHRUUHODWLYHH[HFXWLRQIUHTXHQF\RIWKDWIXQFWLRQZKHUH
∑ wi
= 1
ILLVWKHQXPEHURIWHVWVZKLFKSURGXFHGUREXVWQHVVIDLOXUHIRUIXQFWLRQL WLLVWKHQXPEHURIWHVWVH[HFXWHGIRUIXQFWLRQL
7KLV GHILQLWLRQ SURGXFHV D VRUW RI H[SRVXUH PHWULF LQ ZKLFK WKH IDLOXUH UDWH ZLWKLQ HDFK IXQFWLRQLVZHLJKWHGDQGDYHUDJHGDFURVVDOOIXQFWLRQVWHVWHGIRUDSDUWLFXODU267KLVPHWULF KDV WKH DGYDQWDJH RI UHPRYLQJ WKH HIIHFWV RI GLIIHULQJ QXPEHU RI WHVWV SHU IXQFWLRQ DQG DOVR SHUPLWV FRPSDULQJ 26 LPSOHPHQWDWLRQV ZLWK GLIIHULQJ QXPEHUV RI IXQFWLRQV LPSOHPHQWHG DFFRUGLQJWRDVLQJOHQRUPDOL]HGPHWULF)RUWKHUHVXOWVJLYHQLQ7DEOHDQGWKHUHPDLQGHURI WKLV SDSHU DQ HTXDO ZHLJKWLQJ LV XVHG LH w i = 1 ⁄ N WR SURGXFH D JHQHULFDOO\ DSSOLFDEOH UHVXOW+RZHYHULIDQ26ZHUHEHLQJWHVWHGZLWKUHJDUGWRDVSHFLILFDSSOLFDWLRQZHLJKWLQJV VKRXOGEHGHYHORSHGWRUHIOHFWWKHG\QDPLFIUHTXHQF\RIFDOOLQJHDFKIXQFWLRQWRJLYHDPRUH DFFXUDWHH[SRVXUHPHWULF
2SHUDWLQJ6\VWHP7HVWHG
$,; '81,; '81,; )UHH %6' +38; +38; ,UL[ ,UL[ /LQX[ /\Q[26 1HW%6' 41; 41; 6XQ26 6XQ26
%DOOLVWD 5REXVWQHVV 7HVWV 326,; )XQFWLRQ &DOOV &DWDVWURSKLF
&DWDVWURSKLF
$ERUW )DLOXUHV 5HVWDUW )DLOXUH
&DWDVWURSKLF
&DWDVWURSKLF
&DWDVWURSKLFV
1RUPDOL]HG)DLOXUH5DWH
)LJXUH1RUPDOL]HGUREXVWQHVVIDLOXUHUDWHVIRUILIWHHQ326,;RSHUDWLQJV\VWHPV
.RRSPDQ&6'$¶
)LJXUHVKRZVWKHQRUPDOL]HGIDLOXUHVUDWHVDVFDOFXODWHGIRUHDFKRSHUDWLQJV\VWHP WKLVLV WKH VDPH GDWD DV WKH ULJKWPRVW FROXPQ LQ 7DEOH 2YHUDOO IDLOXUH UDWHV IRU ERWK $ERUW DQG 5HVWDUWIDLOXUHVUDQJHIURPWKHORZRI $,; WRDKLJKRI 41; 7KHPHDQ IDLOXUH UDWH LV ZLWK D VWDQGDUG GHYLDWLRQ RI SHUFHQWDJH SRLQWV $V PHQWLRQHG SUHYLRXVO\ WKHVHDUH UDZ IDLOXUHUDWHV WKDW LQFOXGH QRQH[FHSWLRQDOWHVW FDVHV ZKLFK GLOXWHWKH IDLOXUHUDWHVDQGGRQRWLQFOXGH6LOHQWIDLOXUHV
*HQHUDOL]LQJ%DOOLVWD DSSO\LQJWRRWKHULQWHUIDFHV 7KHLQLWLDODSSURDFKWR%DOOLVWDDQGLWVSUHGHFHVVRUVZDVIRFXVVHGRQWHVWLQJDQLQFUHDVLQJO\ ODUJHUVHWRIRSHUDWLQJV\VWHPFDOOV,WLVRIFRXUVHQDWXUDOWREHVNHSWLFDOWKDWWKHDSSURDFKFDQ VFDOH WR LQFUHDVHG WKRURXJKQHVV DQG RWKHU DSSOLFDWLRQV 7R GDWH LPSOHPHQWDWLRQV KDYH EHHQ FRPSOHWHGWRWKHSURRIRIFRQFHSWSRLQWLQDVLPXODWLRQ$3,DQGDPHFKDQLVPKDVEHHQFUHDWHG IRUSHUIRUPLQJILQHUJUDLQHGWHVWLQJ ([SHULHQFHZLWKGLVWULEXWHGVLPXODWLRQ$3, 7HVWLQJ RSHUDWLQJ V\VWHP FDOOV UHTXLUHV WKH DELOLW\ WR WHVW WUDGLWLRQDO VXEURXWLQHFDOOSOXV SDUDPHWHUOLVWLQWHUIDFHVZLWKVLQJOHFDOOVDQGDPLQLPXPRIV\VWHPVWDWH+RZHYHURWKHU$3,V PD\KDYHVLJQLILFDQWO\GLIIHUHQWUHTXLUHPHQWV7RSUREHWKHH[WHQVLELOLW\RI%DOOLVWDDQHQWLUHO\ GLIIHUHQW$3,LVEHLQJWHVWHGEDVHGRQDGLVWULEXWHGVLPXODWLRQV\VWHP 7KH+LJK/HYHO$UFKLWHFWXUH +/$ 5XQ7LPH,QWHUIDFH 57, LVDQ$3,XVHGIRUDGLVWULEXWHG VLPXODWLRQV\VWHPE\WKH86'HSDUWPHQWRI'HIHQVH>@7KLVV\VWHPLVZULWWHQLQ&LQDQ HQWLUHO\REMHFWRULHQWHGVW\OHDQGKDVVLJQLILFDQWO\PRUHFRPSOLFDWHGSUHFRQGLWLRQVIRUIXQFWLRQ FDOOV'LIIHUHQFHVIURP26WHVWLQJLQFOXGH $QREMHFWRULHQWHGLPSOHPHQWDWLRQPHDQLQJWKDWPHWKRGVDUHWREHWHVWHGUDWKHUWKDQIXQF WLRQFDOOV7KLVFDQODUJHO\EHWUHDWHGDVDV\QWDFWLFGLIIHUHQFH $V\VWHPZKLFKUHTXLUHVVXSSRUWIRUFDOOEDFNVWRREMHFWVIRUWHVWLQJUDWKHUWKDQDVLPSOHFDOO WRDIXQFWLRQWREHWHVWHG7KLVZDVKDQGOHGZLWKDVOLJKWPRGLILFDWLRQWRWKHVWDQGDUG%DOOLVWD FOLHQWWHVWVFDIIROGLQJ 6LJQLILFDQWO\PRUHFRPSOH[GDWDW\SHV,WZDVIHDUHGWKDWDSUROLIHUDWLRQRIGDWDW\SHYDULDQWV ZRXOGLQFUHDVHWKHGLIILFXOW\RIWHVWLQJWKHP+RZHYHULQWKHHQGRQO\GDWDW\SHVZHUH UHTXLUHGIRULQWHUIDFHV$QGRIWKHVHGDWDW\SHVZHUHFRPSOHWHO\UHXVHGIURPRSHUDW LQJV\VWHPWHVWLQJZKLOHWKHUHVWLQKHULWHGH[LVWLQJWHVWFDVHV7KXVSHUIRUPLQJWHVWVRQGDWD W\SHVSURYHGPRGHUDWHO\VFDODEOH3HUKDSVMXVWDVLPSRUWDQWO\WKHVHVDPHGDWDW\SHVZRXOG EHXVHGKHDYLO\E\DQ\+/$DSSOLFDWLRQSURJUDPDQGFRXOGWKXVEHUHXVHGIRU%DOOLVWDWHVW LQJRIVXFKDSSOLFDWLRQSURJUDPVDWDIXWXUHGDWHUHVXOWLQJLQHYHQPRUHHIIHFWLYHVFDOLQJ $V\VWHPZLWKPRGHUDWHO\FRPSOLFDWHGSUHFRQGLWLRQVIRUHDFKDQGHYHU\PHWKRG IRUH[DP SOH WRSHUIRUPDQRSHUDWLRQILUVWDVRFDOOHGIHGHUDWLRQPXVWEHFUHDWHG WKHQDIHGHUDWHPXVW EHFUHDWHGDQGLQVWUXFWHGWRMRLQWKHIHGHUDWLRQ WKHQWKHFRQVWUXFWRUVIRUWKDWIHGHUDWHPXVWEH H[HFXWHGGHSHQGLQJRQGDWDW\SHV 8SRQIXUWKHUH[DPLQDWLRQKRZHYHULWZDVIRXQGWKDW WKHGLIIHUHQWPHWKRGVEHLQJWHVWHGFRXOGEHFROODSVHGLQWRRQO\WZHOYHGLIIHUHQWHTXLYDOHQFH FODVVHVZLWKUHVSHFWWRWHVWVFDIIROGLQJ7KXVRQO\VHWVRIVFDIIROGLQJDUHUHTXLUHG HDFK RQO\DIHZOLQHVRIFRGHORQJ IRUPRGXOHVXQGHUWHVWZKLFKDVZLWKWKHGDWDW\SHVLVD VXEOLQHDUVFDOLQJZLWKUHVSHFWWRQXPEHURIPHWKRGVEHLQJWHVWHG 6XSSRUWIRUHUURUUHSRUWLQJE\WKURZLQJH[FHSWLRQVUDWKHUWKDQHUURUUHWXUQFRGHV7KLVZDV GRQHE\DGGLQJDQH[FHSWLRQKDQGOHUWRWKH%DOOLVWDWHVWKDUQHVVZLWKWKHUHTXLUHPHQWWKDWWKH WHVWHU SURYLGH D ZD\ WR GLVWLQJXLVK EHWZHHQ DFFHSWDEOH H[FHSWLRQ YDOXHV DQG ³XQNQRZQ´ H[FHSWLRQVWKDWZKHQWKURZQDPRXQWWR$ERUWIDLOXUHV
.RRSPDQ&6'$¶
6WUXFWXUHWHVWLQJ 7HVWLQJRIGDWDVWUXFWXUHVFDQTXLFNO\EHFRPHWHGLRXVLIVHSDUDWHGDWDW\SHVPXVWEHFUHDWHGWR WHVW HDFK GLIIHUHQW VWUXFWXUH )RUWXQDWHO\ WKLV FDQ EH RYHUFRPH DXWRPDWLFDOO\ E\ FUHDWLQJ D V\VWHP WKDW FDQ SDUVH VWUXFWXUH GHILQLWLRQV DQG FUHDWH FRPSRVLWH FRQVWUXFWRUV LQ WHUPV RI WKH SULPLWLYHFRPSRQHQWVRIDVWUXFWXUH$VDVLPSOHH[DPSOHDFRPSOH[QXPEHUFDQEHWUHDWHGDV DVWUXFWXUHFRQWDLQLQJWZRIORDWLQJSRLQWQXPEHUVDQGWHVWVIRUWKDWGDWDW\SHFDQEHJHQHUDWHG VLPSO\E\SURYLGLQJWKHVWUXFWXUHLQIRUPDWLRQJLYHQWKDWIORDWLQJSRLQWQXPEHUVFDQDOUHDG\EH WHVWHGDVDEDVHGDWDW\SH ,WPXVWEHNHSWLQPLQGWKDWDGDWDVWUXFWXUHPLJKWEHSRLQWHGWRDVZHOODVUHIHUHQFHGGLUHFWO\ VRWHVWYDOXHVIRUDVWUXFWXUHLQKHULWERWKSRLQWHUWHVWV HJQXOOSRLQWHULQVWHDGRIDSRLQWHUWRWKH VWUXFWXUH DVZHOODVYDULRXVGDWDYDOXHWHVWV HJSRLQWHUWRDVWUXFWXUHZLWKSDUWLFXODUYDOXHV )LQHUJUDLQWHVWLQJGLDOV ORJLFDOVWUXFWXUHV 2QH RI WKH RULJLQDO DQG FXUUHQW JRDOV RI %DOOLVWD LV WR DXWRPDWLFDOO\ JHQHUDWH KDUGHQLQJ ZUDSSHUVEDVHGRQWHVWUHVXOWV:KLOHWKDWZRUNLVVWLOORQJRLQJDQGEH\RQGWKHVFRSHRIWKLV GLVFXVVLRQLWOHGWRDQHDUO\IRFXVRQDQHHGIRUXVLQJWHVWLQJDVDFKDUDFWHUL]DWLRQPHFKDQLVPLQ DGGLWLRQWRPHUHO\DTXDQWLILFDWLRQPHFKDQLVP)RUH[DPSOHLWLVQRWHQRXJKWRVLPSO\VD\WKDW DSDUWLFXODUPRGXOHH[KLELWVUREXVWQHVVIDLOXUHVDWDFHUWDLQUDWHEXWDOVRRQHPXVWNQRZH[DFWO\ ZKDW WULJJHUV WKHVH IDLOXUHV %HLQJ DEOH WR GR WKLV UHTXLUHV IDLUO\ ILQHJUDLQHG WHVWLQJ $V DQ H[DPSOHWKHILOHKDQGOHWHVWVGHVFULEHGHDUOLHUDUHTXLWHFRXUVHJUDLQDQGFHUWDLQO\GRQRWWHVWDOO SRVVLEOHFRPELQDWLRQV IRUH[DPSOHWKH\WHVWDILOHRSHQIRUZULWHZLWKQRGDWDEXWQRWDILOH RSHQIRUZULWHZLWKVRPHGDWDDOUHDG\LQLW ,WFDQEHUHDGLO\VHHQWKDWZLWKRXWVRPHVRUWRI IXUWKHUGHFRPSRVLWLRQWKHQXPEHURISRVVLEOHWHVWFDVHVJURZVLQDFRPELQDWLRQDOH[SORVLRQDQG ORVHVVFDODELOLW\ $GGLWLRQDOO\LWLVGHVLUDEOHWRKDYHDYHU\ODUJHVHWRISRWHQWLDOWHVWVIRU%DOOLVWDWRGUDZXSRQ ZLWKRXWKDYLQJWRPDQXDOO\FUHDWHWRRPDQ\WHVWV$SDUWLFXODUUHDVRQWRZDQWWRGRWKLVLVWR LPSOHPHQWUDQGRPL]HGWHVWLQJIRUWKHSXUSRVHVRIFKDUDFWHUL]DWLRQDVZHOODVDGGLQJDQDELOLW\ WRGLVFRYHUQHZKHXULVWLFDOO\XVHIXOGDWDW\SHYDOXHVIRUWHVWLQJ )RUWXQDWHO\DWHFKQLTXHWRJHQHULFL]HGDWDWHVWLQJLVDYDLODEOHLQWKHIRUPRIGHFRPSRVLQJWKH FRQVWUXFWRUVIRUDQ\SDUWLFXODUGDWDW\SHWHVWYDOXHFUHDWLRQ$QDQDORJ\FDQEHPDGHWRDVHWRI GLDOVWKDWDUHVHWLQDSDUWLFXODUSRVLWLRQIRUDSDUWLFXODUWHVWYDOXH7KHGLDOVDUHRUWKRJRQDO RUDW OHDVWORRVHO\FRXSOHGLQWHUPVRIPHDQLQJIXOVHWVRIYDOXHV DQGHDFKGLDOLVEDFNHGE\DVHWRI HQXPHUDWHGFRQVWUXFWRUVIRUSDUWLFXODUDVSHFWVRIDWHVWYDOXH 7KLVDSSURDFKLVVLPLODUWRWKH FDWHJRU\FKRLFH DSSURDFK GHVFULEHG LQ >@ EXW %DOOLVWD UHTXLUHV QHLWKHU LQWHUFDWHJRU\ FRQVWUDLQWVQRUVSHFLILFDOO\GHVFULEHGWHVWIUDPHV $VDQH[DPSOHVRPH RIWKHGLDOVIRUILOH FUHDWLRQVDQGWKHLUVHWWLQJVLQFOXGH )LOHH[LVWHQFH H[LVWVGRHVQRWH[LVWGHOHWHGDIWHUILOHKDQGOHREWDLQHG )LOHUHDGSHUPLVVLRQV UHDGSHUPLVVLRQUHDGSHUPLVVLRQWRRWKHUWKDQFXUUHQWWDVNQR UHDG SHUPLVVLRQ )LOHDFFHVV DOOFRPELQDWLRQVRIRSHQIRUUHDGZULWHUHDGZULWHHWF 1RWHWKDWWKHDERYHGLDOVDUHLQIOXHQFHGE\DGHVLUHWRGHFRXSOHYDULRXVDVSHFWVRIWKHGDWD YDOXHEXWUHPDLQFRXSOHGWRWKHH[WHQWUHTXLUHGE\FRQYHQLHQWO\ZULWLQJFRQVWUXFWRUV7KXVILOH SHUPLVVLRQVFDQEHGHFRXSOHGEHFDXVHWKH\FDQEHVHWLQGHSHQGHQWO\DIWHUILOHFUHDWLRQEXWILOH DFFHVVPRGHVDUHSODFHGLQWRDVLQJOHGLDOEHFDXVHLWZDVIRXQGWREHDZNZDUGWRPDQLSXODWH WKHPLQGHSHQGHQWO\,QSDUWLFXODUDSSOLFDWLRQVWKHILOHGDWDW\SHPLJKWEHH[SDQGHGWRLQFOXGH RWKHUGLDOVVXFKDVWKHW\SHRIGDWDSUHVHQWLQWKHILOH :KLOHWKHDGGLWLRQRIGLDOVVHHPVWRFRPSOLFDWHWKHWHVWLQJVLWXDWLRQLWWXUQVRXWWKDWWKHUHLVD
.RRSPDQ&6'$¶
XQLI\LQJDSSURDFKDYDLODEOHORJLFDOVWUXFWXUHV(YHU\GDWDW\SHWKDWLVGHFRPSRVHGLQWRGLDOV FDQ EH ORJLFDOO\ WUHDWHG DV D GDWD VWUXFWXUH IRU WHVWLQJ SXUSRVHV WKH RQO\ GLIIHUHQFH IURP D SK\VLFDOVWUXFWXUHSRLQWRIYLHZLVWKDWWKHUHLVQRWHVWIRUDSRLQWHUWRWKDWGDWDVWUXFWXUHVLQFHLW RQO\H[LVWVORJLFDOO\IRUWKHWHVWSURJUDPQRWSK\VLFDOO\ZLWKLQWKH0X7 6RWRFRQWLQXHWKH ILOH KDQGOH H[DPSOH WHVWLQJ ILOH KDQGOHV LV GRQH E\ GHFODULQJ D ILOH KDQGOH WR EH D ORJLFDO VWUXFWXUHFRQWDLQLQJDVHWRIGLDOVDQGWKHQH[SORULQJYDULRXVFRPELQDWLRQVRIGLDOVLQH[DFWO\ WKHVDPHZD\WKDW%DOOLVWDZRXOGWHVWFRPELQDWLRQVRIYDOXHVZLWKLQDSK\VLFDOGDWDVWUXFWXUH 3RWHQWLDOLPSURYHPHQWVLQFKDUDFWHUL]DWLRQ %H\RQG VLPSO\ PHDVXULQJ IDLOXUH UDWHV %DOOLVWD WHVWLQJ KDV WKH SRWHQWLDO WR FKDUDFWHUL]H IDLOXUHSDWWHUQVWKURXJKWKHXVHRISDWWHUQHGWHVWLQJRQILQHJUDLQGLDOEDVHGUHSUHVHQWDWLRQVRI GDWDYDOXHV3UHYLRXVZRUNLQWKHDUHDRIWHVWH[SORUDWLRQLQFOXGHVWKH$(7*V\VWHP> @ZKLFK SHUPLWVWHVWHUVWRVSHFLI\ZKLFKVHWVRISDUDPHWHUVLQDPXOWLSDUDPHWHUIXQFWLRQFDOOVDUHFORVHO\ FRXSOHGIXQFWLRQDOO\DQGWKXVVKRXOGEHWHVWHGLQFRQFHUW7KH72)8V\VWHP>@WDNHVWKLVD VWHSIXUWKHUE\DGGLQJLQWHUDFWLRQZHLJKWLQJVWRWKHLGHQWLILHGSDUDPHWHUWXSOHV &OHDUO\ WKH WHVWLQJ DSSURDFKHV IURP $(7* DQG 72)8 FDQ EH DSSOLHG LQ SDUW WR %DOOLVWD WHVWLQJLQWKHIRUPRIHQFRXUDJLQJWHVWLQJRIRQHRUDIHZSDUDPHWHUVDWDWLPH,WLVSUREDEO\ XQGHVLUDEOHWRKDYHDSURJUDPPHUJLYHH[SOLFLWKLQWVWR%DOOLVWDZLWKUHVSHFWWRFRXSOLQJ WKLV UHGXFHVVFDODELOLW\DQGWKHUHLVDOZD\VWKHFKDQFHWKDWWKHWHVWHUZLOOIDLOWRVHHDVXEWOHFRXSOLQJ DQGSURYLGHDQLQDFFXUDWHKLQW +RZHYHUKHXULVWLFVDUHEHLQJGHYHORSHGWRSHUPLW%DOOLVWDWR DXWRPDWLFDOO\GLVFRYHUZKDWFRXSOLQJVPD\EHSUHVHQWWRGRH[SORUDWLRQDURXQGJHQHUDOUHJLRQV RILQWHUHVWLQWKHERXQGDU\EHWZHHQWHVWVWKDWVXFFHHGDQGWKDWIDLODQGWRJHQHUDOO\GR³VPDUW´ WHVWLQJWRFKDUDFWHUL]HIDLOXUHVZLWKDUHDVRQDEOHQXPEHURIWHVWV 7KHFXUUHQWPHFKDQLVPIRUPDNLQJ%DOOLVWDDYDLODEOHLVDVDZHEEDVHGWHVWLQJVHUYLFHUDWKHU WKDQ DV D SLHFH RI VRIWZDUH WKDW LV H[HFXWHG ORFDOO\ 3DUW RI WKH UHDVRQ IRU WKLV VWUDWHJ\ LV WR HQDEOHGDWDFROOHFWLRQWKDWFDQEHXVHGWRLPSURYHWKHTXDOLW\RIFKDUDFWHUL]DWLRQDSSURDFKHV
3RWHQWLDODSSOLFDWLRQVWRIDXOWWROHUDQFH 7UDGLWLRQDO IDXOW WROHUDQFH WHFKQLTXHV DUH RIWHQ EDVHG RQ WKH QRWLRQ RI FUHDWLQJ GHSHQGDEOH V\VWHPV RXW RI LQGLYLGXDOO\ XQGHSHQGDEOH FRPSRQHQWV 7KH JHQHUDO LGHD LV WKDW LI LQGLYLGXDO FRPSRQHQW IDLOXUH UDWHV DUH NQRZQ WKHQ D YDULHW\ RI WHFKQLTXHV FDQ EH XVHG WR HQVXUH WKDW D FRPSRVLWHV\VWHPLVKLJKO\GHSHQGDEOH,WLVRIFRXUVHXQOLNHO\WKDW%DOOLVWDWHVWLQJZLOOSURGXFH KLJKO\DFFXUDWHUHOLDELOLW\HVWLPDWHVLQWHUPVRIIDLOXUHVSHUPLOOLRQRSHUDWLQJKRXUVDVLVGRQHIRU ILHOGHGKDUGZDUH1RQHWKHOHVVLWLVSUREDEO\DZRUWK\JRDOWRSURGXFHVRPHUHODWLYHPHDVXUHRI VRIWZDUH FRPSRQHQW UHOLDELOLW\ DQG D UDWLQJ RI H[FHSWLRQ KDQGOLQJ DELOLW\ PLJKW EH D XVHIXO VWDUWLQJSRLQW 7KHUH DUH IRXU DUHDV LQ ZKLFK D %DOOLVWD WHVWLQJ PLJKW SURYLGH LPSURYHG TXDQWLILFDWLRQ IRU IDXOWWROHUDQFHRIVRIWZDUHLQWHQVLYHV\VWHPV GHWHFWLQJV\VWHPFUDVKHVGHWHFWLQJYLRODWLRQVRI IDLOVWRSDVVXPSWLRQVSURYLGLQJLQIRUPDWLRQIRUFUHDWLQJVRIWZDUHZUDSSHUVDQGSURYLGLQJDQ DVVHVVPHQWRIVRIWZDUHGLYHUVLW\ ,Q JHQHUDO LW LV FRQVLGHUHG D JRRG LGHD WR EXLOG V\VWHPV WKDW DUH LPSRVVLEOH WR FUDVK IURP ³QRUPDO´VRIWZDUHDSSOLFDWLRQV7KLVLVDQH[WHQVLRQRIWKHWUDGLWLRQDOIDXOWWROHUDQFHDSSURDFK RIKDYLQJD³KDUGFRUH´WKDWLVDVVPDOODVSRVVLEOHDQGXSRQZKLFKUHVRXUFHVDUHODYLVKHGLQ ERWK GHVLJQ DQG LPSOHPHQWDWLRQ WR HQVXUH LWV GHSHQGDELOLW\ 2XWVLGH RI WKDW KDUG FRUH RI D SURFHVVRURURSHUDWLQJV\VWHPNHUQHOLWLVWREHH[SHFWHGWKDWEXJVZLOORFFXURQDUHJXODUEDVLV JHQHUDWLQJ H[FHSWLRQDO VLWXDWLRQV WKDW PXVW EH KDQGOHG UREXVWO\ ZLWK UHVSHFW WR WKH V\VWHP DUFKLWHFWXUH 7KH SDUWLFXODUDUHD RI HPSKDVLV IRU %DOOLVWD LV RQ H[HUFLVLQJ H[FHSWLRQ KDQGOLQJ
.RRSPDQ&6'$¶
WHFKQLTXHV VLQFH LW LV DVVXPHG WKDW QRUPDO FRUUHFW YDOXHV DUH ZHOO H[HUFLVHG E\ QRUPDO RSHUDWLRQDOWHVWLQJ 7HVWIRURYHUDOOV\VWHPFUDVKVLWXDWLRQV 7KHPRVWREYLRXVDSSOLFDWLRQRI%DOOLVWDWRIDXOWWROHUDQFHLVWRIHUUHWRXWZD\VWRFRPSOHWHO\ FUDVKDV\VWHPIURPXQSULYLOHJHGDSSOLFDWLRQVRIWZDUH$VWKHSUHYLRXVUHVXOWVLQGLFDWH%DOOLVWD FDQORFDWHVRPHZD\VWRFUDVKDQRSHUDWLQJV\VWHPHYHQZLWKWKHFXUUHQWUDWKHUVLPSOLVWLFWHVWV DYDLODEOH $SSOLFDWLRQ RI D VLPLODU PHWKRGRORJ\ VKRXOG KHOS TXDQWLI\ WKH H[WHQW WR ZKLFK H[FHSWLRQDOFRQGLWLRQVFDQWULFNOHGRZQWKURXJKDQDSSOLFDWLRQSURJUDPWRUHDFKWKHRSHUDWLQJ V\VWHPDVZHOO 7KHUH DUHIRXUDSSURDFKHV WKDW FDQLPSURYH WKHHIIHFWLYHQHVVRI%DOOLVWD WRLGHQWLI\ V\VWHP FUDVKHV7KHEDVLFWHFKQLTXHVUHTXLUHGDUHWKRVHGLVFXVVHGLQWKHSUHYLRXVVHFWLRQ 8VHILQHUJUDLQWHVWLQJXVLQJWKHGLDODSSURDFKWRGHFRPSRVLQJSDUDPHWHUWHVWYDOXHV 7HVW SULPLWLYH IXQFWLRQV ZLWKLQ WKH RSHUDWLQJ V\VWHP UDWKHU WKDQ MXVW WKH 326,;VSHFLILHG IXQFWLRQV 7HVWFRQFXUUHQWIXQFWLRQFDOOVWRLGHQWLI\WLPLQJGHSHQGHQWSUREOHPVVXFKDVUHVRXUFHORFN LQJEXJV 7HVWIRUWKHUHVXOWVRIUHVRXUFHH[KDXVWLRQHIIHFWVVXFKEHKDYLQJJUDFHIXOO\ZKHQWKHSURFHVV WDEOHLVIXOORUZKHQDPHPRU\OHDNLVHQFRXQWHUHG 7HVWIRUYLRODWLRQVRIIDLOVWRSDVVXPSWLRQ $ VHFRQGZD\ WRDSSO\ %DOOLVWD WR PHDVXULQJ DQG LPSURYLQJ )DXOW 7ROHUDQFH LV WR XVHLW WR GHWHFWSUREOHPVZLWKUHVSHFWWR5HVWDUWDQG6LOHQWIDLOXUHV,WLVFRPPRQWREDVHIDXOWWROHUDQFH RQDIDLOIDVWDVVXPSWLRQ+RZHYHUD5HVWDUWIDLOXUHPHDQVWKDWDWDVNLV³IDLOVORZ´LQWKDWLWV IDLOXUH LV QRW UHFRJQL]HG XQWLO W\SLFDOO\ D WLPHRXW SHULRG KDV HODSVHG IDLOXUH WR GHWHFW D SHULRGLF ³KHDUWEHDW´ PHVVDJH IDLOXUH WR UHVHW D ZDWFKGRJ WLPHU HWF )XUWKHUPRUH D 6LOHQW IDLOXUH PHDQV WKDW D V\VWHP KDV IDLOHG VRPHKRZ EXW WKDW WKH IDLOXUH LV XQGHWHFWHG IURP VRPH SHULRGRIWLPH,WPLJKWEHWKDWWKH6LOHQWIDLOXUHLVQRWDFWLYDWHGRULWPLJKWEHWKDWLWUHVXOWVLQD ³WLPH ERPE´ LQVLGH WKH V\VWHP VXFK DV D 18// SRLQWHU WR D GDWD VWUXFWXUH WKDW LV QRW GHUHIHUHQFHGH[FHSWLQDQHUURUKDQGOLQJURXWLQH 5HVWDUWIDLOXUHVDUHPRGHUDWHO\HDV\WRGHDOZLWKDQGEDVLFDOO\DUHVHDUFKHGIRULQWKHVDPH ZD\WKDWFDWDVWURSKLFV\VWHPIDLOXUHVDUHVHDUFKHGIRUDVGLVFXVVHGDERYH$SDUWLFXODUO\QDVW\ UHVWDUW IDLOXUH WKDW ZDV IRXQG RQ D UHDO V\VWHP ZDV LQ $,; LQ ZKLFK WKH FRUH GXPS IDFLOLW\ ZDLWHGXQWLOHQRXJKGLVNVSDFHRQDQHDUO\IXOOGLVNZDVDYDLODEOHWRGXPSFRUHHYHQLIFRUH GXPSLQJZDVGLVDEOHG WKHFKHFNIRUGLVNVSDFHFDPHEHIRUHWKHFKHFNIRUWKHPD[LPXPDPRXQW WRGXPSZKLFKLQRXUFDVHZDVVHWWR]HUR 6LOHQWIDLOXUHVFDQEHLGHQWLILHGE\XVLQJPXOWLYHUVLRQFRPSDULVRQDPRQJGLIIHUHQW26WHVW UHVXOWV7KLVVFKHPHGHWHUPLQHVGHFODUHVDUHVXOWWREHD6LOHQWIDLOXUHLIWKDWWHVWFDVHRQWKDW26 UHWXUQVQRHUURULQGLFDWLRQZKLOHWKDWVDPHWHVWFDVHUHVXOWVLQDQHUURULQGLFDWLRQRQDQRWKHU26 7KLVDSSURDFKZRUNVZLWKUHDVRQDEOHDFFXUDF\LQSUDFWLFH DSSUR[LPDWHO\DFFXUDWH >@ EXWRIFRXUVHUHTXLUHVWKDWWKHUHEHPXOWLSOHLPSOHPHQWDWLRQVDYDLODEOH,PSURYLQJSHUIRUPDQFH IRU6LOHQWIDLOXUHVWDNHVRQHRIWZRDSSURDFKHV,QVRPHFDVHVKDUGZDUHLVDYDLODEOHWKDWFRXOG GHWHFWD6LOHQWIDLOXUHEXWLVQRWEHLQJXVHG IRUH[DPSOHEHFDXVHPHPRU\SURWHFWLRQLVWXUQHG RII IRU SDJH ]HUR RI PHPRU\ WR VXSSRUW OHJDF\ VRIWZDUH ,Q RWKHU FDVHV LW LV VLPSO\ WKDW WKH H[FHSWLRQDO SDUDPHWHUV DUH QRW EHLQJ FKHFNHG DQG LQ DGGLWLRQ HYDGH GHWHFWLRQ E\ KDUGZDUH PHFKDQLVPV
.RRSPDQ&6'$¶
*HQHUDWLQJVRIWZDUHZUDSSHUV $ SRVVLEOH ZD\ WR LPSURYH VRIWZDUH IDXOW WROHUDQFH LV E\ XVLQJ VRIWZDUH ³ZUDSSHUV´ WKDW HQFDSVXODWHWKHPRGXOHVRIDVRIWZDUHV\VWHP:KLOHLWLVDHVWKHWLFDOO\PRUHSOHDVLQJWRIL[DQ\ H[FHSWLRQKDQGOLQJGHILFLHQFLHVLQDVRIWZDUHPRGXOHGLUHFWO\WKHUHDUHWLPHVZKHQWKDWLVQRW SRVVLEOH HLWKHU EHFDXVH RI ODFN RI DFFHVV WR WKH VRXUFH FRGH RU WLPHEXGJHW FRQVWUDLQWV 7KHUHIRUH LW PD\ EH DWWUDFWLYH WR DXWRPDWLFDOO\ RU VHPLDXWRPDWLFDOO\ JHQHUDWH VRIWZDUH ZUDSSHUV WKDW DUH LQWHUSRVHG EHWZHHQ WKH $3, LPSOHPHQWDWLRQ DQG WKH DSSOLFDWLRQ 7KHVH ZUDSSHUVFDQDXWRPDWLFDOO\FKHFNIRUH[FHSWLRQDOFRQGLWLRQVDQGWDNHDFWLRQWRUHSRUWWKHPRULQ VRPHFDVHVUHFRYHUIURPWKHP :KLOH D JHQHULF FDSDELOLW\ WR FUHDWH XQLYHUVDO VRIWZDUH ZUDSSHUV UHPDLQV D UHVHDUFK LVVXH WKHUHDUHVHYHUDOVWUDWHJLHVWKDWPD\EHHPSOR\HGZLWKLQDZUDSSHUVXFKDVUHSRUWLQJHUURUUHWXUQ FRGHV SHUIRUPLQJ UHWU\ DQG LQYRNLQJ DOWHUQDWH DOJRULWKPV $ SDUWLFXODUO\ LQWHUHVWLQJ XVH RI ZUDSSHUVLQFRQQHFWLRQZLWKUREXVWQHVVWHVWLQJLVWRXVHDUREXVWQHVVWHVWLQJUHVXOWDVDZDUQLQJ WKDWDSDUWLFXODURSHUDWLRQLVKD]DUGRXVDQGWRSHUIRUPDFKHFNSRLQWDJDLQVWWKHQHHGIRUDODWHU UROOEDFNLIWKDWRSHUDWLRQVKRXOGIDLO7KLVZRXOGXVHIDLOXUHSUHGLFWLRQVIURPUREXVWQHVVWHVWLQJ WR OHVVHQ WKH W\SLFDO FRVW RI FKHFNSRLQWLQJ WR WKRVH WLPHV ZKHQ LW ZRXOG DSSHDU WR EH PRVW IUXLWIXO 7KH ;HSW DSSURDFK >@ XVHV VRIWZDUH ³ZUDSSHUV´ DURXQG SURFHGXUH FDOOV DV D ZD\ WR HQFDSVXODWHHUURUFKHFNLQJDQGHUURUKDQGOLQJZLWKLQWKHFRQWH[WRIDUHDGDEOHSURJUDP*LYHQ WKDWWKHIDXOWWROHUDQFHFRPPXQLW\KDVIRXQGWKDWWUDQVLHQWV\VWHPIDLOXUHUDWHVIDURXWQXPEHU PDQLIHVWDWLRQVRISHUPDQHQWV\VWHPIDXOWVGHVLJQHUURUVLQSUDFWLFHHYHQDVVLPSOHDVWUDWHJ\DV UHWU\LQJ IDLOHG RSHUDWLRQV IURP ZLWKLQ D VRIWZDUH ZUDSSHU KDV WKH SRWHQWLDO WR VLJQLILFDQWO\ LPSURYHV\VWHPUREXVWQHVV2QHFRXOGWKHUHIRUHHQYLVLRQXVLQJWKHUHVXOWVRI%DOOLVWDWHVWLQJWR SURYLGHLQIRUPDWLRQIRUHUURUFKHFNLQJZLWKLQDQ;HSWVRIWZDUHZUDSSHU 3RWHQWLDOXVHDVPHDVXUHRIGLYHUVLW\IRUPXOWLYHUVLRQVRIWZDUHV\VWHPV $ PHWKRG IRU XVLQJ %DOOLVWD IRU )DXOW 7ROHUDQFH WKDW GRHV QRW GLUHFWO\ LQYROYH UREXVWQHVV WHVWLQJ SHU VH LV LQ WKH DUHD RI DVVHVVLQJ LPSOHPHQWDWLRQ GLYHUVLW\ IRU PXOWLYHUVLRQ V\VWHPV 2QH ZD\ WR LPSURYH GHSHQGDELOLW\ LV WR XVH PXOWLSOH LQGHSHQGHQWO\GHYHORSHG YHUVLRQV RI VRIWZDUHLQWKHKRSHWKDWGLIIHUHQWYHUVLRQVDUHGLYHUVHLQQDWXUHDQGWKXVKDYHGLIIHULQJIDLOXUH PRGHV >@>@ 7KLV LV D ZD\ RI H[SORLWLQJ LQKHUHQW GHVLJQ GLYHUVLW\ LQ VHSDUDWHO\ GHVLJQHG V\VWHPV LQ DQ DQDORJ\ WR H[SORLWLQJ GLYHUVLW\ DPRQJ PXOWLSOH LQGHSHQGHQWO\PDQXIDFWXUHG KDUGZDUHVXEV\VWHPV 2QHLVVXHZLWKXVLQJPXOWLYHUVLRQVRIWZDUHLVZKHWKHURUQRWVXFKVRIWZDUHLVWUXO\GLYHUVH 7KHUH DUH VWXGLHV DQG DUJXPHQWV IRU ERWK VLGHV RI WKLV LVVXH HJ >@>@>@>@>@>@ +RZHYHU %DOOLVWD WHVWLQJ JLYHV D ZD\ WR VKHG DGGLWLRQDO OLJKW RQWR WKH GLYHUVLW\ TXHVWLRQ ,Q SDUWLFXODULWFDQPHDVXUHWKHGLYHUVLW\RIPXOWLSOHVRIWZDUHYHUVLRQVZLWKUHVSHFWWRH[FHSWLRQ KDQGOLQJ FDSDELOLWLHV 'LYHUVLW\ ZLWK UHVSHFW WR FRUUHFWQHVV ZRXOG KLQJH RQ LPSOHPHQWLQJ VRIWZDUHDVVXUDQFHDELOLWLHVGLVFXVVHGODWHU ,WFDQGRVRE\FRPSDULQJWKHUHVXOWVRILGHQWLFDO WHVW FDVHV RQ PXOWLSOH LPSOHPHQWDWLRQV DQG GHWHFWLQJ LQVWDQFHV LQ ZKLFK DW OHDVW RQH YHUVLRQ GHWHFWHGDQH[FHSWLRQLQGLFDWLQJWKDWDQRWLRQDOPXOWLYHUVLRQV\VWHPFRXOGKDYHGHWHFWHGVXFK DQH[FHSWLRQDQGUHVSRQGHGJUDFHIXOO\HYHQLIDOOV\VWHPVGLGQRW7KLVLVDELWGLIIHUHQWWKDQ WUDGLWLRQDO PXOWLYHUVLRQ YRWLQJLQWKDWLW VXIILFHVIRU DVLQJOH YHUVLRQ RI VRIWZDUH WRGHWHFW DQ H[FHSWLRQ ZKHUHDV YRWLQJ W\SLFDOO\ UHTXLUHV D PDMRULW\ RI YHUVLRQV WR DJUHH RQ D FRUUHFW FRPSXWDWLRQDOUHVXOW 0XOWLYHUVLRQDVVHVVPHQWRIRSHUDWLQJV\VWHPVZDVSHUIRUPHGZLWK%DOOLVWDWHVWGDWD>@DQG IRXQGWKDW26NHUQHOFDOOVDUHPRGHUDWHO\GLYHUVH EXWQRWFRPSOHWHO\GLYHUVH +RZHYHULWZDV
.RRSPDQ&6'$¶
)LJXUH2YHUDOODGMXVWHGIDLOXUHUDWHVIRUWKLUWHHQ326,;RSHUDWLQJV\VWHPVXVLQJ PXOWLYHUVLRQIDLOXUHGHWHFWLRQFRPSDULVRQ DOVR IRXQG WKDW & OLEUDU\ IXQFWLRQ FDOOV GLVSOD\ D UHODWLYHO\ SRRU OHYHO RI GLYHUVLW\ )LJXUH VKRZVDVFDWWHUSORWRIWKHUHVXOWVDWWDLQHGE\XVLQJDOOFRPELQDWLRQVRIPXOWLYHUVLRQGHWHFWLRQ DPRQJWKLUWHHQRSHUDWLQJV\VWHPVDFURVVIXQFWLRQFDOOV>@$VFDQEHVHHQWKHRYHUDOO26 LQWHUIDFHFDQQRWEHLPSURYHGEH\RQGDIDLOXUHUDWHHYHQLIDOOWKLUWHHQ26YHUVLRQVDUHXVHG IRUFRPSDULVRQ+RZHYHULIWKHV\VWHPFDOOVXEVHWLVFRQVLGHUHGLQLVRODWLRQ LHWKRVHFDOOVQRW SDUW RI WKH & OLEUDU\ WKHQ WKH IDLOXUH UDWH LV UHGXFHG GUDPDWLFDOO\ EXW QRW HOLPLQDWHG FRPSOHWHO\ YLDFRPSDULVRQ7KLVVXJJHVWVWKDWDWOHDVWZLWKUHVSHFWWRH[FHSWLRQKDQGOLQJWKH &OLEUDU\IXQFWLRQVDUHQRWSDUWLFXODUO\GLYHUVH
3RWHQWLDOJHQHUDOL]DWLRQWR6RIWZDUH$VVXUDQFH 6RIWZDUHDVVXUDQFHUHVHDUFKKDVSURGXFHGPDQ\PHWULFV6RPHRIWKHVHPHWULFVDUHEDVHGRQ PHDVXULQJWKHFRPSOH[LW\RIWKHVRIWZDUHLWVHOI6RPHRIWKHPHWULFVDUHPRUHSURFHVVEDVHG VXFKDVVRIWZDUHVKLSSLQJGHFLVLRQKHXULVWLFV$QGVRPHRIWKHDSSURDFKHVDUHQRWPHWULFEDVHG EXWUDWKHUEDVHGRQDVVXULQJFRPSOHWHFRUUHFWQHVVVXFKDVZLWKIRUPDOPHWKRGV :LWKLQ WKH UHDOP RI VRIWZDUH DVVXUDQFH %DOOLVWD WHVWLQJ LV DNLQ WR DQ DXWRPDWHG EODFN ER[ WHVWLQJDSSURDFK6LQFHEODFNER[WHVWLQJ LVDERXW IXQFWLRQDOLW\WKHTXHVWLRQ EHFRPHVRQHRI KRZPXFKIXQFWLRQDOLW\FDQEHSXWLQWRWKHWHVWLQJZKLOHUHWDLQLQJVFDODELOLW\
.RRSPDQ&6'$¶
5REXVWQHVVPHWULFIRU&276VRIWZDUHFRPSRQHQWOLEUDULHV ,QDODUJHQXPEHURIVRIWZDUHV\VWHPVH[FHSWLRQKDQGOLQJLVGHDOWZLWKLQFRPSOHWHO\LQWKH VRIWZDUH VSHFLILFDWLRQ 7KLV LV GRQH LQWHQWLRQDOO\ DQG XVXDOO\ LQGLFDWHG E\ SKUDVHV VXFK DV D EHKDYLRUEHLQJ³XQGHILQHG´RU³XQVSHFLILHG´ ,QVRPHV\VWHPVUREXVWQHVVLVQRWDQRYHUULGLQJFRQFHUQEXWLQRWKHUV DQGLQSDUWLFXODURQHV LQ ZKLFK VHFXULW\ DQG IDXOW WROHUDQFH DUH VLJQLILFDQW FRQFHUQV LW PD\ EH LPSRUWDQW ,Q WKHVH ODWWHUFDVHV%DOOLVWDWHVWLQJFDQVHUYHWRH[HUFLVHH[FHSWLRQKDQGOLQJFRGHDVDOUHDG\GHVFULEHG DQG FUHDWH DPHWULF IRU H[FHSWLRQKDQGOLQJ DELOLW\ ,W VKRXOG EH QRWHG WKDW UREXVWQHVV IDLOXUHV IRXQGE\%DOOLVWDDUHQRWQHFHVVDULO\³EXJV´RUWKHUHVXOWVRI³GHIHFWV´EHFDXVHWKRVHWHUPVFDQ EHFRQVWUXHGWREHZLWKUHVSHFWWR WKHRULJLQDOVRIWZDUHVSHFLILFDWLRQ ZKLFKPD\LQGHHGVWDWH WKDWEHKDYLRULQDSDUWLFXODUVLWXDWLRQLV³XQGHILQHG´+RZHYHULIDJHQHUDOSXUSRVHVRIWZDUH FRPSRQHQWOLEUDU\LVEHLQJXVHGLQDFULWLFDOV\VWHPLWPD\EHDGYDQWDJHRXVWRVHOHFWDOLEUDU\ ZLWKWKHEHVWUREXVWQHVVSURSHUWLHVRUDWOHDVWKDYHVRPHPHDVXUHDYDLODEOHRIWKHUREXVWQHVVRI WKHOLEUDU\VHOHFWHG $TXDOLW\DVVXUDQFHPHWULFIRUV\VWHPVZLWKULJRURXVH[FHSWLRQKDQGOLQJ 6RPH V\VWHPV DUH VSHFLILFDOO\ GHVLJQHG WR EH UREXVW $Q H[DPSOH LV WKH HDUOLHU RQH RI WKH +/$57,ZKLFKLVGHVLJQHGWRUHSRUWHUURUFRGHVIRUDQ\SRVVLEOHH[FHSWLRQ7HVWLQJUHVXOWVWR GDWHKDYHIRXQGWKDWPRVWH[FHSWLRQVDUHLQIDFWUHSRUWHGEXWPRVWVRIWZDUHPRGXOHVGRKDYHD IHZH[FHSWLRQVWKDWJRXQUHSRUWHG$QGLQDQLQWHUHVWLQJUHVXOWDIHZPRGXOHVZHUHIRXQGWR KDYHDPXFKKLJKHUIDLOXUHUDWHWKDQWKHRWKHUPRGXOHVVXJJHVWLQJDUHDVRISDUWLFXODUFRQFHUQ ZLWKUHVSHFWWRV\VWHPUREXVWQHVV 7KXVDQDGGLWLRQDOXVHRI%DOOLVWDWHVWLQJIRUVRIWZDUHDVVXUDQFHLVD4XDOLW\$VVXUDQFHFKHFN RQVRIWZDUHWKDWLVVSHFLILFDOO\ZULWWHQWRLPSOHPHQWULJRURXVH[FHSWLRQKDQGOLQJ2QHZD\WR HPSOR\WKLVWHVWLQJLVWRJHQHUDWHEXJOLVWVWKDWPXVWEHIL[HG%XWHVSHFLDOO\LQFRVWVHQVLWLYH V\VWHPV LQ FDQ EH XVHG LQVWHDG WR ILQG WKH SDUWLFXODU PRGXOHV WKDW IRU VRPH UHDVRQ KDYH IDU ZRUVHWKDQDYHUDJHUREXVWQHVVDQGFRQFHQWUDWHDYDLODEOHUHVRXUFHVRQWKHPILUVW7KXV%DOOLVWD WHVWLQJ PLJKW EH FRQVLGHUHG D VFUHHQLQJ WRRO IRU ZKLFK DUHDV WR FRQFHQWUDWH FRUUHFWLYH HIIRUWV XSRQHYHQLILWGRHVQRWILQGDOOH[LVWLQJSUREOHPV $GDSWDWLRQWREODFNER[WHVWLQJZLWKFXVWRPL]HGFRUQHUYDOXHVHWF $SRWHQWLDOO\KLJKLPSDFWEXWDOVRYHU\GLIILFXOWH[WHQVLRQRI%DOOLVWDWHVWLQJZRXOGEHWRXVH LWWRSDUWLDOO\DXWRPDWHWUDGLWLRQDOVRIWZDUHWHVWLQJ ,QWUDGLWLRQDOEODFNER[WHVWLQJHYHU\PRGXOHLVWHVWHGZLWKUHVSHFWWRDVSHFLILFDWLRQ7KLV DSSURDFKWHQGVWRVFDOHSRRUO\EHFDXVHH[WUDZRUNPXVWEHSHUIRUPHGWRGHILQHDVSHFLILFDWLRQ IRUHYHU\PRGXOHWREHWHVWHGWKXVVFDOLQJHIIRUWOLQHDUO\ZLWKWKHVL]HRIWKH$3,WREHWHVWHG 7KHREYLRXVZD\WRDWWDLQVFDODELOLW\LVWRVRPHKRZJHQHULFL]HIXQFWLRQDOLW\VRWKDWLWEHFRPHV PRUHREMHFWRULHQWHG$QGLQIDFWWKDWDSSURDFKZDVWDNHQZLWKSUH%DOOLVWDUREXVWQHVVWHVWLQJ >@EXWGLGQRWDFKLHYHVFDODELOLW\,WLVSRVVLEOHKRZHYHUWKDWLQOLJKWRIZRUNRQ%DOOLVWDWKH JHQHULFIXQFWLRQDOLW\DSSURDFKFRXOGEHPDGHWRZRUN $QRWKHU DSSURDFK LV WR OLPLW FRUUHFWQHVV WHVWLQJ WR SURSHUWLHV WKDW FDQ EH UHSUHVHQWHG DV D OLQHDUVXSHUSRVLWLRQRIWHVWFDVHVIRULQGLYLGXDOSDUDPHWHUV,QWKLVDSSURDFKFRUUHFWQHVVWHVWV ZLWKSRVWFRQGLWLRQDVVHUWLRQVZRXOGEHDGGHGWR%DOOLVWDWHVWLQJDQGWKHSRVWFRQGLWLRQVZRXOG EHWHVWHGDIWHUHDFKWHVWZDVUXQ,WLVGLIILFXOWWRVHHKRZWKLVDSSURDFKFRXOGEHPDGHWRGHDO ZLWKFKDQJHVWRV\VWHPVWDWHFDXVHGE\DIXQFWLRQEHFDXVHWKHFKDQJHVZRXOGRIQHFHVVLW\EH UHODWHGWRIXQFWLRQDOLW\%XWWKHDVVHUWLRQVFRXOGWHVWIRULQYDULDQWVDFURVVFKDQJHVVXFKDVD
.RRSPDQ&6'$¶
VWULQJUHPDLQLQJQXOOWHUPLQDWHGDIWHUDQ\RSHUDWLRQ $ VLPSOHU OHVV DPELWLRXV H[WHQVLRQ ZRXOG EH WR LQFOXGHG FXVWRPL]HG WHVW YDOXHV IRU ERXQGDU\WHVWLQJIRUDSDUWLFXODUPRGXOHRUGDWDW\SHLQDGGLWLRQWRFXUUHQW%DOOLVWDUREXVWQHVV WHVWV )RU H[DPSOH D VWUXFW FRQWDLQLQJ DQ ;@$YL]LHQLV$7KH1YHUVLRQDSSURDFKWRIDXOWWROHUDQWVRIWZDUH,(((7UDQVDFWLRQVRQ6RIWZDUH (QJLQHHULQJ6( 'HF >@ $YL]LHQLV $ /\X 0 6FKXW] : §,Q 6HDUFK RI (IIHFWLYH 'LYHUVLW\ $ 6L[/DQJXDJH 6WXG\ RI )DXOW7ROHUDQW)OLJKW&RQWURO6RIWZDUH¨7KH(LJKWHHQWK,QWHUQDWLRQDO6\PSRVLXPRQ)DXOW7ROHUDQW &RPSXWLQJS >@ %DUWRQ - &]HFN ( 6HJDOO = 6LHZLRUHN ' )DXOW LQMHFWLRQ H[SHULPHQWV XVLQJ ),$7 ,((( 7UDQVDFWLRQVRQ&RPSXWHUV $SU >@ %ULOOLDQW 66 .QLJKW -& /HYHVRQ 1* §$QDO\VLV RI )DXOWV LQ DQ 19HUVLRQ 6RIWZDUH ([SHULPHQW¨,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ )HE >@%HL]HU%%ODFN%R[7HVWLQJ1HZ@%L\DQL5 36DQWKDQDP§72)8 7HVW2SWLPL]HUIRU)XQFWLRQDO8VDJH¨6RIWZDUH(QJLQHHULQJ 7HFKQLFDO%ULHI ,%07-:DWVRQ5HVHDUFK&HQWHU >@ &DUUHWWH * &5$6+0( 5DQGRP LQSXW WHVWLQJ QR IRUPDO SXEOLFDWLRQ DYDLODEOH KWWS SHRSOHGHOSKLFRPJMFFUDVKPHKWPODFFHVVHG-XO\ >@ &KHQ / $YL]LHQLV $ §19HUVLRQ 3URJUDPPLQJ $ )DXOW 7ROHUDQFH $SSURDFK WR 5HOLDELOLW\ RI 6RIWZDUH2SHUDWLRQ¨7KH(LJKWK,QWHUQDWLRQDO6\PSRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJS
.RRSPDQ&6'$¶
> @&RKHQ''DODO6)UHGPDQ0 3DWWRQ*§7KH$(7*6\VWHP DQDSSURDFKWRWHVWLQJEDVHG RQFRPELQDWRULDOGHVLJQ¨,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ -XO >@&ULVWLDQ)ODYLX([FHSWLRQ+DQGOLQJDQG7ROHUDQFHRI6RIWZDUH)DXOWV,Q 0LFKDHO5/\X (G 6RIWZDUH)DXOW7ROHUDQFH&KLFKHVWHU :LOH\S&K >@ 'DKPDQQ - )XMLPRWR 5 :HDWKHUO\ 5 §7KH 'HSDUWPHQW RI 'HIHQVH +LJK /HYHO $UFKLWHFWXUH¨ 3URFHHGLQJV RI WKH :LQWHU 6LPXODWLRQ &RQIHUHQFH :LQWHU &RQIHUHQFH %RDUG RI 'LUHFWRUV6DQ'LHJR&$S >@ 'H9DOH - .RRSPDQ 3 §&RPSDULQJ WKH 5REXVWQHVV RI 326,; 2SHUDWLQJ 6\VWHPV¨ WK 6\PSRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJ )7&6 -XQH LQSUHVV >@ 'H9DOH - .RRSPDQ 3 *XWWHQGRUI ' §%DOOLVWD 6RIWZDUH 5REXVWQHVV 7HVWLQJ 6HUYLFH¨ 7HVWLQJ&RPSXWHU6RIWZDUHFRQIHUHQFH 7&6 -XQH LQSUHVV >@ 'LQJPDQ & 3RUWDEOH 5REXVWQHVV %HQFKPDUNV 3K' 7KHVLV (OHFWULFDO DQG &RPSXWHU (QJLQHHULQJ'HSDUWPHQW&DUQHJLH0HOORQ8QLYHUVLW\3LWWVEXUJK3$0D\ >@(FNKDUGW'( /HH/'§)XQGDPHQWDOGLIIHUHQFHVLQWKHUHOLDELOLW\RI1PRGXODUUHGXQGDQF\ DQG1YHUVLRQSURJUDPPLQJ¨-RXUQDORI6\VWHPVDQG6RIWZDUH 6HSW >@+DUW]0:DONHU( 0DKDU',QWURGXFWLRQWR6RIWZDUH5HOLDELOLW\ $6WDWHRIWKH$UW5HYLHZ 5HOLDELOLW\$QDO\VLV&HQWHUUHSRUW6:5(/5RPH1@,(((6WDQGDUGIRU,QIRUPDWLRQ7HFKQRORJ\3RUWDEOH2SHUDWLQJ6\VWHP,QWHUIDFH 326,; 3DUW 6\VWHP$SSOLFDWLRQ3URJUDP,QWHUIDFH $3, $PHQGPHQW 5HDOWLPH([WHQVLRQ>&/DQJXDJH@,((( 6WGE ,(((&RPSXWHU6RFLHW\ > @ -RQHV ( HG 7KH $SROOR /XQDU 6XUIDFH -RXUQDO $SROOR OXQDU ODQGLQJ HQWULHV DW WLPHV DQG 1DWLRQDO $HURQDXWLFV DQG 6SDFH $GPLQLVWUDWLRQ :DVKLQJWRQ '& >@ .DQDZDWL * .DQDZDWL 1 $EUDKDP - )(55$5, D WRRO IRU WKH YDOLGDWLRQ RI V\VWHP GHSHQGDELOLW\SURSHUWLHV ,(((:RUNVKRSRQ)DXOW7ROHUDQW3DUDOOHODQG'LVWULEXWHG6\VWHPV $PKHUVW0$86$S-XO\ >@ .QLJKW -& /HYHVRQ 1* §$ UHSO\ WR WKH FULWLFLVPV RI WKH .QLJKW DQG /HYHVRQ H[SHULPHQW¨ 6,*62)76RIWZDUH(QJLQHHULQJ1RWHV -DQ >@.RRSPDQ36XQJ-'LQJPDQ&6LHZLRUHN' 0DU]7&RPSDULQJ2SHUDWLQJ6\VWHPV 8VLQJ 5REXVWQHVV %HQFKPDUNV 3URFHHGLQJV 6\PSRVLXP RQ 5HOLDEOH DQG 'LVWULEXWHG 6\VWHPV 'XUKDP1&S 2FW >@ .URSS 1 .RRSPDQ 3 6LHZLRUHN ' $XWRPDWHG 5REXVWQHVV 7HVWLQJ RI 2IIWKH6KHOI 6RIWZDUH&RPSRQHQWVWK)DXOW7ROHUDQW&RPSXWLQJ6\PSRVLXPS -XQH >@/LRQV-/ FKDLUPDQ $ULDQH)OLJKW)DLOXUH UHSRUWE\WKHLQTXLU\ERDUG(XURSHDQ6SDFH $JHQF\3DULV-XO\ >@/\X05§6RIWZDUH5HOLDELOLW\0HDVXUHPHQWVLQ19HUVLRQ6RIWZDUH([HFXWLRQ(QYLURQPHQW¨ 3URFHHGLQJVRIWKH7KLUG,QWHUQDWLRQDO6\PSRVLXPRQ6RIWZDUH5HOLDELOLW\DQG(QJLQHHULQJS >@ /\X 0 HG +DQGERRN RI 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ ,((( &RPSXWHU 6RFLHW\ 3UHVV 0F*UDZ+LOO >@0F&DEH7-6WUXFWXUHGWHVWLQJ $VRIWZDUHWHVWLQJPHWKRGRORJ\XVLQJWKHF\FORPDWLFFRPSOH[LW\ PHWULF1DW%XU6WDQG:DVKLQJWRQ'&86$'HF >@0F$OOLVWHU')6XQ&9RXN0$§5HOLDELOLW\RI9RWLQJLQ)DXOW7ROHUDQW6RIWZDUH6\VWHPV
.RRSPDQ&6'$¶
IRU6PDOO2XWSXW6SDFHV¨,(((7UDQVDFWLRQVRQ5HOLDELOLW\ 'HF > @ 0LOOHU %3 )UHGULNVHQ / 6R % §$Q HPSLULFDO VWXG\ RI WKH UHOLDELOLW\ RI 8QL[ XWLOLWLHV &RPPXQLFDWLRQVRIWKH$&0 'HF >@0LOOHU%.RVNL'/HH&0DJDQW\90XUWK\51DWDUDMDQ$ 6WHLGO-)X]]5HYLVLWHG $ 5HH[DPLQDWLRQ RI WKH 5HOLDELOLW\ RI 81,; 8WLOLWLHV DQG 6HUYLFHV &RPSXWHU 6FLHQFH 7HFKQLFDO 5HSRUW8QLYRI:LVFRQVLQ0DGLVRQ0D\ >@ 0XNKHUMHH $ 6LHZLRUHN '3 §0HDVXULQJ VRIWZDUH GHSHQGDELOLW\ E\ UREXVWQHVV EHQFKPDUNLQJ,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ -XQH >@1XPHJD%RXQGV&KHFNHUKWWS ZZZQXPHJDFRPSURGXFWVDHGYFVKWPODFFHVVHG >@ 2VWUDQG 7- %DOFHU 0- §7KH FDWHJRU\SDUWLWLRQ PHWKRG IRU VSHFLI\LQJ DQG JHQHUDWLQJ IXQFWLRQDOWHVWV¨&RPPXQLFDWLRQVRIWKH$&0 -XQH >@3XULI\KWWS ZZZSXUHDWULDFRPSURGXFWVSXULI\LQGH[KWPODFFHVVHG >@5DP3 5DQG'.§6DWDQ GRXEOHHGJHGVZRUG¨&RPSXWHU -XQH >@6SDQEDXHU6§3HQWLXP%XJ0HHWWKH,()ODZ¨3&:RUOGS)HE >@ 7VDL 7 5 ,\HU 0HDVXULQJ )DXOW 7ROHUDQFH ZLWK WKH )7$3( )DXOW ,QMHFWLRQ 7RRO 3URFHHGLQJV (LJKWK ,QWHUQDWLRQDO &RQIHUHQFH RQ 0RGHOLQJ 7HFKQLTXHV DQG 7RROV IRU &RPSXWHU 3HUIRUPDQFH(YDOXDWLRQ+HLGHOEHUJ*HUPDQ\6SULQJHU9HUODJS6HSW >@ 9R .3 :DQJ
Research Showcase @ CMU Institute for Software Research
School of Computer Science
1998
Toward a Scalable Method for Quantifying Aspects of Fault Tolerance, Software Assurance, and Computer Security Philip Koopman Carnegie Mellon University
Follow this and additional works at: http://repository.cmu.edu/isr
This Conference Proceeding is brought to you for free and open access by the School of Computer Science at Research Showcase @ CMU. It has been accepted for inclusion in Institute for Software Research by an authorized administrator of Research Showcase @ CMU. For more information, please contact [email protected].
.RRSPDQ&6'$¶
,(((3RVWSURFHHGLQJVRIWKH&RPSXWHU6HFXULW\'HSHQGDELOLW\DQG$VVXUDQFH )URP 1HHGVWR6ROXWLRQV &6'$ 1RYHPEHU :DVKLQJWRQ'&
7RZDUGD6FDODEOH0HWKRGIRU4XDQWLI\LQJ$VSHFWVRI )DXOW7ROHUDQFH6RIWZDUH$VVXUDQFHDQG&RPSXWHU6HFXULW\ 3KLOLS.RRSPDQ 'HSDUWPHQWRI(OHFWULFDODQG&RPSXWHU(QJLQHHULQJ ,QVWLWXWHIRU&RPSOH[(QJLQHHUHG6\VWHPV &DUQHJLH0HOORQ8QLYHUVLW\3LWWVEXUJK3$ NRRSPDQ#FPXHGX $EVWUDFW 4XDQWLWDWLYH DVVHVVPHQW WRROV DUH XUJHQWO\ QHHGHG LQ WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUHDVVXUDQFHDQGFRPSXWHUVHFXULW\$VVHVVPHQWPHWKRGVW\SLFDOO\HPSOR\HGLQ YDULRXV FRPELQDWLRQV DUH IDXOW LQMHFWLRQ IRUPDO YHULILFDWLRQ DQG WHVWLQJ +RZHYHU WKHVHPHWKRGVDUHH[SHQVLYHEHFDXVHWKH\DUHODERULQWHQVLYHZLWKFRVWVVFDOLQJDWOHDVW OLQHDUO\ZLWKWKHQXPEHURIVRIWZDUHPRGXOHVWHVWHG$GGLWLRQDOO\WKH\DUHVXEMHFWWR KXPDQODSVHVDQGRYHUVLJKWVEHFDXVHWKH\UHTXLUHWZRGLIIHUHQWUHSUHVHQWDWLRQVIRUHDFK V\VWHPDQGWKHQEDVHUHVXOWVRQDGLUHFWRUDQLQGLUHFWUHSUHVHQWDWLRQFRPSDULVRQ 7KH%DOOLVWDSURMHFWKDVIRXQGWKDWUREXVWQHVVWHVWLQJIRUPVDQLFKHLQZKLFKVFDODEOH TXDQWLWDWLYH DVVHVVPHQWFDQ EH DFKLHYHGDWORZFRVW 7KLV VFDODELOLW\ VWHPVIURP WZR WHFKQLTXHV DVVRFLDWLQJ VWDWHVHWWLQJ LQIRUPDWLRQ ZLWK WHVW FDVHV EDVHG RQ GDWD W\SHV DQGXVLQJRQHJHQHULFEXWQDUURZEHKDYLRUDOVSHFLILFDWLRQIRUDOOPRGXOHV*LYHQWKDW WKLVDSSURDFKKDVVXFFHHGHGLQFRPSDULQJWKHUREXVWQHVVRIYDULRXVRSHUDWLQJV\VWHPV LWLVQDWXUDOWRDVNLILWFDQEHPDGHPRUHJHQHUDOO\DSSOLFDEOH ,WDSSHDUVWKDW%DOOLVWDOLNHWHVWLQJFDQEHXVHGLQWKHIDXOWWROHUDQFHDUHDWRPHDVXUH WKHJHQHULFUREXVWQHVVRIDYDULHW\RI$3,LPSOHPHQWDWLRQVDQGLQSDUWLFXODUWRLGHQWLI\ UHSURGXFLEOHZD\VWRFUDVKDQGKDQJVRIWZDUH,QVRIWZDUHDVVXUDQFHLWFDQEHXVHGDV DTXDOLW\FKHFNRQH[FHSWLRQKDQGOLQJDQGLQSDUWLFXODUDVDPHDQVWRDXJPHQWEODFN ER[WHVWLQJ$SSO\LQJLWWRFRPSXWHUVHFXULW\DSSHDUVPRUHSUREOHPDWLFEXWPLJKWEH SRVVLEOHLIWKHUHLVDZD\WRRUWKRJRQDOO\GHFRPSRVHYDULRXVDVSHFWVRIVHFXULW\UHOHYDQW V\VWHPVWDWHLQWRDQDORJVRI%DOOLVWDGDWDW\SHV :KLOH %DOOLVWDOLNH WHVWLQJ LV QR VXEVWLWXWH IRU WUDGLWLRQDO PHWKRGV LW FDQ VHUYH WR SURYLGH D XVHIXO TXDOLW\ DVVXUDQFHFKHFN WKDW DXJPHQWV H[LVWLQJ SUDFWLFH DW UHODWLYHO\ ORZFRVW$OWHUQDWHO\LWFDQVHUYHWRTXDQWLI\WKHH[WHQWRISRWHQWLDOSUREOHPVHQDEOLQJ EHWWHULQIRUPHGGHFLVLRQVE\ERWKGHYHORSHUVDQGFXVWRPHUV :KHQ\RXFDQPHDVXUHZKDW\RXDUHVSHDNLQJDERXWDQGH[SUHVV LWLQQXPEHUV\RXNQRZVRPHWKLQJDERXWLW EXWZKHQ\RXFDQQRW PHDVXUH LW ZKHQ \RX FDQ QRW H[SUHVV LW LQ QXPEHUV \RXU NQRZOHGJH LV RI D PHDJHU DQG XQVDWLVIDFWRU\ NLQG LW PD\ EH WKH EHJLQQLQJ RI NQRZOHGJH EXW \RX KDYH VFDUFHO\ LQ \RXU WKRXJKW DGYDQFHGWRWKHVWDJHRIVFLHQFH ²/RUG.HOYLQ
.RRSPDQ&6'$¶
,QWURGXFWLRQ 2QH RI WKH PRUH GLIILFXOW DVSHFWV RI ZRUNLQJ LQ WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUH DVVXUDQFH DQG FRPSXWHU VHFXULW\ LV TXDQWLI\LQJ UHVXOWV 4XDQWLILFDWLRQ LV DEVROXWHO\ HVVHQWLDO DWWDLQDPDWXUHFDSDELOLW\IRUFUHDWLQJV\VWHPVRIDQ\NLQG$QG\HWGHVSLWHVLJQLILFDQWUHVHDUFK HIIRUWV WKH DWWDLQPHQW RI TXDQWLILFDWLRQ LV SURYLQJ HOXVLYH LQ WKH DUHD RI VRIWZDUHLQWHQVLYH V\VWHPV:KLOHWKHUHLVQRGRXEWWKDWSURJUHVVKDVEHHQPDGHLQWKHDUHDRIJHQHUDOVRIWZDUH GHYHORSPHQW PHWKRGRORJ\ WKH IRFXV KDV EHHQ RQ WKH SULPDU\ DWWULEXWHV RI SURGXFWLYLW\ DQG VRIWZDUH FRUUHFWQHVV /HVV DWWHQWLRQ KDV EHHQ SDLG WR ZKDW DUH RIWHQ FRQVLGHUHG VHFRQGDU\ DWWULEXWHVVXFKDVIDXOWWROHUDQFHVRIWZDUHVDIHW\DQGVHFXULW\7KHVHWKUHHDUHDVVHHPWRVKDUH DFRPPRQSUREOHPLQWKDWLWLVGLIILFXOWWRDWWDLQTXDQWLILFDWLRQODUJHO\EHFDXVHRIWKHLQWHUDFWLRQ RIIRXUSUREOHPV )LUVWWKHSURSHUWLHVRIVRIWZDUHLQWHQVLYHV\VWHPVWKDWQHHGWREHPHDVXUHGDUHW\SLFDOO\YHU\ GHSHQGHQW RQ WKH GHWDLOV RI WKH V\VWHP VSHFLILFDWLRQ RU XVDJH ([DPSOHV LQFOXGH IXQFWLRQDO GHIHFWUDWHVZKHWKHUDV\VWHPLV³VDIH´DQGZKHWKHUDV\VWHPLV³VHFXUH´7KLVPHDQVWKDWDQ\ PHDVXUHPHQW WRRO PXVW EH SURYLGHG ZLWK D ZHDOWK RI LQIRUPDWLRQ DERXW WKH V\VWHP DQG LQ JHQHUDOWKLVUHTXLUHVDFFHVVWRDVXEVWDQWLDOSRUWLRQRIWKHV\VWHPVSHFLILFDWLRQ&RPSOLFDWLQJ WKH VLWXDWLRQ LV WKDW WKH VSHFLILFDWLRQ RU LQGHHG HYHQ WKH GHWDLOHG LPSOHPHQWDWLRQ PD\ EH XQDYDLODEOHIRUVRIWZDUHFRPSRQHQWVSXUFKDVHGRIIWKHVKHOI2EWDLQLQJVXFKLQIRUPDWLRQFRVWV WLPH DQG PRQH\ 7R LOOXVWUDWH KRZ GLIILFXOW WKLV PDNHV WKLQJV FRPSDUH WKH GLIILFXOW\ RI GHWHUPLQLQJ ZKHWKHU D FRPSXWHU V\VWHP LV VHFXUH WR GHWHUPLQLQJ LWV VSHHG DW H[HFXWLQJ D SUHSDFNDJHGEHQFKPDUNVXLWH 6HFRQGWKHGHVLUHGPHDVXUHPHQWRXWFRPHLVRIWHQGHVLUHGWREHSHUIHFWLRQ DWWDLQLQJD]HUR GHIHFW UDWH FRPSOHWHO\ VDIH RU QHYHU IDLOV (YHQ WKRXJK VXFK SHUIHFWLRQ LV JHQHUDOO\ XQDWWDLQDEOH LQ SUDFWLFH LW PHDQV WKDW WHVWV DUH IRFXVVHG RQ ILQGLQJ GHYLDWLRQV IURP D VSHFLILFDWLRQRIVRPHVRUW7KLVLVLQVKDUSFRQWUDVWWRPRUHWUDGLWLRQDOFRPSXWHUV\VWHPPHWULFV VXFKDVFRPSXWDWLRQVSHHGRUDFKLHYHGQHWZRUNEDQGZLGWK,WDOVRUDLVHVLVVXHVLIDVLVRIWHQ DOZD\V" WKHFDVHWKHVSHFLILFDWLRQLWVHOILVLPSHUIHFW 7KLUGLWLVFRPPRQIRUWKHVDPHWHVWVWKDWDUHXVHGWRPHDVXUHWKHV\VWHPWREHXVHGGLUHFWO\ WRIL[DQ\VKRUWFRPLQJVIRXQG)RUH[DPSOHDSURMHFWPHDVXULQJEXJUHSRUWVSHUZHHNRIWHVWLQJ ZRXOGQDWXUDOO\IHHGDOOEXJUHSRUWVWRWKHPDLQWHQDQFHJURXSIRUFRUUHFWLRQ7KXVLWLVFRPPRQ IRU WKH PHDVXUHPHQW SURFHVV LWVHOI WR EH FRXSOHG WR FKDQJHV LQ WKH V\VWHP EHLQJ PHDVXUHG :KLOH LW LV FOHDU WKDW WUDGLWLRQDO VSHHG PHDVXUHPHQWV UHVXOW LQ V\VWHPV EHLQJ RSWLPL]HG WR SHUIRUP EHWWHU WKH VLWXDWLRQ LV GLIIHUHQW KHUH LQ WKDW WKHUH LV QR IXQGDPHQWDO TXDQWLW\ WR EH PHDVXUHG VXFKDVDSDUWLFXODUZRUNORDGWRFRPSOHWH 5DWKHUGHYLDWLRQVIURPSHUIHFWLRQDUH EHLQJERWKPHDVXUHGDQGFRUUHFWHGE\WKHVDPHSURFHVV )RXUWKDQGSHUKDSVPRVWLPSRUWDQWIURPDSUDFWLFDOLPSOHPHQWDWLRQSRLQWRIYLHZLVWKDWWKH SURFHVVRIPHDVXULQJ³EDGWKLQJV´VXFKDVEXJVLVRIWHQVHHQDVDGHVWUXFWLYHDFWDVRSSRVHGWR PHDVXULQJ ³JRRG WKLQJV´ VXFK DV SHUIRUPDQFH 7KH RXWFRPH RI D PHWULF YDOXH RI OHVV WKDQ SHUIHFWLRQFDQEHVHHQDVDODEHOLPSO\LQJGHIHFWLYHJRRGVZKHUHDVWKHRXWFRPHRIDSRVLWLYH PHWULFVXFKDVVSHHGLVPHUHO\DPHDVXUHPHQWRIDQLQKHUHQWSURSHUW\ 'HVSLWHWKHVHSUREOHPVLWLVLPSHUDWLYHWKDWZHILQGZD\VWRTXDQWLI\SURSHUWLHVRIV\VWHPV WKDW ZH ZLVK WR XQGHUVWDQG DQG WR LPSURYH RYHU WLPH 2I FRXUVH WKHUH LV DOZD\V WKH ULVN RI LQYHQWLQJQXPEHUVMXVWIRUWKHVDNHRIKDYLQJQXPEHUVVRLWLVDOVRLPSRUWDQWWRILQGPHWULFVWKDW DUH DW OHDVW FRUUHODWHG WR DFWXDO GHVLUDEOH SURSHUWLHV 7KXV LW ZRXOG EH GHVLUDEOH WR KDYH D PHDVXUHPHQW WRRO RU PHWKRGRORJ\ WKDW GRHV QRW UHTXLUH PXFK GHWDLOHG LQIRUPDWLRQ DERXW WKH V\VWHP LWLV PHDVXULQJGRHV QRW DWWHPSW WR PHDVXUH TXDQWLWLHV WKDW GHYLDWH YHU\ VOLJKWO\ IURP SHUIHFWLRQXVHVQRYHOWHVWVIRUHDFKPHDVXUHPHQWRURWKHUZLVHGLVHQWDQJOHVPHDVXUHPHQWIURP GHIHFWFRUUHFWLRQDQGLVVRPHKRZVHHQDVDFRQVWUXFWLYHPHDVXUHPHQWUDWKHUWKDQDGHVWUXFWLYH
.RRSPDQ&6'$¶
DFWLYLW\ 7KHZRUNGLVFXVVHGKHUHLQGRHVQRWVDWLVI\DOOWKHFULWHULDIRUSURYLGLQJKLJKTXDOLW\QXPEHUV WKDWZLOOVWDQGWKHWHVWRIWLPH7KXVZHVWLOOVHHNWKHWUXHQXPEHUVWKDW/RUG.HOYLQH[KRUWVXV WRXVH+RZHYHURQWKHSULQFLSOHWKDWSURJUHVVLQPHDVXUHPHQWLVDJHQHUDOO\ZRUWK\JRDORXU DSSURDFKPLJKWEHFRQVLGHUHGDVWHSLQWKHULJKWGLUHFWLRQ,QSDUWLFXODUWKLVZRUNGHPRQVWUDWHV WKDWLWLVSRVVLEOHLQVRPHFDVHVWRDXWRPDWHWHVWLQJDODUJHDPRXQWRIVRIWZDUHZLWKRXWVLPSO\ WXUQLQJDODUJHWHVWLQJHIIRUWLQWRDVLPLODUO\ODUJHH[HUFLVHLQSURYLGLQJGHWDLOHGLQIRUPDWLRQWR WKHDXWRPDWHGWHVWLQJV\VWHP *HQHUDOH[LVWLQJWHFKQLTXHVIRUVRIWZDUHPHDVXUHPHQW ,WLVRIFRXUVHFXUUHQWO\SRVVLEOHWRPHDVXUHPDQ\SURSHUWLHVRIVRIWZDUH7KHPRVWPDWXUH RIWKHVHDSSURDFKHVZHUHERUQRIWKHVRFDOOHGVRIWZDUHFULVLVWKDWKDVVRPHKRZPDQDJHGWRODVW DWWKLVSRLQWVHYHUDOGHFDGHV7KHVHPHWULFVDWWHPSWWRPHDVXUHSURGXFWLYLW\DQGWKXVHYDOXDWH ZD\VWRLPSURYHWKDWSURGXFWLYLW\7KH\LQFOXGHWKHYHQHUDEOHVRXUFHOLQHVRIFRGH 6/2& WKH ZLGHO\ XVHG F\FORPDWLF FRPSOH[LW\ PHWULFV >@ DQG RWKHU PRUH DGYDQFHG PHWULFV HJ DV GLVFXVVHGLQ>@ :KLOHWKHVHPHWULFVYDULRXVO\XVHPHDVXUHPHQWVRIWKHVRIWZDUHLWVHOIWKH VRIWZDUHLVQRWEHLQJHYDOXDWHGIRULQKHUHQWXWLOLW\,QVWHDGVRIWZDUHLVPHDVXUHGWRGHWHUPLQH FRPSOH[LW\ ZKLFK LV WKHQ XVHG WR HVWLPDWH GHYHORSPHQW DQG PDLQWHQDQFH FRVWV DQG SHUKDSV LQGLUHFWO\WRJHQHUDWHDQHVWLPDWHGGHIHFWUDWH 0RUHUHFHQWO\DZLGHYDULHW\RIPHWULFVDQGPHDVXUHPHQWPHWKRGRORJLHVKDYHEHHQSURSRVHG IRU WKH VRIWZDUH UHOLDELOLW\ DUHD 7KHVH PHWULFV WHQG WR XVH YDU\LQJ FRPELQDWLRQV RI VRIWZDUH FRPSOH[LW\GHYHORSPHQWSKDVHGHIHFWGLVFRYHU\WHVWLQJSKDVHGHIHFWGLVFRYHU\DQGLQVHUYLFH GHIHFWGLVFRYHU\WRPDNHSUHGLFWLRQVDERXWUHVLGXDOGHIHFWUDWHVIRUQHZDQGH[LVWLQJVRIWZDUH V\VWHPV DEURDGGLVFXVVLRQRIVXFKDSSURDFKHVFDQEHIRXQGLQ>@DQG>@ :KLOHVRIWZDUHUHOLDELOLW\PHWULFVDUHGHILQLWHO\DSURGXFWLYHVWHSLQWKHULJKWGLUHFWLRQWKHLU XVHLVQRWZLWKRXWSUREOHPV7HVWLQJEDVHGDSSURDFKHVDPRXQWWRUHTXLULQJDVRIWZDUHWHVWVXLWH GHYHORSPHQWHIIRUWSURSRUWLRQDOWRGHYHORSLQJWKHRULJLQDOVRIWZDUHLQWKHILUVWSODFHDQGWKXV DUHH[SHQVLYH:KLOHLQPRVWFDVHVWHVWLQJRIQRUPDOIXQFWLRQDOLW\LVVHHQDVQHFHVVDU\WKHUHLV DSUREOHPLQWKDWH[FHSWLRQKDQGOLQJFRGHDQGRWKHUSRUWLRQVRIWKHV\VWHPFDQRYHUVKDGRZWKH EDVLF IXQFWLRQDOLW\ LQ WHUPV RI WHVWLQJ FRPSOH[LW\ >@ 7KLV PHDQV WKDW LQ DOO EXW WKH PRVW H[WUHPHVDIHW\FULWLFDOFDVHVOLPLWHGWLPHEXGJHWDQGPDQSRZHURIWHQUHVXOWLQWHVWLQJHIIRUW EHLQJ FRQFHQWUDWHG RQ ³QRUPDO´ IXQFWLRQDOLW\ JLYLQJ VKRUW VKULIW WR GHSHQGDELOLW\ DVSHFWV ,Q RWKHUZRUGVUHOLDELOLW\WHVWLQJFDQIDOOYLFWLPWRWKHSUREOHPVRIFRVWDQGQRWEHLQJVHHQDVD SULPDU\YDOXHDGGHGEXVLQHVVDFWLYLW\EH\RQGDFHUWDLQSRLQW $QDOWHUQDWHDSSURDFKWKDWLVRIWHQFRPELQHGZLWKDWHVWLQJEDVHGDSSURDFKLVRQHEDVHGRQ PRQLWRULQJGHIHFWGLVFRYHU\UDWHV,IRQHDVVXPHVWKDWGHIHFWVDUHGLVFRYHUHGDWORZPDUJLQDO FRVW HJ WKURXJK DQ H[WHQVLYH EHWD WHVWLQJ SURJUDP RU D PDQGDWHG IXQFWLRQDO FRUUHFWQHVV WHVWLQJSURJUDP WKHQVXFKPRQLWRULQJEHFRPHVDUHDVRQDEO\ORZFRVWH[HUFLVH+RZHYHUWKLV DSSURDFK GRHV VXIIHU IURP WKH SRWHQWLDO SUREOHP RI GHOD\LQJ PHDVXUHPHQW IHHGEDFN XQWLO GRZQVWUHDP LQ WKH GHYHORSPHQW F\FOH ZKHQ SUREOHPV DUH SRWHQWLDOO\ PRUH H[SHQVLYH WR IL[ $GGLWLRQDOO\GHIHFWUDWHPRQLWRULQJKDVWKHSUREOHPWKDW SUHVXPDEO\ WKHLPSRUWDQWGHIHFWVDUH IL[HGDOWHULQJWKHV\VWHPXQGHUWHVWRYHUWLPH 5REXVWQHVVWHVWLQJDVDQH[DPSOHRISRWHQWLDOO\JHQHUDODSSURDFK 7KLVSDSHUGHVFULEHVWKHDWWDLQPHQWRIDUHODWLYHO\JHQHULFORZFRVWVFDODEOHSRUWDEOHWHVWLQJ DQG PHDVXUHPHQW PHWKRGRORJ\ IRU UREXVWQHVV ,Q SDUWLFXODU WKH PHWKRGRORJ\ PHDVXUHV WKH UREXVWQHVV RI DQ $SSOLFDWLRQ 3URJUDPPLQJ ,QWHUIDFH $3, ZLWK UHVSHFW WR H[FHSWLRQDO
.RRSPDQ&6'$¶
SDUDPHWHUYDOXHV7KHFXUUHQWUHVXOWVKDYHLPSOLFDWLRQVIRUIDXOWWROHUDQFHVRIWZDUHDVVXUDQFH DQGFRPSXWHUVHFXULW\DQGWKHPHWKRGRORJ\PLJKWEHH[WHQGHGIXUWKHU 5REXVWQHVV ZDV VHOHFWHG IRU PHDVXUHPHQW EHFDXVH V\VWHP FUDVKHV DUH D ZD\ RI OLIH LQ DQ\ UHDOZRUOG V\VWHP QR PDWWHU KRZ FDUHIXOO\ GHVLJQHG 6RIWZDUH LV LQFUHDVLQJO\ EHFRPLQJ WKH VRXUFH RI V\VWHP IDLOXUHV DQG WKH PDMRULW\ RI VRIWZDUH IDLOXUHV LQ SUDFWLFH VHHP WR EH GXH WR SUREOHPV ZLWK UREXVWQHVV >@ 7KLUW\ \HDUV DJR WKH $SROOR PLVVLRQ H[SHULHQFHG WKUHH FRPSXWHU FUDVKHV DQG UHERRWV GXULQJ GHVFHQW WR OXQDU ODQGLQJ FDXVHG E\ H[FHSWLRQDO UDGDU FRQILJXUDWLRQVHWWLQJVWKDWUHVXOWHGLQWKHV\VWHPUXQQLQJRXWRIPHPRU\EXIIHUV> @'HFDGHV ODWHUWKHPDLGHQIOLJKWRIWKH$ULDQHKHDY\OLIWLQJURFNHWZDVORVWGXHWRHYHQWVDULVLQJIURPD IORDWLQJ SRLQWWRLQWHJHU FRQYHUVLRQ H[FHSWLRQ >@ 7RGD\ H[FHSWLRQDO FRQGLWLRQV URXWLQHO\ FDXVHV\VWHPIDLOXUHVLQWHOHFRPPXQLFDWLRQV\VWHPVGHVNWRSFRPSXWHUVDQGHOVHZKHUH*LYHQ WKDW VXFK SUREOHPV KDYH SHUVLVWHG LQ PLVVLRQFULWLFDO VSDFH FRPSXWHUV IRU GHFDGHV LW LV QRW GLIILFXOW WR SUHGLFW WKH\ ZLOO VWLOO EH ZLWK XV VHYHUDO GHFDGHV KHQFH LQ HYHU\GD\ FRPSXWLQJ +RZHYHU WKH SUREOHP WKHQ ZLOO EH WKDW HYHU\GD\ FRPSXWLQJ ZLOO KDYH FRPSOHWHG LWV FKDQJH IURP OX[XU\ WR QHFHVVLW\ PDNLQJ VXFK IDLOXUHV SRWHQWLDOO\ GHYDVWDWLQJ LQVWHDG RI PHUHO\ LQFRQYHQLHQWRUH[SHQVLYH 7KXV WKH %DOOLVWD SURMHFW ZDV FUHDWHG WR PHDVXUH WKH UREXVWQHVV RI V\VWHPV %HFDXVH RSHUDWLQJV\VWHP 26 VRIWZDUHXQGHUOLHVPRVWODUJHV\VWHPVWKHILUVWDSSOLFDWLRQDUHDWKDWZDV FKRVHQ ZDV WR SURGXFH TXDQWLILHG UHVXOWV IRU UREXVWQHVV WHVWLQJ RI IXOOVFDOH RIIWKHVKHOI RSHUDWLQJV\VWHPV$XWRPDWHGWHVWLQJZDVSHUIRUPHGRQILIWHHQ326,;>@RSHUDWLQJV\VWHP YHUVLRQV IURP WHQ GLIIHUHQW YHQGRUV DFURVV D YDULHW\ RI KDUGZDUH SODWIRUPV 0RUH WKDQ RQH PLOOLRQWHVWVZHUHH[HFXWHGLQDOOFRYHULQJXSWRGLVWLQFWIXQFWLRQVDQGV\VWHPFDOOVIRUHDFK 260DQ\RIWKHWHVWVUHVXOWHGLQUREXVWQHVVIDLOXUHVUDQJLQJLQVHYHULW\IURPFRPSOHWHV\VWHP FUDVKHVWRIDOVHLQGLFDWLRQRIVXFFHVVIXORSHUDWLRQLQXQVSHFLILHGFLUFXPVWDQFHV ,Q EULHI WKH %DOOLVWD WHVWLQJ PHWKRGRORJ\ LQYROYHV DXWRPDWLFDOO\ JHQHUDWLQJ VHWV RI H[FHSWLRQDOSDUDPHWHUYDOXHVWREHXVHGLQFDOOLQJVRIWZDUHPRGXOHV7KHUHVXOWVRIWKHVHFDOOV DUHH[DPLQHGWRGHWHUPLQHZKHWKHUWKHPRGXOHGHWHFWHGDQGQRWLILHGWKHFDOOLQJSURJUDPRIDQ HUURU DQG ZKHWKHU WKH WDVN RU HYHQ V\VWHP VXIIHUHG D FUDVK RU KDQJ 7KHUH DUH WZR NH\ DSSURDFKHV WKDW PDNH WKLV DSSURDFK LQH[SHQVLYH DQG VFDODEOH WKH XVH RI DQ REMHFWRULHQWHG WHVWLQJ DSSURDFK EDVHG RQ GDWD W\SHV UDWKHU WKDQ IXQFWLRQDOLW\ DQG WKH IDFW WKDW WHVWLQJ LV SHUIRUPHG IRU D UHODWLYHO\ VLPSOH SURSHUW\ RI UREXVWQHVV UDWKHU WKDQ IRU WKH PRUH FRPSOH[ SURSHUW\RIFRUUHFWQHVV ,Q WKH IROORZLQJ VHFWLRQV ZH VKDOO GLVFXVV SUHYLRXV ZRUN LQ WKLV DUHD WKH %DOOLVWD WHVWLQJ PHWKRGRORJ\ DQG SRWHQWLDO DSSOLFDWLRQV RI WKDW PHWKRGRORJ\ WR WKH DUHDV RI IDXOW WROHUDQFH VRIWZDUHDVVXUDQFHDQGFRPSXWHUVHFXULW\:KLOHSDUWRIWKHVWRU\LVVLPSO\WKHDSSOLFDWLRQRI H[LVWLQJ%DOOLVWDUHVXOWVDQGPHWKRGRORJLHVWRWKHVHDUHDVWKHUHVWRIWKHGLVFXVVLRQZLOOEHPRUH VSHFXODWLYHDQGK\SRWKHVL]HZD\VLQZKLFKWKHDSSURDFKPLJKWEHJHQHUDOL]HGDQGH[WHQGHG
3UHYLRXVZRUN :KLOH WKH %DOOLVWD UREXVWQHVV WHVWLQJ PHWKRG GHVFULEHG LQ WKLV SDSHU LV D IRUP RI VRIWZDUH WHVWLQJLWVKHULWDJHWUDFHVEDFNQRWRQO\WRWKHVRIWZDUHWHVWLQJFRPPXQLW\EXWDOVRWRWKHIDXOW WROHUDQFH FRPPXQLW\ DV D IRUP RI VRIWZDUHEDVHG IDXOW LQMHFWLRQ ,Q VRIWZDUH WHVWLQJ WHUPV %DOOLVWDSHUIRUPVWHVWVIRUUHVSRQVHVWRH[FHSWLRQDOLQSXWFRQGLWLRQV VRPHWLPHVFDOOHG³GLUW\´ WHVWVZKLFKLQYROYHH[FHSWLRQDOVLWXDWLRQVDVRSSRVHGWR³FOHDQ´WHVWVRIFRUUHFWIXQFWLRQDOLW\LQ QRUPDO VLWXDWLRQV 7KH WHVW LGHDV XVHG DUH EDVHG RQ ³EODFN ER[´ RU IXQFWLRQDO WHVWLQJ WHFKQLTXHV>%H]LHU @LQZKLFKRQO\IXQFWLRQDOLW\LVRIFRQFHUQQRWWKHDFWXDOVWUXFWXUHRIWKH VRXUFHFRGH+RZHYHU%DOOLVWDLQLWVFXUUHQWIRUPLVQRWFRQFHUQHGZLWKYDOLGDWLQJIXQFWLRQDOLW\ IRURUGLQDU\RSHUDWLQJFRQGLWLRQVEXWUDWKHUGHWHUPLQLQJ ZKHWKHURUQRWDVRIWZDUHPRGXOHLV
.RRSPDQ&6'$¶
UREXVW ,Q FRPPRQ XVDJH WKH WHUP UREXVWQHVV PLJKW UHIHU WR WKH WLPH EHWZHHQ RSHUDWLQJ V\VWHP FUDVKHVXQGHUVRPHXVDJHSURILOH+RZHYHUWKHDXWKRULWDWLYHGHILQLWLRQRIUREXVWQHVVWKDWZLOO EHXVHGLQWKHFXUUHQWGLVFXVVLRQLV³WKHGHJUHHWRZKLFKDV\VWHPRUFRPSRQHQWFDQIXQFWLRQ FRUUHFWO\ LQ WKH SUHVHQFH RI LQYDOLG LQSXWV RU VWUHVVIXO HQYLURQPHQWDO FRQGLWLRQV´ >@ 7KLV H[SDQGVWKHQRWLRQRIUREXVWQHVVWREHPRUHWKDQFDWDVWURSKLFV\VWHPFUDVKHVDQGHQFRPSDVVHV VLWXDWLRQVLQZKLFKVPDOOUHFRYHUDEOHIDLOXUHVPLJKWDOVRRFFXU%DOOLVWDZRUNFRQFHQWUDWHVRQ WKH SRUWLRQ RI UREXVWQHVV GHDOLQJ ZLWK LQYDOLG LQSXWV :KLOH UREXVWQHVV XQGHU VWUHVVIXO HQYLURQPHQWDOFRQGLWLRQVLVLQGHHGDQLPSRUWDQWLVVXHDGHVLUHWRDWWDLQKLJKO\UHSHDWDEOHUHVXOWV KDVOHGWKH%DOOLVWDSURMHFWWRFRQVLGHURQO\UREXVWQHVVLVVXHVGHDOLQJZLWKDVLQJOHLQYRFDWLRQRI DVRIWZDUHPRGXOHIURPDVLQJOHH[HFXWLRQWKUHDG KRZHYHUDVZLOOEHGLVFXVVHGWKLVLVIDUOHVV OLPLWLQJDQDSSURDFKWKDQRQHPLJKWWKLQN &UDVKPH $QHDUO\DQGZHOONQRZQPHWKRGIRUDXWRPDWLFDOO\WHVWLQJRSHUDWLQJV\VWHPVIRUUREXVWQHVV ZDVWKHGHYHORSPHQWRIWKH&UDVKPHSURJUDP>@&UDVKPHRSHUDWHVE\ZULWLQJUDQGRPL]HGGDWD YDOXHVWRPHPRU\WKHQVSDZQLQJODUJHQXPEHUVRIWDVNVWKDWDWWHPSWHGWRH[HFXWHWKRVHUDQGRP E\WHVDVSURJUDPV:KLOHPDQ\ WDVNVWHUPLQDWHDOPRVWLPPHGLDWHO\GXHWRLOOHJDOLQVWUXFWLRQ H[FHSWLRQVRQRFFDVLRQDVLQJOHWDVNRUDFRQIOXHQFHRIPXOWLSOHWDVNVFDQFDXVHDQRSHUDWLQJ V\VWHP WR IDLO 7KH HIIHFWLYHQHVV RI WKH &UDVKPH DSSURDFK UHOLHV XSRQ VHUHQGLSLW\ LQ RWKHU ZRUGVLIUXQORQJHQRXJKLWPD\HYHQWXDOO\ILQGVRPHZD\WRFUDVKWKHV\VWHP 7KHFUDVKPHDSSURDFKKDVWKHGLVWLQFWDGYDQWDJHWKDWLWLVUHODWLYHO\HDV\WRLPSOHPHQWDQG FDQ EH SRUWHG UHDGLO\ WR D ZLGH YDULHW\ RI V\VWHPV ,W DOVR KDV WKH FRPSHOOLQJ YLUWXH WKDW LW DFWXDOO\ GRHVILQGZD\VWR FUDVKRSHUDWLQJV\VWHPV+RZHYHU DVDWHFKQLTXHLWLVQRWUHDGLO\ DSSOLHGWRRWKHUDSSOLFDWLRQDUHDVDQGGRHVQRWSURGXFHUHSHDWDEOHGHWHUPLQLVWLFUHVXOWVWKDWDUH GHVLUDEOH IRU FRQFLVH EXJ UHSRUWV )LQDOO\ LW LV QRW UHDOO\ GHVLJQHG WR TXDQWLI\ DQG FRPSDUH V\VWHPVVRPXFKDVGLVFRYHUSUREOHPVLQDSDUWLFXODUV\VWHP )DXOWLQMHFWLRQ 2WKHUZRUNLQWKHIDXOWLQMHFWLRQDUHDKDVDOVRWHVWHGDVSHFWVRIUREXVWQHVV7KH)LDWV\VWHP >@XVHVSUREHVSODFHGE\WKHSURJUDPPHUWRDOWHUWKHELQDU\SURFHVVLPDJHLQPHPRU\GXULQJ H[HFXWLRQ 7KH )HUUDUL V\VWHP >@ LV VLPLODU LQ LQWHQW WR ),$7 EXW XVHV VRIWZDUH WUDSV LQ D PDQQHUVLPLODUWRGHEXJJHUEUHDNSRLQWVWRSHUPLWHPXODWLRQRIVSHFLILFV\VWHPOHYHOKDUGZDUH IDXOWV HJGDWDDGGUHVVOLQHVFRQGLWLRQFRGHV 7KH)7$3(V\VWHP>@LQMHFWVIDXOWVLQWRD V\VWHPEHLQJH[HUFLVHGZLWKDUDQGRPZRUNORDGJHQHUDWRUE\XVLQJDSODWIRUPVSHFLILFGHYLFH GULYHUWRLQMHFWWKHIDXOWV:KLOHDOORIWKHVHV\VWHPVKDYHSURGXFHGLQWHUHVWLQJUHVXOWVQRQHZDV LQWHQGHGWRTXDQWLI\UREXVWQHVVRQWKHVFDOHRIDQHQWLUH26$3, 7KH)X]]SURMHFWDWWKH8QLYHUVLW\RI:LVFRQVLQKDVXVHGUDQGRPQRLVH RUIX]] LQMHFWLRQ WR GLVFRYHU UREXVWQHVV SUREOHPV LQ RSHUDWLQJ V\VWHPV 7KDW ZRUN GRFXPHQWHG WKH VRXUFH RI VHYHUDO SUREOHPV > @ DQG WKHQ GLVFRYHUHG WKDW WKH SUREOHPV ZHUH VWLOO SUHVHQW LQ RSHUDWLQJ V\VWHPVVHYHUDO\HDUVODWHU>@7KH)X]]DSSURDFKWHVWHGVSHFLILF26HOHPHQWVDQGLQWHUIDFHV FRPSDUHGWRWKHFRPSOHWHO\UDQGRPDSSURDFKRI&UDVKPH DOWKRXJKLWVWLOOUHOLHGRQUDQGRP GDWDLQMHFWLRQ 7KH %DOOLVWD WHVWLQJ DSSURDFK EXLOGV XSRQ D OHJDF\ RI ZRUN LQ WKH IDXOW LQMHFWLRQ DUHD SHUIRUPHGE\6LHZLRUHNDQG KLVVWXGHQWV DW&DUQHJLH0HOORQ8QLYHUVLW\ 7KLVZRUN KDVORQJ VRXJKWDFRPELQDWLRQRIUHSHDWDELOLW\KLJKOHYHOLPSOHPHQWDWLRQDQGSRUWDELOLW\ TXDQWLILFDWLRQ ZDVQRWDQH[SOLFLWJRDOEXWFDQEHDFFRPSOLVKHGJLYHQWKDWWKHRWKHUJRDOVKDYHEHHQPHW
.RRSPDQ&6'$¶
$ILUVWHIIRUWDW&DUQHJLH0HOORQZDVWKHFUHDWLRQRIDGHVFHQGHQWRIFUDVKPHFDOOHG&08 FUDVKPH,QVWHDGRIH[HFXWLQJUDQGRPGDWD&08FUDVKPHUHVWULFWHGUDQGRPQHVVWRSDUDPHWHU YDOXHVIRUFRQFXUUHQWO\H[HFXWHGV\VWHPFDOOV7KLVKDGWKHHIIHFWRIFRQFHQWUDWLQJWHVWLQJRQWKH 26LQWHUIDFHDQGHOLPLQDWLQJDODUJHQXPEHURIWHVWVWKDWVLPSO\UHVXOWHGLQLOOHJDOLQVWUXFWLRQ H[FHSWLRQV,WVWLOOKRZHYHUUHOLHGRQVHUHQGLSLW\WRGLVFRYHUSUREOHPVDQGZDVQRWUHSHDWDEOH 7KHQH[WJHQHUDWLRQRIZRUNZDVFDOOHGUREXVWQHVVEHQFKPDUNLQJLQZKLFKFDUHIXOO\VHOHFWHG GDWDYDOXHVZHUHVHQWWRVHOHFWHGIXQFWLRQFDOOVZLWKRQHFDOOH[HFXWHGDWDWLPH7KHHIIRUWZDV ODERULQWHQVLYHLQWKDWLWUHTXLUHGGHYHORSLQJDWHVWVXLWHEXWDWWDLQHGUHSHDWDELOLW\$QGSHUKDSV VXUSULVLQJO\HYHQZLWKDVLQJOHFDOOPRGHOLWIRXQGZD\VWRFUDVKRSHUDWLQJV\VWHPVIURPXVHU PRGH LQFOXGLQJD VSDFHTXDOLILHG IDXOW WROHUDQW DHURVSDFH FRPSXWHU>@ (YHQWXDOO\ VHYHUDO RSHUDWLQJV\VWHPVZHUHWHVWHGZLWKDKDQGIXORIFDOOVGHPRQVWUDWLQJSRUWDELOLW\DQGUHYHDOLQJ WKDWUREXVWQHVVIDLOXUHVZHUHSUHYDOHQWDFURVVYDULRXVV\VWHPV>@ 2QFHUHSHDWDELOLW\KDGEHHQDFKLHYHGWKHREYLRXVQH[WVWHSZDVWRJHQHULFL]HWKHWHVWVDQG DWWDLQVFDODELOLW\IRUODUJHU$3,V7KXVZRUNZDVGRQHWRH[WHQGWKHUREXVWQHVVEHQFKPDUNVE\ JHQHULFL]LQJ GLIIHUHQW RSHUDWLRQV ZLWKLQ D IXQFWLRQ DQG DWWHPSWLQJ WR DVVRFLDWH WKHP ZLWK GLIIHUHQWWHVWGDWDW\SHV>@)RUH[DPSOHPDQ\IXQFWLRQVVHUYHWRFUHDWHDGDWDREMHFWZKHWKHU LWEHDILOHDQDOORFDWHGEORFNRIPHPRU\RUDGDWDVWUXFWXUH6LPLODUO\RWKHUIXQFWLRQVPRGLI\ GHOHWHDQGFRS\VXFKREMHFWV7KXVRQHFRXOGWKLQNRIJHQHULFL]LQJWKHIXQFWLRQRIDSDUWLFXODU PRGXOHLQWRRQHRIDSUHVHWFDWHJRU\DQGWKHQGLUHFWWHVWLQJEDVHGRQDFRPELQDWLRQRIJHQHULF IXQFWLRQDQGGDWDW\SH:KLOHWKLVDSSURDFKGLGZRUNDWWHPSWVWRVFDOHLWWRIXOOVL]HV\VWHPV ZHUH XQVXFFHVVIXO EHFDXVH WKH HIIRUW UHTXLUHG WR VSHFLI\ JHQHULF IXQFWLRQDOLW\ VFDOHG OLQHDUO\ ZLWKWKHQXPEHURIIXQFWLRQVEHLQJWHVWHG7KHDSSURDFKUHPDLQVSURPLVLQJEXWLVQRWZLWKRXW IXUWKHULVVXHVWREHUHVROYHG $IWHU DWWHPSWLQJ WR JHQHULFL]H IXQFWLRQV D IUHVK DSSURDFK WR WKH SUREOHP ZDV WDNHQ E\ HOLPLQDWLQJ WKH LQYROYHPHQW RI IXQFWLRQDOLW\ DOWRJHWKHU DQG FUHDWLQJ WHVW FDVHV EDVHG RQ GDWD W\SHLQIRUPDWLRQDORQH7KLVDSSURDFKZDVWKHVRXUFHRIWKH%DOOLVWDWHVWLQJPHWKRGRORJ\>@ ZKLFKDWWDLQHGUHSHDWDELOLW\SRUWDELOLW\DQGVFDODELOLW\DQGLVGHVFULEHGLQPRUHGHWDLOLQODWHU VHFWLRQV $V DQ DGGLWLRQDO EHQHILW WKH DELOLW\ WR SHUIRUP RYHU D PLOOLRQ WHVWV RQ PRUH WKDQ D GR]HQ RSHUDWLQJ V\VWHPV SURYLGHG HQRXJK WHVW GDWD WR FUHDWH D VWUDWHJ\ IRU TXDQWLILFDWLRQ RI UREXVWQHVV 6RIWZDUHWHVWLQJ *LYHQDGHVLUHWRSURYLGHFUDVKPHOLNHWHVWLQJLQDPRUHGHWHUPLQLVWLFUHSHDWDEOHVHWWLQJRQH FDQWXUQWRVRIWZDUHWHVWLQJWRERUURZFRQFHSWV6RIWZDUHWHVWLQJIRUWKHSXUSRVHRIGHWHUPLQLQJ UHOLDELOLW\ LV RIWHQ FDUULHG RXW E\ H[HUFLVLQJ D VRIWZDUH V\VWHP XQGHU UHSUHVHQWDWLYH ZRUNORDG FRQGLWLRQVDQGPHDVXULQJIDLOXUHUDWHV,QDGGLWLRQHPSKDVLVLVRIWHQSODFHGRQFRGHFRYHUDJH DVDZD\RIDVVHVVLQJZKHWKHUDPRGXOHKDVEHHQWKRURXJKO\WHVWHG8QIRUWXQDWHO\WUDGLWLRQDO VRIWZDUH UHOLDELOLW\ WHVWLQJ PD\ QRW XQFRYHU UREXVWQHVV SUREOHPV WKDW RFFXU EHFDXVH RI XQH[SHFWHGLQSXWYDOXHVJHQHUDWHGE\EXJVLQRWKHUPRGXOHVRUEHFDXVHRIDQHQFRXQWHUZLWK DW\SLFDORSHUDWLQJFRQGLWLRQV 6WUXFWXUDO RU ZKLWHER[ WHVWLQJ WHFKQLTXHV DUH XVHIXO IRU DWWDLQLQJ KLJK WHVW FRYHUDJH RI SURJUDPV %XWWKH\W\SLFDOO\IRFXVRQ WKH FRQWUROIORZRI DSURJUDPUDWKHUWKDQKDQGOLQJRI H[FHSWLRQDO GDWD YDOXHV )RU H[DPSOH VWUXFWXUDO WHVWLQJ DVFHUWDLQV ZKHWKHU FRGH GHVLJQHG WR GHWHFWLQYDOLGGDWDLVH[HFXWHGE\DWHVWVXLWHEXWPD\QRWGHWHFWZKHWKHUDFKHFNIRULQYDOLGGDWD LVPLVVLQJIURPWKHSURJUDPDOWRJHWKHU$GGLWLRQDOO\VWUXFWXUDOWHVWLQJW\SLFDOO\UHTXLUHVDFFHVV WRVRXUFHFRGHZKLFKPD\EHXQDYDLODEOHZKHQXVLQJ&276VRIWZDUHFRPSRQHQWV $FRPSOHPHQWDU\DSSURDFKLVEODFNER[WHVWLQJDOVRFDOOHGEHKDYLRUDOWHVWLQJ>@%ODFNER[ WHVWLQJ WHFKQLTXHV DUH GHVLJQHG WR GHPRQVWUDWH FRUUHFW UHVSRQVH WR YDULRXV LQSXW YDOXHV
.RRSPDQ&6'$¶
UHJDUGOHVV RI WKH VRIWZDUH LPSOHPHQWDWLRQ DQG VHHP PRUH DSSURSULDWH IRU UREXVWQHVV WHVWLQJ 7ZRW\SHVRIEODFNER[WHVWLQJDUHSDUWLFXODUO\XVHIXODVVWDUWLQJSRLQWVIRUUREXVWQHVVWHVWLQJ GRPDLQWHVWLQJDQGV\QWD[WHVWLQJ'RPDLQWHVWLQJORFDWHVDQGSUREHVSRLQWVDURXQGH[WUHPDDQG GLVFRQWLQXLWLHVLQWKHLQSXWGRPDLQ6\QWD[WHVWLQJFRQVWUXFWVFKDUDFWHUVWULQJVWKDWDUHGHVLJQHG WRWHVWWKHUREXVWQHVVRIVWULQJOH[LQJDQGSDUVLQJV\VWHPV%RWKW\SHVRIWHVWLQJDUHDPRQJWKH DSSURDFKHVXVHGLQ%DOOLVWD $XWRPDWLFDOO\JHQHUDWLQJVRIWZDUHWHVWVUHTXLUHVWKUHHWKLQJV D0RGXOHXQGHU7HVW 0X7 D PDFKLQHXQGHUVWDQGDEOH VSHFLILFDWLRQ RI FRUUHFW EHKDYLRU DQG DQ DXWRPDWLF ZD\ WR FRPSDUH UHVXOWV RI H[HFXWLQJ WKH 0X7 ZLWK WKH VSHFLILFDWLRQ 8QIRUWXQDWHO\ REWDLQLQJ RU FUHDWLQJ D EHKDYLRUDOVSHFLILFDWLRQIRUD&276RUOHJDF\VRIWZDUHFRPSRQHQWLVRIWHQLPSUDFWLFDOGXHWR XQDYDLODELOLW\RUFRVW )RUWXQDWHO\UREXVWQHVVWHVWLQJQHHGQRWXVHDGHWDLOHGEHKDYLRUDOVSHFLILFDWLRQ,QVWHDGWKH DOPRVW WULYLDO VSHFLILFDWLRQ RI ³GRHVQ¶W FUDVK GRHVQ¶W KDQJ´ VXIILFHV 'HWHUPLQLQJ ZKHWKHU D 0X7PHHWVWKLVVSHFLILFDWLRQLVVWUDLJKWIRUZDUGWKHRSHUDWLQJV\VWHPFDQEHTXHULHGWRVHHLID WHVWSURJUDPWHUPLQDWHVDEQRUPDOO\DQGDZDWFKGRJWLPHUFDQEHXVHGWRGHWHFWKDQJV7KXV UREXVWQHVVWHVWLQJFDQEHSHUIRUPHGRQPRGXOHV WKDWGRQ¶WLQWHQWLRQDOO\FUDVKRUKDQJ LQWKH DEVHQFHRILQGLYLGXDOEHKDYLRUDOVSHFLILFDWLRQV 63(&,),(' %(+$9,25
,1387 63$&(
6+28/' :25.
9$/,' ,13876
5(63216( 63$&( 52%867 23(5$7,21 02'8/( 81'(5 7(67 815(352'8&,%/( )$,/85(
)LJXUH%DOOLVWDKDVDJRDORILGHQWLI\LQJUHSURGXFLEOHIDLOXUHVIRULQYDOLGLQSXWVWRD PRGXOHXQGHUWHVW $Q\H[LVWLQJVSHFLILFDWLRQIRUD0X7PLJKWGHILQHLQSXWVDVIDOOLQJLQWRWKUHHFDWHJRULHV YDOLG LQSXWVLQSXWVZKLFKDUHVSHFLILHGWREHKDQGOHGDVH[FHSWLRQVDQGLQSXWVIRUZKLFKWKHEHKDYLRU LV XQVSHFLILHG )LJXUH %DOOLVWD WHVWLQJ EHFDXVH LW LV QRW FRQFHUQHG ZLWK WKH VSHFLILHG EHKDYLRUFROODSVHVWKHXQVSHFLILHGDQGVSHFLILHGH[FHSWLRQDOLQSXWVLQWRDVLQJOHLQYDOLGLQSXW VSDFH 7KH UREXVWQHVV RI WKH UHVSRQVHV RI WKH 0X7 FDQ EH FKDUDFWHUL]HG DV UREXVW QHLWKHU FUDVKHV QRU KDQJV EXW LV QRW QHFHVVDULO\ FRUUHFW IURP D GHWDLOHG EHKDYLRUDO YLHZ KDYLQJ D UHSURGXFLEOH IDLOXUH D FUDVK RU KDQJ WKDW LV FRQVLVWHQWO\ UHSURGXFHG DQG DQ XQUHSURGXFLEOH IDLOXUH D UREXVWQHVV IDLOXUH WKDW LV QRW UHDGLO\ UHSURGXFLEOH 7KH REMHFWLYH RI %DOOLVWD LV WR LGHQWLI\UHSURGXFLEOHIDLOXUHV ,QVWUXPHQWDWLRQDSSURDFKHV 7KHUH DUH VHYHUDO FRPPHUFLDO SURGXFWV WKDW KHOS LQ GHYHORSLQJ UREXVW FRGH VXFK DV 3XULI\ >@DQG%RXQGVFKHFNHU>@ZKLFKLQVWUXPHQWVRIWZDUHWRGHWHFWH[FHSWLRQDOVLWXDWLRQV7KH\ ZRUN E\ GHWHFWLQJ H[FHSWLRQV WKDW DULVH GXULQJ GHYHORSPHQW WHVWLQJ RU XVDJH RI WKH VRIWZDUH +RZHYHUWKH\DUHQRWDEOHWRILQGUREXVWQHVVIDLOXUHVWKDWPLJKWRFFXULQVLWXDWLRQVZKLFKDUH QRWWHVWHG DQGHYHQZLWKZKDWZRXOGQRUPDOO\EHFDOOHGWHVWFRYHUDJHLWLVXQOLNHO\LQ
.RRSPDQ&6'$¶
SUDFWLFHWKDWHYHU\H[FHSWLRQDOFRQGLWLRQZKLFKZLOOEHHQFRXQWHUHGLQWKHILHOGLVLQFOXGHGLQWKH VRIWZDUHWHVWVXLWH 7KH%DOOLVWDDSSURDFKGLIIHUVIURPDQGFRPSOHPHQWVWKHVHDSSURDFKHVE\ DFWLYHO\ VHHNLQJ RXW UREXVWQHVV IDLOXUHV UDWKHU WKDQ EHLQJ DQ LQVWUXPHQWDWLRQ WRRO LW DFWXDOO\ JHQHUDWHV WHVWV IRU H[FHSWLRQ KDQGOLQJ DELOLW\ DQG IHHGV WKHP GLUHFWO\ LQWR VRIWZDUH PRGXOHV 7KXV%DOOLVWDLVOLNHO\WRILQGUREXVWQHVVIDLOXUHVWKDWZRXOGRWKHUZLVHEHPLVVHGGXULQJQRUPDO VRIWZDUHWHVWLQJHYHQZLWKDYDLODEOHLQVWUXPHQWDWLRQWRROV
7KH%DOOLVWDVFDODEOHUREXVWQHVVWHVWLQJIUDPHZRUN $VRIWZDUHFRPSRQHQWIRURXUSXUSRVHVLVDQ\SLHFHRIVRIWZDUHWKDWFDQEHLQYRNHGDVD SURFHGXUH IXQFWLRQ RU PHWKRG ZLWK D QRQQXOO VHW RI LQSXW SDUDPHWHUV :KLOH WKDW LV QRW D XQLYHUVDO GHILQLWLRQ RI DOO VRIWZDUH LQWHUIDFHV LW LV VXIILFLHQWO\ EURDG WR EH RI LQWHUHVW ,Q WKH %DOOLVWD DSSURDFK UREXVWQHVV WHVWLQJ RI VXFK D VRIWZDUH FRPSRQHQW D 0RGXOH XQGHU 7HVW RU 0X7 FRQVLVWV RI HVWDEOLVKLQJ DQ LQLWLDO V\VWHP VWDWH H[HFXWLQJ D VLQJOH FDOO WR WKH 0X7 GHWHUPLQLQJZKHWKHUDUREXVWQHVVSUREOHPRFFXUUHGDQGWKHQUHVWRULQJV\VWHPVWDWHWRSUHWHVW FRQGLWLRQVLQSUHSDUDWLRQIRUWKHQH[WWHVW$OWKRXJKH[HFXWLQJFRPELQDWLRQVRIFDOOVWRRQHRU PRUH0X7VGXULQJDWHVWFDQEHXVHIXOLQVRPHVLWXDWLRQVZHKDYHIRXQGWKDWHYHQWKLVVLPSOH DSSURDFKRIWHVWLQJDVLQJOHFDOODWDWLPHSURYLGHVDULFKVHWRIWHVWVDQGXQFRYHUVDVLJQLILFDQW QXPEHURIUREXVWQHVVSUREOHPV %DOOLVWDGUDZVXSRQLGHDVIURPWKHDUHDVRIERWKVRIWZDUHWHVWLQJDQGIDXOWLQMHFWLRQ$NH\ LGHDLVWKHXVHRIDQREMHFWRULHQWHGDSSURDFKGULYHQE\SDUDPHWHUOLVWGDWDW\SHLQIRUPDWLRQWR DFKLHYHVFDODELOLW\DQGDXWRPDWHGLQLWLDOL]DWLRQRIV\VWHPVWDWHIRUHDFKWHVWFDVH %DOOLVWDDSSURDFK 7KH %DOOLVWD UREXVWQHVV WHVWLQJ PHWKRGRORJ\ LV EDVHG RQ FRPELQDWLRQDO WHVWV RI YDOLG DQG LQYDOLGSDUDPHWHUYDOXHVIRUV\VWHPFDOOVDQGIXQFWLRQV,QHDFKWHVWFDVHDVLQJOHVRIWZDUH0X7 LV FDOOHG D VLQJOH WLPH WR GHWHUPLQH ZKHWKHU LW LV UREXVW ZKHQ FDOOHG ZLWK D SDUWLFXODU VHW RI SDUDPHWHUYDOXHV7KHVHSDUDPHWHUYDOXHVRUWHVWYDOXHVDUHGUDZQIURPDSRRORIQRUPDODQG H[FHSWLRQDO YDOXHV EDVHG RQ WKH GDWD W\SH RI HDFK DUJXPHQW SDVVHG WR WKH 0X7 $ WHVW FDVH WKHUHIRUHFRQVLVWVRIWKHQDPHRIWKH0X7DQGDWXSOHRIWHVWYDOXHVWKDWDUHSDVVHGDVSDUDPHWHUV LH D WHVW FDVH FRXOG EH GHVFULEHG DV D WXSOH ^0X7BQDPH WHVWBYDOXH WHVWBYDOXH ` FRUUHVSRQGLQJWRDSURFHGXUHFDOORIWKHIRUP 0X7BQDPH WHVWBYDOXHWHVWBYDOXH 7KXV WKHJHQHUDODSSURDFKWR%DOOLVWDWHVWLQJLVWRWHVWWKHUREXVWQHVVRIDVLQJOHFDOOWRD0X7IRUD VLQJOH WXSOH RI WHVW YDOXHV DQG WKHQ UHSHDW WKLV SURFHVV IRU PXOWLSOH WHVW FDVHV WKDW HDFK KDYH GLIIHUHQWFRPELQDWLRQVRIYDOLGDQGLQYDOLGWHVWYDOXHV$GHWDLOHGGLVFXVVLRQIROORZV ,Q%DOOLVWDWHVWVDUHEDVHGRQWKHYDOXHVRISDUDPHWHUVDQGQRWRQWKHEHKDYLRUDOGHWDLOVRIWKH 0X77KHVHWRIWHVWFDVHVXVHGWRWHVWD0X7LVFRPSOHWHO\GHWHUPLQHGE\WKHGDWDW\SHVRIWKH SDUDPHWHUOLVWRIWKH0X7DQGGRHVQRWGHSHQGRQDEHKDYLRUDOVSHFLILFDWLRQ )LJXUHVKRZVWKH%DOOLVWDDSSURDFKWRJHQHUDWLQJWHVWFDVHVIRUD0X7%HIRUHFRQGXFWLQJ WHVWLQJDVHWRIWHVWYDOXHVPXVWEHZULWWHQIRUHDFKGDWDW\SHXVHGLQWKH0X7)RUH[DPSOHLI RQHRUPRUHPRGXOHVWREHWHVWHGUHTXLUHDQLQWHJHUGDWDW\SHDVDQLQSXWSDUDPHWHUWHVWYDOXHV PXVWEHFUHDWHGIRUWHVWLQJLQWHJHUV9DOXHVWRWHVWLQWHJHUVPLJKWLQFOXGHDQG0$;,17 PD[LPXPLQWHJHUYDOXH $GGLWLRQDOO\LIDSRLQWHUGDWDW\SHLVXVHGZLWKLQWKH0X7YDOXHVRI 18//DQG DPRQJ RWKHUV PLJKW EH XVHG $ PRGXOH FDQQRWEHWHVWHG XQWLOWHVW YDOXHVDUH FUHDWHGIRUHDFKRILWVSDUDPHWHUGDWDW\SHV$XWRPDWLFWHVWLQJJHQHUDWHVPRGXOHWHVWFDVHVE\ GUDZLQJIURPSRROVRIGHILQHGWHVWYDOXHV (DFKVHWRIWHVWYDOXHV RQHVHWSHUGDWDW\SH LVLPSOHPHQWHGDVDWHVWLQJREMHFWKDYLQJDSDLU RI FRQVWUXFWRU DQG GHVWUXFWRU IXQFWLRQV IRU HDFK GHILQHG WHVW YDOXH ,QVWDQWLDWLRQ RI D WHVWLQJ
.RRSPDQ&6'$¶
$3, 7(67,1* 2%-(&76
7(67 9$/8(6
ZULWH LQWILOHGHVFRQVWYRLG EXIIHUVL]HBWQE\WHV ),/(
0(025
@ &DWDVWURSKLFIDLOXUHVRFFXUZKHQWKHHQWLUH26EHFRPHVFRUUXSWHGRUWKHPDFKLQHFUDVKHVRU UHERRWV,QRWKHUZRUGVWKLVLVDFRPSOHWHV\VWHPFUDVK7KHVHIDLOXUHVDUHLGHQWLILHGPDQX DOO\ EHFDXVH RI GLIILFXOWLHV HQFRXQWHUHG ZLWK ORVV RI QHZO\ ZULWWHQ FRPPLWWHG ILOH GDWD DFURVVV\VWHPFUDVKHVRQVHYHUDORSHUDWLQJV\VWHPV
.RRSPDQ&6'$¶
5HVWDUW IDLOXUHVRFFXUZKHQDIXQFWLRQ FDOOWRDQ26IXQFWLRQ QHYHUUHWXUQVUHVXOWLQJLQD WDVNWKDWKDVKXQJDQGPXVWEHWHUPLQDWHGXVLQJDFRPPDQGVXFKDVNLOO 7KHVHIDLO XUHVDUHLGHQWLILHGE\DZDWFKGRJWLPHUZKLFKWLPHVRXWDIWHUVHYHUDOVHFRQGVRIZDLWLQJIRUD WHVWFDVHWRFRPSOHWH $ERUW IDLOXUHV WHQG WR EH WKH PRVW SUHYDOHQW DQG UHVXOW LQ DEQRUPDO WHUPLQDWLRQ D FRUH GXPS RIDWDVNFDXVHGE\DVLJQDOJHQHUDWHGZLWKLQDQ26IXQFWLRQ$ERUWIDLOXUHVDUHLGHQ WLILHGE\PRQLWRULQJWKHVWDWXVRIWKHFKLOGSURFHVVH[HFXWLQJWKHWHVWFDVH 6LOHQWIDLOXUHVRFFXUZKHQDQ26UHWXUQVQRLQGLFDWLRQRIHUURURQDQH[FHSWLRQDORSHUDWLRQ ZKLFKFOHDUO\FDQQRWEHSHUIRUPHG IRUH[DPSOHZULWLQJWRDUHDGRQO\ILOH 7KHVHIDLOXUHV DUHQRWGLUHFWO\PHDVXUHGEXWFDQEHLQIHUUHGDVGLVFXVVHGLQ6HFWLRQ +LQGHULQJIDLOXUHVRFFXUZKHQDQLQFRUUHFWHUURUFRGHLVUHWXUQHGIURPD0X7ZKLFKFRXOG PDNHLWPRUHGLIILFXOWWRH[HFXWHDSSURSULDWHHUURUUHFRYHU\+LQGHULQJIDLOXUHVKDYHEHHQ REVHUYHG DV IDLUO\ FRPPRQ IRUPLQJ D VXEVWDQWLDO IUDFWLRQ RI FDVHV ZKLFK UHWXUQHG HUURU FRGHV LQSUHYLRXVZRUN>@EXWDUHQRWIXUWKHUGLVFXVVHGLQWKLVSDSHUGXHWRODFNRIDZD\ WRSHUIRUPDXWRPDWHGLGHQWLILFDWLRQRIWKHVHUREXVWQHVVIDLOXUHV 7KHUHDUHWZRDGGLWLRQDOSRVVLEOHRXWFRPHVRIH[HFXWLQJDWHVWFDVH,WLVSRVVLEOHWKDWDWHVW FDVHUHWXUQVZLWKDQHUURUFRGHWKDWLVDSSURSULDWHIRULQYDOLGSDUDPHWHUVIRUPLQJWKHWHVWFDVH 7KLVLVDFDVHLQZKLFKWKHWHVWFDVHSDVVHVLQRWKHUZRUGVJHQHUDWLQJDQHUURUFRGHLVWKH FRUUHFW UHVSRQVH $GGLWLRQDOO\ LQ VRPH WHVWV WKH 0X7 OHJLWLPDWHO\ UHWXUQV QR HUURU FRGH DQG VXFFHVVIXOO\FRPSOHWHVWKHGHVLUHGRSHUDWLRQ7KLVKDSSHQVZKHQWKHSDUDPHWHUVLQWKHWHVWFDVH KDSSHQ WR EH DOO YDOLG RU ZKHQ LW LV XQUHDVRQDEOH WR H[SHFW WKH 26 WR GHWHFW DQ H[FHSWLRQDO VLWXDWLRQ VXFKDVSRLQWLQJWRDQDGGUHVVSDVWWKHHQGRIDEXIIHUEXWQRVRIDUSDVWDVWRWULJJHUD PHPRU\SURWHFWLRQH[FHSWLRQE\WRXFKLQJDQXQDOORFDWHGYLUWXDOPHPRU\SDJH 2QHRIWKHWUDGHRIIVPDGHWRDWWDLQVFDODELOLW\LQ%DOOLVWDWHVWLQJLVWKDWWKHWHVWKDUQHVVKDVQR ZD\ WR WHOO ZKLFK WHVW FDVHV DUH YDOLG RU LQYDOLG IRU DQ\ SDUWLFXODU 0X7 7KXV VRPH WHVWV UHWXUQLQJ QR HUURU FRGH DUH 6LOHQW IDLOXUHV ZKLOH RWKHUV DUH DFWXDOO\ D VHW RI YDOLG SDUDPHWHU YDOXHV ZKLFK VKRXOG OHJLWLPDWHO\ UHWXUQ ZLWK QR HUURU LQGLFDWLRQ 7KLV KDV WKH HIIHFW RI ³ZDWHULQJ GRZQ´ WKH WHVW FDVHV ZLWK QRQH[FHSWLRQDO WHVWV PDNLQJ WKH UDZ IDLOXUH UDWHV XQGHUHVWLPDWHV:HHVWLPDWHWKDWRIWHVWVDUHRIWKLVQRQH[FHSWLRQDOW\SHIRU26WHVWV EXW HYHQ WKH UDZ IDLOXUH UDWHV DUH VLJQLILFDQW HQRXJK WR PDNH WKH SRLQW WKDW %DOOLVWD WHVWLQJ LV HIIHFWLYHLQILQGLQJUREXVWQHVVIDLOXUHV
5HVXOWV ,QDOOGDWDSRLQWVZHUHFROOHFWHG2SHUDWLQJV\VWHPVZKLFKVXSSRUWHGDOORIWKH VHOHFWHG326,;IXQFWLRQVDQGV\VWHPFDOOVHDFKKDG WRWDOWHVWFDVHVEXWWKRVHVXSSRUWLQJ D VXEVHW RIWKH IXQFWLRQDOLW\WHVWHG KDGIHZHUWHVW FDVHV7KHILQDOQXPEHUZDVGHSHQGHQW RQ ZKLFK IXQFWLRQV ZHUH VXSSRUWHG DQG WKH QXPEHU RI FRPELQDWLRQV RI WHVWV IRU HDFK VXSSRUWHG IXQFWLRQ 0HDVXUHGIDLOXUHUDWHV 7KH FRPSLOHUV DQG OLEUDULHV XVHG WR JHQHUDWH WKH WHVW VXLWH ZHUH WKRVH SURYLGHG E\ WKH 26 YHQGRU,QWKHFDVHRI)UHH%6'1HW%6'/LQX[DQG/\Q[26*18&YHUVLRQZDVXVHG WREXLOGWKHWHVWVXLWH 7KHUH ZHUH ILYH GLIIHUHQW IXQFWLRQV WKDW FRXOG EH PDGH WR FDXVH HQWLUH RSHUDWLQJ V\VWHP FUDVKHVRQVRPH26 HLWKHUDXWRPDWLFUHERRWVRUV\VWHPKDQJV 5HVWDUWIDLOXUHVZHUHUHODWLYHO\ VFDUFH EXWSUHVHQW LQ DOO EXWWZR RSHUDWLQJV\VWHPV $ERUW IDLOXUHV ZHUHFRPPRQ LQGLFDWLQJ WKDW LQ DOO RSHUDWLQJ V\VWHPV LW LV UHODWLYHO\ VWUDLJKWIRUZDUG WR HOLFLW D FRUH GXPS IURP DQ
.RRSPDQ&6'$¶
LQVWUXFWLRQZLWKLQDIXQFWLRQRUV\VWHPFDOO $ERUWIDLOXUHVGRQRWKDYHWRGRZLWKVXEVHTXHQW XVH RI DQ H[FHSWLRQDO YDOXH UHWXUQHG IURP D V\VWHP FDOO WKH\ KDSSHQ LQ UHVSRQVH WR DQ LQVWUXFWLRQZLWKLQWKHYHQGRUSURYLGHGVRIWZDUHLWVHOI $FKHFNZDVPDGHWRHQVXUHWKDW$ERUW IDLOXUHV ZHUH QRW GXH WR FRUUXSWLRQ RI VWDFN YDOXHV DQG VXEVHTXHQW FRUUXSWLRQPLVGLUHFWLRQ RI FDOOLQJSURJUDPRSHUDWLRQ 0X7VWKDWXQGHUZHQWDFDWDVWURSKLFIDLOXUHFRXOGQRWEHFRPSOHWHO\WHVWHGDQGUHVXOWHGLQQR GDWD RQ WKDW 0X7 RWKHU WKDQ WKH SUHVHQFH RI D FDWDVWURSKLF IDLOXUH 6LQFH WKH WHVWLQJ VXLWH LV DXWRPDWHG D V\VWHP FUDVK OHDYHV LW LQ DQ XQUHFRYHUDEOH VWDWH ZLWK UHVSHFW WR WKH IXQFWLRQ LQ TXHVWLRQ )XUWKHU WHVWLQJ D IXQFWLRQ ZKLFK VXIIHUHG D FDWDVWURSKLF WHVW ZRXOG UHTXLUH HLWKHU PDQXDOH[HFXWLRQRILQGLYLGXDOFDVHVRUDGGLQJWULFN\ZDLWLQJDQGV\QFKURQL]DWLRQFRGHLQWRWKH WHVW EHQFKPDUN 0DQXDO H[HFXWLRQ ZDV SHUIRUPHG IRU WKH ,UL[ FDWDVWURSKLF IDLOXUH LQ PXQPDS DQG DOORZHG WKH LGHQWLILFDWLRQ RI WKLV VLQJOH OLQH RI XVHU FRGH ZKLFK FDQ FUDVK WKH HQWLUH26UHTXLULQJDPDQXDOUHERRW PXQPDS PDOORF 0$;,17 1RUPDOL]HGIDLOXUHUDWHV :KLOHLWLVWHPSWLQJWRVLPSO\XVHWKHUDZQXPEHURIWHVWVWKDWIDLODVDFRPSDUDWLYHPHWULF WKLVDSSURDFKLVSUREOHPDWLF0RVW26LPSOHPHQWDWLRQVGRQRWVXSSRUWWKHIXOOVHWRI326,; UHDOWLPHH[WHQVLRQVVRWKHUDZQXPEHURIIDLOXUHVFDQQRWEHXVHGIRUFRPSDULVRQV,QDGGLWLRQ WKHQXPEHURIWHVWVH[HFXWHGSHU0X7LVGHWHUPLQHGE\WKHQXPEHUDQGW\SHVRIWKHDUJXPHQWV 6RDVLQJOH0X7ZLWKDODUJHQXPEHURIWHVWFDVHVFRXOGVLJQLILFDQWO\DIIHFWERWKWKHQXPEHURI IDLOXUHVDQGWKHUDWLRRIIDLOXUHVWRWHVWVH[HFXWHG6LPLODUO\DQ26IXQFWLRQZLWKIHZWHVWFDVHV ZRXOG KDYH PLQLPDO HIIHFW RQ UDZ IDLOXUH UDWHV HYHQ LI LW GHPRQVWUDWHG D ODUJH SHUFHQWDJH RI IDLOXUHV7KXVVRPHVRUWRIQRUPDOL]HGIDLOXUHUDWHPHWULFLVFDOOHGIRUDQGLVUHSRUWHGLQWKHODVW FROXPQRI7DEOH 7DEOH 0HDVXUHGIDLOXUHUDWHVIRUILIWHHQ326,;RSHUDWLQJV\VWHPV
6\VWHP
326,;&DOOV 7HVWHG
&DOOVZLWK &DWDVWURSKLF )DLOXUHV
1XPEHURI 7HVWV
$ERUW )DLOXUHV
5HVWDUW )DLOXUHV
1RUPDOL]HG $ERUW 5HVWDUW5DWH
$,;
)UHH%6'
+38;
+38;
,5,;
,5,;
/LQX[
/\Q[
1HW%6'
'LJLWDO8QL[
'LJLWDO8QL[
41;
41;
6XQ26
6XQ26
.RRSPDQ&6'$¶
:HGHILQHWKHQRUPDOL]HGIDLOXUHUDWHIRUDSDUWLFXODURSHUDWLQJV\VWHPWREH N
∑
F =
fi w i ZLWKDUDQJHRIYDOXHVIURPWRLQFOXVLYHZKHUH ti
i=1
1 QXPEHURIIXQFWLRQVWHVWHG ZLLVDZHLJKWLQJRILPSRUWDQFHRUUHODWLYHH[HFXWLRQIUHTXHQF\RIWKDWIXQFWLRQZKHUH
∑ wi
= 1
ILLVWKHQXPEHURIWHVWVZKLFKSURGXFHGUREXVWQHVVIDLOXUHIRUIXQFWLRQL WLLVWKHQXPEHURIWHVWVH[HFXWHGIRUIXQFWLRQL
7KLV GHILQLWLRQ SURGXFHV D VRUW RI H[SRVXUH PHWULF LQ ZKLFK WKH IDLOXUH UDWH ZLWKLQ HDFK IXQFWLRQLVZHLJKWHGDQGDYHUDJHGDFURVVDOOIXQFWLRQVWHVWHGIRUDSDUWLFXODU267KLVPHWULF KDV WKH DGYDQWDJH RI UHPRYLQJ WKH HIIHFWV RI GLIIHULQJ QXPEHU RI WHVWV SHU IXQFWLRQ DQG DOVR SHUPLWV FRPSDULQJ 26 LPSOHPHQWDWLRQV ZLWK GLIIHULQJ QXPEHUV RI IXQFWLRQV LPSOHPHQWHG DFFRUGLQJWRDVLQJOHQRUPDOL]HGPHWULF)RUWKHUHVXOWVJLYHQLQ7DEOHDQGWKHUHPDLQGHURI WKLV SDSHU DQ HTXDO ZHLJKWLQJ LV XVHG LH w i = 1 ⁄ N WR SURGXFH D JHQHULFDOO\ DSSOLFDEOH UHVXOW+RZHYHULIDQ26ZHUHEHLQJWHVWHGZLWKUHJDUGWRDVSHFLILFDSSOLFDWLRQZHLJKWLQJV VKRXOGEHGHYHORSHGWRUHIOHFWWKHG\QDPLFIUHTXHQF\RIFDOOLQJHDFKIXQFWLRQWRJLYHDPRUH DFFXUDWHH[SRVXUHPHWULF
2SHUDWLQJ6\VWHP7HVWHG
$,; '81,; '81,; )UHH %6' +38; +38; ,UL[ ,UL[ /LQX[ /\Q[26 1HW%6' 41; 41; 6XQ26 6XQ26
%DOOLVWD 5REXVWQHVV 7HVWV 326,; )XQFWLRQ &DOOV &DWDVWURSKLF
&DWDVWURSKLF
$ERUW )DLOXUHV 5HVWDUW )DLOXUH
&DWDVWURSKLF
&DWDVWURSKLF
&DWDVWURSKLFV
1RUPDOL]HG)DLOXUH5DWH
)LJXUH1RUPDOL]HGUREXVWQHVVIDLOXUHUDWHVIRUILIWHHQ326,;RSHUDWLQJV\VWHPV
.RRSPDQ&6'$¶
)LJXUHVKRZVWKHQRUPDOL]HGIDLOXUHVUDWHVDVFDOFXODWHGIRUHDFKRSHUDWLQJV\VWHP WKLVLV WKH VDPH GDWD DV WKH ULJKWPRVW FROXPQ LQ 7DEOH 2YHUDOO IDLOXUH UDWHV IRU ERWK $ERUW DQG 5HVWDUWIDLOXUHVUDQJHIURPWKHORZRI $,; WRDKLJKRI 41; 7KHPHDQ IDLOXUH UDWH LV ZLWK D VWDQGDUG GHYLDWLRQ RI SHUFHQWDJH SRLQWV $V PHQWLRQHG SUHYLRXVO\ WKHVHDUH UDZ IDLOXUHUDWHV WKDW LQFOXGH QRQH[FHSWLRQDOWHVW FDVHV ZKLFK GLOXWHWKH IDLOXUHUDWHVDQGGRQRWLQFOXGH6LOHQWIDLOXUHV
*HQHUDOL]LQJ%DOOLVWD DSSO\LQJWRRWKHULQWHUIDFHV 7KHLQLWLDODSSURDFKWR%DOOLVWDDQGLWVSUHGHFHVVRUVZDVIRFXVVHGRQWHVWLQJDQLQFUHDVLQJO\ ODUJHUVHWRIRSHUDWLQJV\VWHPFDOOV,WLVRIFRXUVHQDWXUDOWREHVNHSWLFDOWKDWWKHDSSURDFKFDQ VFDOH WR LQFUHDVHG WKRURXJKQHVV DQG RWKHU DSSOLFDWLRQV 7R GDWH LPSOHPHQWDWLRQV KDYH EHHQ FRPSOHWHGWRWKHSURRIRIFRQFHSWSRLQWLQDVLPXODWLRQ$3,DQGDPHFKDQLVPKDVEHHQFUHDWHG IRUSHUIRUPLQJILQHUJUDLQHGWHVWLQJ ([SHULHQFHZLWKGLVWULEXWHGVLPXODWLRQ$3, 7HVWLQJ RSHUDWLQJ V\VWHP FDOOV UHTXLUHV WKH DELOLW\ WR WHVW WUDGLWLRQDO VXEURXWLQHFDOOSOXV SDUDPHWHUOLVWLQWHUIDFHVZLWKVLQJOHFDOOVDQGDPLQLPXPRIV\VWHPVWDWH+RZHYHURWKHU$3,V PD\KDYHVLJQLILFDQWO\GLIIHUHQWUHTXLUHPHQWV7RSUREHWKHH[WHQVLELOLW\RI%DOOLVWDDQHQWLUHO\ GLIIHUHQW$3,LVEHLQJWHVWHGEDVHGRQDGLVWULEXWHGVLPXODWLRQV\VWHP 7KH+LJK/HYHO$UFKLWHFWXUH +/$ 5XQ7LPH,QWHUIDFH 57, LVDQ$3,XVHGIRUDGLVWULEXWHG VLPXODWLRQV\VWHPE\WKH86'HSDUWPHQWRI'HIHQVH>@7KLVV\VWHPLVZULWWHQLQ&LQDQ HQWLUHO\REMHFWRULHQWHGVW\OHDQGKDVVLJQLILFDQWO\PRUHFRPSOLFDWHGSUHFRQGLWLRQVIRUIXQFWLRQ FDOOV'LIIHUHQFHVIURP26WHVWLQJLQFOXGH $QREMHFWRULHQWHGLPSOHPHQWDWLRQPHDQLQJWKDWPHWKRGVDUHWREHWHVWHGUDWKHUWKDQIXQF WLRQFDOOV7KLVFDQODUJHO\EHWUHDWHGDVDV\QWDFWLFGLIIHUHQFH $V\VWHPZKLFKUHTXLUHVVXSSRUWIRUFDOOEDFNVWRREMHFWVIRUWHVWLQJUDWKHUWKDQDVLPSOHFDOO WRDIXQFWLRQWREHWHVWHG7KLVZDVKDQGOHGZLWKDVOLJKWPRGLILFDWLRQWRWKHVWDQGDUG%DOOLVWD FOLHQWWHVWVFDIIROGLQJ 6LJQLILFDQWO\PRUHFRPSOH[GDWDW\SHV,WZDVIHDUHGWKDWDSUROLIHUDWLRQRIGDWDW\SHYDULDQWV ZRXOGLQFUHDVHWKHGLIILFXOW\RIWHVWLQJWKHP+RZHYHULQWKHHQGRQO\GDWDW\SHVZHUH UHTXLUHGIRULQWHUIDFHV$QGRIWKHVHGDWDW\SHVZHUHFRPSOHWHO\UHXVHGIURPRSHUDW LQJV\VWHPWHVWLQJZKLOHWKHUHVWLQKHULWHGH[LVWLQJWHVWFDVHV7KXVSHUIRUPLQJWHVWVRQGDWD W\SHVSURYHGPRGHUDWHO\VFDODEOH3HUKDSVMXVWDVLPSRUWDQWO\WKHVHVDPHGDWDW\SHVZRXOG EHXVHGKHDYLO\E\DQ\+/$DSSOLFDWLRQSURJUDPDQGFRXOGWKXVEHUHXVHGIRU%DOOLVWDWHVW LQJRIVXFKDSSOLFDWLRQSURJUDPVDWDIXWXUHGDWHUHVXOWLQJLQHYHQPRUHHIIHFWLYHVFDOLQJ $V\VWHPZLWKPRGHUDWHO\FRPSOLFDWHGSUHFRQGLWLRQVIRUHDFKDQGHYHU\PHWKRG IRUH[DP SOH WRSHUIRUPDQRSHUDWLRQILUVWDVRFDOOHGIHGHUDWLRQPXVWEHFUHDWHG WKHQDIHGHUDWHPXVW EHFUHDWHGDQGLQVWUXFWHGWRMRLQWKHIHGHUDWLRQ WKHQWKHFRQVWUXFWRUVIRUWKDWIHGHUDWHPXVWEH H[HFXWHGGHSHQGLQJRQGDWDW\SHV 8SRQIXUWKHUH[DPLQDWLRQKRZHYHULWZDVIRXQGWKDW WKHGLIIHUHQWPHWKRGVEHLQJWHVWHGFRXOGEHFROODSVHGLQWRRQO\WZHOYHGLIIHUHQWHTXLYDOHQFH FODVVHVZLWKUHVSHFWWRWHVWVFDIIROGLQJ7KXVRQO\VHWVRIVFDIIROGLQJDUHUHTXLUHG HDFK RQO\DIHZOLQHVRIFRGHORQJ IRUPRGXOHVXQGHUWHVWZKLFKDVZLWKWKHGDWDW\SHVLVD VXEOLQHDUVFDOLQJZLWKUHVSHFWWRQXPEHURIPHWKRGVEHLQJWHVWHG 6XSSRUWIRUHUURUUHSRUWLQJE\WKURZLQJH[FHSWLRQVUDWKHUWKDQHUURUUHWXUQFRGHV7KLVZDV GRQHE\DGGLQJDQH[FHSWLRQKDQGOHUWRWKH%DOOLVWDWHVWKDUQHVVZLWKWKHUHTXLUHPHQWWKDWWKH WHVWHU SURYLGH D ZD\ WR GLVWLQJXLVK EHWZHHQ DFFHSWDEOH H[FHSWLRQ YDOXHV DQG ³XQNQRZQ´ H[FHSWLRQVWKDWZKHQWKURZQDPRXQWWR$ERUWIDLOXUHV
.RRSPDQ&6'$¶
6WUXFWXUHWHVWLQJ 7HVWLQJRIGDWDVWUXFWXUHVFDQTXLFNO\EHFRPHWHGLRXVLIVHSDUDWHGDWDW\SHVPXVWEHFUHDWHGWR WHVW HDFK GLIIHUHQW VWUXFWXUH )RUWXQDWHO\ WKLV FDQ EH RYHUFRPH DXWRPDWLFDOO\ E\ FUHDWLQJ D V\VWHP WKDW FDQ SDUVH VWUXFWXUH GHILQLWLRQV DQG FUHDWH FRPSRVLWH FRQVWUXFWRUV LQ WHUPV RI WKH SULPLWLYHFRPSRQHQWVRIDVWUXFWXUH$VDVLPSOHH[DPSOHDFRPSOH[QXPEHUFDQEHWUHDWHGDV DVWUXFWXUHFRQWDLQLQJWZRIORDWLQJSRLQWQXPEHUVDQGWHVWVIRUWKDWGDWDW\SHFDQEHJHQHUDWHG VLPSO\E\SURYLGLQJWKHVWUXFWXUHLQIRUPDWLRQJLYHQWKDWIORDWLQJSRLQWQXPEHUVFDQDOUHDG\EH WHVWHGDVDEDVHGDWDW\SH ,WPXVWEHNHSWLQPLQGWKDWDGDWDVWUXFWXUHPLJKWEHSRLQWHGWRDVZHOODVUHIHUHQFHGGLUHFWO\ VRWHVWYDOXHVIRUDVWUXFWXUHLQKHULWERWKSRLQWHUWHVWV HJQXOOSRLQWHULQVWHDGRIDSRLQWHUWRWKH VWUXFWXUH DVZHOODVYDULRXVGDWDYDOXHWHVWV HJSRLQWHUWRDVWUXFWXUHZLWKSDUWLFXODUYDOXHV )LQHUJUDLQWHVWLQJGLDOV ORJLFDOVWUXFWXUHV 2QH RI WKH RULJLQDO DQG FXUUHQW JRDOV RI %DOOLVWD LV WR DXWRPDWLFDOO\ JHQHUDWH KDUGHQLQJ ZUDSSHUVEDVHGRQWHVWUHVXOWV:KLOHWKDWZRUNLVVWLOORQJRLQJDQGEH\RQGWKHVFRSHRIWKLV GLVFXVVLRQLWOHGWRDQHDUO\IRFXVRQDQHHGIRUXVLQJWHVWLQJDVDFKDUDFWHUL]DWLRQPHFKDQLVPLQ DGGLWLRQWRPHUHO\DTXDQWLILFDWLRQPHFKDQLVP)RUH[DPSOHLWLVQRWHQRXJKWRVLPSO\VD\WKDW DSDUWLFXODUPRGXOHH[KLELWVUREXVWQHVVIDLOXUHVDWDFHUWDLQUDWHEXWDOVRRQHPXVWNQRZH[DFWO\ ZKDW WULJJHUV WKHVH IDLOXUHV %HLQJ DEOH WR GR WKLV UHTXLUHV IDLUO\ ILQHJUDLQHG WHVWLQJ $V DQ H[DPSOHWKHILOHKDQGOHWHVWVGHVFULEHGHDUOLHUDUHTXLWHFRXUVHJUDLQDQGFHUWDLQO\GRQRWWHVWDOO SRVVLEOHFRPELQDWLRQV IRUH[DPSOHWKH\WHVWDILOHRSHQIRUZULWHZLWKQRGDWDEXWQRWDILOH RSHQIRUZULWHZLWKVRPHGDWDDOUHDG\LQLW ,WFDQEHUHDGLO\VHHQWKDWZLWKRXWVRPHVRUWRI IXUWKHUGHFRPSRVLWLRQWKHQXPEHURISRVVLEOHWHVWFDVHVJURZVLQDFRPELQDWLRQDOH[SORVLRQDQG ORVHVVFDODELOLW\ $GGLWLRQDOO\LWLVGHVLUDEOHWRKDYHDYHU\ODUJHVHWRISRWHQWLDOWHVWVIRU%DOOLVWDWRGUDZXSRQ ZLWKRXWKDYLQJWRPDQXDOO\FUHDWHWRRPDQ\WHVWV$SDUWLFXODUUHDVRQWRZDQWWRGRWKLVLVWR LPSOHPHQWUDQGRPL]HGWHVWLQJIRUWKHSXUSRVHVRIFKDUDFWHUL]DWLRQDVZHOODVDGGLQJDQDELOLW\ WRGLVFRYHUQHZKHXULVWLFDOO\XVHIXOGDWDW\SHYDOXHVIRUWHVWLQJ )RUWXQDWHO\DWHFKQLTXHWRJHQHULFL]HGDWDWHVWLQJLVDYDLODEOHLQWKHIRUPRIGHFRPSRVLQJWKH FRQVWUXFWRUVIRUDQ\SDUWLFXODUGDWDW\SHWHVWYDOXHFUHDWLRQ$QDQDORJ\FDQEHPDGHWRDVHWRI GLDOVWKDWDUHVHWLQDSDUWLFXODUSRVLWLRQIRUDSDUWLFXODUWHVWYDOXH7KHGLDOVDUHRUWKRJRQDO RUDW OHDVWORRVHO\FRXSOHGLQWHUPVRIPHDQLQJIXOVHWVRIYDOXHV DQGHDFKGLDOLVEDFNHGE\DVHWRI HQXPHUDWHGFRQVWUXFWRUVIRUSDUWLFXODUDVSHFWVRIDWHVWYDOXH 7KLVDSSURDFKLVVLPLODUWRWKH FDWHJRU\FKRLFH DSSURDFK GHVFULEHG LQ >@ EXW %DOOLVWD UHTXLUHV QHLWKHU LQWHUFDWHJRU\ FRQVWUDLQWVQRUVSHFLILFDOO\GHVFULEHGWHVWIUDPHV $VDQH[DPSOHVRPH RIWKHGLDOVIRUILOH FUHDWLRQVDQGWKHLUVHWWLQJVLQFOXGH )LOHH[LVWHQFH H[LVWVGRHVQRWH[LVWGHOHWHGDIWHUILOHKDQGOHREWDLQHG )LOHUHDGSHUPLVVLRQV UHDGSHUPLVVLRQUHDGSHUPLVVLRQWRRWKHUWKDQFXUUHQWWDVNQR UHDG SHUPLVVLRQ )LOHDFFHVV DOOFRPELQDWLRQVRIRSHQIRUUHDGZULWHUHDGZULWHHWF 1RWHWKDWWKHDERYHGLDOVDUHLQIOXHQFHGE\DGHVLUHWRGHFRXSOHYDULRXVDVSHFWVRIWKHGDWD YDOXHEXWUHPDLQFRXSOHGWRWKHH[WHQWUHTXLUHGE\FRQYHQLHQWO\ZULWLQJFRQVWUXFWRUV7KXVILOH SHUPLVVLRQVFDQEHGHFRXSOHGEHFDXVHWKH\FDQEHVHWLQGHSHQGHQWO\DIWHUILOHFUHDWLRQEXWILOH DFFHVVPRGHVDUHSODFHGLQWRDVLQJOHGLDOEHFDXVHLWZDVIRXQGWREHDZNZDUGWRPDQLSXODWH WKHPLQGHSHQGHQWO\,QSDUWLFXODUDSSOLFDWLRQVWKHILOHGDWDW\SHPLJKWEHH[SDQGHGWRLQFOXGH RWKHUGLDOVVXFKDVWKHW\SHRIGDWDSUHVHQWLQWKHILOH :KLOHWKHDGGLWLRQRIGLDOVVHHPVWRFRPSOLFDWHWKHWHVWLQJVLWXDWLRQLWWXUQVRXWWKDWWKHUHLVD
.RRSPDQ&6'$¶
XQLI\LQJDSSURDFKDYDLODEOHORJLFDOVWUXFWXUHV(YHU\GDWDW\SHWKDWLVGHFRPSRVHGLQWRGLDOV FDQ EH ORJLFDOO\ WUHDWHG DV D GDWD VWUXFWXUH IRU WHVWLQJ SXUSRVHV WKH RQO\ GLIIHUHQFH IURP D SK\VLFDOVWUXFWXUHSRLQWRIYLHZLVWKDWWKHUHLVQRWHVWIRUDSRLQWHUWRWKDWGDWDVWUXFWXUHVLQFHLW RQO\H[LVWVORJLFDOO\IRUWKHWHVWSURJUDPQRWSK\VLFDOO\ZLWKLQWKH0X7 6RWRFRQWLQXHWKH ILOH KDQGOH H[DPSOH WHVWLQJ ILOH KDQGOHV LV GRQH E\ GHFODULQJ D ILOH KDQGOH WR EH D ORJLFDO VWUXFWXUHFRQWDLQLQJDVHWRIGLDOVDQGWKHQH[SORULQJYDULRXVFRPELQDWLRQVRIGLDOVLQH[DFWO\ WKHVDPHZD\WKDW%DOOLVWDZRXOGWHVWFRPELQDWLRQVRIYDOXHVZLWKLQDSK\VLFDOGDWDVWUXFWXUH 3RWHQWLDOLPSURYHPHQWVLQFKDUDFWHUL]DWLRQ %H\RQG VLPSO\ PHDVXULQJ IDLOXUH UDWHV %DOOLVWD WHVWLQJ KDV WKH SRWHQWLDO WR FKDUDFWHUL]H IDLOXUHSDWWHUQVWKURXJKWKHXVHRISDWWHUQHGWHVWLQJRQILQHJUDLQGLDOEDVHGUHSUHVHQWDWLRQVRI GDWDYDOXHV3UHYLRXVZRUNLQWKHDUHDRIWHVWH[SORUDWLRQLQFOXGHVWKH$(7*V\VWHP> @ZKLFK SHUPLWVWHVWHUVWRVSHFLI\ZKLFKVHWVRISDUDPHWHUVLQDPXOWLSDUDPHWHUIXQFWLRQFDOOVDUHFORVHO\ FRXSOHGIXQFWLRQDOO\DQGWKXVVKRXOGEHWHVWHGLQFRQFHUW7KH72)8V\VWHP>@WDNHVWKLVD VWHSIXUWKHUE\DGGLQJLQWHUDFWLRQZHLJKWLQJVWRWKHLGHQWLILHGSDUDPHWHUWXSOHV &OHDUO\ WKH WHVWLQJ DSSURDFKHV IURP $(7* DQG 72)8 FDQ EH DSSOLHG LQ SDUW WR %DOOLVWD WHVWLQJLQWKHIRUPRIHQFRXUDJLQJWHVWLQJRIRQHRUDIHZSDUDPHWHUVDWDWLPH,WLVSUREDEO\ XQGHVLUDEOHWRKDYHDSURJUDPPHUJLYHH[SOLFLWKLQWVWR%DOOLVWDZLWKUHVSHFWWRFRXSOLQJ WKLV UHGXFHVVFDODELOLW\DQGWKHUHLVDOZD\VWKHFKDQFHWKDWWKHWHVWHUZLOOIDLOWRVHHDVXEWOHFRXSOLQJ DQGSURYLGHDQLQDFFXUDWHKLQW +RZHYHUKHXULVWLFVDUHEHLQJGHYHORSHGWRSHUPLW%DOOLVWDWR DXWRPDWLFDOO\GLVFRYHUZKDWFRXSOLQJVPD\EHSUHVHQWWRGRH[SORUDWLRQDURXQGJHQHUDOUHJLRQV RILQWHUHVWLQWKHERXQGDU\EHWZHHQWHVWVWKDWVXFFHHGDQGWKDWIDLODQGWRJHQHUDOO\GR³VPDUW´ WHVWLQJWRFKDUDFWHUL]HIDLOXUHVZLWKDUHDVRQDEOHQXPEHURIWHVWV 7KHFXUUHQWPHFKDQLVPIRUPDNLQJ%DOOLVWDDYDLODEOHLVDVDZHEEDVHGWHVWLQJVHUYLFHUDWKHU WKDQ DV D SLHFH RI VRIWZDUH WKDW LV H[HFXWHG ORFDOO\ 3DUW RI WKH UHDVRQ IRU WKLV VWUDWHJ\ LV WR HQDEOHGDWDFROOHFWLRQWKDWFDQEHXVHGWRLPSURYHWKHTXDOLW\RIFKDUDFWHUL]DWLRQDSSURDFKHV
3RWHQWLDODSSOLFDWLRQVWRIDXOWWROHUDQFH 7UDGLWLRQDO IDXOW WROHUDQFH WHFKQLTXHV DUH RIWHQ EDVHG RQ WKH QRWLRQ RI FUHDWLQJ GHSHQGDEOH V\VWHPV RXW RI LQGLYLGXDOO\ XQGHSHQGDEOH FRPSRQHQWV 7KH JHQHUDO LGHD LV WKDW LI LQGLYLGXDO FRPSRQHQW IDLOXUH UDWHV DUH NQRZQ WKHQ D YDULHW\ RI WHFKQLTXHV FDQ EH XVHG WR HQVXUH WKDW D FRPSRVLWHV\VWHPLVKLJKO\GHSHQGDEOH,WLVRIFRXUVHXQOLNHO\WKDW%DOOLVWDWHVWLQJZLOOSURGXFH KLJKO\DFFXUDWHUHOLDELOLW\HVWLPDWHVLQWHUPVRIIDLOXUHVSHUPLOOLRQRSHUDWLQJKRXUVDVLVGRQHIRU ILHOGHGKDUGZDUH1RQHWKHOHVVLWLVSUREDEO\DZRUWK\JRDOWRSURGXFHVRPHUHODWLYHPHDVXUHRI VRIWZDUH FRPSRQHQW UHOLDELOLW\ DQG D UDWLQJ RI H[FHSWLRQ KDQGOLQJ DELOLW\ PLJKW EH D XVHIXO VWDUWLQJSRLQW 7KHUH DUH IRXU DUHDV LQ ZKLFK D %DOOLVWD WHVWLQJ PLJKW SURYLGH LPSURYHG TXDQWLILFDWLRQ IRU IDXOWWROHUDQFHRIVRIWZDUHLQWHQVLYHV\VWHPV GHWHFWLQJV\VWHPFUDVKHVGHWHFWLQJYLRODWLRQVRI IDLOVWRSDVVXPSWLRQVSURYLGLQJLQIRUPDWLRQIRUFUHDWLQJVRIWZDUHZUDSSHUVDQGSURYLGLQJDQ DVVHVVPHQWRIVRIWZDUHGLYHUVLW\ ,Q JHQHUDO LW LV FRQVLGHUHG D JRRG LGHD WR EXLOG V\VWHPV WKDW DUH LPSRVVLEOH WR FUDVK IURP ³QRUPDO´VRIWZDUHDSSOLFDWLRQV7KLVLVDQH[WHQVLRQRIWKHWUDGLWLRQDOIDXOWWROHUDQFHDSSURDFK RIKDYLQJD³KDUGFRUH´WKDWLVDVVPDOODVSRVVLEOHDQGXSRQZKLFKUHVRXUFHVDUHODYLVKHGLQ ERWK GHVLJQ DQG LPSOHPHQWDWLRQ WR HQVXUH LWV GHSHQGDELOLW\ 2XWVLGH RI WKDW KDUG FRUH RI D SURFHVVRURURSHUDWLQJV\VWHPNHUQHOLWLVWREHH[SHFWHGWKDWEXJVZLOORFFXURQDUHJXODUEDVLV JHQHUDWLQJ H[FHSWLRQDO VLWXDWLRQV WKDW PXVW EH KDQGOHG UREXVWO\ ZLWK UHVSHFW WR WKH V\VWHP DUFKLWHFWXUH 7KH SDUWLFXODUDUHD RI HPSKDVLV IRU %DOOLVWD LV RQ H[HUFLVLQJ H[FHSWLRQ KDQGOLQJ
.RRSPDQ&6'$¶
WHFKQLTXHV VLQFH LW LV DVVXPHG WKDW QRUPDO FRUUHFW YDOXHV DUH ZHOO H[HUFLVHG E\ QRUPDO RSHUDWLRQDOWHVWLQJ 7HVWIRURYHUDOOV\VWHPFUDVKVLWXDWLRQV 7KHPRVWREYLRXVDSSOLFDWLRQRI%DOOLVWDWRIDXOWWROHUDQFHLVWRIHUUHWRXWZD\VWRFRPSOHWHO\ FUDVKDV\VWHPIURPXQSULYLOHJHGDSSOLFDWLRQVRIWZDUH$VWKHSUHYLRXVUHVXOWVLQGLFDWH%DOOLVWD FDQORFDWHVRPHZD\VWRFUDVKDQRSHUDWLQJV\VWHPHYHQZLWKWKHFXUUHQWUDWKHUVLPSOLVWLFWHVWV DYDLODEOH $SSOLFDWLRQ RI D VLPLODU PHWKRGRORJ\ VKRXOG KHOS TXDQWLI\ WKH H[WHQW WR ZKLFK H[FHSWLRQDOFRQGLWLRQVFDQWULFNOHGRZQWKURXJKDQDSSOLFDWLRQSURJUDPWRUHDFKWKHRSHUDWLQJ V\VWHPDVZHOO 7KHUH DUHIRXUDSSURDFKHV WKDW FDQLPSURYH WKHHIIHFWLYHQHVVRI%DOOLVWD WRLGHQWLI\ V\VWHP FUDVKHV7KHEDVLFWHFKQLTXHVUHTXLUHGDUHWKRVHGLVFXVVHGLQWKHSUHYLRXVVHFWLRQ 8VHILQHUJUDLQWHVWLQJXVLQJWKHGLDODSSURDFKWRGHFRPSRVLQJSDUDPHWHUWHVWYDOXHV 7HVW SULPLWLYH IXQFWLRQV ZLWKLQ WKH RSHUDWLQJ V\VWHP UDWKHU WKDQ MXVW WKH 326,;VSHFLILHG IXQFWLRQV 7HVWFRQFXUUHQWIXQFWLRQFDOOVWRLGHQWLI\WLPLQJGHSHQGHQWSUREOHPVVXFKDVUHVRXUFHORFN LQJEXJV 7HVWIRUWKHUHVXOWVRIUHVRXUFHH[KDXVWLRQHIIHFWVVXFKEHKDYLQJJUDFHIXOO\ZKHQWKHSURFHVV WDEOHLVIXOORUZKHQDPHPRU\OHDNLVHQFRXQWHUHG 7HVWIRUYLRODWLRQVRIIDLOVWRSDVVXPSWLRQ $ VHFRQGZD\ WRDSSO\ %DOOLVWD WR PHDVXULQJ DQG LPSURYLQJ )DXOW 7ROHUDQFH LV WR XVHLW WR GHWHFWSUREOHPVZLWKUHVSHFWWR5HVWDUWDQG6LOHQWIDLOXUHV,WLVFRPPRQWREDVHIDXOWWROHUDQFH RQDIDLOIDVWDVVXPSWLRQ+RZHYHUD5HVWDUWIDLOXUHPHDQVWKDWDWDVNLV³IDLOVORZ´LQWKDWLWV IDLOXUH LV QRW UHFRJQL]HG XQWLO W\SLFDOO\ D WLPHRXW SHULRG KDV HODSVHG IDLOXUH WR GHWHFW D SHULRGLF ³KHDUWEHDW´ PHVVDJH IDLOXUH WR UHVHW D ZDWFKGRJ WLPHU HWF )XUWKHUPRUH D 6LOHQW IDLOXUH PHDQV WKDW D V\VWHP KDV IDLOHG VRPHKRZ EXW WKDW WKH IDLOXUH LV XQGHWHFWHG IURP VRPH SHULRGRIWLPH,WPLJKWEHWKDWWKH6LOHQWIDLOXUHLVQRWDFWLYDWHGRULWPLJKWEHWKDWLWUHVXOWVLQD ³WLPH ERPE´ LQVLGH WKH V\VWHP VXFK DV D 18// SRLQWHU WR D GDWD VWUXFWXUH WKDW LV QRW GHUHIHUHQFHGH[FHSWLQDQHUURUKDQGOLQJURXWLQH 5HVWDUWIDLOXUHVDUHPRGHUDWHO\HDV\WRGHDOZLWKDQGEDVLFDOO\DUHVHDUFKHGIRULQWKHVDPH ZD\WKDWFDWDVWURSKLFV\VWHPIDLOXUHVDUHVHDUFKHGIRUDVGLVFXVVHGDERYH$SDUWLFXODUO\QDVW\ UHVWDUW IDLOXUH WKDW ZDV IRXQG RQ D UHDO V\VWHP ZDV LQ $,; LQ ZKLFK WKH FRUH GXPS IDFLOLW\ ZDLWHGXQWLOHQRXJKGLVNVSDFHRQDQHDUO\IXOOGLVNZDVDYDLODEOHWRGXPSFRUHHYHQLIFRUH GXPSLQJZDVGLVDEOHG WKHFKHFNIRUGLVNVSDFHFDPHEHIRUHWKHFKHFNIRUWKHPD[LPXPDPRXQW WRGXPSZKLFKLQRXUFDVHZDVVHWWR]HUR 6LOHQWIDLOXUHVFDQEHLGHQWLILHGE\XVLQJPXOWLYHUVLRQFRPSDULVRQDPRQJGLIIHUHQW26WHVW UHVXOWV7KLVVFKHPHGHWHUPLQHVGHFODUHVDUHVXOWWREHD6LOHQWIDLOXUHLIWKDWWHVWFDVHRQWKDW26 UHWXUQVQRHUURULQGLFDWLRQZKLOHWKDWVDPHWHVWFDVHUHVXOWVLQDQHUURULQGLFDWLRQRQDQRWKHU26 7KLVDSSURDFKZRUNVZLWKUHDVRQDEOHDFFXUDF\LQSUDFWLFH DSSUR[LPDWHO\DFFXUDWH >@ EXWRIFRXUVHUHTXLUHVWKDWWKHUHEHPXOWLSOHLPSOHPHQWDWLRQVDYDLODEOH,PSURYLQJSHUIRUPDQFH IRU6LOHQWIDLOXUHVWDNHVRQHRIWZRDSSURDFKHV,QVRPHFDVHVKDUGZDUHLVDYDLODEOHWKDWFRXOG GHWHFWD6LOHQWIDLOXUHEXWLVQRWEHLQJXVHG IRUH[DPSOHEHFDXVHPHPRU\SURWHFWLRQLVWXUQHG RII IRU SDJH ]HUR RI PHPRU\ WR VXSSRUW OHJDF\ VRIWZDUH ,Q RWKHU FDVHV LW LV VLPSO\ WKDW WKH H[FHSWLRQDO SDUDPHWHUV DUH QRW EHLQJ FKHFNHG DQG LQ DGGLWLRQ HYDGH GHWHFWLRQ E\ KDUGZDUH PHFKDQLVPV
.RRSPDQ&6'$¶
*HQHUDWLQJVRIWZDUHZUDSSHUV $ SRVVLEOH ZD\ WR LPSURYH VRIWZDUH IDXOW WROHUDQFH LV E\ XVLQJ VRIWZDUH ³ZUDSSHUV´ WKDW HQFDSVXODWHWKHPRGXOHVRIDVRIWZDUHV\VWHP:KLOHLWLVDHVWKHWLFDOO\PRUHSOHDVLQJWRIL[DQ\ H[FHSWLRQKDQGOLQJGHILFLHQFLHVLQDVRIWZDUHPRGXOHGLUHFWO\WKHUHDUHWLPHVZKHQWKDWLVQRW SRVVLEOH HLWKHU EHFDXVH RI ODFN RI DFFHVV WR WKH VRXUFH FRGH RU WLPHEXGJHW FRQVWUDLQWV 7KHUHIRUH LW PD\ EH DWWUDFWLYH WR DXWRPDWLFDOO\ RU VHPLDXWRPDWLFDOO\ JHQHUDWH VRIWZDUH ZUDSSHUV WKDW DUH LQWHUSRVHG EHWZHHQ WKH $3, LPSOHPHQWDWLRQ DQG WKH DSSOLFDWLRQ 7KHVH ZUDSSHUVFDQDXWRPDWLFDOO\FKHFNIRUH[FHSWLRQDOFRQGLWLRQVDQGWDNHDFWLRQWRUHSRUWWKHPRULQ VRPHFDVHVUHFRYHUIURPWKHP :KLOH D JHQHULF FDSDELOLW\ WR FUHDWH XQLYHUVDO VRIWZDUH ZUDSSHUV UHPDLQV D UHVHDUFK LVVXH WKHUHDUHVHYHUDOVWUDWHJLHVWKDWPD\EHHPSOR\HGZLWKLQDZUDSSHUVXFKDVUHSRUWLQJHUURUUHWXUQ FRGHV SHUIRUPLQJ UHWU\ DQG LQYRNLQJ DOWHUQDWH DOJRULWKPV $ SDUWLFXODUO\ LQWHUHVWLQJ XVH RI ZUDSSHUVLQFRQQHFWLRQZLWKUREXVWQHVVWHVWLQJLVWRXVHDUREXVWQHVVWHVWLQJUHVXOWDVDZDUQLQJ WKDWDSDUWLFXODURSHUDWLRQLVKD]DUGRXVDQGWRSHUIRUPDFKHFNSRLQWDJDLQVWWKHQHHGIRUDODWHU UROOEDFNLIWKDWRSHUDWLRQVKRXOGIDLO7KLVZRXOGXVHIDLOXUHSUHGLFWLRQVIURPUREXVWQHVVWHVWLQJ WR OHVVHQ WKH W\SLFDO FRVW RI FKHFNSRLQWLQJ WR WKRVH WLPHV ZKHQ LW ZRXOG DSSHDU WR EH PRVW IUXLWIXO 7KH ;HSW DSSURDFK >@ XVHV VRIWZDUH ³ZUDSSHUV´ DURXQG SURFHGXUH FDOOV DV D ZD\ WR HQFDSVXODWHHUURUFKHFNLQJDQGHUURUKDQGOLQJZLWKLQWKHFRQWH[WRIDUHDGDEOHSURJUDP*LYHQ WKDWWKHIDXOWWROHUDQFHFRPPXQLW\KDVIRXQGWKDWWUDQVLHQWV\VWHPIDLOXUHUDWHVIDURXWQXPEHU PDQLIHVWDWLRQVRISHUPDQHQWV\VWHPIDXOWVGHVLJQHUURUVLQSUDFWLFHHYHQDVVLPSOHDVWUDWHJ\DV UHWU\LQJ IDLOHG RSHUDWLRQV IURP ZLWKLQ D VRIWZDUH ZUDSSHU KDV WKH SRWHQWLDO WR VLJQLILFDQWO\ LPSURYHV\VWHPUREXVWQHVV2QHFRXOGWKHUHIRUHHQYLVLRQXVLQJWKHUHVXOWVRI%DOOLVWDWHVWLQJWR SURYLGHLQIRUPDWLRQIRUHUURUFKHFNLQJZLWKLQDQ;HSWVRIWZDUHZUDSSHU 3RWHQWLDOXVHDVPHDVXUHRIGLYHUVLW\IRUPXOWLYHUVLRQVRIWZDUHV\VWHPV $ PHWKRG IRU XVLQJ %DOOLVWD IRU )DXOW 7ROHUDQFH WKDW GRHV QRW GLUHFWO\ LQYROYH UREXVWQHVV WHVWLQJ SHU VH LV LQ WKH DUHD RI DVVHVVLQJ LPSOHPHQWDWLRQ GLYHUVLW\ IRU PXOWLYHUVLRQ V\VWHPV 2QH ZD\ WR LPSURYH GHSHQGDELOLW\ LV WR XVH PXOWLSOH LQGHSHQGHQWO\GHYHORSHG YHUVLRQV RI VRIWZDUHLQWKHKRSHWKDWGLIIHUHQWYHUVLRQVDUHGLYHUVHLQQDWXUHDQGWKXVKDYHGLIIHULQJIDLOXUH PRGHV >@>@ 7KLV LV D ZD\ RI H[SORLWLQJ LQKHUHQW GHVLJQ GLYHUVLW\ LQ VHSDUDWHO\ GHVLJQHG V\VWHPV LQ DQ DQDORJ\ WR H[SORLWLQJ GLYHUVLW\ DPRQJ PXOWLSOH LQGHSHQGHQWO\PDQXIDFWXUHG KDUGZDUHVXEV\VWHPV 2QHLVVXHZLWKXVLQJPXOWLYHUVLRQVRIWZDUHLVZKHWKHURUQRWVXFKVRIWZDUHLVWUXO\GLYHUVH 7KHUH DUH VWXGLHV DQG DUJXPHQWV IRU ERWK VLGHV RI WKLV LVVXH HJ >@>@>@>@>@>@ +RZHYHU %DOOLVWD WHVWLQJ JLYHV D ZD\ WR VKHG DGGLWLRQDO OLJKW RQWR WKH GLYHUVLW\ TXHVWLRQ ,Q SDUWLFXODULWFDQPHDVXUHWKHGLYHUVLW\RIPXOWLSOHVRIWZDUHYHUVLRQVZLWKUHVSHFWWRH[FHSWLRQ KDQGOLQJ FDSDELOLWLHV 'LYHUVLW\ ZLWK UHVSHFW WR FRUUHFWQHVV ZRXOG KLQJH RQ LPSOHPHQWLQJ VRIWZDUHDVVXUDQFHDELOLWLHVGLVFXVVHGODWHU ,WFDQGRVRE\FRPSDULQJWKHUHVXOWVRILGHQWLFDO WHVW FDVHV RQ PXOWLSOH LPSOHPHQWDWLRQV DQG GHWHFWLQJ LQVWDQFHV LQ ZKLFK DW OHDVW RQH YHUVLRQ GHWHFWHGDQH[FHSWLRQLQGLFDWLQJWKDWDQRWLRQDOPXOWLYHUVLRQV\VWHPFRXOGKDYHGHWHFWHGVXFK DQH[FHSWLRQDQGUHVSRQGHGJUDFHIXOO\HYHQLIDOOV\VWHPVGLGQRW7KLVLVDELWGLIIHUHQWWKDQ WUDGLWLRQDO PXOWLYHUVLRQ YRWLQJLQWKDWLW VXIILFHVIRU DVLQJOH YHUVLRQ RI VRIWZDUH WRGHWHFW DQ H[FHSWLRQ ZKHUHDV YRWLQJ W\SLFDOO\ UHTXLUHV D PDMRULW\ RI YHUVLRQV WR DJUHH RQ D FRUUHFW FRPSXWDWLRQDOUHVXOW 0XOWLYHUVLRQDVVHVVPHQWRIRSHUDWLQJV\VWHPVZDVSHUIRUPHGZLWK%DOOLVWDWHVWGDWD>@DQG IRXQGWKDW26NHUQHOFDOOVDUHPRGHUDWHO\GLYHUVH EXWQRWFRPSOHWHO\GLYHUVH +RZHYHULWZDV
.RRSPDQ&6'$¶
)LJXUH2YHUDOODGMXVWHGIDLOXUHUDWHVIRUWKLUWHHQ326,;RSHUDWLQJV\VWHPVXVLQJ PXOWLYHUVLRQIDLOXUHGHWHFWLRQFRPSDULVRQ DOVR IRXQG WKDW & OLEUDU\ IXQFWLRQ FDOOV GLVSOD\ D UHODWLYHO\ SRRU OHYHO RI GLYHUVLW\ )LJXUH VKRZVDVFDWWHUSORWRIWKHUHVXOWVDWWDLQHGE\XVLQJDOOFRPELQDWLRQVRIPXOWLYHUVLRQGHWHFWLRQ DPRQJWKLUWHHQRSHUDWLQJV\VWHPVDFURVVIXQFWLRQFDOOV>@$VFDQEHVHHQWKHRYHUDOO26 LQWHUIDFHFDQQRWEHLPSURYHGEH\RQGDIDLOXUHUDWHHYHQLIDOOWKLUWHHQ26YHUVLRQVDUHXVHG IRUFRPSDULVRQ+RZHYHULIWKHV\VWHPFDOOVXEVHWLVFRQVLGHUHGLQLVRODWLRQ LHWKRVHFDOOVQRW SDUW RI WKH & OLEUDU\ WKHQ WKH IDLOXUH UDWH LV UHGXFHG GUDPDWLFDOO\ EXW QRW HOLPLQDWHG FRPSOHWHO\ YLDFRPSDULVRQ7KLVVXJJHVWVWKDWDWOHDVWZLWKUHVSHFWWRH[FHSWLRQKDQGOLQJWKH &OLEUDU\IXQFWLRQVDUHQRWSDUWLFXODUO\GLYHUVH
3RWHQWLDOJHQHUDOL]DWLRQWR6RIWZDUH$VVXUDQFH 6RIWZDUHDVVXUDQFHUHVHDUFKKDVSURGXFHGPDQ\PHWULFV6RPHRIWKHVHPHWULFVDUHEDVHGRQ PHDVXULQJWKHFRPSOH[LW\RIWKHVRIWZDUHLWVHOI6RPHRIWKHPHWULFVDUHPRUHSURFHVVEDVHG VXFKDVVRIWZDUHVKLSSLQJGHFLVLRQKHXULVWLFV$QGVRPHRIWKHDSSURDFKHVDUHQRWPHWULFEDVHG EXWUDWKHUEDVHGRQDVVXULQJFRPSOHWHFRUUHFWQHVVVXFKDVZLWKIRUPDOPHWKRGV :LWKLQ WKH UHDOP RI VRIWZDUH DVVXUDQFH %DOOLVWD WHVWLQJ LV DNLQ WR DQ DXWRPDWHG EODFN ER[ WHVWLQJDSSURDFK6LQFHEODFNER[WHVWLQJ LVDERXW IXQFWLRQDOLW\WKHTXHVWLRQ EHFRPHVRQHRI KRZPXFKIXQFWLRQDOLW\FDQEHSXWLQWRWKHWHVWLQJZKLOHUHWDLQLQJVFDODELOLW\
.RRSPDQ&6'$¶
5REXVWQHVVPHWULFIRU&276VRIWZDUHFRPSRQHQWOLEUDULHV ,QDODUJHQXPEHURIVRIWZDUHV\VWHPVH[FHSWLRQKDQGOLQJLVGHDOWZLWKLQFRPSOHWHO\LQWKH VRIWZDUH VSHFLILFDWLRQ 7KLV LV GRQH LQWHQWLRQDOO\ DQG XVXDOO\ LQGLFDWHG E\ SKUDVHV VXFK DV D EHKDYLRUEHLQJ³XQGHILQHG´RU³XQVSHFLILHG´ ,QVRPHV\VWHPVUREXVWQHVVLVQRWDQRYHUULGLQJFRQFHUQEXWLQRWKHUV DQGLQSDUWLFXODURQHV LQ ZKLFK VHFXULW\ DQG IDXOW WROHUDQFH DUH VLJQLILFDQW FRQFHUQV LW PD\ EH LPSRUWDQW ,Q WKHVH ODWWHUFDVHV%DOOLVWDWHVWLQJFDQVHUYHWRH[HUFLVHH[FHSWLRQKDQGOLQJFRGHDVDOUHDG\GHVFULEHG DQG FUHDWH DPHWULF IRU H[FHSWLRQKDQGOLQJ DELOLW\ ,W VKRXOG EH QRWHG WKDW UREXVWQHVV IDLOXUHV IRXQGE\%DOOLVWDDUHQRWQHFHVVDULO\³EXJV´RUWKHUHVXOWVRI³GHIHFWV´EHFDXVHWKRVHWHUPVFDQ EHFRQVWUXHGWREHZLWKUHVSHFWWR WKHRULJLQDOVRIWZDUHVSHFLILFDWLRQ ZKLFKPD\LQGHHGVWDWH WKDWEHKDYLRULQDSDUWLFXODUVLWXDWLRQLV³XQGHILQHG´+RZHYHULIDJHQHUDOSXUSRVHVRIWZDUH FRPSRQHQWOLEUDU\LVEHLQJXVHGLQDFULWLFDOV\VWHPLWPD\EHDGYDQWDJHRXVWRVHOHFWDOLEUDU\ ZLWKWKHEHVWUREXVWQHVVSURSHUWLHVRUDWOHDVWKDYHVRPHPHDVXUHDYDLODEOHRIWKHUREXVWQHVVRI WKHOLEUDU\VHOHFWHG $TXDOLW\DVVXUDQFHPHWULFIRUV\VWHPVZLWKULJRURXVH[FHSWLRQKDQGOLQJ 6RPH V\VWHPV DUH VSHFLILFDOO\ GHVLJQHG WR EH UREXVW $Q H[DPSOH LV WKH HDUOLHU RQH RI WKH +/$57,ZKLFKLVGHVLJQHGWRUHSRUWHUURUFRGHVIRUDQ\SRVVLEOHH[FHSWLRQ7HVWLQJUHVXOWVWR GDWHKDYHIRXQGWKDWPRVWH[FHSWLRQVDUHLQIDFWUHSRUWHGEXWPRVWVRIWZDUHPRGXOHVGRKDYHD IHZH[FHSWLRQVWKDWJRXQUHSRUWHG$QGLQDQLQWHUHVWLQJUHVXOWDIHZPRGXOHVZHUHIRXQGWR KDYHDPXFKKLJKHUIDLOXUHUDWHWKDQWKHRWKHUPRGXOHVVXJJHVWLQJDUHDVRISDUWLFXODUFRQFHUQ ZLWKUHVSHFWWRV\VWHPUREXVWQHVV 7KXVDQDGGLWLRQDOXVHRI%DOOLVWDWHVWLQJIRUVRIWZDUHDVVXUDQFHLVD4XDOLW\$VVXUDQFHFKHFN RQVRIWZDUHWKDWLVVSHFLILFDOO\ZULWWHQWRLPSOHPHQWULJRURXVH[FHSWLRQKDQGOLQJ2QHZD\WR HPSOR\WKLVWHVWLQJLVWRJHQHUDWHEXJOLVWVWKDWPXVWEHIL[HG%XWHVSHFLDOO\LQFRVWVHQVLWLYH V\VWHPV LQ FDQ EH XVHG LQVWHDG WR ILQG WKH SDUWLFXODU PRGXOHV WKDW IRU VRPH UHDVRQ KDYH IDU ZRUVHWKDQDYHUDJHUREXVWQHVVDQGFRQFHQWUDWHDYDLODEOHUHVRXUFHVRQWKHPILUVW7KXV%DOOLVWD WHVWLQJ PLJKW EH FRQVLGHUHG D VFUHHQLQJ WRRO IRU ZKLFK DUHDV WR FRQFHQWUDWH FRUUHFWLYH HIIRUWV XSRQHYHQLILWGRHVQRWILQGDOOH[LVWLQJSUREOHPV $GDSWDWLRQWREODFNER[WHVWLQJZLWKFXVWRPL]HGFRUQHUYDOXHVHWF $SRWHQWLDOO\KLJKLPSDFWEXWDOVRYHU\GLIILFXOWH[WHQVLRQRI%DOOLVWDWHVWLQJZRXOGEHWRXVH LWWRSDUWLDOO\DXWRPDWHWUDGLWLRQDOVRIWZDUHWHVWLQJ ,QWUDGLWLRQDOEODFNER[WHVWLQJHYHU\PRGXOHLVWHVWHGZLWKUHVSHFWWRDVSHFLILFDWLRQ7KLV DSSURDFKWHQGVWRVFDOHSRRUO\EHFDXVHH[WUDZRUNPXVWEHSHUIRUPHGWRGHILQHDVSHFLILFDWLRQ IRUHYHU\PRGXOHWREHWHVWHGWKXVVFDOLQJHIIRUWOLQHDUO\ZLWKWKHVL]HRIWKH$3,WREHWHVWHG 7KHREYLRXVZD\WRDWWDLQVFDODELOLW\LVWRVRPHKRZJHQHULFL]HIXQFWLRQDOLW\VRWKDWLWEHFRPHV PRUHREMHFWRULHQWHG$QGLQIDFWWKDWDSSURDFKZDVWDNHQZLWKSUH%DOOLVWDUREXVWQHVVWHVWLQJ >@EXWGLGQRWDFKLHYHVFDODELOLW\,WLVSRVVLEOHKRZHYHUWKDWLQOLJKWRIZRUNRQ%DOOLVWDWKH JHQHULFIXQFWLRQDOLW\DSSURDFKFRXOGEHPDGHWRZRUN $QRWKHU DSSURDFK LV WR OLPLW FRUUHFWQHVV WHVWLQJ WR SURSHUWLHV WKDW FDQ EH UHSUHVHQWHG DV D OLQHDUVXSHUSRVLWLRQRIWHVWFDVHVIRULQGLYLGXDOSDUDPHWHUV,QWKLVDSSURDFKFRUUHFWQHVVWHVWV ZLWKSRVWFRQGLWLRQDVVHUWLRQVZRXOGEHDGGHGWR%DOOLVWDWHVWLQJDQGWKHSRVWFRQGLWLRQVZRXOG EHWHVWHGDIWHUHDFKWHVWZDVUXQ,WLVGLIILFXOWWRVHHKRZWKLVDSSURDFKFRXOGEHPDGHWRGHDO ZLWKFKDQJHVWRV\VWHPVWDWHFDXVHGE\DIXQFWLRQEHFDXVHWKHFKDQJHVZRXOGRIQHFHVVLW\EH UHODWHGWRIXQFWLRQDOLW\%XWWKHDVVHUWLRQVFRXOGWHVWIRULQYDULDQWVDFURVVFKDQJHVVXFKDVD
.RRSPDQ&6'$¶
VWULQJUHPDLQLQJQXOOWHUPLQDWHGDIWHUDQ\RSHUDWLRQ $ VLPSOHU OHVV DPELWLRXV H[WHQVLRQ ZRXOG EH WR LQFOXGHG FXVWRPL]HG WHVW YDOXHV IRU ERXQGDU\WHVWLQJIRUDSDUWLFXODUPRGXOHRUGDWDW\SHLQDGGLWLRQWRFXUUHQW%DOOLVWDUREXVWQHVV WHVWV )RU H[DPSOH D VWUXFW FRQWDLQLQJ DQ ;@$YL]LHQLV$7KH1YHUVLRQDSSURDFKWRIDXOWWROHUDQWVRIWZDUH,(((7UDQVDFWLRQVRQ6RIWZDUH (QJLQHHULQJ6( 'HF >@ $YL]LHQLV $ /\X 0 6FKXW] : §,Q 6HDUFK RI (IIHFWLYH 'LYHUVLW\ $ 6L[/DQJXDJH 6WXG\ RI )DXOW7ROHUDQW)OLJKW&RQWURO6RIWZDUH¨7KH(LJKWHHQWK,QWHUQDWLRQDO6\PSRVLXPRQ)DXOW7ROHUDQW &RPSXWLQJS >@ %DUWRQ - &]HFN ( 6HJDOO = 6LHZLRUHN ' )DXOW LQMHFWLRQ H[SHULPHQWV XVLQJ ),$7 ,((( 7UDQVDFWLRQVRQ&RPSXWHUV $SU >@ %ULOOLDQW 66 .QLJKW -& /HYHVRQ 1* §$QDO\VLV RI )DXOWV LQ DQ 19HUVLRQ 6RIWZDUH ([SHULPHQW¨,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ )HE >@%HL]HU%%ODFN%R[7HVWLQJ1HZ@%L\DQL5 36DQWKDQDP§72)8 7HVW2SWLPL]HUIRU)XQFWLRQDO8VDJH¨6RIWZDUH(QJLQHHULQJ 7HFKQLFDO%ULHI ,%07-:DWVRQ5HVHDUFK&HQWHU >@ &DUUHWWH * &5$6+0( 5DQGRP LQSXW WHVWLQJ QR IRUPDO SXEOLFDWLRQ DYDLODEOH KWWS SHRSOHGHOSKLFRPJMFFUDVKPHKWPODFFHVVHG-XO\ >@ &KHQ / $YL]LHQLV $ §19HUVLRQ 3URJUDPPLQJ $ )DXOW 7ROHUDQFH $SSURDFK WR 5HOLDELOLW\ RI 6RIWZDUH2SHUDWLRQ¨7KH(LJKWK,QWHUQDWLRQDO6\PSRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJS
.RRSPDQ&6'$¶
> @&RKHQ''DODO6)UHGPDQ0 3DWWRQ*§7KH$(7*6\VWHP DQDSSURDFKWRWHVWLQJEDVHG RQFRPELQDWRULDOGHVLJQ¨,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ -XO >@&ULVWLDQ)ODYLX([FHSWLRQ+DQGOLQJDQG7ROHUDQFHRI6RIWZDUH)DXOWV,Q 0LFKDHO5/\X (G 6RIWZDUH)DXOW7ROHUDQFH&KLFKHVWHU :LOH\S&K >@ 'DKPDQQ - )XMLPRWR 5 :HDWKHUO\ 5 §7KH 'HSDUWPHQW RI 'HIHQVH +LJK /HYHO $UFKLWHFWXUH¨ 3URFHHGLQJV RI WKH :LQWHU 6LPXODWLRQ &RQIHUHQFH :LQWHU &RQIHUHQFH %RDUG RI 'LUHFWRUV6DQ'LHJR&$S >@ 'H9DOH - .RRSPDQ 3 §&RPSDULQJ WKH 5REXVWQHVV RI 326,; 2SHUDWLQJ 6\VWHPV¨ WK 6\PSRVLXPRQ)DXOW7ROHUDQW&RPSXWLQJ )7&6 -XQH LQSUHVV >@ 'H9DOH - .RRSPDQ 3 *XWWHQGRUI ' §%DOOLVWD 6RIWZDUH 5REXVWQHVV 7HVWLQJ 6HUYLFH¨ 7HVWLQJ&RPSXWHU6RIWZDUHFRQIHUHQFH 7&6 -XQH LQSUHVV >@ 'LQJPDQ & 3RUWDEOH 5REXVWQHVV %HQFKPDUNV 3K' 7KHVLV (OHFWULFDO DQG &RPSXWHU (QJLQHHULQJ'HSDUWPHQW&DUQHJLH0HOORQ8QLYHUVLW\3LWWVEXUJK3$0D\ >@(FNKDUGW'( /HH/'§)XQGDPHQWDOGLIIHUHQFHVLQWKHUHOLDELOLW\RI1PRGXODUUHGXQGDQF\ DQG1YHUVLRQSURJUDPPLQJ¨-RXUQDORI6\VWHPVDQG6RIWZDUH 6HSW >@+DUW]0:DONHU( 0DKDU',QWURGXFWLRQWR6RIWZDUH5HOLDELOLW\ $6WDWHRIWKH$UW5HYLHZ 5HOLDELOLW\$QDO\VLV&HQWHUUHSRUW6:5(/5RPH1@,(((6WDQGDUGIRU,QIRUPDWLRQ7HFKQRORJ\3RUWDEOH2SHUDWLQJ6\VWHP,QWHUIDFH 326,; 3DUW 6\VWHP$SSOLFDWLRQ3URJUDP,QWHUIDFH $3, $PHQGPHQW 5HDOWLPH([WHQVLRQ>&/DQJXDJH@,((( 6WGE ,(((&RPSXWHU6RFLHW\ > @ -RQHV ( HG 7KH $SROOR /XQDU 6XUIDFH -RXUQDO $SROOR OXQDU ODQGLQJ HQWULHV DW WLPHV DQG 1DWLRQDO $HURQDXWLFV DQG 6SDFH $GPLQLVWUDWLRQ :DVKLQJWRQ '& >@ .DQDZDWL * .DQDZDWL 1 $EUDKDP - )(55$5, D WRRO IRU WKH YDOLGDWLRQ RI V\VWHP GHSHQGDELOLW\SURSHUWLHV ,(((:RUNVKRSRQ)DXOW7ROHUDQW3DUDOOHODQG'LVWULEXWHG6\VWHPV $PKHUVW0$86$S-XO\ >@ .QLJKW -& /HYHVRQ 1* §$ UHSO\ WR WKH FULWLFLVPV RI WKH .QLJKW DQG /HYHVRQ H[SHULPHQW¨ 6,*62)76RIWZDUH(QJLQHHULQJ1RWHV -DQ >@.RRSPDQ36XQJ-'LQJPDQ&6LHZLRUHN' 0DU]7&RPSDULQJ2SHUDWLQJ6\VWHPV 8VLQJ 5REXVWQHVV %HQFKPDUNV 3URFHHGLQJV 6\PSRVLXP RQ 5HOLDEOH DQG 'LVWULEXWHG 6\VWHPV 'XUKDP1&S 2FW >@ .URSS 1 .RRSPDQ 3 6LHZLRUHN ' $XWRPDWHG 5REXVWQHVV 7HVWLQJ RI 2IIWKH6KHOI 6RIWZDUH&RPSRQHQWVWK)DXOW7ROHUDQW&RPSXWLQJ6\PSRVLXPS -XQH >@/LRQV-/ FKDLUPDQ $ULDQH)OLJKW)DLOXUH UHSRUWE\WKHLQTXLU\ERDUG(XURSHDQ6SDFH $JHQF\3DULV-XO\ >@/\X05§6RIWZDUH5HOLDELOLW\0HDVXUHPHQWVLQ19HUVLRQ6RIWZDUH([HFXWLRQ(QYLURQPHQW¨ 3URFHHGLQJVRIWKH7KLUG,QWHUQDWLRQDO6\PSRVLXPRQ6RIWZDUH5HOLDELOLW\DQG(QJLQHHULQJS >@ /\X 0 HG +DQGERRN RI 6RIWZDUH 5HOLDELOLW\ (QJLQHHULQJ ,((( &RPSXWHU 6RFLHW\ 3UHVV 0F*UDZ+LOO >@0F&DEH7-6WUXFWXUHGWHVWLQJ $VRIWZDUHWHVWLQJPHWKRGRORJ\XVLQJWKHF\FORPDWLFFRPSOH[LW\ PHWULF1DW%XU6WDQG:DVKLQJWRQ'&86$'HF >@0F$OOLVWHU')6XQ&9RXN0$§5HOLDELOLW\RI9RWLQJLQ)DXOW7ROHUDQW6RIWZDUH6\VWHPV
.RRSPDQ&6'$¶
IRU6PDOO2XWSXW6SDFHV¨,(((7UDQVDFWLRQVRQ5HOLDELOLW\ 'HF > @ 0LOOHU %3 )UHGULNVHQ / 6R % §$Q HPSLULFDO VWXG\ RI WKH UHOLDELOLW\ RI 8QL[ XWLOLWLHV &RPPXQLFDWLRQVRIWKH$&0 'HF >@0LOOHU%.RVNL'/HH&0DJDQW\90XUWK\51DWDUDMDQ$ 6WHLGO-)X]]5HYLVLWHG $ 5HH[DPLQDWLRQ RI WKH 5HOLDELOLW\ RI 81,; 8WLOLWLHV DQG 6HUYLFHV &RPSXWHU 6FLHQFH 7HFKQLFDO 5HSRUW8QLYRI:LVFRQVLQ0DGLVRQ0D\ >@ 0XNKHUMHH $ 6LHZLRUHN '3 §0HDVXULQJ VRIWZDUH GHSHQGDELOLW\ E\ UREXVWQHVV EHQFKPDUNLQJ,(((7UDQVDFWLRQVRQ6RIWZDUH(QJLQHHULQJ -XQH >@1XPHJD%RXQGV&KHFNHUKWWS ZZZQXPHJDFRPSURGXFWVDHGYFVKWPODFFHVVHG >@ 2VWUDQG 7- %DOFHU 0- §7KH FDWHJRU\SDUWLWLRQ PHWKRG IRU VSHFLI\LQJ DQG JHQHUDWLQJ IXQFWLRQDOWHVWV¨&RPPXQLFDWLRQVRIWKH$&0 -XQH >@3XULI\KWWS ZZZSXUHDWULDFRPSURGXFWVSXULI\LQGH[KWPODFFHVVHG >@5DP3 5DQG'.§6DWDQ GRXEOHHGJHGVZRUG¨&RPSXWHU -XQH >@6SDQEDXHU6§3HQWLXP%XJ0HHWWKH,()ODZ¨3&:RUOGS)HE >@ 7VDL 7 5 ,\HU 0HDVXULQJ )DXOW 7ROHUDQFH ZLWK WKH )7$3( )DXOW ,QMHFWLRQ 7RRO 3URFHHGLQJV (LJKWK ,QWHUQDWLRQDO &RQIHUHQFH RQ 0RGHOLQJ 7HFKQLTXHV DQG 7RROV IRU &RPSXWHU 3HUIRUPDQFH(YDOXDWLRQ+HLGHOEHUJ*HUPDQ\6SULQJHU9HUODJS6HSW >@ 9R .3 :DQJ
