Di

nli
n-di
nli
ncsc413summer
y
Ov er viewofPer formanceEv aluat ion
Comput ersy stems per formance ev aluation def ines standar d met hods f ordesi gni
ng and
conduct ingper f
ormancest udies.Ther ei sawhol ef i
eldofcomput ersci encecal l
edcomput er
syst emsper f ormanceev aluat iont hati sdev otedt oexact lythis.
met hodsusedi nt hedesi gnofper formanceev al uationst udies
1.Anal yt i
cappr oaches:t heuseofmat hemat i
cs,Mar kovchai ns,queuei ngt heor y,Pet r
iNet s,
abst ractmodel s…
2.Si mul at i
onappr oaches:desi gnanduseofcomput ersi mul at ionsandsi mpl ifi
edmodel st o
assessper for mance
3.Exper iment alappr oaches: measur ementanduseofar ealsy st em
Exampl es
Anal y ti
cal Exampl e:Queuei ngTheor y
Queuei ngt heor y:isamat hemat icalt echniquet hatspeci ali
zesi nt heanal ysisofqueues( e.g.,
cust omerar rivalsatabank, jobsar rivingatCPU, et c) .
Gener al diagr am:
char act er i
sticsofaqueuei ngsy st em
Arrival pr ocess( M, G) ii
i.cNumberofser ver s(1t oinfi
ni ty
Ser vicet imepr ocess( M, D, G) iv.Numberofbuf fers( infini
teorf inite)
Exampl enot ation: M/M/ 1, M/ D/ 1
si mpl equeuei ngmodel s:
Ther ear ewel l-
known mat hemat icalr esultsf ort hemean wai ti
ng t imeand t henumberof
cust omer sint hesy stem f orsev er al simpl equeuei ngmodel s.E.g., M/ M/ 1,M/ D/ 1, M/ G/1
-Thesesi mpl emodel scanbecascadedi nser iesandi npar all
elt ocr eatear bit
rari
lyl arge
compl icat edqueuei ngnet wor kmodel s
Ty pesofqueuei ngnet wor kmodel s
1.cl osedqueuei ngnet wor kmodel (finitepop. ).
2.openqueuei ngnet wor kmodel (
inf initepop. ) .
Simul at ionExampl e:TCPThr oughput
-Canuseanexi sti
ngsi mul ationt ool , ordesi gnandbui l
dy ourowncust om simul ator:Exampl e:n-
2net wor ksimul ator
-Adi scr ete-ev entsi mul atorwi thdet ai ledTCPpr ot ocol model s
-Conf i
gur enet workt opol ogyandwor kl oad
OTHERSI MULATI ONI SSUES
1.Si mul ationr unl engt h:choosi ngal ongenoughr unt i
met ogetst atisti
call
ymeani ngfulresults
(equi li
br ium) .
2.Si mul ationst art-upef fect sandendef f
ect s:
deci dinghowmucht o“ chopof f ”att hest artand
endofsi mul ationst ogetpr operr esul t s.
3.Repl ications:ensur er epeat abi lityofr esul t
s,andgai ngr eaterst at i
sti
calconf idencei nt he
resul tsgi ven.
Exper iment alExampl e:Benchmar ki ng
Thedesi gnofaper formancest udyr equi resgr eatcar einexper i
ment al designandmet hodology .
-exper iment al factorstobet ested.
-l
ev els( set ti
ngs)f orthesef act or s.
-perfor mancemet ri
cst obeused.
-exper iment al designt obeused.
FACTORS
Fact orsar ethemai n“component s”t hatar evar i
edi nanexper iment
LEVELS
Lev elsar et hepr eciseset tingsoft hef actorst hatar et obeusedi nanexper i
ment
Exampl e1: reqsi zeS=1KB, 10KB, 1MB.
Exampl e2: num cl i
entsC=10, 20, 30, 40, 50.
PERFORMANCEMETRI CS
Per formancemet r
icsspeci fywhaty ouwantt omeasur ei nyourper formancest udy .Mustchoose

~st
il
lsmar
t,t
hat
'
showwer
oll
yourmet r
icspr operl
yandi nstr
umenty ourexper i
mentaccordingly
Exampl es: responsetime, throughput,
pktl oss.
EXPERI MENTALDESI GN Exper i
mentaldesi gnr ef
erst otheor gani
zationalstr
uct
ureofyour
experiment .Ithassev eralapproaches
EXAMPLES
Onef actoratat i
me:v aryonl yonef act orthroughi t
slevelst oseewhatt heimpacti
son
performance.
Twof actorsatat i
me:v arytwof actorst oseenotonl yt
hei rindivi
dualef f
ect
s,butal
sothei
r
i
nteractionef fects,
ifany .
Fullfactorial:t
ryeverypossi blecombinationoff actorsandl
ev elstoseef ullr
angeof
performancer esult
s.
ENDOFCHAPTER!
!!
Comput erperformanceev al
uat i
on
Comput erper f
ormanceev aluat ion: i
sofv it
alimpor tancei n
thesel ect i
onofcomput ersy stems, thedesi gnof
appl icationsandequi pment ,andanal ysisofexisting
syst ems
Pur poseofEv aluation
Ther ear et hreegener al purposesofper f
ormanceev aluation:
Sel ect ionev aluation-sy stem exi stsel sewhere.
Per formancepr oject i
on-sy stem doesnoty et
exist .
Per formancemoni t
or i
ng-sy stem i noper at
ion
Appl icat i
onsofPer formanceEv aluat i
on
1.Pr ocur ement 3.Capaci typl anni ng
2.Sy stem upgr ade 4.Sy st em desi gn
St epsf oraPer formanceEv aluationSt udy
1.St at ethegoal sanddef inet hesy stem boundar i
es. 6.Sel ectev aluationtechni ques.
2.Li stsy st em servicesandpossi bleout comes. 7.Sel ectt hewor kload.
3.Sel ectper formancemet rics. 8.Desi gnt heexper i
ment s.
4.Li stsy st em andwor kloadpar amet er s. 9.Anal yzeandi nterpretthedat a.
5.Sel ectf actorsandt hei rvalues. 10.Pr esentt her esults.Startov er,
ifnecessar y
.
Per formanceEv al uationMet ri
cs
Differ entper f
ormancemet ricsar euseddependi ngupont heser vi
cesof f
eredbyt hesy st
em:
1.cor rect l
yper formed
•tur nar oundt ime-t het imebet weent hesubmi ssi onofabat chj obandt hecompl et i
onofits
output
•responset ime-t hei nterval bet weenauser '
srequestandt hesy stem response
•thr oughput( orproduct i
vit
y )-t her ate( requestsperuni ttime)atwhi chr equestsar e
serv icedbyt hesy stem
•ut il
izat ionofar esour ce-i sameasur edast hef r
act i
onoft imet her esourcei sbusyser vi
cing
request s
2.incor rect l
yper f
or med
•reliabi l
ity-measur edbyt hepr obabi l
i
t yoferrorsort hemeant imebet weener r
ors
3.unabl et odo
•av ai l
abi li
ty-t hefractionoft ot altimedur i
ngwhi cht hesy stem i satt hedi sposal oftheusers
Char acter ist
icsofPer formancemet ri
cs
•Hi gheri sBet terorHB
•Loweri sBet t
erorLB
•Nomi nal isBestorNB

~st
il
lsmar
t,t
hat
'
showwer
oll
 Cost/perf
ormanceRatio
Acommonl yusedmet ri
cforcompar ingtwoormor esyst
emsi nsy st
em
procurementappl i
cati
on
•costincludesh/wands/ wlicensing,i
nstal
lat
ionandmaintenancecosts
•performancei smeasuredintermsoft hroughputunderagivenresponse
ti
meconst r
ain
ResponseTime
ResponseTi me: defi
nedasthet i
mei nt
ervalbetweenthei
nstanttheinput
ti
ngcommandt
o
aninteracti
vesystem ter
m

Throughput
Throughput :defi
nedast herat
e(request
speruni tti
me)atwhi chrequest
sar eservi
cedbyt
he
system.exampl es:
1.bat chsystems-jobspersecond; 3.CPUs-MI PSorMFLOPS;
2.interact
ivesystems-request
spersecond; 4.net works-packetspersecond(pps)
Capacit
yofaSy stem
•Nomi nalcapaci ty :maxi mum achi ev ablet hr oughputunderi dealwor kloadcondi t
ions(e.
g.,
bandwi dth( inbps)ofan/ w)
•Usabl e capaci ty:maxi mum t hroughputachi ev able wi thoutexceedi ng a pr especifi
ed
responset imel i
mi t
•Kneecapaci ty:thr oughputatt heknee( pointbey ondwhi chr esponset i
mei ncreasesrapidl
y
wi t
hloadbuti ncr easei nthroughputi sssmal l
Perfor manceEv aluat i
onSt udyExampl es
1.Sy st em Def inition:Thegoaloft hecasest udyi st ocompar etheper formanceofdat a
transferusi ngr emot epi pestot hoseofdat at ransf erusi ngr emot eprocedur ecal l
s.
2.Ser vices:Theser vicesof f
er edbyt hesy stem ar et het wot ypesofchannelcal ls-remote
procedur ecal l andr emot epipe.
3.Met rics:Ther esour cesar et hel ocalcomput er( client )
, t
her emot ecomput er(server)
,and
thenet wor kl i
nk.
4.Wor kload:Thewor kloadwi llconsi stofasy nt het i
cpr ogram gener atingt hespeci f
iedtypes
ofchannel request s.
5.Exper iment alDesi gn:Af ullf act orialexper i
ment aldesi gnwi th2^3x11=88exper i
ments
wi l
lbeusedf ort hei ni ti
alstudy .
6.Dat aPr esent ation: Thef i
nal resul tswi l
l bepl ottedasaf unct i
onoft hebl ocksi zen.
CommonMi stakesi nPer formanceEv aluati
on
1.Isthesy stem cor rectlydefinedandt hegoal scl earlyst ated?
2.Ar ethegoal sst atedi nanunbi asedmanner ?
3.Hav eal lthest epsoft heanal y sisf ollowedsy stemat ically?
4.Isthepr obl em cl ear lyunder stoodbef or eanal y zingi t?
5.Ar etheper for mancemet ri
csr elev antf ort hispr obl em?
6.Isthewor kloadcor rectfort hispr oblem?
7.Istheev aluat iont echni queappr opr iate?
8.Isthel i
stofpar amet ersthataf fectper formancecompl ete?

~st
il
lsmar
t,t
hat
'
showwer
oll
9.Haveallpar amet erst hataf fectper f
ormancebeenchosenasf act
orstobevar
ied?
10.I
stheexper iment aldesi gnef ficientintermsoft i
meandr esults?
11.I
sthelev el ofdet ail proper ?
12.I
sthemeasur eddat apr esent edwi t
hanal ysi
sandi nterpretat
ion?
13.I
stheanal ysisst atisti
cal l
ycor rect?
14.Hasthesensi t i
vityanal ysisbeendone?
15.Woulder rorsi nt hei nputcauseani nsignif
icantchangei nt heresul
ts?
16.Havetheout liersi nthei nputandout putbeent reatedpr operly?
17.Havethef ut urechangesi nthesy stem andwor kl
oadbeenmodel ed?
18.Havethev arianceofi nputbeent akenintoaccount ?
19.Havethev arianceoft her esultsbeenanal yzed?
20.I
stheanal ysiseasyt oexpl ain?
21.I
sthepr esent ati
onst ylesui t
abl ef orit
saudi ence?
22.Havether esul t
sbeenpr esent edgr aphical
lyasmuchaspossi bl
e?
23.Aret
heassumpt i
onsandl i
mi tati
onsoft heanalysisclearlydocumented?

24.HOPEYOU'
LLNOTMAKEAMI
STAKEOFMEMORI SINGALLTHEABOVEMI
STAKES?
Justdeyj
oke,
,
,,
,,
hhhhh

Goodl
uck

~st
il
lsmar
t,t
hat
'
showwer
oll