5/8/2016

ABasicIntroductionToNeuralNetworks

ABasicIntroductionToNeuralNetworks
WhatIsANeuralNetwork?
Thesimplestdefinitionofaneuralnetwork,moreproperlyreferredtoasan'artificial'neuralnetwork
(ANN),isprovidedbytheinventorofoneofthefirstneurocomputers,Dr.RobertHechtNielsen.He
definesaneuralnetworkas:
"...acomputingsystemmadeupofanumberofsimple,highlyinterconnectedprocessing
elements,whichprocessinformationbytheirdynamicstateresponsetoexternalinputs.
In"NeuralNetworkPrimer:PartI"byMaureenCaudill,AIExpert,Feb.1989
ANNsareprocessingdevices(algorithmsoractualhardware)thatarelooselymodeledaftertheneuronal
structureofthemamaliancerebralcortexbutonmuchsmallerscales.AlargeANNmighthavehundreds
orthousandsofprocessorunits,whereasamamalianbrainhasbillionsofneuronswithacorresponding
increaseinmagnitudeoftheiroverallinteractionandemergentbehavior.AlthoughANNresearchersare
generallynotconcernedwithwhethertheirnetworksaccuratelyresemblebiologicalsystems,somehave.
Forexample,researchershaveaccuratelysimulatedthefunctionoftheretinaandmodeledtheeyerather
well.
Althoughthemathematicsinvolvedwithneuralnetworkingisnotatrivialmatter,ausercanrathereasily
gainatleastanoperationalunderstandingoftheirstructureandfunction.

TheBasicsofNeuralNetworks
Neuralneworksaretypicallyorganizedinlayers.Layersaremadeupofanumberofinterconnected
'nodes'whichcontainan'activationfunction'.Patternsarepresentedtothenetworkviathe'inputlayer',
whichcommunicatestooneormore'hiddenlayers'wheretheactualprocessingisdoneviaasystemof
weighted'connections'.Thehiddenlayersthenlinktoan'outputlayer'wheretheanswerisoutputas
showninthegraphicbelow.

http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

1/6

5/8/2016

ABasicIntroductionToNeuralNetworks

MostANNscontainsomeformof'learningrule'whichmodifiestheweightsoftheconnections
accordingtotheinputpatternsthatitispresentedwith.Inasense,ANNslearnbyexampleasdotheir
biologicalcounterpartsachildlearnstorecognizedogsfromexamplesofdogs.
Althoughtherearemanydifferentkindsoflearningrulesusedbyneuralnetworks,thisdemonstrationis
concernedonlywithonethedeltarule.Thedeltaruleisoftenutilizedbythemostcommonclassof
ANNscalled'backpropagationalneuralnetworks'(BPNNs).Backpropagationisanabbreviationforthe
backwardspropagationoferror.
Withthedeltarule,aswithothertypesofbackpropagation,'learning'isasupervisedprocessthatoccurs
witheachcycleor'epoch'(i.e.eachtimethenetworkispresentedwithanewinputpattern)througha
forwardactivationflowofoutputs,andthebackwardserrorpropagationofweightadjustments.More
simply,whenaneuralnetworkisinitiallypresentedwithapatternitmakesarandom'guess'astowhatit
mightbe.Itthenseeshowfaritsanswerwasfromtheactualoneandmakesanappropriateadjustmentto
itsconnectionweights.Moregraphically,theprocesslookssomethinglikethis:

http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

2/6

5/8/2016

ABasicIntroductionToNeuralNetworks

Notealso,thatwithineachhiddenlayernodeisasigmoidalactivationfunctionwhichpolarizesnetwork
activityandhelpsittostablize.
Backpropagationperformsagradientdescentwithinthesolution'svectorspacetowardsa'global
minimum'alongthesteepestvectoroftheerrorsurface.Theglobalminimumisthattheoreticalsolution
withthelowestpossibleerror.Theerrorsurfaceitselfisahyperparaboloidbutisseldom'smooth'asis
depictedinthegraphicbelow.Indeed,inmostproblems,thesolutionspaceisquiteirregularwith
numerous'pits'and'hills'whichmaycausethenetworktosettledownina'localminum'whichisnotthe
bestoverallsolution.

http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

3/6

5/8/2016

ABasicIntroductionToNeuralNetworks

Sincethenatureoftheerrorspacecannotbeknownaprioi,neuralnetworkanalysisoftenrequiresa
largenumberofindividualrunstodeterminethebestsolution.Mostlearningruleshavebuiltin
mathematicaltermstoassistinthisprocesswhichcontrolthe'speed'(Betacoefficient)andthe
'momentum'ofthelearning.Thespeedoflearningisactuallytherateofconvergencebetweenthecurrent
solutionandtheglobalminimum.Momentumhelpsthenetworktoovercomeobstacles(localminima)in
theerrorsurfaceandsettledownatorneartheglobalminiumum.
Onceaneuralnetworkis'trained'toasatisfactorylevelitmaybeusedasananalyticaltoolonotherdata.
Todothis,theusernolongerspecifiesanytrainingrunsandinsteadallowsthenetworktoworkin
forwardpropagationmodeonly.Newinputsarepresentedtotheinputpatternwheretheyfilterintoand
areprocessedbythemiddlelayersasthoughtrainingweretakingplace,however,atthispointtheoutput
isretainedandnobackpropagationoccurs.Theoutputofaforwardpropagationrunisthepredicted
modelforthedatawhichcanthenbeusedforfurtheranalysisandinterpretation.
Itisalsopossibletoovertrainaneuralnetwork,whichmeansthatthenetworkhasbeentrainedexactly
torespondtoonlyonetypeofinputwhichismuchlikerotememorization.Ifthisshouldhappenthen
learningcannolongeroccurandthenetworkisreferedtoashavingbeen"grandmothered"inneural
networkjargon.Inrealworldapplicationsthissituationisnotveryusefulsinceonewouldneeda
separategrandmotherednetworkforeachnewkindofinput.

HowDoNeuralNetworksDifferFromConventionalComputing?
Tobetterunderstandartificialneuralcomputingitisimportanttoknowfirsthowaconventional'serial'
computerandit'ssoftwareprocessinformation.Aserialcomputerhasacentralprocessorthatcan
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

4/6

5/8/2016

ABasicIntroductionToNeuralNetworks

addressanarrayofmemorylocationswheredataandinstructionsarestored.Computationsaremadeby
theprocessorreadinganinstructionaswellasanydatatheinstructionrequiresfrommemoryaddresses,
theinstructionisthenexecutedandtheresultsaresavedinaspecifiedmemorylocationasrequired.Ina
serialsystem(andastandardparalleloneaswell)thecomputationalstepsaredeterministic,sequential
andlogical,andthestateofagivenvariablecanbetrackedfromoneoperationtoanother.
Incomparison,ANNsarenotsequentialornecessarilydeterministic.Therearenocomplexcentral
processors,rathertherearemanysimpleoneswhichgenerallydonothingmorethantaketheweighted
sumoftheirinputsfromotherprocessors.ANNsdonotexecuteprogramedinstructionstheyrespondin
parallel(eithersimulatedoractual)tothepatternofinputspresentedtoit.Therearealsonoseparate
memoryaddressesforstoringdata.Instead,informationiscontainedintheoverallactivation'state'ofthe
network.'Knowledge'isthusrepresentedbythenetworkitself,whichisquiteliterallymorethanthesum
ofitsindividualcomponents.

WhatApplicationsShouldNeuralNetworksBeUsedFor?
Neuralnetworksareuniversalapproximators,andtheyworkbestifthesystemyouareusingthemto
modelhasahightolerancetoerror.Onewouldthereforenotbeadvisedtouseaneuralnetworkto
balanceone'schequebook!Howevertheyworkverywellfor:
capturingassociationsordiscoveringregularitieswithinasetofpatterns
wherethevolume,numberofvariablesordiversityofthedataisverygreat
therelationshipsbetweenvariablesarevaguelyunderstoodor,
therelationshipsaredifficulttodescribeadequatelywithconventionalapproaches.

WhatAreTheirLimitations?
Therearemanyadvantagesandlimitationstoneuralnetworkanalysisandtodiscussthissubjectproperly
wewouldhavetolookateachindividualtypeofnetwork,whichisn'tnecessaryforthisgeneral
discussion.Inreferencetobackpropagationalnetworkshowever,therearesomespecificissuespotential
usersshouldbeawareof.
Backpropagationalneuralnetworks(andmanyothertypesofnetworks)areinasensetheultimate
'blackboxes'.Apartfromdefiningthegeneralarchetectureofanetworkandperhapsinitially
seedingitwitharandomnumbers,theuserhasnootherrolethantofeeditinputandwatchittrain
andawaittheoutput.Infact,ithasbeensaidthatwithbackpropagation,"youalmostdon'tknow
whatyou'redoing".Somesoftwarefreelyavailablesoftwarepackages(NevProp,bp,Mactivation)
doallowtheusertosamplethenetworks'progress'atregulartimeintervals,butthelearningitself
progressesonitsown.Thefinalproductofthisactivityisatrainednetworkthatprovidesno
equationsorcoefficientsdefiningarelationship(asinregression)beyondit'sowninternal
mathematics.Thenetwork'IS'thefinalequationoftherelationship.
Backpropagationalnetworksalsotendtobeslowertotrainthanothertypesofnetworksand
sometimesrequirethousandsofepochs.Ifrunonatrulyparallelcomputersystemthisissueisnot
reallyaproblem,butiftheBPNNisbeingsimulatedonastandardserialmachine(i.e.asingle
SPARC,MacorPC)trainingcantakesometime.ThisisbecausethemachinesCPUmustcompute
thefunctionofeachnodeandconnectionseparately,whichcanbeproblematicinverylarge
http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

5/6

5/8/2016

ABasicIntroductionToNeuralNetworks

networkswithalargeamountofdata.However,thespeedofmostcurrentmachinesissuchthat
thisistypicallynotmuchofanissue.

WhatAreTheirAdvantagesOverConventionalTechniques?
Dependingonthenatureoftheapplicationandthestrengthoftheinternaldatapatternsyoucangenerally
expectanetworktotrainquitewell.Thisappliestoproblemswheretherelationshipsmaybequite
dynamicornonlinear.ANNsprovideananalyticalalternativetoconventionaltechniqueswhichare
oftenlimitedbystrictassumptionsofnormality,linearity,variableindependenceetc.BecauseanANN
cancapturemanykindsofrelationshipsitallowstheusertoquicklyandrelativelyeasilymodel
phenomenawhichotherwisemayhavebeenverydifficultorimposibletoexplainotherwise.
ForfurtherinformationonneuralnetworksontheWWW.
BackToMainPage
Seeanexampleofaneuralnetwork.

http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html

6/6