Supplementarymaterial:Thesocialperceptualsalience Effect 1

SupplementaryMaterial:TheSocialPerceptualSalience Effect 1

Runninghead: SUPPLEMENTARYMATERIAL:THESOCIALPERCEPTUAL SALIENCEEFFECT

SupplementaryMaterial:TheSocialPerceptualSalience Effect

MartinP.Inderbitzin1,AlbertoBetella1,AntonioLanatà2,EnzoP.Scilingo2,UlyssesBernardet1,PaulF.M.J.Verschure1,3

1 Laboratory forSynthetic, PerceptiveandEmotiveSystems,TechnologyDepartment, Universitat PompeuFabra, Barcelona,Spain.

2 InterdepartmentalResearchCenter E.Piaggio, FacultyofEngineering,Universityof

Pisa, Italy.

3 CatalanInstituteforResearchandAdvancedStudies(ICREA),Barcelona,Spain.

March7,2012

AuthorNote

Correspondenceconcerningthispapershould beaddressed to:

PaulF.M.J.Verschure,Synthetic, Perceptive,Emotive,CognitiveSystemsGroup

Universitat PompeuFabra, RocBoronat 138,08018 Barcelona,Spain. E-mail:,Phone:(0034)935421372

SupplementaryMaterial:TheSocialPerceptualSalience Effect 2

SupplementaryMaterial:TheSocialPerceptualSalienceEffect

PreprocessingofPhysiologicalData

Therawdata wascollected frombiosensors andpre-processed.Each signalwas segmentedaccordingtothetimedurationofthestimulatingepochs.Themethodsfordataanalysis ofthephysiological signals arebasedonpreviouslypublished algorithms. Detaileddescriptionscanbefoundin(Valenza, Lanata,Scilingo,2011).

HeartRate Variability(HRV)

Electrocardiogram(ECG)waspre-filtered through aMovingAverage Filter(MAF)inordertoextract andsubtractthebaseline.Since HRVrefers tothechangeovertimeof

theHeartRate (HR).Weadaptedanautomaticalgorithmtodetect theQ-,R-andS-wave formsoftheECGsignal(QRScomplex)(PanTompkins, 1985).

ThetimeintervalbetweentwosuccessiveQRScomplexesisdefined astheR-wave

toR-wave (RR)interval(tR−R).Thereaftertheheartrate(HR)isdefined as:

HR=

60

tR−R

(1)

BecausetheHRisatimeseriessequenceofnon-uniformRRintervals,were-sampledthesignalusingthealgorithmofBerger etal.(2007)

Respiration(RSP)

Weidentifiedthebaseline andremovedmovementartifacts. Additionallywefilteredthesignalusingatenthorderlow-passfiniteimpulse responsefilter(FIR)withacut-offfrequencyof1HzapproximatedbyButterworthpolynomial.

SupplementaryMaterial:TheSocialPerceptualSalience Effect 3

ElectrodermalResponse (EDR)

TheEDRsignalwasfilteredusinga2.5Hzlow-passFIRfilter.Becauseithas beenshownthattheenergy ofthetoniccomponentisinthefrequencyband from0to0.05Hzandtheenergyofthephasiccomponentintheband from0.05to1-2Hz(IshchenkoShev’ev,1989),weappliedatwelveleveldecompositionwavelet filterinordertoidentifytwomainresponsecomponentsinthesebands.Approximationatlevel1ofthefilterwasthetoniccomponentandsubsequentdetailswerethephasiccomponent.

StandardFeatureSetIdentification

Wecalculatedallthefeaturesforeachneutral aswellasforeachstimulationsession.Weused43standard featuresand8featuresextractedusingnon-linear dynamicmethods whicharedescribedinthenextsection.Thestandard featuresetwasderived fromthefollowingcomponents ofthesignal: timeseries, statistics,frequencydomain andgeometricanalysis.

HeartRateVariability(HRV)HRVfeaturesweredecomposedinfeaturesdescribing boththetimeandfrequencydomain. Definingthebeat-to-beat timewindow(NN),wecalculatedthemeanoftheNN(MNN)andthestandard deviationoftheNN(SDNN).Additionallytherootmeansquareofsuccessivedifferencesofintervals (RMSSD)andthenumberofsuccessivedifferencesofintervalswhichdifferbymorethan50ms(pNN50%)wascalculated.

Thetriangular index,thatrefers tothemorphological changesoftheHRV,wasderivedfromthehistogram ofRRintervals byperformingatriangular interpolation overtheNNwindows(TINN).

Webasedallfeaturesextractedinthefrequencydomain onthePower Spectral

Density(PSD)oftheHRV.Three mainspectralcomponentsweredistinguishedina

SupplementaryMaterial:TheSocialPerceptualSalience Effect 4

spectrumcalculatedfromshort-term recordings:VeryLowFrequency(VLF),LowFrequency(LF),andHighFrequency(HF)components.WeadditionallycalculatedtheLF/HFRatiowhichshouldgiveinformationabout theSympatho-Vagalbalance(Camm etal.,1996).

Respiration(RSP)BydefiningatimewindowWtherespiration rate(RSPR) was calculatedasthefrequencycorrespondingtothemaximumspectralmagnitude.We identifiedthemaximum(MAXRSP)andtheminimum(MINRSP)valueofbreathingamplitudeandtheirdifference(DMMRSP)tocharacterizethedifferencesbetweeninspiratoryandexpiratoryphase(range orgreatestbreath).

ElectrodermalResponse(EDR) Weapplied thesamestandardmethodsasusedforthe

RSP signalprocessing toidentifyboththetonicandthephasic EDRthecentralfrequency,meanandstandard deviationoftheamplitude. Additionallywecalculatedthemaximumpeak andtherelativelatency fromthebeginning oftheinteraction phase,frequency(rate)andmagnitude(max)ofthemaximumcomponentofthephasicEDR.

Non-LinearDynamicMethods forFeatureExtraction

Thehere documented non-linear dynamic methods forfeatureextraction arebasedonthestudyofValenza etal.(2011).

Webasedouranalysis ontheso-calledembeddingprocedure.Embeddingofatimeseriesxt=(x1,x2,..., xN)isrealized bycreating asetofvectorsXi such that

Xi=[xi,xi+4,xi+24,..., xi+(m−1)4](2)

where4isthedelayinnumberofsamplesandmisthenumberofsamplesofthearray

Xi.InorderthatthevectorXi representsthevaluesthatrevealthetopological

SupplementaryMaterial:TheSocialPerceptualSalience Effect 5

relationshipbetweensubsequent pointsinthetimeseries, wemustdefinethedimensionofmandXi andthedelay∆.Wecanrepresentthetemporal evolutionofthesystem byprojecting thevectorsXi ontoatrajectorythrough amultidimensionalspace,oftenreferred toasphasespaceorstatespace.TheRecurrence Plot(RP)visualizesalltimesatwhichastateofthedynamicalsystem recurs (Marwan,CarmenRomano, Thiel,Kurths,2007). Higherdimensionalphasespacescanbevisualized byprojecting themintotwoorthree dimensionalsub-spaces(Eckmann,Kamphorst,Ruelle, 1987). When

astateattimeirecurs alsoattimej,theelement (i,j)ofasquaredmatrixNxN issetto

1,0otherwise.Thisrepresentation iscalledrecurrence plot(RP).Wecanmathematically

expresssuch anRPas

Ri,j=Θ(i−||xi−xj||)

wherexi Rm,i,j=1,....,N;Nisthenumberofconsideredstatesxi,εiisathreshold distance,||.||anormandΘ(.)theHeaviside functionwhichisdefined as:



1,ifz≥0

H(z)=

0,ifz0

(3)

Wechosetheoptimalvalueofεi(Schinkel, Dimigen,Marwan,2008)asfollowing:

i=0.1∗APD(4)

whereAPD isaveragedphasespacediameterofdata xi.

Toquantifythenumberanddurationofrecurrences ofadynamical systempresentedbyitsstatespacetrajectorytheRecurrence Quantification Analysis(RQA)canbeapplied (Zbilut &WebberJr,2006). Inthisstudywecalculatedthefollowingfeatures:

RecurrenceRate (RR)isthepercentageofrecurrence pointsinanRPandit corresponds tothecorrelationsum:

RR=

N

XRi,j

N2

i,j=1

(5)

SupplementaryMaterial:TheSocialPerceptualSalience Effect 6

whereNisthenumberofpointsonthephasespacetrajectory.

Thedeterminism(DET)isdefined asthepercentageofrecurrence pointswhichformdiagonal lines:

DET =

N

PlP(l)

l=lmin

N

PRi,j

i,j=1

(6)

whereP(l)isthehistogram ofthelengths lofthediagonal lines.

Laminarity(LAM)isthepercentageofrecurrence pointswhichformverticallines:

LAM=

N

PυP(υ)

υ=υmin

N

PυP(υ)

υ=1

(7)

whereP(υ)isthehistogram ofthelengths υofthediagonal lines.

Trapping TimeTTistheaveragelengthoftheverticallines:

TT=

N

PυP(υ)

υ=υmin

N

PP(υ)

υ=υmin

(8)

Ratio(RATIO)istheratiobetweenDET andRR:

RATIO=

DET

RR

(9)

Averaged diagonal linelength(L)istheaveragelengthofthediagonal lines:

N

PlP(l)

L= l=lmin

PP(l)

l=lmin

(10)

Entropy(ENTR)istheShannon entropyoftheprobabilitydistributionofthe

diagonal linelengths p(l):

N

ENTR=−Xp(l)lnp(l)(11)

l=lmin

SupplementaryMaterial:TheSocialPerceptualSalience Effect 7

Longest diagonal line(Lmax) Thelengthofthelongest diagonal line:

Lmax=max({li;i=1,..., Nl})(12)

whereNl isthenumberofdiagonal linesintherecurrenceplot.

Ithas beenshownthatApproximate Entropy (ApEn)canbeusedtomeasurethe complexityorirregularityofthesignal(Fusheng, Bo,Qingyu,2000;Richman Moorman,2000). SmallvaluesofApEnindicate amoreregular signal,lagervaluesahighirregularone.

TocomputetheApENfirstasetoflengthmvectorsuj isformed:

uj=(RRj,RRj+1,..., RRj+m−1),(13)

wherej=1,2,..., N−m+1,mistheembeddingdimension,andNisthenumberof measuredRRintervals. Themaximumabsolutedifferencebetweenthecorresponding elementsdefines thedistancebetweenthesevectors:

d(uj,uk)=max

n=0,...,m−1

{|RRj+n−RRk+n|}(14)

Foreachuj therelativenumberofvectorsuk forwhichd(uj,uk)≤riscalculated.r

isthetolerancevalue. Theindexisdenoted withCm(r)andcanbewrittenintheform:

j (r)=

nbrof{uk|d(uj,uk)≤r}

N−m+1∀k

(15)

Duetothenormalization,thevalueofCm(r)issmaller orequal to1.ThevalueofCm(r)

jj

isatleast 1/(N−m+1)since uj isalsoincluded inthecount. Theaveragednatural

logarithmofeachCm(r)yieldsto:

Φm(r)=1

N−m+1

N−m+1

X

j=1

lnCm(r).(16)

Theapproximateentropyfinallycanbecalculatedas:

ApEn(m,r,N)=Φm(r)−Φm+1(r)(17)

SupplementaryMaterial:TheSocialPerceptualSalience Effect 8

Three parametersareinfluencingthevalueofApEn:thelengthmofthevectorsuj,thetolerancer,andthedata lengthN.Inthisworkwehavechosenm=2.ThelengthNofthedata alsoaffectsApEn.AsNincreasestheApEnapproachesitsasymptoticvalue.Thetolerancerhas astrongeffectonApEnandshould beselectedasafractionoftheStandard DeviationofallNormal-to-Normal (SDNN)intervals,i.e.thestandard deviationoftheintervals betweensuccessivenormalQRScomplexes.Acommon selectionforrisr=0.2·SDNN,whichisalsousedhere.

Featurereductionstrategy

Weobtainedahigh-dimensionalfeaturespace,thatwereduced byapplyingthePrincipalComponentAnalysis(PCA)method (Jolliffe,2002). Weimplementedthisapproach bymeansoftheSingular ValueDecomposition(SVD).Each trainingsetvectorcanbeapproximatedbytakingonlythefirstfewk,where,k≤r,PrincipalComponents.Thismathematicalmethod isbasedonthe lineartransformationofthedifferentvariablesinprincipalcomponents whichcouldbeassembledinclusters.

Classification

ForclassificationaNearest MeanClassifier (NMC))isused.Thisclassifier usesthe similaritybetweenpatternstodecide onagoodclassification.Thequestionishowtodefinesimilarity.NMCdefines thefeaturesofaclassasavectorandrepresentstheclasswiththemeanoftheelementsofthisvector. Thus,anyunlabeledvectoroffeatureswillbeclassifiedastheclasswiththenearestmeanvalue. Templatematching usesatemplatefordefiningclasslabels, andtriestofindthemostsimilartemplateforclassification.

Theclassificationtaskwasevaluatedusingtheconfusion matrix.Thegeneric element rij oftheconfusion matrixindicateshowfrequently apattern belonging tothestimulusclassiwasclassifiedasbelonging totheresponseclassj.Thematrixhas toberead bycolumns. Weused80%ofthefeaturedatasetfortrainingandtheremaining 20%

SupplementaryMaterial:TheSocialPerceptualSalience Effect 9

forthetesting phase.Inordertoobtainunbiasedclassificationresults, weperformed

40-foldcross-validationsteps.Thisprocedure allowedustoconsiderthedistributionoftheclassificationresults asGaussian.Theclassificationisdescribedbythemeanandstandarddeviations among the40confusion matrices(SeeTable1).

SupplementaryMaterial:TheSocialPerceptualSalience Effect 10

Questionnaire

Wewouldliketoevaluateinmoredetailhowyouperceivedtheinteraction withtherealperson ortheavataratDIFFERENTdistances.’Veryclose’wastheinteraction distance of0.5meter. ’Close’thedistance of1.2meters.Pleasemarkwithacrossyourpersonal experience.Thanks.

VirtualInteraction

Iperceivedthecloseinteraction withtheVIRTUALavataras:

Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

Iperceivedtheverycloseinteraction withtheVIRTUALavataras: Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

Physical Interaction

Iperceivedthecloseinteraction withthePHYSICALperson as:

Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

Iperceivedtheverycloseinteraction withthePHYSICALperson as: Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

SupplementaryMaterial:TheSocialPerceptualSalience Effect 11

References

Berger, R.,Akselrod,S.,Gordon, D.,Cohen, R.(2007). Anefficientalgorithmforspectralanalysis ofheartratevariability.Biomedical Engineering,IEEETransactionson(9),900–904.

Camm, A.,Malik,M.,Bigger,J.,Breithardt, G.,Cerutti,S.,Cohen, R.,etal.(1996). Heartratevariability:standardsofmeasurement, physiological interpretation,andclinicaluse. Circulation,93(5),1043–1065.

Eckmann,J.,Kamphorst,S.,Ruelle,D.(1987). Recurrenceplotsofdynamical systems.

EPL(EurophysicsLetters), 4,973.

Fusheng,Y.,Bo,H.,Qingyu,T.(2000). Approximate Entropyanditsapplication in biosignalanalysis.Nonlinear biomedical signalprocessing,72.

Ishchenko,A.,Shev’ev,P. (1989). Automatedcomplexformultiparameteranalysisofthegalvanicskinresponsesignal. Biomedical Engineering,23(3),113–117.

Jolliffe,I.(2002). Principalcomponentanalysis.WileyOnlineLibrary.

Marwan,N.,CarmenRomano, M.,Thiel,M.,Kurths,J.(2007). Recurrence plotsfortheanalysisofcomplexsystems.Physics Reports, 438(5-6),237–329.

Pan, J.,Tompkins, W.(1985). Areal-time QRSdetectionalgorithm. IEEETransactionsonBiomedical Engineering,230–236.

Richman,J.,Moorman, J.(2000). Physiological time-seriesanalysisusingapproximateentropyandsampleentropy. American Journal ofPhysiology-HeartandCirculatoryPhysiology,278(6),H2039.

Schinkel,S.,Dimigen,O.,Marwan,N.(2008). Selectionofrecurrencethresholdforsignaldetection.TheEuropean PhysicalJournal-SpecialTopics,164(1),45–53.

Valenza,G.,Lanata, A.,Scilingo,E.P. (2011). TheRoleofNonlinear Dynamics in AffectiveValence andArousalRecognition.IEEETransactiononAffectiveComputing,1–14.

SupplementaryMaterial:TheSocialPerceptualSalience Effect 12

Zbilut,J.,WebberJr,C.(2006). Recurrence quantification analysis.WileyOnline

Library.

SupplementaryMaterial:TheSocialPerceptualSalience Effect 13

Table1

NMCclassifier -Physiological signal classificationbasedona20componentfeatureset.Therowsare "responseclass" whilethecolumnsare "stimulusclass". Thetablemust bereadcolumn-wise.

NMC / Physical Intimate / VirtualIntimate
Physical Intimate / 88.23(21.5) / 23.53(25.3)
VirtualIntimate / 11.76(21.5) / 76.47(25.3)
Physical Neutral / Physical Intimate
Physical Neutral / 98.75(7.9) / 0.0(0.0)
Physical Intimate / 1.25(7.9) / 100.0(0.0)
VirtualNeutral / VirtualIntimate
VirtualNeutral / 79.15(25.0) / 12.50(21.9)
VirtualIntimate / 20.84(25.0) / 87.50(21.9)
VirtualNeutral / Physical Neutral
VirtualNeutral / 60.00(35.7) / 48.33(33.4)
Physical Neutral / 40.00(35.7) / 51.66(33.4)