1. No category

Same-different and A-not A tests with sensR
Christine Borgen Linander
A huge thank to a former colleague of mine Rune H B Christensen.
DTU Compute
Section for Statistics
Technical University of Denmark
chjo@dtu.dk
August 20th 2015
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
1 / 23
© Christine Borgen Linander (DTU)
Outline
Same-different and A-not A tests
Sensometrics Summer School
2 / 23
Same-different protocol
Outline
Same-Different and the Degree-of-Difference tests
2 products — 2 confusable stimuli:
A Chocolate bar (standard)
1
B Chocolate bar with less saturated fat
Same-different protocol
Setting:
2
The A-not A protocol
3
Measures of sensitivity
4
The A-not A with sureness protocol
One pair of samples evaluated at each trial
Question: Are the samples the same or different?
Stimuli:
Same stimuli pairs: AA and BB
Different stimuli pairs: AB and BA
Same-Different test:
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
3 / 23
Same
Different
© Christine Borgen Linander (DTU)
Degree-of-Difference test:
Same
2
3
4
Different
Same-different and A-not A tests
Sensometrics Summer School
4 / 23
Same-different protocol
Same-different protocol
Characteristics of the DOD test
Giving answers — τ criteria and the decision rule
Same-Different:
A
B
τ
An unspecified test (like Triangle, Duo-Trio, Tetrad)
A
Intensity
Intensity
B
Only 2 samples compared at each trial
|B − A| > τ
|B − A| < τ
→
→
”different”
”same”
τ
Easily understood test (by consumers) (O’Mahony and Rousseau, 2002)
No prior knowledge of products required (unlike A-not A)
Degree of difference:
Response bias (like A-not A)
4
3
A
2
1
2
3
B
4
Intensity
τ1
Rating scale:
τ2
1
2
3
4
τ3
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
5 / 23
© Christine Borgen Linander (DTU)
Same-different protocol
0.5
0.4
0.4
A
B
Same-different example
different
− τ3
same
− τ2 − τ1
τ1
AA, BB
τ2
different
τ3
0.3
0.2
0.2
σ2 = 2
0.1
0.1
AB, BA
0.0
0.0
0
6 / 23
Difference distributions
0.5
0.3
Sensometrics Summer School
Same-different protocol
Thurstonian model for the DOD test
Thurstonian distributions:
Same-different and A-not A tests
δ
0
Examples in R — difference and similarity assessments.
Sample
δ
Probability of answer in the j th category:
Same
Different
P (”j ”|Same-pair) = fs (τ )
Response
“Same” “Different”
8
5
4
9
Total
13
13
P (”j ”|Different-pair) = fd (τ , δ)
Maximum likelihood estimation of parameters:
likelihood ∼ fs (τ ) + fd (τ , δ)
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
7 / 23
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
8 / 23
The A-not A protocol
The A-not A protocol
The A-not A protocol
Example: the A-not A test
Example data:
Situation:
Sample
2 products: A and B (“not A”)
A
Not-A
Assessors are familiarized with A samples (and sometimes B samples
as well)
Assessors are served one sample — either A or B
Response
“A” “Not-A”
26
29
14
41
Total
55
55
Null hypothesis, H0 : products are similar
Alternative hypothesis, HA : products are different
Question: Is the sample an A or a not A sample?
Known as the “yes-no” method in Signal Detection Theory (Macmillan and
Creelman, 2005)
Problem: There are many tests to choose from.
What is the p-value?
Can we reject H0 ?
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
9 / 23
© Christine Borgen Linander (DTU)
The A-not A protocol
Same-different and A-not A tests
Sensometrics Summer School
10 / 23
The A-not A protocol
Estimation of d 0 with sensR
0.4
The Thurstonian model for the A-not A test
Not A
A
0.2
> library(sensR)
> AnotA(26, 55, 14, 55)
Call:
0.1
Density
0.3
Estimation of d 0 with sensR:
AnotA(x1 = 26, n1 = 55, x2 = 14, n2 = 55)
0.0
Results for the A-Not A test:
0
θ
Estimate Std. Error
Lower
Upper
P-value
d-prime 0.591838 0.2492597 0.1032979 1.080378 0.01431559
d'
Psychological continuuum
d 0 : Sensory difference
θ: Decision threshold
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
11 / 23
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
12 / 23
The A-not A protocol
The A-not A protocol
Likelihood confidence intervals for A-not A tests
Similarity testing
> (ana <- AnotA(26, 55, 14, 55))
Results for the A-Not A test:
Aim:
Prove that products are identical Prove that products are identical
Establish similarity within a similarity bound at some α-level
Estimate Std. Error
Lower
Upper
P-value
d-prime 0.591838 0.2492597 0.1032979 1.080378 0.01431559
How:
Interchange the roles of the hypotheses:
Standard normal based confidence intervals: CI95% = d 0 ± 1.96se(d 0 )
Example:
H0 : d 0 is larger than 1
HA : d 0 is less than 1
Call:
AnotA(x1 = 26, n1 = 55, x2 = 14, n2 = 55)
Improved likelihood based confidence intervals:
> confint(ana)
2.5 %
97.5 %
threshold -0.4007727 0.2627993
d.prime
0.1063875 1.0842269
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Huge practical challenge:
How to choose the similarity bound?
Sensometrics Summer School
13 / 23
© Christine Borgen Linander (DTU)
The A-not A protocol
Same-different and A-not A tests
Sensometrics Summer School
14 / 23
Measures of sensitivity
Similarity testing with d 0 for A-not A tests
Discrimination measures in equal-variance models
Use d 0 for similarity testing:
d 0 = (µ2 − µ1 )/σ is the (relative) distance between normal
distributions
Hypotheses:
H0 : d 0 is larger than 1
HA : d 0 is less than 1
λ = 2Φ(−d 0 /2) is the distribution overlap (0 < λ ≤ 1)
√
Sensitivity, S = P (x1 < x2 ) = Φ(d 0 / 2) is the probability that a
random sample from the low-intensity distribution has a lower
intensity than a random sample from the high-intensity distribution
p-value = P (Z < (d 0 − d00 )/se(d 0 )|H0 ):
>
>
>
>
## The Wald statistic:
statistic <- (0.592 - 1)/ 0.249
## Compute p-value:
pnorm(statistic, lower.tail=TRUE)
overlap 1.0 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.05 0.01
d.prime 0.0 0.25 0.51 0.77 1.05 1.35 1.68 2.07 2.56 3.29 3.92 5.15
AUC
0.5 0.60 0.69 0.78 0.85 0.91 0.95 0.98 0.99 1.00 1.00 1.00
[1] 0.05065307
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
15 / 23
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
16 / 23
Measures of sensitivity
The A-not A with sureness protocol
The Receiver Operating Characteristic (ROC) curve
Basics of the A-not A with sureness protocol
Answers are given on a ’sureness’ scale with J categories
What is a ROC curve?
A visual description of the discriminative ability
The model assumes J − 1 thresholds are adopted by the assessors
A central concept in Signal Detection Theory (A plot of False positive
ratio ∼ True positive ratio (alt. Hit rate ∼ False-alarm Rate)
Multinomial response: several ordered response categories
Many parameters: Thresholds, θj and effect, δ
The real number of interest — the area under the ROC curve, AUC:
√
S = AUC = Φ(d 0 / 2)
Table: Soup data
Product
Reference
Test
Examples in R
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
17 / 23
Sure
134
101
’Reference’
Not Sure Guess
162
66
101
51
© Christine Borgen Linander (DTU)
The A-not A with sureness protocol
’Not Reference’
Guess Not Sure Sure
41
122
222
57
157
653
Same-different and A-not A tests
Sensometrics Summer School
18 / 23
The A-not A with sureness protocol
NOT A
A : σ1 = 1
NOT A : σ2 = 1.3
θ1
θ2
δ σ
0
Psychological continuum
0.0
0.0
0.1
0.1
0.2
0.2
0.3
0.3
A
Unequal variances model
0.4
0.4
Thurstonian model for the A-not A with sureness protocol
θ3
δ σk
0
Psychological continuum
Multiple thresholds (θ) parameters.
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
19 / 23
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
20 / 23
The A-not A with sureness protocol
The A-not A with sureness protocol
An unequal-variance model in practice
Cumulative link models for A-not A with sureness
δ
Reference
products
N(0, 1)
Christensen showed that the A-not A protocol (with and without
sureness) is a version of a cumulative link model:
θj − δ(prodi )
P (Si ≤ θj ) = Φ
σ(prodi )
Test
products
N(δ, σ22)
where σ is the ratio of scales (std. dev).
θ1 θ2 θ3 θ4 θ5
This provides (optimal) ML estimates of the parameters, standard
errors etc.
Sensory intensity
Product
Reference
Test
Sure
132
96
’Reference’
Not Sure Guess
161
65
99
50
profile likelihood confidence intervals available with confint.
’Not Reference’
Guess Not Sure
Sure
41
121
219
57
156
650
> fm1 <- clm(sureness ~ prod, data=my_data, link="probit")
> summary(fm1) ## print d-prime etc.
> confint(fm1) ## likelihood confidence interval for d-prime
Table: Discrimination of packet soup (Christensen, Cleaver and Brockhoff, 2011)
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
21 / 23
The A-not A with sureness protocol
Discrimination measures in unequal-variance models
d 0 = (µ2 − µ1 )/σ (no change)
Distribution overlap: use the overlap function.
p
Sensitivity: S = Φ(d 0 / 1 + σ22 ) where σ2 is the scale ratio of the
high-intensity distribution relative to the low-intensity distribution.
All measureas are ’equivalent’ in the equal-variance model, but not so in
the unequal-variance model.
In the unequal-variance model d 0 can be a poor measure of discrimination.
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
23 / 23
© Christine Borgen Linander (DTU)
Same-different and A-not A tests
Sensometrics Summer School
22 / 23