This presentation will review the basics in how to perform a between-subjects ANOVA in R using the aov function and the afex package. I will go through this using a generated dataset.

But before running this code, you will need to load the following necessary package libraries. If you don’t have the packages installed, you will need to install them before continuing. To install them, delete the “#” in the following lines of code before running it.

#install.packages(plyr)
#install.packages(ggplot2)
#install.packages(afex)
#install.packages(lsmeans)

library(plyr)   
library(ggplot2)  
library(afex)  
library(lsmeans)

1 Generating a dataset

The created data set for a hypothetical 2x3 between-subjects experimental design where participants are asked to read a passage on different platforms and answer ten comprehension questions about the passage. The purpose of this hypothetical experiment is to examine the effect that reading platform has on text comprehension as a function of age.

  • IV 1: Reading Platform (Paper, Kindle, Computer)
  • IV 2: Age Group (for simplicity, the levels are just Old and Young. If you would like to examine age as a continuous variable, you can run a regression analysis. See Chapter 12 for more information on the basics of performing a regression analyses in R)
  • DV: Comprehension Score (1-10)

Here, I am creating a dataframe called “Comprehension.Data”, inserting 60 participants into the dataframe, adding reading platform as a between-subjects variable with 20 participants per level, adding age group as another between-subjects variable with 10 participants in each level. Then, I am filling the dataframe with 10 participants per condition with 6 conditions. Specifying the means and standard deviations for each condition. See Chapter 7 for more information on the rnorm function.

set.seed(42) #this is make sure you get the same values randomly each time

N.2=10 #sample size of age 
N.3=20 #sample size of reading platform

Comprehension.Data <- data.frame( 
 Subject = seq(1,N.3*3), 
 Platform = c(rep("Paper", N.3), rep("Kindle", N.3), rep("Computer", N.3)), 
 Age = rep(c(rep("Old",N.2), rep("Young", N.2)),3), 
 Comprehension = c(rnorm(N.2,10,.5),rnorm(N.2,8,.5), 
                    rnorm(N.2,6,.5),rnorm(N.2,7,.5),
                    rnorm(N.2,2,.5),rnorm(N.2,6,.5))) 

Note: this dataset is in long-format. If your dataset is in wide-format, you have to “reshape” the data (See Chapter 8 for more information).

Here we can view the structure of the dataframe.

str(Comprehension.Data) 
## 'data.frame':    60 obs. of  4 variables:
##  $ Subject      : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ Platform     : Factor w/ 3 levels "Computer","Kindle",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ Age          : Factor w/ 2 levels "Old","Young": 1 1 1 1 1 1 1 1 1 1 ...
##  $ Comprehension: num  10.69 9.72 10.18 10.32 10.2 ...

Here we can view the first few lines of the data. You can also look at the entire dataframe by removing head, and just calling for Comprehension.Data

head(Comprehension.Data)
##   Subject Platform Age Comprehension
## 1       1    Paper Old     10.685479
## 2       2    Paper Old      9.717651
## 3       3    Paper Old     10.181564
## 4       4    Paper Old     10.316431
## 5       5    Paper Old     10.202134
## 6       6    Paper Old      9.946938

Let’s look at our means for each of the six conditions. We can use the tapply function to do this. (See Chapter 3 for more information on the different apply functions).

tapply(Comprehension.Data$Comprehension,
        list(Comprehension.Data$Platform,Comprehension.Data$Age), mean)
##                Old    Young
## Computer  1.989892 6.009197
## Kindle    5.910960 6.818048
## Paper    10.273648 7.918272

We can also look at the means collapsed across each independent variable. Here are the means for the two Age levels collapsed across Platform.

tapply(Comprehension.Data$Comprehension,
        list(Comprehension.Data$Age), mean)
##      Old    Young 
## 6.058167 6.915172

And here are the means for the three Platform levels collapsed across Age.

tapply(Comprehension.Data$Comprehension,
        list(Comprehension.Data$Platform), mean)
## Computer   Kindle    Paper 
## 3.999545 6.364504 9.095960

2 What is the aov function?

This is a built-in R function that allows you to run an Analysis of Variance (ANOVA). This function defaults to running a Type I Sum of Squares.

You can use the help section to see a description of the aov function where it will display the arguments that go into this function. Putting a “?” in front of a function is a way to easily find more information on the function.

?aov

2.1 Running an ANOVA using the aov function

Here, I am just inserting the names of my variables into the function, and specifying the dataset I am working with.

Model.aov.1<-aov(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data)

summary(Model.aov.1)            
## 
## Error: Subject
##          Df Sum Sq Mean Sq
## Platform  1  207.4   207.4
## 
## Error: Within
##              Df Sum Sq Mean Sq F value   Pr(>F)    
## Platform      2  58.47   29.23   89.25  < 2e-16 ***
## Age           1   6.29    6.29   19.20 5.59e-05 ***
## Platform:Age  2 101.61   50.81  155.10  < 2e-16 ***
## Residuals    53  17.36    0.33                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

If you would like to specify Type II or III Sum of Squares using the aov function, you can do that by adjusting the line of code as in the following ways. For more information on different types of Sum of Squares, refer to the following paper:

  • Langsrud, Ø. (2003), ANOVA for Unbalanced Data: Use Type II Instead of Type III Sums of Squares, Statistics and Computing, 13, 163-167.
summary(Model.aov.1, type="II") #Type II Sum of Squares
## 
## Error: Subject
##          Df Sum Sq Mean Sq
## Platform  1  207.4   207.4
## 
## Error: Within
##              Df Sum Sq Mean Sq F value   Pr(>F)    
## Platform      2  58.47   29.23   89.25  < 2e-16 ***
## Age           1   6.29    6.29   19.20 5.59e-05 ***
## Platform:Age  2 101.61   50.81  155.10  < 2e-16 ***
## Residuals    53  17.36    0.33                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
summary(Model.aov.1, type="III") #Type III Sum of Squares 
## 
## Error: Subject
##          Df Sum Sq Mean Sq
## Platform  1  207.4   207.4
## 
## Error: Within
##              Df Sum Sq Mean Sq F value   Pr(>F)    
## Platform      2  58.47   29.23   89.25  < 2e-16 ***
## Age           1   6.29    6.29   19.20 5.59e-05 ***
## Platform:Age  2 101.61   50.81  155.10  < 2e-16 ***
## Residuals    53  17.36    0.33                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Note: I created a balanced design (equal number of observations in each condition), therefore the results of the Type I, II, and III Sum of Squares are the same.

3 What is the afex package?

The afex (“Analysis of Factorial Experiments”) package is an alternative to using the aov function to run an ANOVA in R.

There are three different functions in the afex package related to calculating an ANOVA:

  • aov_car (This is the main function we will focus on for this tutorial)
  • aov_ez (same as aov_car, but has a different layout, easier version if you don’t understand how to specify your error term)
  • aov_4 (wrapper function for lmer function in the lme4 package for mixed-effects models)

You can use the help section to get a more detailed description of the afex package functions.

?aov_car

3.1 Using the aov_car function

The aov_car function defaults to Type III Sum of Squares, which is the default in SPSS. It also gives you the generalized eta-squared, not the partial eta-squared. Additionally, this function autocorrects for violations of sphericity, and has a better output format than the built-in aov function.

Model.aov.2<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data)

summary(Model.aov.2) #Defaults to type III Sum of Squares
## Anova Table (Type 3 tests)
## 
## Response: Comprehension
##              num Df den Df     MSE       F     ges    Pr(>F)    
## Platform          2     54 0.33949 383.199 0.93418 < 2.2e-16 ***
## Age               1     54 0.33949  32.452 0.37537 5.202e-07 ***
## Platform:Age      2     54 0.33949 149.652 0.84716 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We can have it show us the partial eta-squared if we want. This is what SPSS outputs for us. For more information on the differences between generalized eta-squared and partial eta-squared, refer to the following paper:

  • Levine, T.R. & Hullett, C.R. (2002). Eta Squared, Partial Eta Squared and the Misreporting of Effect Size in Communication Research. Human Communication Research, 28, 612-625.
Model.aov.3<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data,
                     anova_table = list(es = "pes"))

summary(Model.aov.3)
## Anova Table (Type 3 tests)
## 
## Response: Comprehension
##              num Df den Df     MSE       F     pes    Pr(>F)    
## Platform          2     54 0.33949 383.199 0.93418 < 2.2e-16 ***
## Age               1     54 0.33949  32.452 0.37537 5.202e-07 ***
## Platform:Age      2     54 0.33949 149.652 0.84716 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We can also have it show us an output table like what you would get in SPSS by adding return="univariate".

Model.aov.4<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data,
                     anova_table = list(es = "pes"),
                     return="univariate")

Model.aov.4
## Anova Table (Type III tests)
## 
## Response: dv
##               Sum Sq Df  F value    Pr(>F)    
## (Intercept)  2524.61  1 7436.541 < 2.2e-16 ***
## Platform      260.18  2  383.199 < 2.2e-16 ***
## Age            11.02  1   32.452 5.202e-07 ***
## Platform:Age  101.61  2  149.652 < 2.2e-16 ***
## Residuals      18.33 54                       
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

4 Follow-up tests

In the hypothetical dataset, we have a significant main effect of Age (we can see from our Age mean table above that the Young group had significantly higher comprehension than the Old group), a significant main effect of Platform, and a significant Age by Platform interaction. We are going to first follow-up the significant main effect of Platform because this variable contains more than two levels.

We will follow-up this main effect by performing a pairwise analysis on fitted means to see where the significant differences between these levels lie within the Platform factor.

FittedMeans.Platform<-lsmeans(Model.aov.2, ~Platform)
FittedMeans.Platform
##  Platform   lsmean        SE df lower.CL upper.CL
##  Computer 3.999545 0.1302858 54 3.738337 4.260752
##  Kindle   6.364504 0.1302858 54 6.103297 6.625711
##  Paper    9.095960 0.1302858 54 8.834753 9.357167
## 
## Results are averaged over the levels of: Age 
## Confidence level used: 0.95
pairs(FittedMeans.Platform) 
##  contrast           estimate        SE df t.ratio p.value
##  Computer - Kindle -2.364959 0.1842519 54 -12.835  <.0001
##  Computer - Paper  -5.096415 0.1842519 54 -27.660  <.0001
##  Kindle - Paper    -2.731456 0.1842519 54 -14.825  <.0001
## 
## Results are averaged over the levels of: Age 
## P value adjustment: tukey method for comparing a family of 3 estimates

We can do this using the Tukey correction.

pairs(FittedMeans.Platform, adjust="Tukey") 
##  contrast           estimate        SE df t.ratio p.value
##  Computer - Kindle -2.364959 0.1842519 54 -12.835  <.0001
##  Computer - Paper  -5.096415 0.1842519 54 -27.660  <.0001
##  Kindle - Paper    -2.731456 0.1842519 54 -14.825  <.0001
## 
## Results are averaged over the levels of: Age 
## P value adjustment: tukey method for comparing a family of 3 estimates

We can also do this using the Bonferroni correction.

pairs(FittedMeans.Platform, adjust="bon") 
##  contrast           estimate        SE df t.ratio p.value
##  Computer - Kindle -2.364959 0.1842519 54 -12.835  <.0001
##  Computer - Paper  -5.096415 0.1842519 54 -27.660  <.0001
##  Kindle - Paper    -2.731456 0.1842519 54 -14.825  <.0001
## 
## Results are averaged over the levels of: Age 
## P value adjustment: bonferroni method for 3 tests

These pairwise comparisons reveal that comprehension score significantly differed between all Platform groups (Paper, Kindle, and Computer).

We also need to follow-up the significant Age by Platform interaction by performing pairwise effects for Age on Platform. We will do this using the Tukey correction.

FittedMeans.AgebyPlatform <- lsmeans(Model.aov.2, ~ Age|Platform) 

FittedMeans.AgebyPlatform
## Platform = Computer:
##  Age      lsmean        SE df lower.CL  upper.CL
##  Old    1.989892 0.1842519 54 1.620490  2.359295
##  Young  6.009197 0.1842519 54 5.639794  6.378600
## 
## Platform = Kindle:
##  Age      lsmean        SE df lower.CL  upper.CL
##  Old    5.910960 0.1842519 54 5.541557  6.280363
##  Young  6.818048 0.1842519 54 6.448645  7.187451
## 
## Platform = Paper:
##  Age      lsmean        SE df lower.CL  upper.CL
##  Old   10.273648 0.1842519 54 9.904246 10.643051
##  Young  7.918272 0.1842519 54 7.548869  8.287674
## 
## Confidence level used: 0.95
pairs(FittedMeans.AgebyPlatform, adjust="tukey")
## Platform = Computer:
##  contrast      estimate        SE df t.ratio p.value
##  Old - Young -4.0193046 0.2605715 54 -15.425  <.0001
## 
## Platform = Kindle:
##  contrast      estimate        SE df t.ratio p.value
##  Old - Young -0.9070877 0.2605715 54  -3.481  0.0010
## 
## Platform = Paper:
##  contrast      estimate        SE df t.ratio p.value
##  Old - Young  2.3553767 0.2605715 54   9.039  <.0001

5 Plotting the results

Here, I will plot the results on a bar graph using the ggplot2 package. This is a very basic graph, but if you want more information on how to do more fancy plots, refer to Chapters 10 & 11.

Plot.ComprehensionData <-ggplot(data = Comprehension.Data, aes(x = Platform, y =Comprehension, fill=Age))+
  coord_cartesian(ylim = c(1,12))+ #specifying the y-axis scale
  geom_bar(stat='identity', position=position_dodge(width=1))+
  xlab("Platform Type")+ylab("Comprehension Score")+ #labelling the x-axis and y-axis
  theme_bw()+
  theme(panel.grid.major=element_blank(),
        panel.grid.minor=element_blank())+
  scale_fill_grey()

Plot.ComprehensionData #displaying the graph 

LS0tDQp0aXRsZTogIkNoYXB0ZXIgMjA6IEJldHdlZW4tU3ViamVjdHMgQU5PVkEgaW4gUiINCmF1dGhvcjogIkFuZHJpYW5hIENocmlzdG9mYWxvcyINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDoNCiAgICB0aGVtZTogY2VydWxlYW4NCiAgICBoaWdobGlnaHQ6IHRleHRtYXRlDQogICAgZm9udHNpemU6IDhwdA0KICAgIHRvYzogdHJ1ZQ0KICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQ0KICAgIGNvZGVfZG93bmxvYWQ6IHRydWUNCiAgICB0b2NfZmxvYXQ6DQogICAgICBjb2xsYXBzZWQ6IGZhbHNlDQotLS0NCg0KYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9DQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpDQpgYGANCg0KVGhpcyBwcmVzZW50YXRpb24gd2lsbCByZXZpZXcgdGhlIGJhc2ljcyBpbiBob3cgdG8gcGVyZm9ybSBhIGJldHdlZW4tc3ViamVjdHMgQU5PVkEgaW4gUiB1c2luZyB0aGUgYW92IGZ1bmN0aW9uIGFuZCB0aGUgYWZleCBwYWNrYWdlLiBJIHdpbGwgZ28gdGhyb3VnaCB0aGlzIHVzaW5nIGEgZ2VuZXJhdGVkIGRhdGFzZXQuIA0KDQpCdXQgYmVmb3JlIHJ1bm5pbmcgdGhpcyBjb2RlLCB5b3Ugd2lsbCBuZWVkIHRvIGxvYWQgdGhlIGZvbGxvd2luZyBuZWNlc3NhcnkgcGFja2FnZSBsaWJyYXJpZXMuIElmIHlvdSBkb24ndCBoYXZlIHRoZSBwYWNrYWdlcyBpbnN0YWxsZWQsIHlvdSB3aWxsIG5lZWQgdG8gaW5zdGFsbCB0aGVtIGJlZm9yZSBjb250aW51aW5nLiBUbyBpbnN0YWxsIHRoZW0sIGRlbGV0ZSB0aGUgIiMiIGluIHRoZSBmb2xsb3dpbmcgbGluZXMgb2YgY29kZSBiZWZvcmUgcnVubmluZyBpdC4gDQpgYGB7ciwgbWVzc2FnZT1GQUxTRX0NCiNpbnN0YWxsLnBhY2thZ2VzKHBseXIpDQojaW5zdGFsbC5wYWNrYWdlcyhnZ3Bsb3QyKQ0KI2luc3RhbGwucGFja2FnZXMoYWZleCkNCiNpbnN0YWxsLnBhY2thZ2VzKGxzbWVhbnMpDQoNCmxpYnJhcnkocGx5cikgICANCmxpYnJhcnkoZ2dwbG90MikgIA0KbGlicmFyeShhZmV4KSAgDQpsaWJyYXJ5KGxzbWVhbnMpDQpgYGANCg0KIyBHZW5lcmF0aW5nIGEgZGF0YXNldA0KVGhlIGNyZWF0ZWQgZGF0YSBzZXQgZm9yIGEgaHlwb3RoZXRpY2FsIDJ4MyBiZXR3ZWVuLXN1YmplY3RzIGV4cGVyaW1lbnRhbCBkZXNpZ24gd2hlcmUgcGFydGljaXBhbnRzIGFyZSBhc2tlZCB0byByZWFkIGEgcGFzc2FnZSBvbiBkaWZmZXJlbnQgcGxhdGZvcm1zIGFuZCBhbnN3ZXIgdGVuIGNvbXByZWhlbnNpb24gcXVlc3Rpb25zIGFib3V0IHRoZSBwYXNzYWdlLiBUaGUgcHVycG9zZSBvZiB0aGlzIGh5cG90aGV0aWNhbCBleHBlcmltZW50IGlzIHRvIGV4YW1pbmUgdGhlIGVmZmVjdCB0aGF0IHJlYWRpbmcgcGxhdGZvcm0gaGFzIG9uIHRleHQgY29tcHJlaGVuc2lvbiBhcyBhIGZ1bmN0aW9uIG9mIGFnZS4NCg0KLSBJViAxOiBSZWFkaW5nIFBsYXRmb3JtIChQYXBlciwgS2luZGxlLCBDb21wdXRlcikNCi0gSVYgMjogQWdlIEdyb3VwIChmb3Igc2ltcGxpY2l0eSwgdGhlIGxldmVscyBhcmUganVzdCBPbGQgYW5kIFlvdW5nLiBJZiB5b3Ugd291bGQgbGlrZSB0byBleGFtaW5lIGFnZSBhcyBhIGNvbnRpbnVvdXMgdmFyaWFibGUsIHlvdSBjYW4gcnVuIGEgcmVncmVzc2lvbiBhbmFseXNpcy4gU2VlIENoYXB0ZXIgMTIgZm9yIG1vcmUgaW5mb3JtYXRpb24gb24gdGhlIGJhc2ljcyBvZiBwZXJmb3JtaW5nIGEgcmVncmVzc2lvbiBhbmFseXNlcyBpbiBSKQ0KLSBEVjogQ29tcHJlaGVuc2lvbiBTY29yZSAoMS0xMCkNCg0KSGVyZSwgSSBhbSBjcmVhdGluZyBhIGRhdGFmcmFtZSBjYWxsZWQgIkNvbXByZWhlbnNpb24uRGF0YSIsIGluc2VydGluZyA2MCBwYXJ0aWNpcGFudHMgaW50byB0aGUgZGF0YWZyYW1lLCBhZGRpbmcgcmVhZGluZyBwbGF0Zm9ybSBhcyBhIGJldHdlZW4tc3ViamVjdHMgdmFyaWFibGUgd2l0aCAyMCBwYXJ0aWNpcGFudHMgcGVyIGxldmVsLCBhZGRpbmcgYWdlIGdyb3VwIGFzIGFub3RoZXIgYmV0d2Vlbi1zdWJqZWN0cyB2YXJpYWJsZSB3aXRoIDEwIHBhcnRpY2lwYW50cyBpbiBlYWNoIGxldmVsLiBUaGVuLCBJIGFtIGZpbGxpbmcgdGhlIGRhdGFmcmFtZSB3aXRoIDEwIHBhcnRpY2lwYW50cyBwZXIgY29uZGl0aW9uIHdpdGggNiBjb25kaXRpb25zLiBTcGVjaWZ5aW5nIHRoZSBtZWFucyBhbmQgc3RhbmRhcmQgZGV2aWF0aW9ucyBmb3IgZWFjaCBjb25kaXRpb24uIFNlZSBDaGFwdGVyIDcgZm9yIG1vcmUgaW5mb3JtYXRpb24gb24gdGhlIGBgYHJub3JtYGBgIGZ1bmN0aW9uLiANCmBgYHtyLCBlY2hvPVRSVUV9DQpzZXQuc2VlZCg0MikgI3RoaXMgaXMgbWFrZSBzdXJlIHlvdSBnZXQgdGhlIHNhbWUgdmFsdWVzIHJhbmRvbWx5IGVhY2ggdGltZQ0KDQpOLjI9MTAgI3NhbXBsZSBzaXplIG9mIGFnZSANCk4uMz0yMCAjc2FtcGxlIHNpemUgb2YgcmVhZGluZyBwbGF0Zm9ybQ0KDQpDb21wcmVoZW5zaW9uLkRhdGEgPC0gZGF0YS5mcmFtZSggDQogU3ViamVjdCA9IHNlcSgxLE4uMyozKSwgDQogUGxhdGZvcm0gPSBjKHJlcCgiUGFwZXIiLCBOLjMpLCByZXAoIktpbmRsZSIsIE4uMyksIHJlcCgiQ29tcHV0ZXIiLCBOLjMpKSwgDQogQWdlID0gcmVwKGMocmVwKCJPbGQiLE4uMiksIHJlcCgiWW91bmciLCBOLjIpKSwzKSwgDQogQ29tcHJlaGVuc2lvbiA9IGMocm5vcm0oTi4yLDEwLC41KSxybm9ybShOLjIsOCwuNSksIA0KICAgICAgICAgICAgICAgICAgICBybm9ybShOLjIsNiwuNSkscm5vcm0oTi4yLDcsLjUpLA0KICAgICAgICAgICAgICAgICAgICBybm9ybShOLjIsMiwuNSkscm5vcm0oTi4yLDYsLjUpKSkgDQpgYGANCk5vdGU6IHRoaXMgZGF0YXNldCBpcyBpbiBsb25nLWZvcm1hdC4gSWYgeW91ciBkYXRhc2V0IGlzIGluIHdpZGUtZm9ybWF0LCB5b3UgaGF2ZSB0byAicmVzaGFwZSIgdGhlIGRhdGEgKFNlZSBDaGFwdGVyIDggZm9yIG1vcmUgaW5mb3JtYXRpb24pLg0KDQpIZXJlIHdlIGNhbiB2aWV3IHRoZSBzdHJ1Y3R1cmUgb2YgdGhlIGRhdGFmcmFtZS4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpzdHIoQ29tcHJlaGVuc2lvbi5EYXRhKSANCmBgYA0KDQpIZXJlIHdlIGNhbiB2aWV3IHRoZSBmaXJzdCBmZXcgbGluZXMgb2YgdGhlIGRhdGEuIFlvdSBjYW4gYWxzbyBsb29rIGF0IHRoZSBlbnRpcmUgZGF0YWZyYW1lIGJ5IHJlbW92aW5nIGBgYGhlYWRgYGAsIGFuZCBqdXN0IGNhbGxpbmcgZm9yIGBgYENvbXByZWhlbnNpb24uRGF0YWBgYA0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCmhlYWQoQ29tcHJlaGVuc2lvbi5EYXRhKQ0KYGBgDQoNCkxldCdzIGxvb2sgYXQgb3VyIG1lYW5zIGZvciBlYWNoIG9mIHRoZSBzaXggY29uZGl0aW9ucy4gV2UgY2FuIHVzZSB0aGUgYGBgdGFwcGx5YGBgIGZ1bmN0aW9uIHRvIGRvIHRoaXMuIChTZWUgQ2hhcHRlciAzIGZvciBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBkaWZmZXJlbnQgYXBwbHkgZnVuY3Rpb25zKS4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQp0YXBwbHkoQ29tcHJlaGVuc2lvbi5EYXRhJENvbXByZWhlbnNpb24sDQogICAgICAgIGxpc3QoQ29tcHJlaGVuc2lvbi5EYXRhJFBsYXRmb3JtLENvbXByZWhlbnNpb24uRGF0YSRBZ2UpLCBtZWFuKQ0KYGBgDQoNCldlIGNhbiBhbHNvIGxvb2sgYXQgdGhlIG1lYW5zIGNvbGxhcHNlZCBhY3Jvc3MgZWFjaCBpbmRlcGVuZGVudCB2YXJpYWJsZS4gSGVyZSBhcmUgdGhlIG1lYW5zIGZvciB0aGUgdHdvIEFnZSBsZXZlbHMgY29sbGFwc2VkIGFjcm9zcyBQbGF0Zm9ybS4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQp0YXBwbHkoQ29tcHJlaGVuc2lvbi5EYXRhJENvbXByZWhlbnNpb24sDQogICAgICAgIGxpc3QoQ29tcHJlaGVuc2lvbi5EYXRhJEFnZSksIG1lYW4pDQpgYGANCg0KQW5kIGhlcmUgYXJlIHRoZSBtZWFucyBmb3IgdGhlIHRocmVlIFBsYXRmb3JtIGxldmVscyBjb2xsYXBzZWQgYWNyb3NzIEFnZS4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQp0YXBwbHkoQ29tcHJlaGVuc2lvbi5EYXRhJENvbXByZWhlbnNpb24sDQogICAgICAgIGxpc3QoQ29tcHJlaGVuc2lvbi5EYXRhJFBsYXRmb3JtKSwgbWVhbikNCmBgYA0KDQojIFdoYXQgaXMgdGhlIGFvdiBmdW5jdGlvbj8NClRoaXMgaXMgYSBidWlsdC1pbiBSIGZ1bmN0aW9uIHRoYXQgYWxsb3dzIHlvdSB0byBydW4gYW4gQW5hbHlzaXMgb2YgVmFyaWFuY2UgKEFOT1ZBKS4gVGhpcyBmdW5jdGlvbiBkZWZhdWx0cyB0byBydW5uaW5nIGEgVHlwZSBJIFN1bSBvZiBTcXVhcmVzLg0KDQpZb3UgY2FuIHVzZSB0aGUgaGVscCBzZWN0aW9uIHRvIHNlZSBhIGRlc2NyaXB0aW9uIG9mIHRoZSBhb3YgZnVuY3Rpb24gd2hlcmUgaXQgd2lsbCBkaXNwbGF5IHRoZSBhcmd1bWVudHMgdGhhdCBnbyBpbnRvIHRoaXMgZnVuY3Rpb24uIFB1dHRpbmcgYSAiPyIgaW4gZnJvbnQgb2YgYSBmdW5jdGlvbiBpcyBhIHdheSB0byBlYXNpbHkgZmluZCBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBmdW5jdGlvbi4gDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KP2Fvdg0KYGBgDQoNCiMjUnVubmluZyBhbiBBTk9WQSB1c2luZyB0aGUgYW92IGZ1bmN0aW9uDQpIZXJlLCBJIGFtIGp1c3QgaW5zZXJ0aW5nIHRoZSBuYW1lcyBvZiBteSB2YXJpYWJsZXMgaW50byB0aGUgZnVuY3Rpb24sIGFuZCBzcGVjaWZ5aW5nIHRoZSBkYXRhc2V0IEkgYW0gd29ya2luZyB3aXRoLiANCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpNb2RlbC5hb3YuMTwtYW92KENvbXByZWhlbnNpb24gfiBQbGF0Zm9ybSpBZ2UgKyBFcnJvcihTdWJqZWN0KSwgZGF0YT1Db21wcmVoZW5zaW9uLkRhdGEpDQoNCnN1bW1hcnkoTW9kZWwuYW92LjEpICAgICAgICAgICAgDQpgYGANCg0KSWYgeW91IHdvdWxkIGxpa2UgdG8gc3BlY2lmeSBUeXBlIElJIG9yIElJSSBTdW0gb2YgU3F1YXJlcyB1c2luZyB0aGUgYW92IGZ1bmN0aW9uLCB5b3UgY2FuIGRvIHRoYXQgYnkgYWRqdXN0aW5nIHRoZSBsaW5lIG9mIGNvZGUgYXMgaW4gdGhlIGZvbGxvd2luZyB3YXlzLiBGb3IgbW9yZSBpbmZvcm1hdGlvbiBvbiBkaWZmZXJlbnQgdHlwZXMgb2YgU3VtIG9mIFNxdWFyZXMsIHJlZmVyIHRvIHRoZSBmb2xsb3dpbmcgcGFwZXI6DQoNCi0gTGFuZ3NydWQsIMOYLiAoMjAwMyksIEFOT1ZBIGZvciBVbmJhbGFuY2VkIERhdGE6IFVzZSBUeXBlIElJIEluc3RlYWQgb2YgVHlwZSBJSUkgU3VtcyBvZiBTcXVhcmVzLCBTdGF0aXN0aWNzIGFuZCBDb21wdXRpbmcsIDEzLCAxNjMtMTY3LiANCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpzdW1tYXJ5KE1vZGVsLmFvdi4xLCB0eXBlPSJJSSIpICNUeXBlIElJIFN1bSBvZiBTcXVhcmVzDQoNCnN1bW1hcnkoTW9kZWwuYW92LjEsIHR5cGU9IklJSSIpICNUeXBlIElJSSBTdW0gb2YgU3F1YXJlcyANCmBgYA0KTm90ZTogSSBjcmVhdGVkIGEgYmFsYW5jZWQgZGVzaWduIChlcXVhbCBudW1iZXIgb2Ygb2JzZXJ2YXRpb25zIGluIGVhY2ggY29uZGl0aW9uKSwgdGhlcmVmb3JlIHRoZSByZXN1bHRzIG9mIHRoZSBUeXBlIEksIElJLCBhbmQgSUlJIFN1bSBvZiBTcXVhcmVzIGFyZSB0aGUgc2FtZS4gDQoNCiMgV2hhdCBpcyB0aGUgYWZleCBwYWNrYWdlPyANClRoZSBhZmV4ICgiQW5hbHlzaXMgb2YgRmFjdG9yaWFsIEV4cGVyaW1lbnRzIikgcGFja2FnZSBpcyBhbiBhbHRlcm5hdGl2ZSB0byB1c2luZyB0aGUgYW92IGZ1bmN0aW9uIHRvIHJ1biBhbiBBTk9WQSBpbiBSLiAgDQoNClRoZXJlIGFyZSB0aHJlZSBkaWZmZXJlbnQgZnVuY3Rpb25zIGluIHRoZSBhZmV4IHBhY2thZ2UgcmVsYXRlZCB0byBjYWxjdWxhdGluZyBhbiBBTk9WQToNCg0KLSBhb3ZfY2FyIChUaGlzIGlzIHRoZSBtYWluIGZ1bmN0aW9uIHdlIHdpbGwgZm9jdXMgb24gZm9yIHRoaXMgdHV0b3JpYWwpDQotIGFvdl9leiAoc2FtZSBhcyBhb3ZfY2FyLCBidXQgaGFzIGEgZGlmZmVyZW50IGxheW91dCwgZWFzaWVyIHZlcnNpb24gaWYgeW91IGRvbid0IHVuZGVyc3RhbmQgaG93IHRvIHNwZWNpZnkgeW91ciBlcnJvciB0ZXJtKQ0KLSBhb3ZfNCAod3JhcHBlciBmdW5jdGlvbiBmb3IgbG1lciBmdW5jdGlvbiBpbiB0aGUgbG1lNCBwYWNrYWdlIGZvciBtaXhlZC1lZmZlY3RzIG1vZGVscykNCg0KWW91IGNhbiB1c2UgdGhlIGhlbHAgc2VjdGlvbiB0byBnZXQgYSBtb3JlIGRldGFpbGVkIGRlc2NyaXB0aW9uIG9mIHRoZSBhZmV4IHBhY2thZ2UgZnVuY3Rpb25zLiANCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQo/YW92X2Nhcg0KYGBgDQoNCiMjVXNpbmcgdGhlIGFvdl9jYXIgZnVuY3Rpb24NClRoZSBhb3ZfY2FyIGZ1bmN0aW9uIGRlZmF1bHRzIHRvIFR5cGUgSUlJIFN1bSBvZiBTcXVhcmVzLCB3aGljaCBpcyB0aGUgZGVmYXVsdCBpbiBTUFNTLiBJdCBhbHNvIGdpdmVzIHlvdSB0aGUgZ2VuZXJhbGl6ZWQgZXRhLXNxdWFyZWQsIG5vdCB0aGUgcGFydGlhbCBldGEtc3F1YXJlZC4gQWRkaXRpb25hbGx5LCB0aGlzIGZ1bmN0aW9uIGF1dG9jb3JyZWN0cyBmb3IgdmlvbGF0aW9ucyBvZiBzcGhlcmljaXR5LCBhbmQgaGFzIGEgYmV0dGVyIG91dHB1dCBmb3JtYXQgdGhhbiB0aGUgYnVpbHQtaW4gYW92IGZ1bmN0aW9uLg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCk1vZGVsLmFvdi4yPC1hb3ZfY2FyKENvbXByZWhlbnNpb24gfiBQbGF0Zm9ybSpBZ2UgKyBFcnJvcihTdWJqZWN0KSwgZGF0YT1Db21wcmVoZW5zaW9uLkRhdGEpDQoNCnN1bW1hcnkoTW9kZWwuYW92LjIpICNEZWZhdWx0cyB0byB0eXBlIElJSSBTdW0gb2YgU3F1YXJlcw0KYGBgDQoNCldlIGNhbiBoYXZlIGl0IHNob3cgdXMgdGhlIHBhcnRpYWwgZXRhLXNxdWFyZWQgaWYgd2Ugd2FudC4gVGhpcyBpcyB3aGF0IFNQU1Mgb3V0cHV0cyBmb3IgdXMuIEZvciBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBkaWZmZXJlbmNlcyBiZXR3ZWVuIGdlbmVyYWxpemVkIGV0YS1zcXVhcmVkIGFuZCBwYXJ0aWFsIGV0YS1zcXVhcmVkLCByZWZlciB0byB0aGUgZm9sbG93aW5nIHBhcGVyOg0KDQotIExldmluZSwgVC5SLiAmIEh1bGxldHQsIEMuUi4gKDIwMDIpLiBFdGEgU3F1YXJlZCwgUGFydGlhbCBFdGEgU3F1YXJlZCBhbmQgdGhlIE1pc3JlcG9ydGluZyBvZiBFZmZlY3QgU2l6ZSBpbiBDb21tdW5pY2F0aW9uIFJlc2VhcmNoLiBIdW1hbiBDb21tdW5pY2F0aW9uICAgIFJlc2VhcmNoLCAyOCwgNjEyLTYyNS4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpNb2RlbC5hb3YuMzwtYW92X2NhcihDb21wcmVoZW5zaW9uIH4gUGxhdGZvcm0qQWdlICsgRXJyb3IoU3ViamVjdCksIGRhdGE9Q29tcHJlaGVuc2lvbi5EYXRhLA0KICAgICAgICAgICAgICAgICAgICAgYW5vdmFfdGFibGUgPSBsaXN0KGVzID0gInBlcyIpKQ0KDQpzdW1tYXJ5KE1vZGVsLmFvdi4zKQ0KYGBgDQoNCldlIGNhbiBhbHNvIGhhdmUgaXQgc2hvdyB1cyBhbiBvdXRwdXQgdGFibGUgbGlrZSB3aGF0IHlvdSB3b3VsZCBnZXQgaW4gU1BTUyBieSBhZGRpbmcgYGBgcmV0dXJuPSJ1bml2YXJpYXRlImBgYC4NCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpNb2RlbC5hb3YuNDwtYW92X2NhcihDb21wcmVoZW5zaW9uIH4gUGxhdGZvcm0qQWdlICsgRXJyb3IoU3ViamVjdCksIGRhdGE9Q29tcHJlaGVuc2lvbi5EYXRhLA0KICAgICAgICAgICAgICAgICAgICAgYW5vdmFfdGFibGUgPSBsaXN0KGVzID0gInBlcyIpLA0KICAgICAgICAgICAgICAgICAgICAgcmV0dXJuPSJ1bml2YXJpYXRlIikNCg0KTW9kZWwuYW92LjQNCmBgYA0KDQojRm9sbG93LXVwIHRlc3RzDQpJbiB0aGUgaHlwb3RoZXRpY2FsIGRhdGFzZXQsIHdlIGhhdmUgYSBzaWduaWZpY2FudCBtYWluIGVmZmVjdCBvZiBBZ2UgKHdlIGNhbiBzZWUgZnJvbSBvdXIgQWdlIG1lYW4gdGFibGUgYWJvdmUgdGhhdCB0aGUgWW91bmcgZ3JvdXAgaGFkIHNpZ25pZmljYW50bHkgaGlnaGVyIGNvbXByZWhlbnNpb24gdGhhbiB0aGUgT2xkIGdyb3VwKSwgYSBzaWduaWZpY2FudCBtYWluIGVmZmVjdCBvZiBQbGF0Zm9ybSwgYW5kIGEgc2lnbmlmaWNhbnQgQWdlIGJ5IFBsYXRmb3JtIGludGVyYWN0aW9uLiBXZSBhcmUgZ29pbmcgdG8gZmlyc3QgZm9sbG93LXVwIHRoZSBzaWduaWZpY2FudCBtYWluIGVmZmVjdCBvZiBQbGF0Zm9ybSBiZWNhdXNlIHRoaXMgdmFyaWFibGUgY29udGFpbnMgbW9yZSB0aGFuIHR3byBsZXZlbHMuIA0KDQpXZSB3aWxsIGZvbGxvdy11cCB0aGlzIG1haW4gZWZmZWN0IGJ5IHBlcmZvcm1pbmcgYSBwYWlyd2lzZSBhbmFseXNpcyBvbiBmaXR0ZWQgbWVhbnMgdG8gc2VlIHdoZXJlIHRoZSBzaWduaWZpY2FudCBkaWZmZXJlbmNlcyBiZXR3ZWVuIHRoZXNlIGxldmVscyBsaWUgd2l0aGluIHRoZSBQbGF0Zm9ybSBmYWN0b3IuDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KRml0dGVkTWVhbnMuUGxhdGZvcm08LWxzbWVhbnMoTW9kZWwuYW92LjIsIH5QbGF0Zm9ybSkNCkZpdHRlZE1lYW5zLlBsYXRmb3JtDQoNCnBhaXJzKEZpdHRlZE1lYW5zLlBsYXRmb3JtKSANCmBgYA0KDQpXZSBjYW4gZG8gdGhpcyB1c2luZyB0aGUgVHVrZXkgY29ycmVjdGlvbi4gDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KcGFpcnMoRml0dGVkTWVhbnMuUGxhdGZvcm0sIGFkanVzdD0iVHVrZXkiKSANCmBgYA0KDQpXZSBjYW4gYWxzbyBkbyB0aGlzIHVzaW5nIHRoZSBCb25mZXJyb25pIGNvcnJlY3Rpb24uIA0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCnBhaXJzKEZpdHRlZE1lYW5zLlBsYXRmb3JtLCBhZGp1c3Q9ImJvbiIpIA0KYGBgDQpUaGVzZSBwYWlyd2lzZSBjb21wYXJpc29ucyByZXZlYWwgdGhhdCBjb21wcmVoZW5zaW9uIHNjb3JlIHNpZ25pZmljYW50bHkgZGlmZmVyZWQgYmV0d2VlbiBhbGwgUGxhdGZvcm0gZ3JvdXBzIChQYXBlciwgS2luZGxlLCBhbmQgQ29tcHV0ZXIpLiANCg0KV2UgYWxzbyBuZWVkIHRvIGZvbGxvdy11cCB0aGUgc2lnbmlmaWNhbnQgQWdlIGJ5IFBsYXRmb3JtIGludGVyYWN0aW9uIGJ5IHBlcmZvcm1pbmcgcGFpcndpc2UgZWZmZWN0cyBmb3IgQWdlIG9uIFBsYXRmb3JtLiBXZSB3aWxsIGRvIHRoaXMgdXNpbmcgdGhlIFR1a2V5IGNvcnJlY3Rpb24uDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KRml0dGVkTWVhbnMuQWdlYnlQbGF0Zm9ybSA8LSBsc21lYW5zKE1vZGVsLmFvdi4yLCB+IEFnZXxQbGF0Zm9ybSkgDQoNCkZpdHRlZE1lYW5zLkFnZWJ5UGxhdGZvcm0NCg0KcGFpcnMoRml0dGVkTWVhbnMuQWdlYnlQbGF0Zm9ybSwgYWRqdXN0PSJ0dWtleSIpDQpgYGANCg0KI1Bsb3R0aW5nIHRoZSByZXN1bHRzDQpIZXJlLCBJIHdpbGwgcGxvdCB0aGUgcmVzdWx0cyBvbiBhIGJhciBncmFwaCB1c2luZyB0aGUgZ2dwbG90MiBwYWNrYWdlLiBUaGlzIGlzIGEgdmVyeSBiYXNpYyBncmFwaCwgYnV0IGlmIHlvdSB3YW50IG1vcmUgaW5mb3JtYXRpb24gb24gaG93IHRvIGRvIG1vcmUgZmFuY3kgcGxvdHMsIHJlZmVyIHRvIENoYXB0ZXJzIDEwICYgMTEuIA0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NClBsb3QuQ29tcHJlaGVuc2lvbkRhdGEgPC1nZ3Bsb3QoZGF0YSA9IENvbXByZWhlbnNpb24uRGF0YSwgYWVzKHggPSBQbGF0Zm9ybSwgeSA9Q29tcHJlaGVuc2lvbiwgZmlsbD1BZ2UpKSsNCiAgY29vcmRfY2FydGVzaWFuKHlsaW0gPSBjKDEsMTIpKSsgI3NwZWNpZnlpbmcgdGhlIHktYXhpcyBzY2FsZQ0KICBnZW9tX2JhcihzdGF0PSdpZGVudGl0eScsIHBvc2l0aW9uPXBvc2l0aW9uX2RvZGdlKHdpZHRoPTEpKSsNCiAgeGxhYigiUGxhdGZvcm0gVHlwZSIpK3lsYWIoIkNvbXByZWhlbnNpb24gU2NvcmUiKSsgI2xhYmVsbGluZyB0aGUgeC1heGlzIGFuZCB5LWF4aXMNCiAgdGhlbWVfYncoKSsNCiAgdGhlbWUocGFuZWwuZ3JpZC5tYWpvcj1lbGVtZW50X2JsYW5rKCksDQogICAgICAgIHBhbmVsLmdyaWQubWlub3I9ZWxlbWVudF9ibGFuaygpKSsNCiAgc2NhbGVfZmlsbF9ncmV5KCkNCg0KUGxvdC5Db21wcmVoZW5zaW9uRGF0YSAjZGlzcGxheWluZyB0aGUgZ3JhcGggDQpgYGANCg0KPHNjcmlwdD4NCiAgKGZ1bmN0aW9uKGkscyxvLGcscixhLG0pe2lbJ0dvb2dsZUFuYWx5dGljc09iamVjdCddPXI7aVtyXT1pW3JdfHxmdW5jdGlvbigpew0KICAoaVtyXS5xPWlbcl0ucXx8W10pLnB1c2goYXJndW1lbnRzKX0saVtyXS5sPTEqbmV3IERhdGUoKTthPXMuY3JlYXRlRWxlbWVudChvKSwNCiAgbT1zLmdldEVsZW1lbnRzQnlUYWdOYW1lKG8pWzBdO2EuYXN5bmM9MTthLnNyYz1nO20ucGFyZW50Tm9kZS5pbnNlcnRCZWZvcmUoYSxtKQ0KICB9KSh3aW5kb3csZG9jdW1lbnQsJ3NjcmlwdCcsJ2h0dHBzOi8vd3d3Lmdvb2dsZS1hbmFseXRpY3MuY29tL2FuYWx5dGljcy5qcycsJ2dhJyk7DQoNCiAgZ2EoJ2NyZWF0ZScsICdVQS05ODg3ODc5My0xJywgJ2F1dG8nKTsNCiAgZ2EoJ3NlbmQnLCAncGFnZXZpZXcnKTsNCg0KPC9zY3JpcHQ+