This presentation will review the basics in how to perform a between-subjects ANOVA in R using the aov function and the afex package. I will go through this using a generated dataset.
But before running this code, you will need to load the following necessary package libraries. If you don’t have the packages installed, you will need to install them before continuing. To install them, delete the “#” in the following lines of code before running it.
#install.packages(plyr)
#install.packages(ggplot2)
#install.packages(afex)
#install.packages(emmeans)
library(plyr)
library(ggplot2)
library(afex)
library(emmeans)
Generating a dataset
The created data set for a hypothetical 2x3 between-subjects experimental design where participants are asked to read a passage on different platforms and answer ten comprehension questions about the passage. The purpose of this hypothetical experiment is to examine the effect that reading platform has on text comprehension as a function of age.
- IV 1: Reading Platform (Paper, Kindle, Computer)
- IV 2: Age Group (for simplicity, the levels are just Old and Young. If you would like to examine age as a continuous variable, you can run a regression analysis. See Chapter 12 for more information on the basics of performing a regression analyses in R)
- DV: Comprehension Score (1-10)
Here, I am creating a dataframe called “Comprehension.Data”, inserting 60 participants into the dataframe, adding reading platform as a between-subjects variable with 20 participants per level, adding age group as another between-subjects variable with 10 participants in each level. Then, I am filling the dataframe with 10 participants per condition with 6 conditions. Specifying the means and standard deviations for each condition. See Chapter 7 for more information on the rnorm function.
set.seed(42) #this is make sure you get the same values randomly each time
N.2=10 #sample size of age
N.3=20 #sample size of reading platform
Comprehension.Data <- data.frame(
Subject = seq(1,N.3*3),
Platform = c(rep("Paper", N.3), rep("Kindle", N.3), rep("Computer", N.3)),
Age = rep(c(rep("Old",N.2), rep("Young", N.2)),3),
Comprehension = c(rnorm(N.2,10,.5),rnorm(N.2,8,.5),
rnorm(N.2,6,.5),rnorm(N.2,7,.5),
rnorm(N.2,2,.5),rnorm(N.2,6,.5)))
Note: this dataset is in long-format. If your dataset is in wide-format, you have to “reshape” the data (See Chapter 8 for more information).
Here we can view the structure of the dataframe.
str(Comprehension.Data)
## 'data.frame': 60 obs. of 4 variables:
## $ Subject : int 1 2 3 4 5 6 7 8 9 10 ...
## $ Platform : Factor w/ 3 levels "Computer","Kindle",..: 3 3 3 3 3 3 3 3 3 3 ...
## $ Age : Factor w/ 2 levels "Old","Young": 1 1 1 1 1 1 1 1 1 1 ...
## $ Comprehension: num 10.69 9.72 10.18 10.32 10.2 ...
Here we can view the first few lines of the data. You can also look at the entire dataframe by removing head, and just calling for Comprehension.Data.
head(Comprehension.Data)
## Subject Platform Age Comprehension
## 1 1 Paper Old 10.685479
## 2 2 Paper Old 9.717651
## 3 3 Paper Old 10.181564
## 4 4 Paper Old 10.316431
## 5 5 Paper Old 10.202134
## 6 6 Paper Old 9.946938
Let’s look at our means for each of the six conditions. We can use the tapply function to do this. (See Chapter 3 for more information on the different apply functions).
tapply(Comprehension.Data$Comprehension,
list(Comprehension.Data$Platform,Comprehension.Data$Age), mean)
## Old Young
## Computer 1.989892 6.009197
## Kindle 5.910960 6.818048
## Paper 10.273648 7.918272
We can also look at the means collapsed across each independent variable. Here are the means for the two Age levels collapsed across Platform.
tapply(Comprehension.Data$Comprehension,
list(Comprehension.Data$Age), mean)
## Old Young
## 6.058167 6.915172
And here are the means for the three Platform levels collapsed across Age.
tapply(Comprehension.Data$Comprehension,
list(Comprehension.Data$Platform), mean)
## Computer Kindle Paper
## 3.999545 6.364504 9.095960
What is the aov function?
This is a built-in R function that allows you to run an Analysis of Variance (ANOVA). This function defaults to running a Type I Sum of Squares.
You can use the help section to see a description of the aov function where it will display the arguments that go into this function. Putting a “?” in front of a function is a way to easily find more information on the function.
?aov
Running an ANOVA using the aov function
Here, I am just inserting the names of my variables into the function, and specifying the dataset I am working with.
Model.aov.1<-aov(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data)
summary(Model.aov.1)
##
## Error: Subject
## Df Sum Sq Mean Sq
## Platform 1 207.4 207.4
##
## Error: Within
## Df Sum Sq Mean Sq F value Pr(>F)
## Platform 2 58.47 29.23 89.25 < 2e-16 ***
## Age 1 6.29 6.29 19.20 5.59e-05 ***
## Platform:Age 2 101.61 50.81 155.10 < 2e-16 ***
## Residuals 53 17.36 0.33
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
If you would like to specify Type II or III Sum of Squares using the aov function, you can do that by using the afex package below:
- Langsrud, Ø. (2003). ANOVA for Unbalanced Data: Use Type II Instead of Type III Sums of Squares. Statistics and Computing, 13, 163-167.
What is the afex package?
The afex (“Analysis of Factorial Experiments”) package is an alternative to using the aov function to run an ANOVA in R.
There are three different functions in the afex package related to calculating an ANOVA:
- aov_car (This is the main function we will focus on for this tutorial)
- aov_ez (same as aov_car, but has a different layout, easier version if you don’t understand how to specify your error term)
- aov_4 (wrapper function for lmer function in the lme4 package for mixed-effects models)
You can use the help section to get a more detailed description of the afex package functions.
?aov_car
Using the aov_car function
The aov_car function defaults to Type III Sum of Squares, which is the default in SPSS. It also gives you the generalized eta-squared, not the partial eta-squared. Additionally, this function autocorrects for violations of sphericity in RM and Mixed designs (see Chapter 21), and has a better output format than the built-in aov function as it provides effect sizes. In this case our “Age” variable was observed and not manipulated so we have passed that into the function as it will impact our effect size estimates.
Model.aov.2<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data, observed = "Age")
summary(Model.aov.2) #Defaults to type III Sum of Squares
## Anova Table (Type 3 tests)
##
## Response: Comprehension
## num Df den Df MSE F ges Pr(>F)
## Platform 2 54 0.33949 383.199 0.66519 < 2.2e-16 ***
## Age 1 54 0.33949 32.452 0.08412 5.202e-07 ***
## Platform:Age 2 54 0.33949 149.652 0.77589 < 2.2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
We can have it show us the partial eta-squared if we want. This is what SPSS outputs for us. For more information on the differences between generalized eta-squared and partial eta-squared, refer to the following paper:
- Levine, T.R. & Hullett, C.R. (2002). Eta Squared, Partial Eta Squared and the Misreporting of Effect Size in Communication Research. Human Communication Research, 28, 612-625.
Note: PES and GES would be the same in fully-between subjects designs when all the IVs are manipulated.
Model.aov.3<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data,
anova_table = list(es = "pes"))
summary(Model.aov.3)
## Anova Table (Type 3 tests)
##
## Response: Comprehension
## num Df den Df MSE F pes Pr(>F)
## Platform 2 54 0.33949 383.199 0.93418 < 2.2e-16 ***
## Age 1 54 0.33949 32.452 0.37537 5.202e-07 ***
## Platform:Age 2 54 0.33949 149.652 0.84716 < 2.2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
We can also have it show us an output table like what you would get in SPSS by adding return="univariate", but it will not show you Partial Eta Squared.
Model.aov.4<-aov_car(Comprehension ~ Platform*Age + Error(Subject), data=Comprehension.Data,
anova_table = list(es = "pes"),
return="univariate")
Model.aov.4
## Anova Table (Type III tests)
##
## Response: dv
## Sum Sq Df F value Pr(>F)
## (Intercept) 2524.61 1 7436.541 < 2.2e-16 ***
## Platform 260.18 2 383.199 < 2.2e-16 ***
## Age 11.02 1 32.452 5.202e-07 ***
## Platform:Age 101.61 2 149.652 < 2.2e-16 ***
## Residuals 18.33 54
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Follow-up tests
Main Effects
In the hypothetical dataset, we have a significant main effect of Age (we can see from our Age mean table above that the Young group had significantly higher comprehension than the Old group), a significant main effect of Platform, and a significant Age by Platform interaction. We are going to first follow-up the significant main effect of Platform because this variable contains more than two levels.
We will follow-up this main effect by performing a pairwise analysis on fitted means to see where the significant differences between these levels lie within the Platform factor. Note below that it automatically applied the Tukey HSD correction for multiple corrections.
FittedMeans.Platform<-emmeans(Model.aov.2, ~Platform)
FittedMeans.Platform
## Platform emmean SE df lower.CL upper.CL
## Computer 3.999545 0.1302858 54 3.738337 4.260752
## Kindle 6.364504 0.1302858 54 6.103297 6.625711
## Paper 9.095960 0.1302858 54 8.834753 9.357167
##
## Results are averaged over the levels of: Age
## Confidence level used: 0.95
pairs(FittedMeans.Platform)
## contrast estimate SE df t.ratio p.value
## Computer - Kindle -2.364959 0.1842519 54 -12.835 <.0001
## Computer - Paper -5.096415 0.1842519 54 -27.660 <.0001
## Kindle - Paper -2.731456 0.1842519 54 -14.825 <.0001
##
## Results are averaged over the levels of: Age
## P value adjustment: tukey method for comparing a family of 3 estimates
We can also do this using the Bonferroni correction.
pairs(FittedMeans.Platform, adjust="bon")
## contrast estimate SE df t.ratio p.value
## Computer - Kindle -2.364959 0.1842519 54 -12.835 <.0001
## Computer - Paper -5.096415 0.1842519 54 -27.660 <.0001
## Kindle - Paper -2.731456 0.1842519 54 -14.825 <.0001
##
## Results are averaged over the levels of: Age
## P value adjustment: bonferroni method for 3 tests
These pairwise comparisons reveal that comprehension score significantly differed between all Platform groups (Paper, Kindle, and Computer).
Interaction
We also need to follow-up the significant Age by Platform interaction by performing pairwise effects for Age on Platform. We will do this using no pvalue correction as there are only 2 levels within each groups (and emmeans reads this as 1 test per family).
FittedMeans.AgebyPlatform <- emmeans(Model.aov.2, ~ Age|Platform)
FittedMeans.AgebyPlatform
## Platform = Computer:
## Age emmean SE df lower.CL upper.CL
## Old 1.989892 0.1842519 54 1.620490 2.359295
## Young 6.009197 0.1842519 54 5.639794 6.378600
##
## Platform = Kindle:
## Age emmean SE df lower.CL upper.CL
## Old 5.910960 0.1842519 54 5.541557 6.280363
## Young 6.818048 0.1842519 54 6.448645 7.187451
##
## Platform = Paper:
## Age emmean SE df lower.CL upper.CL
## Old 10.273648 0.1842519 54 9.904246 10.643051
## Young 7.918272 0.1842519 54 7.548869 8.287674
##
## Confidence level used: 0.95
pairs(FittedMeans.AgebyPlatform)
## Platform = Computer:
## contrast estimate SE df t.ratio p.value
## Old - Young -4.019305 0.2605715 54 -15.425 <.0001
##
## Platform = Kindle:
## contrast estimate SE df t.ratio p.value
## Old - Young -0.907088 0.2605715 54 -3.481 0.0010
##
## Platform = Paper:
## contrast estimate SE df t.ratio p.value
## Old - Young 2.355377 0.2605715 54 9.039 <.0001
For additional information on follow-ups for between-subjects designs see: http://www.alexanderdemos.org/ANOVA9.html
Plotting the results
Here, I will plot the results on a bar graph using the ggplot2 package. This is a very basic graph, but if you want more information on how to do more fancy plots, refer to Chapters 10 & 11 or http://www.alexanderdemos.org/ANOVA8.html#analysis_in_r.
Plot.ComprehensionData <-ggplot(data = Comprehension.Data, aes(x = Platform, y =Comprehension, fill=Age))+
coord_cartesian(ylim = c(1,12))+ #specifying the y-axis scale
geom_bar(stat='identity', position=position_dodge(width=1))+
xlab("Platform Type")+ylab("Comprehension Score")+ #labelling the x-axis and y-axis
theme_bw()+
theme(panel.grid.major=element_blank(),
panel.grid.minor=element_blank())+
scale_fill_grey()
Plot.ComprehensionData #displaying the graph
LS0tDQp0aXRsZTogIkNoYXB0ZXIgMjA6IEJldHdlZW4tU3ViamVjdHMgQU5PVkEgaW4gUiINCmF1dGhvcjogIkFuZHJpYW5hIENocmlzdG9mYWxvcyINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDoNCiAgICB0aGVtZTogY2VydWxlYW4NCiAgICBoaWdobGlnaHQ6IHRleHRtYXRlDQogICAgZm9udHNpemU6IDhwdA0KICAgIHRvYzogdHJ1ZQ0KICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQ0KICAgIGNvZGVfZG93bmxvYWQ6IHRydWUNCiAgICB0b2NfZmxvYXQ6DQogICAgICBjb2xsYXBzZWQ6IGZhbHNlDQotLS0NCg0KYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9DQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpDQpgYGANCg0KVGhpcyBwcmVzZW50YXRpb24gd2lsbCByZXZpZXcgdGhlIGJhc2ljcyBpbiBob3cgdG8gcGVyZm9ybSBhIGJldHdlZW4tc3ViamVjdHMgQU5PVkEgaW4gUiB1c2luZyB0aGUgYW92IGZ1bmN0aW9uIGFuZCB0aGUgYWZleCBwYWNrYWdlLiBJIHdpbGwgZ28gdGhyb3VnaCB0aGlzIHVzaW5nIGEgZ2VuZXJhdGVkIGRhdGFzZXQuIA0KDQpCdXQgYmVmb3JlIHJ1bm5pbmcgdGhpcyBjb2RlLCB5b3Ugd2lsbCBuZWVkIHRvIGxvYWQgdGhlIGZvbGxvd2luZyBuZWNlc3NhcnkgcGFja2FnZSBsaWJyYXJpZXMuIElmIHlvdSBkb24ndCBoYXZlIHRoZSBwYWNrYWdlcyBpbnN0YWxsZWQsIHlvdSB3aWxsIG5lZWQgdG8gaW5zdGFsbCB0aGVtIGJlZm9yZSBjb250aW51aW5nLiBUbyBpbnN0YWxsIHRoZW0sIGRlbGV0ZSB0aGUgIiMiIGluIHRoZSBmb2xsb3dpbmcgbGluZXMgb2YgY29kZSBiZWZvcmUgcnVubmluZyBpdC4NCg0KYGBge3IsIG1lc3NhZ2U9RkFMU0V9DQojaW5zdGFsbC5wYWNrYWdlcyhwbHlyKQ0KI2luc3RhbGwucGFja2FnZXMoZ2dwbG90MikNCiNpbnN0YWxsLnBhY2thZ2VzKGFmZXgpDQojaW5zdGFsbC5wYWNrYWdlcyhlbW1lYW5zKQ0KDQpsaWJyYXJ5KHBseXIpICAgDQpsaWJyYXJ5KGdncGxvdDIpICANCmxpYnJhcnkoYWZleCkgIA0KbGlicmFyeShlbW1lYW5zKQ0KYGBgDQoNCiMgR2VuZXJhdGluZyBhIGRhdGFzZXQNClRoZSBjcmVhdGVkIGRhdGEgc2V0IGZvciBhIGh5cG90aGV0aWNhbCAyeDMgYmV0d2Vlbi1zdWJqZWN0cyBleHBlcmltZW50YWwgZGVzaWduIHdoZXJlIHBhcnRpY2lwYW50cyBhcmUgYXNrZWQgdG8gcmVhZCBhIHBhc3NhZ2Ugb24gZGlmZmVyZW50IHBsYXRmb3JtcyBhbmQgYW5zd2VyIHRlbiBjb21wcmVoZW5zaW9uIHF1ZXN0aW9ucyBhYm91dCB0aGUgcGFzc2FnZS4gVGhlIHB1cnBvc2Ugb2YgdGhpcyBoeXBvdGhldGljYWwgZXhwZXJpbWVudCBpcyB0byBleGFtaW5lIHRoZSBlZmZlY3QgdGhhdCByZWFkaW5nIHBsYXRmb3JtIGhhcyBvbiB0ZXh0IGNvbXByZWhlbnNpb24gYXMgYSBmdW5jdGlvbiBvZiBhZ2UuDQoNCi0gSVYgMTogUmVhZGluZyBQbGF0Zm9ybSAoUGFwZXIsIEtpbmRsZSwgQ29tcHV0ZXIpDQotIElWIDI6IEFnZSBHcm91cCAoZm9yIHNpbXBsaWNpdHksIHRoZSBsZXZlbHMgYXJlIGp1c3QgT2xkIGFuZCBZb3VuZy4gSWYgeW91IHdvdWxkIGxpa2UgdG8gZXhhbWluZSBhZ2UgYXMgYSBjb250aW51b3VzIHZhcmlhYmxlLCB5b3UgY2FuIHJ1biBhIHJlZ3Jlc3Npb24gYW5hbHlzaXMuIFNlZSBDaGFwdGVyIDEyIGZvciBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBiYXNpY3Mgb2YgcGVyZm9ybWluZyBhIHJlZ3Jlc3Npb24gYW5hbHlzZXMgaW4gUikNCi0gRFY6IENvbXByZWhlbnNpb24gU2NvcmUgKDEtMTApDQoNCkhlcmUsIEkgYW0gY3JlYXRpbmcgYSBkYXRhZnJhbWUgY2FsbGVkICJDb21wcmVoZW5zaW9uLkRhdGEiLCBpbnNlcnRpbmcgNjAgcGFydGljaXBhbnRzIGludG8gdGhlIGRhdGFmcmFtZSwgYWRkaW5nIHJlYWRpbmcgcGxhdGZvcm0gYXMgYSBiZXR3ZWVuLXN1YmplY3RzIHZhcmlhYmxlIHdpdGggMjAgcGFydGljaXBhbnRzIHBlciBsZXZlbCwgYWRkaW5nIGFnZSBncm91cCBhcyBhbm90aGVyIGJldHdlZW4tc3ViamVjdHMgdmFyaWFibGUgd2l0aCAxMCBwYXJ0aWNpcGFudHMgaW4gZWFjaCBsZXZlbC4gVGhlbiwgSSBhbSBmaWxsaW5nIHRoZSBkYXRhZnJhbWUgd2l0aCAxMCBwYXJ0aWNpcGFudHMgcGVyIGNvbmRpdGlvbiB3aXRoIDYgY29uZGl0aW9ucy4gU3BlY2lmeWluZyB0aGUgbWVhbnMgYW5kIHN0YW5kYXJkIGRldmlhdGlvbnMgZm9yIGVhY2ggY29uZGl0aW9uLiBTZWUgQ2hhcHRlciA3IGZvciBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBgYGBybm9ybWBgYCBmdW5jdGlvbi4gDQoNCmBgYHtyLCBlY2hvPVRSVUV9DQpzZXQuc2VlZCg0MikgI3RoaXMgaXMgbWFrZSBzdXJlIHlvdSBnZXQgdGhlIHNhbWUgdmFsdWVzIHJhbmRvbWx5IGVhY2ggdGltZQ0KDQpOLjI9MTAgI3NhbXBsZSBzaXplIG9mIGFnZSANCk4uMz0yMCAjc2FtcGxlIHNpemUgb2YgcmVhZGluZyBwbGF0Zm9ybQ0KDQpDb21wcmVoZW5zaW9uLkRhdGEgPC0gZGF0YS5mcmFtZSggDQogU3ViamVjdCA9IHNlcSgxLE4uMyozKSwgDQogUGxhdGZvcm0gPSBjKHJlcCgiUGFwZXIiLCBOLjMpLCByZXAoIktpbmRsZSIsIE4uMyksIHJlcCgiQ29tcHV0ZXIiLCBOLjMpKSwgDQogQWdlID0gcmVwKGMocmVwKCJPbGQiLE4uMiksIHJlcCgiWW91bmciLCBOLjIpKSwzKSwgDQogQ29tcHJlaGVuc2lvbiA9IGMocm5vcm0oTi4yLDEwLC41KSxybm9ybShOLjIsOCwuNSksIA0KICAgICAgICAgICAgICAgICAgICBybm9ybShOLjIsNiwuNSkscm5vcm0oTi4yLDcsLjUpLA0KICAgICAgICAgICAgICAgICAgICBybm9ybShOLjIsMiwuNSkscm5vcm0oTi4yLDYsLjUpKSkgDQpgYGANCg0KTm90ZTogdGhpcyBkYXRhc2V0IGlzIGluIGxvbmctZm9ybWF0LiBJZiB5b3VyIGRhdGFzZXQgaXMgaW4gd2lkZS1mb3JtYXQsIHlvdSBoYXZlIHRvICJyZXNoYXBlIiB0aGUgZGF0YSAoU2VlIENoYXB0ZXIgOCBmb3IgbW9yZSBpbmZvcm1hdGlvbikuDQoNCkhlcmUgd2UgY2FuIHZpZXcgdGhlIHN0cnVjdHVyZSBvZiB0aGUgZGF0YWZyYW1lLg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCnN0cihDb21wcmVoZW5zaW9uLkRhdGEpIA0KYGBgDQoNCkhlcmUgd2UgY2FuIHZpZXcgdGhlIGZpcnN0IGZldyBsaW5lcyBvZiB0aGUgZGF0YS4gWW91IGNhbiBhbHNvIGxvb2sgYXQgdGhlIGVudGlyZSBkYXRhZnJhbWUgYnkgcmVtb3ZpbmcgYGBgaGVhZGBgYCwgYW5kIGp1c3QgY2FsbGluZyBmb3IgYGBgQ29tcHJlaGVuc2lvbi5EYXRhYGBgLg0KDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KaGVhZChDb21wcmVoZW5zaW9uLkRhdGEpDQpgYGANCg0KTGV0J3MgbG9vayBhdCBvdXIgbWVhbnMgZm9yIGVhY2ggb2YgdGhlIHNpeCBjb25kaXRpb25zLiBXZSBjYW4gdXNlIHRoZSBgYGB0YXBwbHlgYGAgZnVuY3Rpb24gdG8gZG8gdGhpcy4gKFNlZSBDaGFwdGVyIDMgZm9yIG1vcmUgaW5mb3JtYXRpb24gb24gdGhlIGRpZmZlcmVudCBhcHBseSBmdW5jdGlvbnMpLg0KDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KdGFwcGx5KENvbXByZWhlbnNpb24uRGF0YSRDb21wcmVoZW5zaW9uLA0KICAgICAgICBsaXN0KENvbXByZWhlbnNpb24uRGF0YSRQbGF0Zm9ybSxDb21wcmVoZW5zaW9uLkRhdGEkQWdlKSwgbWVhbikNCmBgYA0KDQpXZSBjYW4gYWxzbyBsb29rIGF0IHRoZSBtZWFucyBjb2xsYXBzZWQgYWNyb3NzIGVhY2ggaW5kZXBlbmRlbnQgdmFyaWFibGUuIEhlcmUgYXJlIHRoZSBtZWFucyBmb3IgdGhlIHR3byBBZ2UgbGV2ZWxzIGNvbGxhcHNlZCBhY3Jvc3MgUGxhdGZvcm0uDQoNCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQp0YXBwbHkoQ29tcHJlaGVuc2lvbi5EYXRhJENvbXByZWhlbnNpb24sDQogICAgICAgIGxpc3QoQ29tcHJlaGVuc2lvbi5EYXRhJEFnZSksIG1lYW4pDQpgYGANCg0KQW5kIGhlcmUgYXJlIHRoZSBtZWFucyBmb3IgdGhlIHRocmVlIFBsYXRmb3JtIGxldmVscyBjb2xsYXBzZWQgYWNyb3NzIEFnZS4NCg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCnRhcHBseShDb21wcmVoZW5zaW9uLkRhdGEkQ29tcHJlaGVuc2lvbiwNCiAgICAgICAgbGlzdChDb21wcmVoZW5zaW9uLkRhdGEkUGxhdGZvcm0pLCBtZWFuKQ0KYGBgDQoNCiMgV2hhdCBpcyB0aGUgYW92IGZ1bmN0aW9uPw0KVGhpcyBpcyBhIGJ1aWx0LWluIFIgZnVuY3Rpb24gdGhhdCBhbGxvd3MgeW91IHRvIHJ1biBhbiBBbmFseXNpcyBvZiBWYXJpYW5jZSAoQU5PVkEpLiBUaGlzIGZ1bmN0aW9uIGRlZmF1bHRzIHRvIHJ1bm5pbmcgYSBUeXBlIEkgU3VtIG9mIFNxdWFyZXMuDQoNCllvdSBjYW4gdXNlIHRoZSBoZWxwIHNlY3Rpb24gdG8gc2VlIGEgZGVzY3JpcHRpb24gb2YgdGhlIGFvdiBmdW5jdGlvbiB3aGVyZSBpdCB3aWxsIGRpc3BsYXkgdGhlIGFyZ3VtZW50cyB0aGF0IGdvIGludG8gdGhpcyBmdW5jdGlvbi4gUHV0dGluZyBhICI/IiBpbiBmcm9udCBvZiBhIGZ1bmN0aW9uIGlzIGEgd2F5IHRvIGVhc2lseSBmaW5kIG1vcmUgaW5mb3JtYXRpb24gb24gdGhlIGZ1bmN0aW9uLiANCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQo/YW92DQpgYGANCg0KIyMgUnVubmluZyBhbiBBTk9WQSB1c2luZyB0aGUgYW92IGZ1bmN0aW9uDQpIZXJlLCBJIGFtIGp1c3QgaW5zZXJ0aW5nIHRoZSBuYW1lcyBvZiBteSB2YXJpYWJsZXMgaW50byB0aGUgZnVuY3Rpb24sIGFuZCBzcGVjaWZ5aW5nIHRoZSBkYXRhc2V0IEkgYW0gd29ya2luZyB3aXRoLiANCg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCk1vZGVsLmFvdi4xPC1hb3YoQ29tcHJlaGVuc2lvbiB+IFBsYXRmb3JtKkFnZSArIEVycm9yKFN1YmplY3QpLCBkYXRhPUNvbXByZWhlbnNpb24uRGF0YSkNCg0Kc3VtbWFyeShNb2RlbC5hb3YuMSkgICAgICAgICAgICANCmBgYA0KDQpJZiB5b3Ugd291bGQgbGlrZSB0byBzcGVjaWZ5IFR5cGUgSUkgb3IgSUlJIFN1bSBvZiBTcXVhcmVzIHVzaW5nIHRoZSBhb3YgZnVuY3Rpb24sIHlvdSBjYW4gZG8gdGhhdCBieSB1c2luZyB0aGUgYGFmZXhgIHBhY2thZ2UgYmVsb3c6DQoNCi0gTGFuZ3NydWQsIMOYLiAoMjAwMykuIEFOT1ZBIGZvciBVbmJhbGFuY2VkIERhdGE6IFVzZSBUeXBlIElJIEluc3RlYWQgb2YgVHlwZSBJSUkgU3VtcyBvZiBTcXVhcmVzLiAqU3RhdGlzdGljcyBhbmQgQ29tcHV0aW5nKiwgMTMsIDE2My0xNjcuIA0KDQojIFdoYXQgaXMgdGhlIGFmZXggcGFja2FnZT8gDQpUaGUgYWZleCAoIkFuYWx5c2lzIG9mIEZhY3RvcmlhbCBFeHBlcmltZW50cyIpIHBhY2thZ2UgaXMgYW4gYWx0ZXJuYXRpdmUgdG8gdXNpbmcgdGhlIGFvdiBmdW5jdGlvbiB0byBydW4gYW4gQU5PVkEgaW4gUi4gIA0KDQpUaGVyZSBhcmUgdGhyZWUgZGlmZmVyZW50IGZ1bmN0aW9ucyBpbiB0aGUgYWZleCBwYWNrYWdlIHJlbGF0ZWQgdG8gY2FsY3VsYXRpbmcgYW4gQU5PVkE6DQoNCi0gYW92X2NhciAoVGhpcyBpcyB0aGUgbWFpbiBmdW5jdGlvbiB3ZSB3aWxsIGZvY3VzIG9uIGZvciB0aGlzIHR1dG9yaWFsKQ0KLSBhb3ZfZXogKHNhbWUgYXMgYW92X2NhciwgYnV0IGhhcyBhIGRpZmZlcmVudCBsYXlvdXQsIGVhc2llciB2ZXJzaW9uIGlmIHlvdSBkb24ndCB1bmRlcnN0YW5kIGhvdyB0byBzcGVjaWZ5IHlvdXIgZXJyb3IgdGVybSkNCi0gYW92XzQgKHdyYXBwZXIgZnVuY3Rpb24gZm9yIGxtZXIgZnVuY3Rpb24gaW4gdGhlIGxtZTQgcGFja2FnZSBmb3IgbWl4ZWQtZWZmZWN0cyBtb2RlbHMpDQoNCllvdSBjYW4gdXNlIHRoZSBoZWxwIHNlY3Rpb24gdG8gZ2V0IGEgbW9yZSBkZXRhaWxlZCBkZXNjcmlwdGlvbiBvZiB0aGUgYWZleCBwYWNrYWdlIGZ1bmN0aW9ucy4gDQoNCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQo/YW92X2Nhcg0KYGBgDQoNCiMjIFVzaW5nIHRoZSBhb3ZfY2FyIGZ1bmN0aW9uDQpUaGUgYW92X2NhciBmdW5jdGlvbiBkZWZhdWx0cyB0byBUeXBlIElJSSBTdW0gb2YgU3F1YXJlcywgd2hpY2ggaXMgdGhlIGRlZmF1bHQgaW4gU1BTUy4gSXQgYWxzbyBnaXZlcyB5b3UgdGhlIGdlbmVyYWxpemVkIGV0YS1zcXVhcmVkLCBub3QgdGhlIHBhcnRpYWwgZXRhLXNxdWFyZWQuIEFkZGl0aW9uYWxseSwgdGhpcyBmdW5jdGlvbiBhdXRvY29ycmVjdHMgZm9yIHZpb2xhdGlvbnMgb2Ygc3BoZXJpY2l0eSBpbiBSTSBhbmQgTWl4ZWQgZGVzaWducyAoc2VlIENoYXB0ZXIgMjEpLCBhbmQgaGFzIGEgYmV0dGVyIG91dHB1dCBmb3JtYXQgdGhhbiB0aGUgYnVpbHQtaW4gYW92IGZ1bmN0aW9uIGFzIGl0IHByb3ZpZGVzIGVmZmVjdCBzaXplcy4gSW4gdGhpcyBjYXNlIG91ciAiQWdlIiB2YXJpYWJsZSB3YXMgb2JzZXJ2ZWQgYW5kIG5vdCBtYW5pcHVsYXRlZCBzbyB3ZSBoYXZlIHBhc3NlZCB0aGF0IGludG8gdGhlIGZ1bmN0aW9uIGFzIGl0IHdpbGwgaW1wYWN0IG91ciBlZmZlY3Qgc2l6ZSBlc3RpbWF0ZXMuICANCg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NCk1vZGVsLmFvdi4yPC1hb3ZfY2FyKENvbXByZWhlbnNpb24gfiBQbGF0Zm9ybSpBZ2UgKyBFcnJvcihTdWJqZWN0KSwgZGF0YT1Db21wcmVoZW5zaW9uLkRhdGEsIG9ic2VydmVkID0gIkFnZSIpDQoNCnN1bW1hcnkoTW9kZWwuYW92LjIpICNEZWZhdWx0cyB0byB0eXBlIElJSSBTdW0gb2YgU3F1YXJlcw0KYGBgDQoNCldlIGNhbiBoYXZlIGl0IHNob3cgdXMgdGhlIHBhcnRpYWwgZXRhLXNxdWFyZWQgaWYgd2Ugd2FudC4gVGhpcyBpcyB3aGF0IFNQU1Mgb3V0cHV0cyBmb3IgdXMuIEZvciBtb3JlIGluZm9ybWF0aW9uIG9uIHRoZSBkaWZmZXJlbmNlcyBiZXR3ZWVuIGdlbmVyYWxpemVkIGV0YS1zcXVhcmVkIGFuZCBwYXJ0aWFsIGV0YS1zcXVhcmVkLCByZWZlciB0byB0aGUgZm9sbG93aW5nIHBhcGVyOg0KDQotIExldmluZSwgVC5SLiAmIEh1bGxldHQsIEMuUi4gKDIwMDIpLiBFdGEgU3F1YXJlZCwgUGFydGlhbCBFdGEgU3F1YXJlZCBhbmQgdGhlIE1pc3JlcG9ydGluZyBvZiBFZmZlY3QgU2l6ZSBpbiBDb21tdW5pY2F0aW9uIFJlc2VhcmNoLiAqSHVtYW4gQ29tbXVuaWNhdGlvbiBSZXNlYXJjaCosIDI4LCA2MTItNjI1LiANCg0KKk5vdGU6IFBFUyBhbmQgR0VTIHdvdWxkIGJlIHRoZSBzYW1lIGluIGZ1bGx5LWJldHdlZW4gc3ViamVjdHMgZGVzaWducyB3aGVuIGFsbCB0aGUgSVZzIGFyZSBtYW5pcHVsYXRlZC4qDQoNCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpNb2RlbC5hb3YuMzwtYW92X2NhcihDb21wcmVoZW5zaW9uIH4gUGxhdGZvcm0qQWdlICsgRXJyb3IoU3ViamVjdCksIGRhdGE9Q29tcHJlaGVuc2lvbi5EYXRhLA0KICAgICAgICAgICAgICAgICAgICAgYW5vdmFfdGFibGUgPSBsaXN0KGVzID0gInBlcyIpKQ0KDQpzdW1tYXJ5KE1vZGVsLmFvdi4zKQ0KYGBgDQoNCldlIGNhbiBhbHNvIGhhdmUgaXQgc2hvdyB1cyBhbiBvdXRwdXQgdGFibGUgbGlrZSB3aGF0IHlvdSB3b3VsZCBnZXQgaW4gU1BTUyBieSBhZGRpbmcgYGBgcmV0dXJuPSJ1bml2YXJpYXRlImBgYCwgYnV0IGl0IHdpbGwgbm90IHNob3cgeW91IFBhcnRpYWwgRXRhIFNxdWFyZWQuIA0KDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KTW9kZWwuYW92LjQ8LWFvdl9jYXIoQ29tcHJlaGVuc2lvbiB+IFBsYXRmb3JtKkFnZSArIEVycm9yKFN1YmplY3QpLCBkYXRhPUNvbXByZWhlbnNpb24uRGF0YSwNCiAgICAgICAgICAgICAgICAgICAgIGFub3ZhX3RhYmxlID0gbGlzdChlcyA9ICJwZXMiKSwNCiAgICAgICAgICAgICAgICAgICAgIHJldHVybj0idW5pdmFyaWF0ZSIpDQpNb2RlbC5hb3YuNA0KYGBgDQoNCiMgRm9sbG93LXVwIHRlc3RzDQojIyBNYWluIEVmZmVjdHMNCg0KSW4gdGhlIGh5cG90aGV0aWNhbCBkYXRhc2V0LCB3ZSBoYXZlIGEgc2lnbmlmaWNhbnQgbWFpbiBlZmZlY3Qgb2YgQWdlICh3ZSBjYW4gc2VlIGZyb20gb3VyIEFnZSBtZWFuIHRhYmxlIGFib3ZlIHRoYXQgdGhlIFlvdW5nIGdyb3VwIGhhZCBzaWduaWZpY2FudGx5IGhpZ2hlciBjb21wcmVoZW5zaW9uIHRoYW4gdGhlIE9sZCBncm91cCksIGEgc2lnbmlmaWNhbnQgbWFpbiBlZmZlY3Qgb2YgUGxhdGZvcm0sIGFuZCBhIHNpZ25pZmljYW50IEFnZSBieSBQbGF0Zm9ybSBpbnRlcmFjdGlvbi4gV2UgYXJlIGdvaW5nIHRvIGZpcnN0IGZvbGxvdy11cCB0aGUgc2lnbmlmaWNhbnQgbWFpbiBlZmZlY3Qgb2YgUGxhdGZvcm0gYmVjYXVzZSB0aGlzIHZhcmlhYmxlIGNvbnRhaW5zIG1vcmUgdGhhbiB0d28gbGV2ZWxzLiANCg0KV2Ugd2lsbCBmb2xsb3ctdXAgdGhpcyBtYWluIGVmZmVjdCBieSBwZXJmb3JtaW5nIGEgcGFpcndpc2UgYW5hbHlzaXMgb24gZml0dGVkIG1lYW5zIHRvIHNlZSB3aGVyZSB0aGUgc2lnbmlmaWNhbnQgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGVzZSBsZXZlbHMgbGllIHdpdGhpbiB0aGUgUGxhdGZvcm0gZmFjdG9yLiBOb3RlIGJlbG93IHRoYXQgaXQgYXV0b21hdGljYWxseSBhcHBsaWVkIHRoZSBUdWtleSBIU0QgY29ycmVjdGlvbiBmb3IgbXVsdGlwbGUgY29ycmVjdGlvbnMuIA0KDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KRml0dGVkTWVhbnMuUGxhdGZvcm08LWVtbWVhbnMoTW9kZWwuYW92LjIsIH5QbGF0Zm9ybSkNCkZpdHRlZE1lYW5zLlBsYXRmb3JtDQoNCnBhaXJzKEZpdHRlZE1lYW5zLlBsYXRmb3JtKSANCmBgYA0KDQpXZSBjYW4gYWxzbyBkbyB0aGlzIHVzaW5nIHRoZSBCb25mZXJyb25pIGNvcnJlY3Rpb24uIA0KDQpgYGB7ciwgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFfQ0KcGFpcnMoRml0dGVkTWVhbnMuUGxhdGZvcm0sIGFkanVzdD0iYm9uIikgDQpgYGANCg0KVGhlc2UgcGFpcndpc2UgY29tcGFyaXNvbnMgcmV2ZWFsIHRoYXQgY29tcHJlaGVuc2lvbiBzY29yZSBzaWduaWZpY2FudGx5IGRpZmZlcmVkIGJldHdlZW4gYWxsIFBsYXRmb3JtIGdyb3VwcyAoUGFwZXIsIEtpbmRsZSwgYW5kIENvbXB1dGVyKS4gDQoNCiMjIEludGVyYWN0aW9uDQoNCldlIGFsc28gbmVlZCB0byBmb2xsb3ctdXAgdGhlIHNpZ25pZmljYW50IEFnZSBieSBQbGF0Zm9ybSBpbnRlcmFjdGlvbiBieSBwZXJmb3JtaW5nIHBhaXJ3aXNlIGVmZmVjdHMgZm9yIEFnZSBvbiBQbGF0Zm9ybS4gV2Ugd2lsbCBkbyB0aGlzIHVzaW5nIG5vIHB2YWx1ZSBjb3JyZWN0aW9uIGFzIHRoZXJlIGFyZSBvbmx5IDIgbGV2ZWxzIHdpdGhpbiBlYWNoIGdyb3VwcyAoYW5kIGVtbWVhbnMgcmVhZHMgdGhpcyBhcyAxIHRlc3QgcGVyIGZhbWlseSkuDQoNCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0V9DQpGaXR0ZWRNZWFucy5BZ2VieVBsYXRmb3JtIDwtIGVtbWVhbnMoTW9kZWwuYW92LjIsIH4gQWdlfFBsYXRmb3JtKSANCg0KRml0dGVkTWVhbnMuQWdlYnlQbGF0Zm9ybQ0KDQpwYWlycyhGaXR0ZWRNZWFucy5BZ2VieVBsYXRmb3JtKQ0KYGBgDQoNCipGb3IgYWRkaXRpb25hbCBpbmZvcm1hdGlvbiBvbiBmb2xsb3ctdXBzIGZvciBiZXR3ZWVuLXN1YmplY3RzIGRlc2lnbnMgc2VlOiBodHRwOi8vd3d3LmFsZXhhbmRlcmRlbW9zLm9yZy9BTk9WQTkuaHRtbCoNCg0KIyBQbG90dGluZyB0aGUgcmVzdWx0cw0KDQpIZXJlLCBJIHdpbGwgcGxvdCB0aGUgcmVzdWx0cyBvbiBhIGJhciBncmFwaCB1c2luZyB0aGUgZ2dwbG90MiBwYWNrYWdlLiBUaGlzIGlzIGEgdmVyeSBiYXNpYyBncmFwaCwgYnV0IGlmIHlvdSB3YW50IG1vcmUgaW5mb3JtYXRpb24gb24gaG93IHRvIGRvIG1vcmUgZmFuY3kgcGxvdHMsIHJlZmVyIHRvIENoYXB0ZXJzIDEwICYgMTEgb3IgaHR0cDovL3d3dy5hbGV4YW5kZXJkZW1vcy5vcmcvQU5PVkE4Lmh0bWwjYW5hbHlzaXNfaW5fci4NCg0KYGBge3IsIGVjaG89VFJVRSwgbWVzc2FnZT1GQUxTRX0NClBsb3QuQ29tcHJlaGVuc2lvbkRhdGEgPC1nZ3Bsb3QoZGF0YSA9IENvbXByZWhlbnNpb24uRGF0YSwgYWVzKHggPSBQbGF0Zm9ybSwgeSA9Q29tcHJlaGVuc2lvbiwgZmlsbD1BZ2UpKSsNCiAgY29vcmRfY2FydGVzaWFuKHlsaW0gPSBjKDEsMTIpKSsgI3NwZWNpZnlpbmcgdGhlIHktYXhpcyBzY2FsZQ0KICBnZW9tX2JhcihzdGF0PSdpZGVudGl0eScsIHBvc2l0aW9uPXBvc2l0aW9uX2RvZGdlKHdpZHRoPTEpKSsNCiAgeGxhYigiUGxhdGZvcm0gVHlwZSIpK3lsYWIoIkNvbXByZWhlbnNpb24gU2NvcmUiKSsgI2xhYmVsbGluZyB0aGUgeC1heGlzIGFuZCB5LWF4aXMNCiAgdGhlbWVfYncoKSsNCiAgdGhlbWUocGFuZWwuZ3JpZC5tYWpvcj1lbGVtZW50X2JsYW5rKCksDQogICAgICAgIHBhbmVsLmdyaWQubWlub3I9ZWxlbWVudF9ibGFuaygpKSsNCiAgc2NhbGVfZmlsbF9ncmV5KCkNCg0KUGxvdC5Db21wcmVoZW5zaW9uRGF0YSAjZGlzcGxheWluZyB0aGUgZ3JhcGggDQpgYGANCg0KDQoNCjxzY3JpcHQ+DQogIChmdW5jdGlvbihpLHMsbyxnLHIsYSxtKXtpWydHb29nbGVBbmFseXRpY3NPYmplY3QnXT1yO2lbcl09aVtyXXx8ZnVuY3Rpb24oKXsNCiAgKGlbcl0ucT1pW3JdLnF8fFtdKS5wdXNoKGFyZ3VtZW50cyl9LGlbcl0ubD0xKm5ldyBEYXRlKCk7YT1zLmNyZWF0ZUVsZW1lbnQobyksDQogIG09cy5nZXRFbGVtZW50c0J5VGFnTmFtZShvKVswXTthLmFzeW5jPTE7YS5zcmM9ZzttLnBhcmVudE5vZGUuaW5zZXJ0QmVmb3JlKGEsbSkNCiAgfSkod2luZG93LGRvY3VtZW50LCdzY3JpcHQnLCdodHRwczovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9hbmFseXRpY3MuanMnLCdnYScpOw0KDQogIGdhKCdjcmVhdGUnLCAnVUEtOTg4Nzg3OTMtMScsICdhdXRvJyk7DQogIGdhKCdzZW5kJywgJ3BhZ2V2aWV3Jyk7DQoNCjwvc2NyaXB0Pg==