Random permutations and t-test

Let’s assume that we have two groups (= datasets) of observations. And we are interested if two sets of data are significantly different from each other. In first place we will compare the means of each group and for these two means we will calculate Welch two sample t-test.  I will not go into details on how t-test is being calculated, just briefly mentioning H0 and H1 hypothesis. A test statistic either exactly follows or closely approximates a t-distribution under the null hypothesis, respectively.  For each case, degrees of freedom are calculated.

Our test data for calculating difference between means of each group is:

#b is group Black and w is group White
b <- c(30.24,22.40,23.52,29.12,30.24,34.72,26.88,23.52,22.40,
w <- c(23.52,24.64,16.80,13.44,23.52,17.92,21.28,16.80,24.64,

#calculate difference between means
#Welch two sample t-test

Difference in Mean is 4.9 (4,903111) and T-Test shows statistically significant difference between two groups (p = 0.007, df = 39.1, t = 3.65)

	Welch Two Sample t-test

data:  b and w
t = 3.6582, df = 39.113, p-value = 0.0007474

Now let presume that the degrees of freedom arise from residuals from sum of squares in case of T-test and can be understood also as before and after conditions of calculations. So it is an independent ways by which a dynamic system can move, without violating any constraint.

So by computations of random permutation we can test if this 4.9 difference in mean is a small or big difference. And remember, by staying in boundaries of original degrees of freedom, we can demonstrate what and how statistically significant our difference really is.

In this case we will replace and switch couple of numbers (labels) between the groups in order to show, that this statistical significance has little or no importance.

Following is R code for swapping the values, storing difference in mean and plotting the values.

diff_data <- list()
for(i in 1:5000){
 b <- c(30.24,22.40,23.52,29.12,30.24,34.72,26.88,23.52,22.40,21.28,
 w <- c(23.52,24.64,16.80,13.44,23.52,17.92,21.28,16.80,24.64,26.88,
 #create permutation
 b_r <- sample(b,5)
 w_r <- sample(w,5)
 b_new <- replace(b, b_r, w_r)
 w_new <- replace(w, w_r, b_r)
 #diff<- round((mean(b_new, na.rm=TRUE)-mean(w_new, na.rm=TRUE)), digits=2)
 x <- round((mean(b_new, na.rm=TRUE)-mean(w_new, na.rm=TRUE)), digits=2)
 y <- 1
 diff_data[[i]] <- c(x,y)
diff_data_v <- as.data.frame(do.call("rbind", diff_data))
diff <- data.frame(diff,0)
diff_2 <- rbind(diff, c(4.903111, 175))
 geom_histogram(data=diff_data_v, aes(x=V1), color="brown", 
binwidth = 0.05)+
 geom_line(data=diff_2, aes(x=diff, y=X0), color= 'REd', size =1)+
 labs(title="Random Permutation test",x="AVG Diff", y = "Count")

Little explanation to the code. Loop is doing 5000 permutations of in batches of 5 replacements between the groups and calculating the new mean.


Plot clearly shows number of occurrences when the differences between means was so high (4.9) and hence statistically significant. This occurred only in ca. 0,02% of cases, making or repeating this test with permutations relatively questionable if is would be statistically significant next time.

d49 <- diff_data_v$V1 >= 4.9


So watch out next time you do t-test.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s