University of Illinois Urbana-Champaign

Asymptotically robust permutation tests for two-sample goodness-of-fit testing problems

Olivares Gonzalez, Mauricio

Content Files
OLIVARESGONZALEZ-DISSERTATION-2022.pdf

Permalink

https://hdl.handle.net/2142/115521

Description

Title
Asymptotically robust permutation tests for two-sample goodness-of-fit testing problems
Author(s)
Olivares Gonzalez, Mauricio
Issue Date
2022-04-14
Director of Research (if dissertation) or Advisor (if thesis)
Chung, EunYi
Doctoral Committee Chair(s)
Chung, EunYi
Committee Member(s)
  • Koenker, Roger
  • Shao, Xiaofeng
  • Lee, Ji Hyung
Department of Study
Economics
Discipline
Economics
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • Permutation Tests
  • Goodness-of-fit Testing
  • Heterogeneous Treatment Effects
  • Covariate-Adaptive Randomization
Abstract
The three chapters of this dissertation introduce novel permutation tests for two relevant hypotheses in economics: heterogeneous treatment effects and equality of distributions under covariate-adaptive randomization (CAR). Permutation tests have several attractive properties. First, they are exact in finite samples under the randomization hypothesis. Second, they are easy to implement since they are nonparametric and their construction is the same regardless of the test statistic. Third, they are asymptotically as powerful as standard optimal procedures for many testing problems. Last, even when the randomization hypothesis does not hold, permutation tests could still be asymptotically valid for general non-exchangeable settings if we replace the test statistic used in the permutation test by a test statistic that is asymptotically distribution-free. Despite these advantages, the direct application of permutation-based inference to the problems we investigate in this dissertation is not automatic. The reason is two-fold. First, the randomization hypothesis is hard to justify in the testing problems we consider. Second, studentization in higher dimensions is difficult since we consider two-sample goodness-of-fit hypotheses with associated Kolmogorov-Smirnov-type test statistics. Thus, establishing the asymptotic validity of the permutation test is not straightforward either. This work provides asymptotic arguments to handle these difficulties by leveraging empirical process techniques. Throughout this thesis, we will invoke martingale transformations and prepivoting to restore the asymptotic pivotality of the test statistic. The main contribution of this work is to show these techniques provide a path forward to carry on asymptotically valid inference for testing heterogeneous treatment effects and equality of distributions under CAR. Below are the individual abstracts for each chapter. Chapter 1: Robust Permutation Test for Equality of Distributions under Covariate- Adaptive Randomization Though stratified randomization achieves more balance on baseline covariates than pure randomization, it does affect the way we conduct inference. This paper considers the classical two-sample goodness-of-fit testing problem in randomized controlled trials (RCTs) when the researcher employs a particular type of stratified randomization—covariate-adaptive randomization (CAR). When testing the null hypothesis of equality of distributions between experimental groups in this setup, we first show that stratification leaves a mark on the test statistic’s limit distribution, making it difficult, if not impossible, to obtain critical values. We instead propose an alternative approach to conducting inference based on a permutation test that i) is asymptotically exact in the sense that the limiting rejection probability under the null hypothesis equals the nominal α level, ii) is viable under relatively weak assumptions commonly satisfied in practice, iii) works for randomization schemes that are popular among empirically oriented researchers, and iv) is applicable without requiring knowledge about the pre-treatment covariates used for stratification. The proposed test’s main idea is that by transforming the original statistic by one minus its bootstrap p-value, it becomes asymptotically uniformly distributed on [0,1]. Thus, the transformed test statistic—also called prepivoted—has a fixed limit distribution that is free of unknown parameters, effectively removing the effect of stratification. Consequently, a permutation test based on the prepivoted statistic produces a test whose limiting rejection probability equals the nominal level. We present further numerical evidence of the proposed test’s advantages in a Monte Carlo exercise, showing our permutation test outperforms the existing alternatives. We illustrate our method’s empirical relevance by revisiting a field experiment by Butler and Broockman (2011) on the effect of race on state legislators' responsiveness to help their constituents register to vote during elections in the United States. Lastly, we provide the companion RATest R package to facilitate and encourage applying our test in empirical research. Chapter 2: Permutation Test for Heterogeneous Treatment Effects with a Nuisance Parameter This paper proposes an asymptotically valid permutation test for heterogeneous treatment effects in the presence of an estimated nuisance parameter. Not accounting for the estimation error of the nuisance parameter results in statistics that depend on the particulars of the data generating process, and the resulting permutation test fails to control the Type 1 error, even asymptotically. In this paper we consider a permutation test based on the martingale transformation of the empirical process to render an asymptotically pivotal statistic, effectively nullifying the effect associated with the estimation error on the limiting distribution of the statistic. Under weak conditions, we show that the permutation test based on the martingale-transformed statistic results in the asymptotic rejection probability of α in general while retaining the exact control of the test level when testing for the more restrictive sharp null. We also show how our martingale-based permutation test extends to testing whether there exists treatment effect heterogeneity within subgroups defined by observable covariates. Our approach comprises testing the joint null hypothesis that treatment effects are constant within mutually exclusive subgroups while allowing the treatment effects to vary across subgroups. Monte Carlo simulations show that the permutation test presented here performs well in finite samples, and is comparable to those existing in the literature. To gain further understanding of the test to practical problems, we investigate the gift exchange hypothesis in the context of two field experiments from Gneezy and List (2006). Lastly, we provide the companion RATest R package to facilitate and encourage the application of our test in empirical research. Chapter 3: Quantile-based Test for Heterogeneous Treatment Effects One way to look at the distributional effects of a policy intervention comprises estimating the quantile treatment effect at different quantiles. We exploit this idea and develop a new permutation test for heterogeneous treatment effects based on a modified quantile process. To establish the asymptotic validity of our test, we transform the test statistic using a martingale transformation so that its limit behavior is distribution-free. Numerical evidence shows our permutation test outmatches other popular quantile-based tests in terms of size and power performance. We discuss a fast implementation algorithm and illustrate our method using experimental data from a welfare reform.
Graduation Semester
2022-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/115521
Copyright and License Information
Copyright 2022 Mauricio Olivares Gonzalez

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois