Asymptotically robust permutation tests for two-sample goodness-of-fit testing problems

Olivares Gonzalez, Mauricio

Permalink

https://hdl.handle.net/2142/115521 Copy

Description

Title

Asymptotically robust permutation tests for two-sample goodness-of-fit testing problems

Author(s)

Olivares Gonzalez, Mauricio

Issue Date

2022-04-14

Director of Research (if dissertation) or Advisor (if thesis)

Chung, EunYi

Doctoral Committee Chair(s)

Chung, EunYi

Committee Member(s)

• Koenker, Roger
• Shao, Xiaofeng
• Lee, Ji Hyung

Department of Study

Economics

Discipline

Economics

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Permutation Tests
• Goodness-of-fit Testing
• Heterogeneous Treatment Effects
• Covariate-Adaptive Randomization

Abstract

The three chapters of this dissertation introduce novel permutation tests for two relevant hypotheses in economics: heterogeneous treatment effects and equality of distributions under covariate-adaptive randomization (CAR). Permutation tests have several attractive properties. First, they are exact in finite samples under the randomization hypothesis. Second, they are easy to implement since they are nonparametric and their construction is the same regardless of the test statistic. Third, they are asymptotically as powerful as standard optimal procedures for many testing problems. Last, even when the randomization hypothesis does not hold, permutation tests could still be asymptotically valid for general non-exchangeable settings if we replace the test statistic used in the permutation test by a test statistic that is asymptotically distribution-free. Despite these advantages, the direct application of permutation-based inference to the problems we investigate in this dissertation is not automatic. The reason is two-fold. First, the randomization hypothesis is hard to justify in the testing problems we consider. Second, studentization in higher dimensions is difficult since we consider two-sample goodness-of-fit hypotheses with associated Kolmogorov-Smirnov-type test statistics. Thus, establishing the asymptotic validity of the permutation test is not straightforward either. This work provides asymptotic arguments to handle these difficulties by leveraging empirical process techniques. Throughout this thesis, we will invoke martingale transformations and prepivoting to restore the asymptotic pivotality of the test statistic. The main contribution of this work is to show these techniques provide a path forward to carry on asymptotically valid inference for testing heterogeneous treatment effects and equality of distributions under CAR. Below are the individual abstracts for each chapter. Chapter 1: Robust Permutation Test for Equality of Distributions under Covariate- Adaptive Randomization Though stratified randomization achieves more balance on baseline covariates than pure randomization, it does affect the way we conduct inference. This paper considers the classical two-sample goodness-of-fit testing problem in randomized controlled trials (RCTs) when the researcher employs a particular type of stratified randomization—covariate-adaptive randomization (CAR). When testing the null hypothesis of equality of distributions between experimental groups in this setup, we first show that stratification leaves a mark on the test statistic’s limit distribution, making it difficult, if not impossible, to obtain critical values. We instead propose an alternative approach to conducting inference based on a permutation test that i) is asymptotically exact in the sense that the limiting rejection probability under the null hypothesis equals the nominal α level, ii) is viable under relatively weak assumptions commonly satisfied in practice, iii) works for randomization schemes that are popular among empirically oriented researchers, and iv) is applicable without requiring knowledge about the pre-treatment covariates used for stratification. The proposed test’s main idea is that by transforming the original statistic by one minus its bootstrap p-value, it becomes asymptotically uniformly distributed on [0,1]. Thus, the transformed test statistic—also called prepivoted—has a fixed limit distribution that is free of unknown parameters, effectively removing the effect of stratification. Consequently, a permutation test based on the prepivoted statistic produces a test whose limiting rejection probability equals the nominal level. We present further numerical evidence of the proposed test’s advantages in a Monte Carlo exercise, showing our permutation test outperforms the existing alternatives. We illustrate our method’s empirical relevance by revisiting a field experiment by Butler and Broockman (2011) on the effect of race on state legislators' responsiveness to help their constituents register to vote during elections in the United States. Lastly, we provide the companion RATest R package to facilitate and encourage applying our test in empirical research. Chapter 2: Permutation Test for Heterogeneous Treatment Effects with a Nuisance Parameter This paper proposes an asymptotically valid permutation test for heterogeneous treatment effects in the presence of an estimated nuisance parameter. Not accounting for the estimation error of the nuisance parameter results in statistics that depend on the particulars of the data generating process, and the resulting permutation test fails to control the Type 1 error, even asymptotically. In this paper we consider a permutation test based on the martingale transformation of the empirical process to render an asymptotically pivotal statistic, effectively nullifying the effect associated with the estimation error on the limiting distribution of the statistic. Under weak conditions, we show that the permutation test based on the martingale-transformed statistic results in the asymptotic rejection probability of α in general while retaining the exact control of the test level when testing for the more restrictive sharp null. We also show how our martingale-based permutation test extends to testing whether there exists treatment effect heterogeneity within subgroups defined by observable covariates. Our approach comprises testing the joint null hypothesis that treatment effects are constant within mutually exclusive subgroups while allowing the treatment effects to vary across subgroups. Monte Carlo simulations show that the permutation test presented here performs well in finite samples, and is comparable to those existing in the literature. To gain further understanding of the test to practical problems, we investigate the gift exchange hypothesis in the context of two field experiments from Gneezy and List (2006). Lastly, we provide the companion RATest R package to facilitate and encourage the application of our test in empirical research. Chapter 3: Quantile-based Test for Heterogeneous Treatment Effects One way to look at the distributional effects of a policy intervention comprises estimating the quantile treatment effect at different quantiles. We exploit this idea and develop a new permutation test for heterogeneous treatment effects based on a modified quantile process. To establish the asymptotic validity of our test, we transform the test statistic using a martingale transformation so that its limit behavior is distribution-free. Numerical evidence shows our permutation test outmatches other popular quantile-based tests in terms of size and power performance. We discuss a fast implementation algorithm and illustrate our method using experimental data from a welfare reform.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Mauricio Olivares Gonzalez

Owning Collections