Ray Block1, Charles Crabtree2, John B Holbein3, J Quin Monson4. 1. Department of Political Science, Pennsylvania State University, University Park, PA 16802. 2. Department of Government, Dartmouth College, Hanover, NH 03755. 3. Frank Batten School of Leadership & Public Policy, University of Virginia, Charlottesville, VA 22904. 4. Department of Political Science, Brigham Young University, Provo, UT 84602.
Mitterer (1) raises important issues in response to ref. 2, but his conclusions are incorrect and incomplete.How We Conceptualize and Design Experiments to Test for Discrimination Matters a Great DealWe agree that experiments using names as racial signals to study discrimination should consider what other characteristics beyond race people infer from names. Mitterer (1) does not note, however, that other modes of manipulating race may be even more “bundled” treatments (3). Moreover, partialling out perceptions that covary with racially distinctive names might not be desirable as it potentially decreases ecological validity and blocks potential mechanisms driving discrimination (4, 5).We also agree that multiple names should be used. Our work used five times the number of names commonly used. These names were preregistered prior to data collection. Future work should continue to use multiple names but should do so with care (6).Analyses of Discrimination-Based Experiments Should be Executed with Fidelity to the Experimental DesignMitterer (1) asserts that we assumed names had fixed effects. We did not. We included two checks that examine differences in the individual names we employed (see figures S6 and S8 of ref. 2). These show patterns consistent with Black names receiving lower response rates. Our approach is desirable as it allows us to actually estimate individual name effects.Even if we assume Mitterer’s approach (1) is best, his analysis is incorrectly executed. His model erroneously treats our data as if it were collected with a between-subjects design when we actually used a within-subjects design. Mitterer ignores the considerable statistical-power gains that come from a within-subjects design (7, 8). Mitterer’s model with individual fixed effects—as is standard in within-subjects designs (8)—shows robust evidence for discrimination—with effect sizes that are substantively and statistically indistinguishable from those provided in our original paper.*,We are ambivalent as to whether Mitterer’s approach (1) is best. We acknowledge model-based uncertainty. However, any analysis must correctly account for experimental design.