Publications and preprints
Time-uniform confidence bands for the CDF under nonstationarity
P. Mineiro, S. R. Howard | NeurIPS 2023 | arXiv
Estimation of the complete distribution of a random variable is a useful primitive for both manual and automated decision making. This problem has received extensive attention in the i.i.d. setting, but the arbitrary data dependent setting remains largely unaddressed. Consistent with known impossibility results, we present computationally felicitous time-uniform and value-uniform bounds on the CDF of the running averaged conditional distribution of a real-valued random variable which are always valid and sometimes trivial, along with an instance-dependent convergence guarantee. The importance-weighted extension is appropriate for estimating complete counterfactual distributions of rewards given controlled experimentation data exhaust, e.g., from an A/B test or a contextual bandit.
Sequential estimation of quantiles with applications to A/B-testing and best-arm identification.
S. R. Howard, A. Ramdas. | Bernoulli, 2022 | Public link
We derive confidence sequences for sequentially estimating quantiles of any distribution, with coverage guarantees holding non-asymptotically and uniformly over all sample sizes. We also present non-asymptotic, time-uniform concentration bounds for the empirical distribution function, a quantile best-arm identification algorithm, sequential Kolmogorov-Smirnov tests, and procedures for sequential quantile A/B tests.
The uniform general signed rank test and its design sensitivity.
S. R. Howard, S. D. Pimentel. | Biometrika, 2021.
A sensitivity analysis in an observational study tests whether the qualitative conclusions of an analysis would change if we were to allow for the possibility of limited bias due to confounding. We propose a new class of non-asymptotic, distribution-free tests, the uniform general signed rank tests, for observational studies with paired data. This class includes uniform versions of the sign test and the Wilxcoxon signed-rank test. Our tests adaptively choose a subpopulation which exhibits the strongest treatment effect, controlling for this adaptivity using a uniform martingale concentration argument, and achieving superior results to existing tests in a sensitivity analysis, especially with large samples.
Time-uniform, nonparametric, nonasymptotic confidence sequences.
S. R. Howard, A. Ramdas, J. McAuliffe, J. Sekhon. | Annals of Statistics, 2021.
A confidence sequence is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon; they are fundamental tools for sequential experimentation. We develop confidence sequences whose widths go to zero, with non-asymptotic coverage guarantees under nonparametric conditions. In particular, we derive an empirical-Bernstein bound growing at an iterated-logarithm rate which allows for non-asymptotic, sequential estimation of the mean of any bounded distribution, with confidence interval widths scaling appropriately with the estimated standard deviation. We apply our methods to covariance matrix estimation and to estimation of sample average treatment effect under the Neyman-Rubin potential outcomes model.
Time-uniform Chernoff bounds via nonnegative supermartingales.
S. R. Howard, A. Ramdas, J. McAuliffe, J. Sekhon. | Probability Surveys, 2020.
We give a powerful, general formulation of the Cramér-Chernoff method for exponential concentration inequalities which unifies and strengthens dozens of inequalities from the literature. These include classical results for concentration of scalar sums and martingales, more recent matrix concentration results, self-normalized inequalities, and results for Banach-space valued martingales and continuous-time processes. Such results are of broad theoretical interest, but are particularly useful as the foundation of a variety of methods for the analysis of sequential experiments.
Informal writings
Augmented inverse propensity weighting for randomized experiments
Draft in progress
Code
Confidence sequences and uniform boundaries
2019+ | GitHub | with Ian Waudby-Smith
A library implementing many techniques from my research to computing confidence sequences for means, quantiles, and cumulative distribution functions. Written in C++ with Python and R wrappers.
Quantile best-arm identification simulations
2019 | GitHub
This repository contains all code to reproduce the simulations and plots from the paper "Sequential estimation of quantiles with applications to A/B-testing and best-arm identification". Of particular interest is an efficient C++ simulation harness for quantile best-arm identification.
Talks
Topics from experimentation in the tech industry
November 16, 2023 | Berkeley Stat 158 (Design of Experiments) | slides
Augmented inverse propensity weighting for randomized experiments
March 14, 2023 | DSCO 2023 Data Science Conference | slides
Confidence sequences for sequential experimentation
August 5, 2022 | Netflix Statistics, Methodology, and Engineering talk | slides
Nonasymptotic confidence sequences for sequential estimation of treatment effects in randomized trials
June 30, 2021 | UCSF Department of Biostatistics | slides
Nonparametric generalizations of the sequential probability ratio test
November 4, 2019 | Asilomar Conference on Signals, Systems, and Computers | slides
Confidence sequences for sequential experimentation and best-arm identification
September 5, 2019 | Adobe Research | slides
Uniform, non-asymptotic confidence sequences for sequential estimation of average treatment effect
June 25, 2019 | WNAR/IMS/JR Annual Meeting | slides
Randomized experiments, A/B tests and sequential monitoring
April 26, 2018 | Guest lecture, Principles and Techniques of Data Science (DS 100), UC Berkeley | slides | Jupyter notebook (co-authored with Jake Soloff)
Moving to Wyoming (Dependency Management in Python)
December 12, 2013 | Thumbtack Tech Talk | video + transcript | slides
Distributed Web Crawling with Python and AWS
March 1, 2012 | Thumbtack Tech Talk | slides
Music
complicated or not
2014 | Bandcamp
Recorded in late 2013/early 2014 while I was playing with a couple of awesome musicians in Little Heart. I played the lead guitar parts, wrote some of them, and coiled and uncoiled a lot of microphone cables.
Classical guitar recordings
Valse Choro by Francis Kleynjans
Miami by Gerard Montreuil
Prelude by George Handel