Bootstrapping (statistics)
Bootstrapping in statistics is a powerful resampling technique that enables researchers to estimate various statistics by repeatedly drawing samples from an existing dataset, with replacement. This method generates numerous simulated samples, facilitating the estimation of summary statistics, construction of confidence intervals, calculation of standard errors, and execution of hypothesis tests. Originating from the work of statistician Bradley Efron in 1979, bootstrapping has gained popularity as computational capabilities have advanced, allowing for extensive repetition of the sampling process. Unlike traditional hypothesis testing, which relies on specific equations and sample properties, bootstrapping uses the sample data as its own population, creating a more accessible and interpretable approach to statistical analysis. This technique is especially beneficial in applied machine learning, where it helps assess a model's predictive performance on new data. By transforming a single sample into multiple simulated representations, bootstrapping provides insights into the variability of estimates and supports robust statistical inference.
On this Page
Bootstrapping (statistics)
In statistics, bootstrapping is a resampling method in which a small sample is “bootstrapped” from a larger sample to create multiple simulated samples. In short, bootstrapping is a means of estimating statistics by sampling an existing dataset with replacement. It is often used to estimate summary statistics, construct confidence intervals, calculate standard errors, or perform hypothesis testing for various kinds of sample statistics. While bootstrapping itself is generally a simple process, it often requires such a great degree of repetition that it can only be performed with the aid of a computer. In comparison to traditional methods of hypothesis testing, bootstrapping typically produces results that are easier to interpret and valid under a wider array of possible conditions. Importantly, bootstrapping is commonly used in applied machine learning (ML) to estimate an ML model’s skill when making predictions based on data that is not part of the original training data.
Background
Statistics is a scientific field focused primarily on the development and study of methods used for the collection, analysis, interpretation, and presentation of empirical data. As a discipline, statistics is segmented into two distinct categories: descriptive and inferential. While both types of statistics offer some sort of glimpse into what is going on within a specific population or dataset, each takes a different approach and ultimately yields a different type of information.
Descriptive statistics offers a direct description of what is going on in a population or dataset. This description is delivered through various numerical measures that reveal key insights about a given data set. These measures include familiar concepts like the average, which is a measure of the center of a dataset and includes the mean, median, mode, or midrange of the set in question. Some other concepts that are part of descriptive statistics include the spread of a dataset, the five number summary, and measurements like skewness and kurtosis. The exploration of relationships between paired data and the graphical presentation of statistical results are also both important parts of descriptive statistics. Researchers use these and other tools to find patterns in data that help them to better understand that data. Importantly, however, descriptive statistics can only yield information about the specific population or dataset being studied and cannot be used to make generalizations about other populations or datasets.
In inferential statistics, researchers use complex mathematical calculations to make inferences about a larger population based on a small sample taken from that population. Inferential statistics is useful because it allows researchers to study the relationships between variables in a given sample and subsequently make predictions about how those variables will ultimately relate to a larger population. Inferential statistics is also useful because examining each member of a large population is often not a realistic goal. As a result, the best alternative is usually to take a small sample of that population, analyze it, and extrapolate information about the larger population based on an analysis of the sample. The two main types of inferential statistics are confidence intervals and tests of hypothesis. The former provides a range of values for an unknown parameter of the population through measurement of a sample, with this measurement being expressed both in terms of an interval and the degree of confidence that the parameter lies within the interval. In the latter, a researcher makes a claim about a given population through the analysis of a statistical sample.
Overview
Bootstrapping is a commonly used technique in inferential statistics. The concept of bootstrapping was first suggested by American statistician Bradley Efron in 1979. As computing power improved and the cost of computers decreased from that time forward, bootstrapping came into increasingly widespread use. The term “bootstrapping” was inspired by the traditional notion of pulling oneself up by the bootstraps to overcome seemingly impossible problems. This name is appropriate; by estimating a population statistic simply by reusing the same sample, bootstrapping allows people to do something that seems like it should be impossible.
In the simplest terms, bootstrapping is a statistical procedure in which a single dataset is repeatedly resampled to produce numerous simulated samples. Statisticians use bootstrapping to construct confidence samples, calculate standard errors, and perform hypothesis tests. However, bootstrapping differs from traditional hypothesis testing in some ways. Traditional hypothesis testing relies on equations that use the properties of samples, the experimental design, and a test statistic to estimate sampling distributions. Bootstrapping, on the other hand, simply resamples sample data numerous times to create simulated samples that can in turn be used to graph sample distributions. These sample distributions then serve as the basis for hypothesis testing and confidence intervals.
For example, if researchers want to determine the average size of all families residing in Philadelphia, they can construct a random sample of Philadelphian families to accomplish their goal since they cannot possibly examine every family in Philadelphia. If a total of 1.5 million families are in Philadelphia, the researchers might put together a sample group of the hundred families and find that these families have a mean of four members. To determine how close this estimate is to the truth, the researchers must turn to bootstrapping. As part of the bootstrapping process, the researchers use the sample as its own population and resample over and over again from the original three-hundred-family sample. By subsequently using the differences between the re-sampled family sizes and the already established mean family size of the three hundred families in the original sample, the researchers can infer the accuracy or inaccuracy of the mean family size of four in Philadelphia.
It is important to remember that the bootstrapping process does not yield new data but instead uses a sample group as a proxy for a larger population and draws random samples from that group. In short, the sample group effectively functions as a smaller representation of the larger population. Bootstrapping works because the resampling process produces numerous samples that could have been drawn by a full study. Further, the simulated samples created through bootstrapping together provide researchers with a reliable estimate of the variability between different random samples taken from the same population. Through the use of these ranges, bootstrapping allows researchers to build confidence intervals and conduct hypothesis testing.
Bibliography
Brownlee, Jason. “A Gentle Introduction to the Bootstrap Method.” Machine Learning Mastery, 25 May 2018, machinelearningmastery.com/a-gentle-introduction-to-the-bootstrap-method. Accessed 9 Sept. 2020.
Frost, Jim. “Introduction to Bootstrapping in Statistics with an Example.” Statistics by Jim, 2020, statisticsbyjim.com/hypothesis-testing/bootstrapping. Accessed 9 Sept. 2020.
Glen, Stephanie. “Bootstrap Sample: Definition, Example.” Statistics How To, 2020, www.statisticshowto.com/bootstrap-sample. Accessed 9 Sept. 2020.
JJ. “Bootstrapping in Statistics.” Medium, 3 Feb. 2017, medium.com/human-in-a-machine-world/bootstrapping-in-statistics-281217ffeb75. Accessed 9 Sept. 2020.
Joseph, Trist’n. “Bootstrapping Statistics. What It Is and Why It’s Used.” Toward Data Science, 17 June 2020, towardsdatascience.com/bootstrapping-statistics-what-it-is-and-why-its-used-e2fa29577307. Accessed 9 Sept. 2020.
Pezzullo, John. “The Bootstrap Method for Standard Errors and Confidence Intervals.” Dummies.com, 2020, www.dummies.com/education/science/biology/the-bootstrap-method-for-standard-errors-and-confidence-intervals. Accessed 9 Sept. 2020.
Taylor, Courtney. “The Difference Between Descriptive and Inferential Statistics.” ThoughtCo., 17 Feb. 2020, www.thoughtco.com/differences-in-descriptive-and-inferential-statistics-3126224. Accessed 9 Sept. 2020.
Taylor, Courtney. “What Is Bootstrapping in Statistics?” ThoughtCo., 13 Jan. 2019, www.thoughtco.com/what-is-bootstrapping-in-statistics-3126172. Accessed 9 Sept. 2020.