Why Use “Ensemble Approaches” for Forecasting?

If you wanted to predict how much corn would be sold at a farmer’s market stand this summer, you might ask multiple people to make educated guesses using different methods. One person might be a farmer who knows how the crops are doing in the region this year, one might be the person who has operated the stand for the past 3 years and has a good understanding of the seasonal purchasing patterns, and one might be a statistician with a table of the last 5 years’ of weekly sales.

Each would give a different answer. The more people you ask, the more different answers you would get. You might throw away guesses that seem totally unrealistic (unless there is good reason for a dramatic swing in corn sales this year), and then average the rest of the guesses to make your own now-well-informed guess. This type of forecasting - using a variety of methods to come up with a combined prediction - is called “Ensemble Learning.”

I gave a talk last week at the North Carolina Association for Institutional Research conference (NCAIR) called “The HelioCampus Ensemble Approach to Admissions & Enrollment Predictions,” in which I explained 5 different methods we used to predict the number of applications, admissions, pre-matriculations (deposits + deferred deposits), and enrollments for a HelioCampus Client’s incoming freshman class.

For this institution, we initially decided to use multiple methods (curve-fitting/regression, ARIMA, exponential smoothing, etc.) in order to have a variety of models to evaluate for predictive power at the end of the admissions cycle. Since we didn’t know how each model would do making “live” predictions, we wanted to have several running throughout the cycle.

Most were time-series forecasts, but one model used factors such as high school GPA and distance from campus in order to predict each admitted student’s likelihood to enroll, then aggregated those predictions to estimate how large the incoming class might be. The remaining models each used a single input - the count of applications, for example - summarized by day, week, or month. Time-series methods find trends and patterns in the prior years’ data and use that along with the current-year trends to forecast the counts for each week or month, which we can add up to get a forecasted total.

Averaging the numeric output of multiple predictive models is the simplest version of ensemble learning. More sophisticated ensemble models can learn to weight each forecast differently based on past accuracy throughout each part of the cycle, but follow the same principle of combining multiple results to get a “consensus” result. Some classification ensembles “vote” on an output.

Ensemble models are often known to outperform a single model by both increasing the diversity of possible outcomes, and by reducing the effect of the more extreme predictions. Of course, if one model is an extreme outlier compared to all of the others, we would need to evaluate whether it should be included at all.

Because we are also tracking the predictions each model generates each week, we will be able to look back at the end of the cycle and determine which models worked best with this institution’s data, and when each model got closest to the final actual counts, so next year we know when we might start using them to make decisions.

When we plotted the forecasts in a user-facing dashboard, we chose to shade the area between the lowest and the highest forecast value gray to depict the range. The end-user can then visualize this projected range of end-results in comparison to prior years’ results, to get a sense where their counts might end up for the year. But one of the models was more optimistic than the others, so the displayed range wasn’t telling the whole story.

In order to clarify which outcome within the range was most likely, we added a plot of the average of the 5 different models, which would appear as a dashed line. This line usually ended up in the lower half of the displayed range, giving a better indication that most of the models agreed on a less-optimistic outcome.

20 weeks before the term starts, the range of Pre-Matriculation forecast values is still wide.

Detail zoom view: The average forecast (dashed line) lies between the two prior years’ final counts.

By displaying both the full range of predictions and the average prediction to the end-user, we give them a sense of the possible outputs as well as indicating where the “ensembled” output lies within that range. This helps communicate the uncertainty inherent in the forecasting, while also giving more confidence in the prediction when multiple models built using different approaches agree.