It’s been more than a week now since our College Football Playoff model finished its job for the year, and for those interested in the modeling side of this enterprise, as well as for our own sake in the thought-gathering department, it seemed worthwhile to do a post-mortem. As a quick update on this side of the business as a whole—we remain convinced that it’s a core part of the website’s best path forward, and we want to dedicate the time necessary to make it happen, but due to this being a part-time endeavor for all of us, that time hasn’t been as easily available this year as it’s been in years past. We do hope to have the NHL model published again for this year soon, and we’d like to have both (men’s and women’s) college basketball models up by the morning after the College Football Playoff’s championship, if not sooner. We’re trying not to make any promises, though. It’s been a bad year from us in the promise-keeping department with this stuff.
The College Football Playoff model is one of our favorites because it’s 1) relatively simple, 2) easy to upkeep, and 3) something we seem to be uniquely good at? I’m always surprised at the numbers ESPN puts out there, and I’m always surprised at the conflicts between their Allstate Playoff Predictor and FPI. Our model is straightforward compared to what they’re capable of doing, with their education and technical resources. And yet they were quite confident in 2020 about USC’s playoff prospects.
Our purpose in ripping on ESPN here isn’t to degrade anyone’s work over there. We have no reason to believe they’re anything but wonderful people. We do see an opportunity, though, and the fact they don’t publish their back testing gives us an opportunity to prove our model’s worth to non-ESPN outlets. To speak more plainly: If we can figure out our marketing, we have a big opportunity with this model.
There are three main challenges, then, for the offseason work on this guy.
The first is to bring back its probabilities, rather than just use it to react to rankings. The reason we didn’t get there this year is we wanted to be sure we were doing it as well as possible, and in a replicable manner. We know how to do this, and we have some ideas for doing it especially well, but it’s going to take time. Time is our first obstacle.
The second is to market the thing. We need to make it easily accessible, fun and easy to navigate, and we need to get people to know about it before they need to know about it. Beat writers and bloggers share our NIT Bracketology when the time comes. We need to let the relevant parties know we have worthwhile College Football Playoff reflections to share too.
The third is refinement. The model works spectacularly, but it has room to grow. We need to figure out hard-and-fast rules for how FPA* is applied and unapplied, and whether to use it at all (we were set for our FPA and non-FPA rankings to disagree on the final four, should Georgia have beaten Bama, and that’s not a situation we much like because we weren’t sure which to trust). We need to look at historic FPA and see how much we can predict its movements, taking those movements out of FPA and putting them in separate variables that are not strictly reactive to rankings.
Beyond those three, there are of course things we’d like to better. We’d like to expand back into the FCS. We’d like to figure out how to incorporate transfers into our Movelor system (which we’ve never even finished yet, but that’s back to the first main challenge). We’d like to add full bowl projections, rather than just playoff probabilities. Overall, though, we have a good thing here, and we need to steer into it.
One question, I suppose, for those who care about our models, is what you would like to see? We’ve been told a FiveThirtyEight-like interactive element where you can see how different teams winning and losing affects the playoff would be a good touch, and we’d like to roll that out with all our models. What else do you want, though? And how would you like us to present it? If you have thoughts, as always, let us know.
We’re a long way from doing the work again on this—we’ve got basketball season, and the start of auto racing, and college baseball and softball but perhaps a few others. So we’re a ways away from doing much real work again on this model. Which, of course, is why we’re writing this. Anyway, let us know if you have requests or thoughts. Bark.
*FPA is Forgiveness/Punishment Adjustment, our model that reflects how the committee is treating a team compared to the model’s expectations. It’s how our model reacts to each week’s rankings.
