TNG Year 1 in Review
How did we do? A look back at our preseason picks, regular season/playoff performance, and what we'll be working on this offseason
It’s hard to believe that the 2022 season, and our first season at TNG, has come and gone (read more about our background here if you haven’t already). It was a great year and we’re happy to be back in the saddle. With the dust settled, we thought it would be good to take some time to review how we did, and then we wanted to share what we’re planning to do to improve in the offseason. We’ll close by introducing a couple of thoughts we have for the site/pod going forward.
Preseason Picks
Way back in late May, we started putting together our preseason content, which you can find archived here: TNG 2022 Season Preview. In our previews, we put together a Preseason Top 50, District/Region Picks, Playoff Projections, and more based on the data we’d collected back through the 2008 season and through studying each team’s returning talent, applying adjustments to each team’s offense and defense ratings to create a preseason Elo rating for each team. With those preseason Elo ratings set, we could then simulate the season to provide all of the projections featured. If you want to read more about how we arrived at our ratings, check out our Ratings Origin and Methodology.
So, how did we do with our Preseason picks?
We picked 23 of 32 District Champions
Correctly projected 101 of 128 playoff teams; 18 of our misses were teams picked in 4th place, and 5 of our misses were teams picked in 3rd
Each of our Preseason Top 6 made at least the Semifinals
We identified 14 of 16 Quarterfinalists and 7 of 8 Semifinalists
14 teams in our Preseason Top 19 made at least the Quarterfinals
State Champs Duncanville and DeSoto were Preseason #3 and #4
DeSoto was our Preseason pick as Division 2 Champion
We were pretty floored with how well our preseason picks held up, especially in terms of the bracket projections, because in order to nail those we had to have each district correctly modeled to be able to get teams into the correct brackets to make the bracket projections relevant. We struggled a bit with our picks in Region 2, but that is notoriously the most volatile of the four regions just due to the usual parity among the teams involved. There’s definitely room for improvement.
In-Season Model Performance
Our model picked each of the 1,490 games involving 6A teams in the 2022 season and correctly picked the winner in 78.2% of those contests (1,165 winners). If we follow Calpreps’ lead and only consider the games played after September 15th, our model picked 80.9% of the correct winners.
Another way to evaluate the accuracy of our model is to look at Brier Score. Brier Score, in short, measures the accuracy of our game-by-game probabilistic forecasts. If the model picks a team as a 92% favorite and they win, the Brier score for that prediction is 0.006. But, if that 92% favorite were to lose, the Brier score would be 0.846 for that game. The lower the Brier score, the better. An average Brier Score of 0.00 would represent perfect forecasting accuracy, 0.50 would be akin to random guesses, and a score approaching 1.00 would mean you’re getting it wrong every single time. fivethirtyeight.com has long been the standard for Elo modeling in the NFL and they usually achieve a Brier score in the range of 0.20-0.22.
For the 2022 season, our model achieved a Brier Score of 0.15.
It’s not surprising that our model would score better than fivethirtyeight’s NFL model due to how much more difficult it is to predict NFL games and due to how little relative separation there is between teams in the NFL when compared to the average game between 6A high school opponents. The NFL has very few “gimmies” when it comes to projecting winners, but the fact that our model is in range of fivethirtyeight’s is confirmation that we’re on a good track.
Playoff Performance
If we isolate the playoffs, how did the model do each round, in terms of wins and losses, as the matchups got tougher and tougher compared to the regular season?
Bi-District Round: 51-13 (79.7%)
Area Round: 27-5 (84.4%) *best Area round in our dataset
Regional Semifinals: 16-0 (100%) *best Reg Semi round in our dataset
State Quarterfinals: 8-0 (100%) *best State Qtrs round in our dataset
State Semifinals: 1-3 (25%)
State Championships: 1-1 (50%)
Overall playoff performance: 104-22 (82.5%)
When the playoff brackets were initially set, we used our model and each team’s Elo rating at the end of the season to project the brackets all the way through to the state championships in our TNG Bracketology: 2022 Playoff Bracket Odds post. How did that hold up?
In Division 1, our start-of-the-playoffs odds model got 7 of 8 Quarterfinalists, 4 of 4 Semifinalists, but just 1 of 2 finalists and did not pick the correct champion.
In Division 2, our start-of-the-playoffs odds model got 8 of 8 Quarterfinalists, 4 of 4 Semifinalists, but were shutout in the finals. Congrats to Vandegrift and DeSoto there!
We entered each of our simulated brackets in the 6ATexasFootball.com annual Bracket Challenge and finished 4th on the D1 leaderboard and 11th on the D2 leaderboard. That’s not bad for a purely mathematical model with no week-to-week human influence, but we hope to finish higher next year of course!
Where do we go from here?
As thrilled as we are with how the model held up throughout our first “live” season, the wheels are already turning on some potential improvements for the 2023 season. Here are some of the things we’re considering:
We took steps this year to better tune our starting Elo ratings for each team, but we’re going to go even further this next season. We hope to put more work into studying/identifying returning rosters, transfers, coaching changes, etc. to better capture offseason changes within programs.
We also want to start all teams with less initial Elo separation than we did this year to allow teams to establish their Elo rating more organically. We noticed several instances this year where teams’ ratings were either unnecessarily hampered or artificially boosted for too long due to their preseason rating.
The early season 5A vs 6A games gave us fits despite attempting to get basic Elo ratings for all 5A opponents. We’re thinking about building out a more robust 5A dataset (gulp)
We will eventually build out more score history to tune our model. This year’s model was based on data going back to 2008, which represented ~15,000 games but with more data, we can possibly tune it even better. Also, having more data could allow us to do some cool historical plots, capturing more team history like we did in our preseason TNG Timeline features.
Aside from the model tweaks above, we will be looking into moving our Podcast to YouTube in addition to Spotify and Apple. On YouTube, we should be able to draw from a wider audience since pretty much everyone uses YouTube and it has a better search feature than the traditional pod sites. We’ll also be able to incorporate some graphics to go along with our commentary, which should add to the overall experience.
We’re also looking into moving the more graphical portions of the site and many of the weekly features we published this year to a website, leaving this Substack for more writing/analysis focused posts and updates. We’d hope to be able to put much of our data into tables that could be sortable and better-presented than the static images we’re limited to here, but that’s all going to take some learning on our part!
With all of that said, we truly cannot thank you enough for making this inaugural year at TNG so successful. Since opening the site in late May, we grew from 0 to 290 subscribers in 7 months! Our 21 published podcasts received 1,309 downloads! We are truly humbled by the response and that y’all would choose to spend some of your time reading and listening to our stuff.
We are especially honored to count a number of coaches among our amazing 290 subscribers. Coaches, we are your biggest fans and we are grateful for all of the work you put in with your kids and communities. Y’all change lives daily. I know for a fact that the three of us wouldn’t be who we are today without having played Texas high school football under great leaders like you.
We hope everyone has a wonderful holiday season…have a Merry Christmas and a Happy New Year!