Skip to content

US Development Graphs

I posted these graphs on Twitter a few days ago with basically zero commentary, so I thought I should at least describe what I did in a tad more detail.

I make these graphs because they get at an aspect of something the US Ski Team leadership talks about fairly often: being “on track” for top international results. They are not meant to represent any actual artifact or analysis that the US Ski Team actually uses for making decisions. They are merely my translation of the general concept. There are lots of different ways you could actually approach this idea and end up with somewhat different conclusions.

First we need a definition of “top international results”. For distance events I took that to be finishing in the top ten in WC/WSC/OWG/TdS events at least three times. For sprints, reaching the semifinals in WC/WSC/OWG/TdS events at least three times.

Then I took those “successful” skiers and plotted a band that represents the 10th-90th percentile of their race FIS points by age. So this band represents what the bulk of these skier’s race FIS points looked like at different ages.

Then I plotted on top the actual race FIS points from some of the top US skiers according to the most recent FIS points ranking lists for distance and sprint. I skipped a few people who I believe retired, and generally tried to focus on younger skiers in places. But mostly I was just running down that ranking list.

Finally, the red trend line is tracking the median result for each skier. I was a tad lazy and didn’t tweak the smoother too rigorously for each skier, which results in some poor (overfit) results in a handful of cases. Those are the red lines that are really, really squiggly. If I were taking this more seriously, I’d replace those with a more rigid linear fit, or maybe just omit the trend line for those skiers entirely.

National Team Selection & Age

Springtime means national team nominations, which is always one of people’s favorite things to argue about. This has become particularly true in recent years with some national teams having a fairly explicit preference for younger skiers built into their criteria. It seems like every year there are 1-2 skiers pushing 30 (or beyond) that have skied well that season but are passed over for younger competitors who have equal or even worse results to boast of.

National team selections, along with Olympic and World Championship team selections, are delicate and fraught decisions, and so I try to write very carefully on the topic. So be prepared for lots of qualifications and caveats. If you wanted a straight answer, you shouldn’t have asked a statistician!

One of the first problems with looking at patterns between age and international skiing success is defining “success”. Is it podium results? Top 10? Scoring World Cup points? Is it doing any of those repeatedly, and if so, how many times? This is the first subjective decision point in this ostensibly objective exercise.

For the purposes of this post I’m going to go with success being a top ten result in a distance race (WC/OWG/WSC/TdS) or a making a sprint final (top 6). Why? Because this is my blog and so I get to do whatever I want. Let’s start with some simple descriptive stuff. At what age to skiers tend to achieve these things?

Starting with distance:

The way to read this is to start on the y-axis and move horizontally across. So start at 50% and move across to see that around 50% have gotten their first top 10 distance result by the age of 25. 75% of men have gotten their first top 10 by around age 26 and women closer to 28. Another way to put this is that “only” around 25% of people with a top ten distance result manage to do that for the first time past age 26-28.

Now, 25% is a minority, but you might also say it’s a pretty big minority. Big enough that it isn’t really surprising when it does happen. (Does this kind of language feel familiar to certain probabilistic forecasts of a certain US presidential election?)

Before we get into more technicalities and caveats, lets look at the same thing for sprinting:

These curves are pretty similar, maybe shifted back by a year or two. So here around 50% of skiers have their first sprint final by age 23-24 and around 75% have achieved that by around age 26.

An interesting thing to think about with this data is what direction the causation (if any) is going in. Or more precisely, what specifically is it about being 25-30 that makes it harder to land a strong result? Do older skiers have an inherently lower ceiling on their future improvement? Or are there structural barriers that give them fewer opportunities to improve?

It is certainly true (or at least plausible) that on average your potential to improve decreases as you age. I mean, I’m 41; trust me, getting older is a thing. But what if because everyone “knows” that older skiers have a lower ceiling they are subtly but systematically given fewer opportunities that artificially contributes to their lack of success?

I’m not trying to be inflammatory here. Consider the case of Martin Johnsrud Sundby, who was left off the Norwegian national team this year. Norway probably could field a pretty darn competitive national team comprised entirely of skiers over the age of 27-28, at least from the perspective of us in North America. But in an environment filled with talented skiers under 23 (again, compared to North America) why on earth would you?

So what if this pattern toward success at an early age is being driven by the hyper-competitive environments of Norway, Russia, Sweden, etc. where if you haven’t had World Cup success by age 24 there are 5-10 more super talented 22 year olds biting at your heels for a chance?

The problem with this is that if you’re looking for a data set of skiers who have plugged away at competing on the World Cup at a middling level for a long time to see how many of them suddenly get much better past age 28, that is a very small and peculiar set of skiers. Hard to do “unbiased” analysis on those folks. (I mean, I have looked at it, and these folks really do rarely improve, but it’s such an oddball group of skiers I’d hesitate to generalize much from them.)

Now that I’ve pummeled you with that barrage of caveats, let’s actually look at what really does happen after a skier’s first top 10 distance result or their first sprint final. First up, distance:

For people who got their first top 10 WC/OWG/WSC/TdS result between 2005-2015, this shows the number of subsequent top 10s they had over the next five years. The points at the bottom get pretty crowded (and I didn’t want to make you think about a log scale) so I’ve highlighted all the 0’s by coloring them blue and dropping them down slightly. So the blue dots are people who were one and done.

So this is why things look pretty grim for older skiers, particularly men. Look at that drop off for the men at age 26-27! The women have a handful of folks who developed into regular top 10 skiers past age 25, but they are a distinct minority. Next up, sprinting:

More or less the same story, but shifted 1-2 years younger.

Again, one of the things I worry about when I look at plots like these is what I was rambling on about before. How much of this pattern isn’t because skiers can’t improve dramatically after the age of 25, but instead that so many hyper-competitive skiing nations can afford to be really brutal about tossing them aside because they have a seemingly endless supply of talented 22 year olds right behind them?

Like everything else, it’s probably not one or the other, but a mix of both. This is where we start to incorporate other sources of information. For example, I’m not an exercise physiologist but I can imagine evidence from that field either in support of or against skiers having a lower ceiling past age 25 that is independent of race result patterns. Also, coaches will typically have knowledge about a specific athlete that isn’t captured by knowledge about the “average athlete”. This can be a dangerous source of information, but it is real and valuable. But now we’ve strayed past what could reasonably be encoded into a concrete national team selection criteria.

Don’t say I didn’t warn you! Ask a statistician and you’ll likely get a lot of graphs and data that mostly lead to more questions.

Do US Women Have Better Results Earlier In The Season?

I received an email recently asking the above question, and since responding to these sorts of things is often kind of challenging in an email I figured I’d just write it up here. I’m sure I’ve written about this sort of thing before, but it was probably a long time ago. Regardless, I’ve been looking at skiing results data for long enough now that I’ll admit to having some pretty strong priors when people ask me questions like this. Specifically, my gut reaction is to respond that whatever pattern you’re responding to is probably not going to be particularly evident if we start looking at it systematically. Then again, I’m sort of a statistical Debbie Downer so take that with a grain of salt.

Setting my depressing priors aside, let’s see what we see. So let’s start by taking all US women’s major international results over a sizable chunk of recent history, say since the 2013-2014 season. We’ll exclude pursuit starts, since they rather famously don’t compare well with anything else, and let’s also exclude events in USA/CAN to avoid having to worry too much about events with enormously expanded American starters. (This last one isn’t a huge number anyway.)

The FIS race calendar is mostly pretty predictable, but it does shift around from year to year. But let’s be simple about this and divide the races up by month. However, I’m going make a special allowance for the Tour de Ski and basically consider it a “month” by itself. Probably I’d guess that people who’ve done it might be ok with labelling it as a month’s worth of racing, anyway. So our “months” are Nov, Dec, TdS, Jan, Feb and Mar. That’s a little weird, but hopefully it makes sense.

Again, simple is the best place to start, so let’s just count up American women top 3 and top 10 results in those time periods since 2014:

Three things immediately jump out: a noticeable lack of top results in November (i.e. Ruka), an apparent spike in good distance results during the Tour de Ski and a very consistent rate of top results in sprinting after November.

Yes, yes, I hear you all shouting at me through the internet about needing to look at rates not counts, so let’s divide all these by the total number of results (i.e. how many US women finished a race in that time period):

See, now that’s kind of interesting! It’s almost like things completely reversed between the distance and sprint results. We still see pretty low rates of top results in November, but now the rate of top distance results is the more stable of the two and the rate of top sprint results has an obvious peak during the Tour de Ski.

Circling back to the original question, my Debbie Downer-ism was mostly correct in that there isn’t some very obvious peak in Nov/Dec that steadily declines throughout the season. But that doesn’t mean this was all a waste, of course.

What’s going on here, I think, is two things: I suspect the US may have made the Tour de Ski a focus for their season in ways that perhaps other nations have not, and if you do focus on the Tour and finish it you’re likely to be rewarded with some significantly reduced field strengths and sizes in the later stages.

Frankly, what’s most interesting to me about all this is the relative lack of success at the Ruka races in November. I don’t know if that’s the result of intentional strategic choices or if perhaps the races and formats haven’t really suited as many of the US women as other events have. Perhaps I’ll have to write a follow up that digs into that a little further.

Chinese World Cup Performance

I got another friendly homework assignment from The Devon Kershaw Show today. This one was pretty simple, though, as Jason was just wondering about the performance of Chinese skiers on the World Cup (or similar level events). This was prompted by Chunxue Chi finishing 28th in today’s 10k freestyle race in Oestersund.

Jason was curious whether this was one of the best performances by a Chinese skier on the circuit, and the answer is: kind of.

The Chinese women have had a handful of roughly equivalent distance performances, but not recently. In general they’ve had more success with their male sprinters. In this particular case, though, Chunxue Chi finished 24th just a few weeks ago in Nove Mesto. Still, it hasn’t been very common to see the Chinese women scoring points in distance events.

Tagged ,

Unusual WC Sprint Qualifying Time Gaps

The recent episode of the Devon Kershaw Show by FasterSkier (which I’ve been enjoying quite a lot) included some discussion of the noticeably large time gaps in the men’s classic sprint World Cup in Oberstdorf, Germany. Beginning at around 25:18 they discussed a variety of aspects of the large time gaps. I thought that some of the things they noted would benefit from some cursory looks at some data. In no particular order…

“We’ve seen this before from Klaebo”

Indeed we have. Over the past decade or so when Klaebo has won a sprint qualification the margin has averaged around 1.65% in classic and 1.12% in freestyle. His margin in Oberstdorf was 1.31%. So this was not even particularly extreme for him, in terms of a gap over second place.

30th place is 15 second back

That is a remarkable time gap for a World Cup sprint qualification, for sure. But how unusual is it for much of the top 30 field to be that far off the pace? Let’s take all major international (WC, OWG, WSC, and TdS) sprint races over the past 10 seasons and plot the top 30 qualifying %-back values for each race as a line.

That’s definitely unusual. One of the things that’s interesting about gauging performance in individual sports like xc skiing is that there’s a subtle distinction between a particular outcome arising from one person skiing really slowly and another person skiing really fast. And of course it could be a combination of the two.

Devon and Jason discussed the fact that if you’re 30th and 15 seconds back in qualification, it seems really unlikely that you have a realistic shot at winning. But I think that underplays somewhat how unlikely it is for anyone to win a WC sprint race when you qualify that slowly, regardless of the time gap.

Over ten seasons, men qualifying in 25th-30th have reached the podium only 16 times, or around 1.92% of the time. On the women’s side it has happened only 7 times or 0.8% of the time. Actually winning is obviously even rarer: 5 times for the men and only once for the women, for 0.6% and 0.1% respectively. So the folks qualifying in the back of the field are very unlikely to win, even with “normal” time gaps.

Of course, it is easy to get confused about the causal relationship here. Fast qualifying times will tend to be correlated with all sorts of other fitness characteristics that lead to good performance in the heats. It’s not the fast qualifying time itself that leads to better performances.

Let’s return, though, to the unusual time gaps themselves. Certainly some specific talented sprinters had rough days (Ustiogov and Iversen were mentioned in the podcast). But if you’re sitting in 30th, 15 seconds out in this particular case, should you be saying to yourself, “Geez, I skied pretty badly!” or should you be saying, “Geez, the top 3-4 guys really just had another gear today!”. (Or obviously somewhere in between.)

This is one of the problems with measuring performance based on only the winner. Devon talked (correctly, I think) in the podcast about how if you’re 15 seconds back from the winner in qualification that’s a pretty good signal that you’re not going to win. But, it is a potentially quite misleading signal about how you skied relative to your own performance history!

One approach I happen to like to use as an additional perspective is to calculate percent back values based on the median, or middle, skier. What do the percent back curves I plotted above look like if I recalculate them based on the skier who qualified 15th?

Now which part of the red curve looks more unusual? This look at the race would at least suggest that the top 4-5 men were the ones who had unusual races, rather than the the field as a whole. The back of the qualifying field seems a bit more behind 15th place than usual, but not nearly as dramatically.

Obviously, this doesn’t change the calculus for how close you are to winning. If you’re racing against people like Klaebo who can put down winning margins like this, it is small comfort to be told that you didn’t really ski any slower than you usually might. But it is useful to keep in mind when you see time gaps like this that it is quite possible that the folks in 25th skied just as fast as they did yesterday, a month ago, a year ago (maybe faster!). In other words, it can be a signal of how much work you have to go, but not necessarily that the work you’ve done so far isn’t helping or even making you slower.

How fast do podium finishers qualify?

I’m glad you asked! Let’s look at that with the following plot:

ECDF stands for “empirical cumulative distribution function” which is quite a mouthful and isn’t a graph that normal folks interact with regularly, but don’t panic.

This just graphs the cumulative proportion of podium finishers (y-axis) who qualify with a given percent back or better (x-axis). So if you pick a spot on the y-axis, say 25%, and move horizontally until you hit the curve you look down on the y-axis and see that ~25% of podium finishers qualified with a percent back of ~0.4% or better. Don’t forget the “or better”.

Similarly, ~50% of sprint podium finishers qualify at ~1.4% back or better. And so on. The vertical line each curve starts with represents the fact that ~20% of podium finishers actually win qualification (0% back).

Looking at this it seems that 1.5-2% back might be a good target to think about if you want to be landing on the podium in World Cup sprint races. If you can’t qualify at least that close, you’re probably not setting yourself up for a good shot at a podium.

Tagged , , , ,

Diversity of Nationalities on the World Cup

Norwegians don’t win everything, but sometimes it can sure feel like it. A very common thing ecologists measure and study is species diversity in an ecosystem. We can do the same thing with nationalities finishing in the top 3, 10, etc in major international races. As you might imagine, there are quite a few ways to measure species diversity. I’m not interested in delving into the minutiae of all these measures, so we’ll just use a nice, boring Shannon index:

Higher values indicate more diversity (among nationalities) for athletes who win, finish in the top 3, 10 etc. The Shannon diversity index was calculated for each group for each season. Note that my data is incomplete for parts of the 1980s.

The index values bounce around a fair bit from year to year in a way that seems like noise, but some larger trends also stand out. For instance, the mid-00s in women’s distance events was notably more diverse for the top finishers. I suspect a major driver of this may be 1-2 very talented female skiers from nations like Poland and the Czech Republic during that time period that you may know of. But since 2010, national diversity in women’s distance events at basically all levels has decreased.

Men’s distance events also seemed to see slightly more diversity during the 00s, at least at the podium level and above, but this also seems to be receding back to historical levels.

The diversity among women’s sprinting results seems to have been gradually declining for the past 15 years or so at all levels. Trends in the men’s sprint panel seem less apparent to me; mostly flat, maybe declining very recently.

One interesting thing that does stand out to me in all cases is the relative stability of the diversity of nationalities among the top 30 over the entire time range. Before I made this graph I probably wouldn’t have been too shocked if it had turned out that the 1980s had vastly less diversity than now, but that isn’t really the case it seems.

US World Cup Progress

I got an email the other day asking if I thought that the US had been improving in international races lately. There are obviously lots of ways to measure the progress of a national ski program, but the only one I really can comment on is simply race results. So these are the US major international race results (WC, OWG, WSC and TdS) over the past decade or so. I’ve removed pursuit (i.e. handicap) start distance races, as they tend not to be very representative of how athletes perform in any other circumstance.

The women’s sprint and distance results have obviously improved, as they transitioned from one skier doing consistently well (Kikkan) to a handful of skiers doing well. The men’s sprint results have declined slightly, most likely correlated with Andrew Newell getting older and then retiring. The men’s distance results have been pretty consistently in the 25th-75th range with very occasional top 10-15 results, basically always in mass start events.

So the short answer isn’t much different from what you might say without all the number crunching: the women have improved quite a bit, while the men’s results have been either treading water or getting slightly worse, in the aggregate.