Putting a number on 328 olympiad problems

Most of my preparation time as a tutor does not go into teaching. It goes into finding the question. A student learns more from one problem at the edge of what they can do than from ten they can already handle, and the edge moves every few weeks. Finding that problem means remembering which of thirty years of olympiad papers is the right level for this student today.

So I built something to do it, and the interesting part turned out to be the measurement.

What "difficulty" should mean

The obvious approach is to label problems easy, medium and hard. This is useless within a week: a student who finds "medium" comfortable has no next step, and two mediums can be half an hour apart in reality.

Chess has a better answer. Ratings are not labels, they are predictions: the gap between two ratings tells you the probability of each result. Treat a problem as the opponent and the same idea works, which is exactly how chess sites rate their puzzles. A rating then answers the question I actually care about: what are the chances this student cracks this problem?

That gives every question a number on the same scale as every student, and it makes the tool's job precise. "Three geometry questions I have an even chance at" becomes a search, not a judgement call.

Where the starting numbers came from

Ratings are only useful once they are calibrated, and a new system has no attempts to learn from. The British Mathematical Olympiad has thirty-three years of papers but almost no published statistics, so the starting difficulties had to come from what the UK Mathematics Trust does release: the markers' reports.

Nine of those reports exist online. Two of them, from 2016/17 and 2017/18, give the average mark for each question, which is close to a direct difficulty measurement. The other seven give a histogram of total scores, drawn as a chart. Those charts turned out to be vector graphics rather than images, so the exact bar heights could be read out of the PDF: for 2019/20 the extracted counts sum to precisely the 1,475 candidates the report states.

A total-score histogram seems like weak evidence about individual questions, but it is not. BMO is marked "10 minus / 0 plus": a candidate who cracks a question scores 7 to 10, and one who does not scores 0 to 3. Totals therefore cluster near multiples of ten, and each year's histogram is effectively a count of how many questions each candidate solved. Fit a model where ability varies across candidates and difficulty rises through the paper, and the shape of that distribution pins down how much harder each question is than the last.

The fit says about 1.39 logits per question position, which on the scale I settled on is roughly 173 points. Anchoring it so that a typical modern Round 1 Q1 sits at 1200 and a Round 2 Q4 at 2500 gives a ladder for all 328 questions, adjusted for the year and for how hard each individual paper turned out to be.

Three things the data said that I did not expect

Question order is only a rough guide. In 2017/18 the average mark on Q3 was 7.2, higher than Q1's 6.8 and far above Q2's 4.2. The paper's intended order and the order candidates experienced were different, which is worth remembering before assuming Q5 is harder than Q4.

Papers vary far more year to year than over decades. Older papers are easier, but mildly: the long trend is about nine points a year. Individual papers swing several times that. The qualifying mark for Round 2 was 43 in two consecutive years and then 31, a twelve-mark collapse at the top of the cohort in a single year.

Round 2 looks easier than its reputation. Each year, 70% of qualified entrants reach the Merit threshold on Round 2, and those same students mostly did not solve Round 1's last question. Taken literally, Round 2 Q1 behaves like a Round 1 Q4 under exam conditions. I overrode that with judgement, because a Merit mark of 6 to 10 can be assembled from partial credit rather than one solved problem, but it was not what I expected to find.

What chess already knows

Having decided to rate problems like opponents, I got the update rule wrong in an instructive way.

My first version let a new student's uncertainty run far wider than any chess system allows, and combined it with the standard approximation used in Elo. That approximation is fine for a settled rating and badly overconfident for an unknown one. The result: I solved a question I give to beginners and my rating leapt 800 points.

Chess has guarded against every part of that for decades. Glicko starts an unrated player with a wide uncertainty but caps it, and never allows more. FIDE limits how far one game can move you, and sets a first rating from a whole event rather than a single result. There is even a rule for lopsided pairings, where any difference beyond 400 points counts as exactly 400 — which is precisely the situation here, since a student can open any question they like, however far above them.

Adopting all of it, unchanged, fixed the problem. The same solve is now worth 64 points to an unrated student and 11 to an established one.

The part chess does not have to solve

One problem is genuinely ours. Chess players are mostly stable in strength; students improve every month. If question ratings are free to drift, "students got better" and "questions got easier" are mathematically identical, and a student who improves all year can show a flat rating.

The fix is to notice that a question is a fixed object. Its difficulty never changes, so any common movement across the whole catalogue is the population changing, not the questions, and it belongs in student ratings. So the catalogue's average difficulty is held exactly where the calibration put it, while individual questions still move relative to each other. The yardstick stays still and the students move against it.

What this is for

None of it matters unless it saves the hour I spend choosing questions, and that will only be clear once students are using it. The ratings that came out of the markers' reports are a starting guess: real attempts will correct them, question by question, and I expect several to move a long way.

The worked-out version is Mathelo, currently in private beta with my own students.