Using an ML model to Improve Predictions ✨🔮

TLDR: As of 24/01/2023, 12PM AEDT, the default view of 100 Warm Tunas will display results with an ML model applied to them, in their new order. Results which have changed ranks due to the adjustment will be marked with text “✨ ML Model Applied”, along with the original rank:

The vote counts displayed on the prediction will continue to be the true number of votes collected. You can opt out of ML applied ordering by using the controls overlay.

We hope you enjoy these improvements to 100 Warm Tunas. If you want to show your appreciation for the work that I do, you can help contribute to the project by buying me a coffee ☕️.

---

A Deep Dive into 100 Warm Tunas ML Model

100 Warm Tunas collects voting slips posted by people to various social networks, and tallies the votes for each song to produce an overall prediction.

This method of collecting data has a potential downside - selection bias. Selection bias occurs when the sample of data that is collected is not representative of the entire population. In this case, the sample of voting slips that is collected only represents a small percentage of the total number of votes, which means that the results may not be representative of the entire population of voters

Over the last 6 years, we’ve seen a handful of patterns emerge in the predictions which 100 Warm Tunas provides. Many of our users also are aware of this. For example, 100 Warm Tunas will frequently rank Australian rock tracks significantly higher than they place on Hottest 100 day.

Having 6 years of historical data puts us in a unique position. We have information about how the algorithm predicted songs and can compare this to how the song actually placed (error). This information can be used to train an ML model to predict error in new, unseen datasets and adjust for biases that may exist in the data that we collect.

So first, you might be wondering what ML is? Machine learning (ML) is a method of teaching computers to learn from data, without being explicitly programmed. It is a type of artificial intelligence that allows computers to automatically improve their performance with experience. The algorithm is presented with a dataset, which includes input data and the corresponding correct output. The computer uses this dataset to learn mappings and patterns from inputs (features) to outputs (labels). Once the model has learned this mapping, it can be used to make predictions on new, unseen data.

For 100 Warm Tunas, we can train a model to attempt to predict the error based on previous countdown results. This is considered a regression problem (the aim is to predict the output of a continuous value). Contrast this with a classification problem, where the aim is to select a class from a list of classes.

For every countdown, we have the following data:

Using this data, we can then extrapolate the number of votes required for a given song to have been predicted in the correct position:

However, in this instance, vote_count_error is absolute. We must normalise this to be relative to the vote count in order for neural network to correctly identify patterns.

In order to actually build a model, we must also have some inputs (features). We’ve used some of the following features to train the model:

• Artist (To account for patterns of artist bias found in the sample)
• Artist Genre (To account for patterns of genre bias found in the sample)
• Duration
• Tempo
• Predicted rank
• Various Spotify Attributes

With these inputs (features), and known good outputs (labels), we can now train a model. Once the model is trained, it can then be tested on a validation set, which was subtracted from the dataset prior to training in order to test its performance.

The following diagram demonstrates how closely the model is able to predict the vote count error for the previously extracted validation set. The closer to the blue line, the more accurate the predicted adjustment is:

We can also take a look generally at how all of the training data looks when plotted:

Finally, as one last test, we re-train the model, except this time reserving the 2021 countdown data as our validation set:

Armed with a trained model, we can then apply it to our unseen data (2022’s predictions) which we have no known good/expected output (vote_count_error):

For every song, we use the model to produce a prediction of vote count error:

The predicted vote count error is then applied to each of the songs known vote_count in the prediction to produce a new, adjusted vote count. For the top 15 predicted tracks, the adjustment is applied 100%. Between 15 and 25 the adjustment is blended into the mean of whole adjustment dataset. Finally, the remaining tracks (ranks >25) adjusted by the mean of the whole adjustment dataset:

Doing this produces a new, reordered prediction which hopefully has some of the inherent collection bias factored out. It’s important to note that while this model can make predictions about vote count error, it is not perfect and the predictions should be taken with a grain of salt. The model is only as good as the data it was trained on, and factors such as changing public taste cannot be accounted for.

As of 24/01/2023, 12PM AEDT, the default view of 100 Warm Tunas will display ranks in their ML applied order. Results which have changed ranks due to the adjustment will be marked with text ✨ ML Model Applied, along with the original rank.

The vote counts displayed on the prediction will continue to be the true number of votes collected.

You can opt out of ML applied ordering by using the controls overlay:

We hope you enjoy these improvements to 100 Warm Tunas. If you want to show your appreciation for the work that I do, you can help contribute to the project by buying me a coffee ☕️.

Your support is greatly appreciated and helps to keep 100 Warm Tunas running for years to come.