Back to basics: how data cleaning makes all the difference

A simple example like this can help everyone understand the importance of good data. But in the real world, we’ve noticed that a lot of scientists don’t pay much attention to it. They usually focus more on the model they’re feeding with the data set. And those who do want to check their data sets, but don’t know where to start, won’t find much help online. We found some articles that described parts of the process, but none that discussed the entire process in detail. So we thought it was high time to create a place that explains how to get a good data set in just a few steps.

We talk about those steps in detail in our Tech Talk, and this blog will give you a good idea. And you’ll see: it’s really quite simple.

Now that you’ve identified the holes in your data, you want to know why they’re missing. There are three possible reasons. We use the example of taking a cat to a veterinarian to be weighed. The data can be incomplete in one of the following three ways:

- Missing completely at random: the cat can’t be weighed due to external factors, like the scale’s battery being empty or recharged.

- Missing at random: The cat was sick, and didn’t come to the appointment. The data have a column for ‘sick’, and if that column is true, then the ‘weight’ column is missing a value. That missing value is due to the complete column ‘sick’.

- Missing not at random: the cat’s weight hasn’t been filled in, because the owner was embarrassed at how fat it is. So the data are missing on purpose.

Finally, you want to test if the completed data set is actually valuable. You can do that using a test- and training set. We’ll explain how to do that exactly in our Tech Talk.

But the most important takeaway is: that we need to be aware of our responsibility for the quality of data. We’re not just responsible for developing models, but also the cleanliness of our data sets. Even a minor data cleanup can make a huge step forwards in our results.

When you work with data sets, the question you face is always whether or not to fill in the gaps. If you don’t fill them in, should you just delete the missing data? And if you do want to fill them in, how would you do it? Should you even be worrying about missing data? In their Tech Talk, Frances and Andre answer all of these questions and more. But to get you started: yes, you should always be worried about missing data. What you do about it, though, is up to you.