Why Mean Squared Error Hates Outliers by Devansh Sep, 2024 Medium

Why Mean Squared Error Hates Outliers | by Devansh | Sep, 2024 | Medium #

Excerpt #
Too many ML folk blindly use MSE (Mean Squared Error) as the default error function without thinking about the consequences. Small changes in your error functions can have HUGE impacts on your AI…

Why you should not blindly use error functions #

[

Devansh

]( https://machine-learning-made-simple.medium.com/?source=post_page-----b601e3b116f0--------------------------------)

Too many ML folk blindly use MSE (Mean Squared Error) as the default error function without thinking about the consequences. Small changes in your error functions can have HUGE impacts on your AI training behavior.

To illustrate, we’ll discuss two well-known error functions- MSE and MAE (absolute error).

Mathematically, these are different because-

-)MSE squares the errors (differences between predicted and actual values), which means that larger errors are heavily penalized.

-)MAE simply takes the absolute value of the errors. All errors are treated proportionally to their size.

This leads to some pretty cool outcomes. For one, “MSE really hates outliers, but ignores small mistakes”. Let’s talk about why-

When your error is small (<1), then squaring it reduces the value (0.5 *0.5 =0.25). When the error is large, squaring it increases the error (2*2=4). Functions like MAE (or AE) will flip this dynamic.

This can be a big difference maker in your AI training. Here’s an example of when MAE becomes better (look at image two). In it, while both LS (least squares- another way calling MSE) and AE (Angular Embeddings, which are non-convex like MAE) can reconstruct the ground truth with no issues in normal conditions, AE absolutely dog-walks LS when we add in more outliers-

“AE with a quadratic criterion in the complex domain is remarkably robust to outliers, unlike its real domain counterpart LS. (a) A groundtruth surface with a height range of 1 and a maximal gradient magnitude of 0.16. (b) Gaussian noise with ¼ 0:05 is added to the pairwise height differences between pixels within a city-block distance of 2. Integration of these differences by LS and AE shows comparable reconstructions, both with a standard error of 0.0085. © Uniformly distributed outliers of values 3 are added to 10 percent of these noisy height differences. The standard error becomes 0.1666 for LS and 0.0184 for AE. The surface reconstructed by LS is barely recognizable, while that by AE remains a close approximation to the ground-truth surface”

These are just some of the many design choices you should consider when designing your AI.

If you’ve developed an interest in Angular Embeddings, check out our guide to improving AI Embedding with Geometry here-

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Many companies have a learning budget, and you can expense your subscription through that budget. You can use the following for an email template.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.