profitably wrong

I’ve come to realize that my career so far has been built on being “profitably wrong.” I think this is interesting because the usual approaches are being “profitably fast” (optimizing) or “profitably better” (improving), and most people think of any kind of wrongness as being a terrible thing. But sometimes the best way to optimize or improve is approximating!

The definitions of “profitably” has changed as I’ve worked on different things, as has the specific type of “wrongness”. A couple specific ways accepting “wrongness” have been profitable for me include:

• being simpler: approximating very complex neural models with very simple mathematical models
• being faster: re-using outdated matrix factorizations to still improve a Newton iterate toward a solution
• being tractable: naive bayes makes an independence assumption that is demonstrably false, but can still be useful

This type of idea seems to pop up everywhere for me:

• HOGWILD: a parallel stochastic gradient descent algorithm that gets unreal speed by disregarding classic concurrency correctness mechanisms (read: locks) but works anyway
• bloom filter: gives up the ability to always answer correctly to instead operate in a fixed, small amount of memory.
• HyperLogLog: similarly gives an approximate unique count in a fixed, small amount of memory overhead. it’s used in redis and at google.
• p2 algorithm: estimates quantiles without storing all observations.

These ideas were all initially hard for me to wrap my head around, after spending several years learning how to prove that this sorting algorithm or that one actually sorts a list. But I’m completely sold on their utility and have come to see it as my own little niche.

DAYDREAM

What if you knew a secret to solving whatever problem you work on? What good is a secret? It’s good if it helps you solve that problem better than other people working on it.

What’s the worst that happens if you’re wrong? What usually happens when you’re wrong?

APPROXIMATION

One way to invent secrets is by approximating reality – inventing a model. It’s not good enough to just invent a model though. That model must have some predictive power.

The key in each of my examples is that predictive power:

• simple neural models exhibit some properties of much more complex, chemically-based models.
• a Newton-like algorithm that can estimate a better iterate.

QUANTIFY THE ERROR IN YOUR APPROXIMATION

It’s important to understand the errors you might make with your approximation. At a very basic level, you need to understand whether the approximation incurs more cost than it earns to decide whether to use it or not.

One starting place for this is simply writing down the eventual outcome for false positives and false negatives. Another great approach is calculating an exact solution for an easier (relatively!) problem and comparing against the approximation.

FIND THE APPROXIMATION’S SWEET SPOT

All approximations have some range of validity, outside of which they aren’t a reasonable model of reality anymore. Sometimes, they’re very well-defined and carefully explored; HOGWILD, for example, needs a certain sparsity structure to avoid destroying the effectiveness of the parallelism with very frequent concurrent writes. Other times, they’re simply not well-understood (megacomplex systems under study in medicine, for example) or obscured by an assumption of the model: assuming linearity of some portion of a function, assuming an incompressible fluid, etc. It’s important to have some way of knowing whether your predictions/approximations are adding value or simply garbage.

FIX IT LATER / EVENTUAL CORRECTNESS

Sometimes, approximations apply even to problems that seem like they warrant the utmost correctness. Think money movement or automated vehicle control systems. On the contrary, lots of applications like these really just rely on some notion of eventual correctness. (For varying definitions of “eventual”, of course!) When you think from the “what’s the worst that could happen” or the “crap, it happened anyway because of something else” angles, it turns out that you’d just issue a refund or steer back into your lane or whatever. And, in fact, the “profitably wrong” mindset might enable you to decide on that corrective action sooner than later!

WHEN TO AVOID / WARNINGS

Sometimes, correctness is king. If you’re moving money you might be able to tolerate “wrongness” in your “fraud/not fraud likelihood” calculation, but you probably aren’t willing to tolerate the same on your balance adjustment method. Even if you’re controlling a spacecraft, a single wacky control recommendation might be okay, as long as there’s a bulletproof watchdog/contingency plan when you get several in a row. Hopefully, you know which type of system you’re working on!

It’s tempting to think because it’s only an approximation, you be sloppy in your implementation of it. But bugs in approximating code are much harder to nail down: did your code make a mistake because of a bug, or because the model (your simplifying assumptions) was wrong? My world-weary, hard-won advice is that it’s worth as much rigor as you can muster in approximating code.

I hope I’ve inspired you to embrace the idea of having an error rate! I’d love to hear from people about ways you:

1. have been profitably wrong?
2. have tried to be profitably wrong and failed?