As the members of UAI climb up the maturity curve and share their success around using analytics in their business, there are still road bumps ahead. I recently read an article related to machine learning, “The machine learning reproducibility crisis”, by analytics thought leader Pete Warden. This article highlights tissues with existing analytics ability and lack of replicating and training machine learning models to deliver value over time through a single or multiple use cases or purposes.
Here are a few interesting quotes from the article:
- “It’s hard to explain to people who haven’t worked with machine learning, but we’re still back in the dark ages when it comes to tracking changes and rebuilding models from scratch.”
- “Why does this all matter? I’ve had several friends contact me about their struggles reproducing published models as baselines for their own papers. If they can’t get the same accuracy that the original authors did, how can they tell if their new approach is an improvement? It’s also clearly concerning to rely on models in production systems if you don’t have a way of rebuilding them to cope with changed requirements or platforms.”
- “It’s not all doom and gloom, there are some notable efforts around reproducibility happening in the community. One of my favorites is the TensorFlow Benchmarks project Toby Boyd’s leading. He’s made it his team’s mission not only to lay out exactly how to train some of the leading models from scratch with high training speed on a lot of different platforms, but also ensures that the models train to the expected accuracy.”
Those are some of the highlights within the article, but I suggest you check it out yourself and discuss with your teams.
You can access the article here: The machine learning reproducibility crisis
More information on Pete Warden available here.
