Baseline modeling is a necessary part of model validation. In our expert opinion, it should be required before model deployment. There are many baseline modeling types and in this episode, we're discussing their use cases, strengths, and weaknesses. We're sure you'll appreciate a fresh take on how to improve your modeling practices.Show notesIntroductions and news: why reporting and visibility is a good thing for AI 0:03Spoiler alert: Providing visibility to AI bias audits does NOT mean exposing trade secrets. Some reports claim otherwise.Discussion about AI regulation in the context of current events and how regulation is playing out between Boeing and the FAA (tbc)Understanding baseline modeling for machine learning 7:41Establishing baselines allows us to understand how models perform relative to simple rules-based models, aka heuristics.Reporting results without baselines to compare against is like giving a movie a rating of 5 without telling the listener that you were using a 10-point scale.Baseline modeling comparisons are part of rigorous model validations and should always be conducted during early model development and final production deployment.Pairs with analyses of theoretical upper bounds for modeling performance to show how your technique scores between acceptable worst and best case performance.We often find complex models being deployed in the real world that haven’t proven their value over simpler and explainable baseline modelsClassification baselines and model performance comparison 19:40Uniform Random Selection - simulate how your model does against a baseline model that guesses classes randomly like a dice.Most Frequent Class (MFC) - the most telling test and often the most telling test in the case of highly skewed data with inappropriate metrics.Single-feature modeling - Validates how much the complex signal from your data and model improves over a bare minimum explainable model.And more…Exploring regression and more advanced baselines for modeling 24:11Regression baselines: mean, median mode, Single-variable linear regression, Lag 1, and Least 5% re-interpretationAdvanced baselines in language and visionConclusions 35:39Baseline modeling is a necessary part of model validationThere are differing flavors of baselines that are appropriate for all types of modelingBaselines are needed to establish fair and realistic lower bounds for performanceIf your model can’t perform significantly better than a baseline consider scrapping the model and trying a new approachWhat did you think? Let us know.Do you have a question or a discussion topic for the AI Fundamentalists? Connect with them to comment on your favorite topics: LinkedIn - Episode summaries, shares of cited articles, and more. YouTube - Was it something that we said? Good. Share your favorite quotes. Visit our page - see past episodes and submit your feedback! It continues to inspire future episodes.

Ver todos los episodios