Manager: Last week you told me that the model was ready.
Data Scientist: Yes, but on Tuesday I found a bug. Column X in the data had a lookahead bias due to some quirks. So now I am retraining the model. The existing version is complete junk and should not be trusted. But we’ll get the new model tomorrow.
Manager: And will it be ready then?
Data Scientist: Yes, unless there is another issue.
Manager: And when will we know for sure that the model can be trusted?
Data Scientist: Never. Unless you try it on real people.
Manager: So are you telling me, we have to risk company’s money and reputation because you cannot figure out if the model is working?
Data Scientist: Well, no. We can just test it for a few more weeks until we feel confident. Or you can take a chance and deploy it today for a small set of users.
Manager: I don’t like how “take a chance” sounds. Why was our other model ready in a week?
Data Scientist: Because, I guess, Jake was not as careful and just didn’t test his model so thoroughly. Don’t you remember how last summer we had a disaster with his other model?
Manager: Oh, that was a one-off thing. It happened because the data vendor didn’t tell us one detail.
Data Scientist: You see! My testing is to make sure it doesn’t happen.Manager: I still don’t know why you are saying that a model is never ready. Jake’s models are always ready on time, and I need your model to be ready by next week. I have to go now. Bye.