What would ideal explainability even look like? For example, in deep learning if we knew what individual neurons were doing, would this solve any problems? Or even if we could specifically know why the model makes a particular error, the solution is still just to throw more training data at the model. Maybe increased explainability could help with more rational design of model architectures and training methods, but that has never been such a huge bottleneck in my experience.
SHAP, LIME, saliency maps, and the like I have always felt to be pretty hand wavy, inaccurate, and they don’t provide much insight for what developers need. I feel like they only exist to placate AI/ML skeptics so we can claim we sort of know what is going on with our models only when someone cares.
Am I missing something here? What specific efficiencies or insights would a developer benefit from if we had more explainable AI/ML models?
Also, I feel like this sounds a lot like generative AI, and explainability might not be necessary. Or maybe one could say that in the real world people want to have better assurance that things they do are based on a good underlying model. Images and text are easier to evaluate than predictive, speculative decisions about housing markets.
It is the reason why there is little to no application of AI models in the most serious of use-cases, especially in high risk environment(s) like transportation, finance, medical systems, legal applications. Even if there are, they are either extremely limited or have to be forced to comply with very strict regulations due to their untrustworthiness. There is a lot of money and lives at risk here if it goes wrong.
Tesla FSD is a great example of this and I guarantee that not many here trust the system to safely drive them without their hands on the wheel, end-to-end every day since it can get confused without explanation as to why it hallucinates. Not even Wall Street trusts AI black-boxes like ChatGPT to use it which also regularly hallucinates its answers [0]. Once again, even if they do use it, is in a very limited non-serious applications.
As you can see, it especially affects 'deep neural networks' in general which is what FSD and ChatGPT use and their black box nature makes their outputs highly suspicious and untrustworthy for anything serious. Hence the influx of 'chatbots', BS generators, etc on the internet.
[0] https://news.bloomberglaw.com/banking-law/wall-street-banks-...
On the other hand AI's used for specific applications and purposes can be tested thoroughly, you know that FSD sucks because it was tested and shown not to do it's intended work properly. You can test a medical imaging interpretation AI by comparing the results to a human doctor, and you can test test an in house climate system controlled by AI by judging its performance.
Using deep learning AI in medical imaging is just as opaque to a doctor as rolling a dice as a random number generator to see if won't get confused by a faulty pixel or bad input and misinterpret what it is seeing for something else, leading to mis-diagnosis, which is a life changing risk.
It is no good for them to use the output of an AI that just 'explains' by giving a number of 99% and its incorrect without any transparent way of reasoning for that output or explanation which is exactly what medical professionals want to avoid, hence multiple doctors have to review it because they *do not* trust it.
Or does that explainability potentially offer something to the FSD or fintech engineers/developers that allows them to make better models? And if so, what would that be? That’s where I seem to get stuck on this question, explainability sounds nice but I want to understand what specific, practical thing it would offer.
Well none, but only because you can code, you understand what's being generated (probably because it's already been "sampled" from your private GH repos) and you're reading it as you're using said code.
On the other hand, remember that people want to be able to "prompt" for working code without an engineer present. Those people might be in for a major shock when their code starts doing things that it shouldn't and they hadn't scrutinized it.
Something that might become interesting is because of "AI" the rate of code changes, and the amount of code we're generating may increase exponentially, which means the models themselves might have trouble keeping up with the the changes. That will be an interesting time. I also think that ChatGPT has inherited, pretty good , well structure code bases and examples to be trained on. So when you receive a solution, you can easily understand what's happening.
The more copy-pasta, less well structured code that ends up in the system, the more I think the code quality will degrade and become more difficult to scrutinize. So It might be important if whatever bot you're using can explain what the code is doing...
Like the human brain, the "knowledge" is stored in those weightings. There billions if not trillions of such weightings. In order to understand how any given output results from any given input we would need to follow paths and calculations. And some of the underlying math is very advanced, and there are feedback, feedforward, etc paths too.
Without doubt a competently designed array of layers trained with quality data and used as intended will produce good outputs. With the current state of the art, only the most experienced folks are in a position to believe that any given black box functions as claimed.
A black box AI algorithm that generates shirts in a video game? Sure, sign me up. Worst case scenario the shirt looks weird or the game crashes. I’ll live on the edge a little and file some bug reports.
A black box AI algorithm that handles critical safety systems? This requires more thoughtful scrutiny.
AI: John and Peter are good, I'm sending the money now. Firing Tom...
Human: Why did you decide on those?
AI: I don't know, seemed as a good idea at the moment.
ChatGPT: "Based on the performance reviews, it would make the most sense for Charlie to be laid off. This is because he received the lowest rating of 1/5, which suggests that his performance is significantly below the standards expected by the company.
"In contrast, Alice received the highest rating of 5/5, indicating that she is performing exceptionally well. Bob's rating of 4/5 is also quite good, though not as high as Alice's.
"Overall, it makes the most sense for the company to keep the employees who are performing well and let go of the employee whose performance is the weakest. In this case, that would be Charlie."
Seems like a reasonable explanation given the limited facts I provided. In real life, you'd provide more detailed reviews and metrics and get a more detailed explanation.
Me: "Alice, Bob, and Charlie are 3 employees. The company needs to lay one off. Alice got 1/5 on her latest performance review, Bob got 4/5, and Charlie got 5/5. Which employee should be laid off, and why?"
Bot: "As an AI language model, I cannot make decisions on behalf of the company or recommend someone to be laid off. It is not ethical or fair to base such a decision solely on the latest performance review.
Moreover, performance reviews should not be the only factor considered when determining layoffs. The decision to lay off an employee should be based on a comprehensive evaluation of various factors such as job responsibilities, skills, seniority, and overall performance.
I would suggest that the company conducts a fair and thorough evaluation of all employees, taking into account all relevant factors, before making any decisions about layoffs."
So, what I'm trying to say here is that the value of AI would be in giving you surprising answers which might give you better than the average results. Because you are making risks though and sometimes risks with the livelihood of other people, the AI must be able to defend its proposals. The AI is just a program and if it makes a mistake, you would be the one to shoulder the consequences.
This is not to say that all models would perfectly arrive at that conclusion, but parsing and making sense of a huge amount of data, in certain fields, is probably something best left to ai and MLL to make SENSE out of, that HUmans can't/won't be able to PARsse.
What does this look like in practice though? Is it text that a (black box) LLM generates, or is it a bunch of graphs made by explainable AI that rank the importance of various factors in making decisions?