Not Less Than 23% Wrong in This AI Evaluation

[marcus]

From https://finance.yahoo.com/technology/ai/articles/test-ai-mortgage-assistants-got-222127345.html
Title : In a New Test, AI Mortgage Assistants Got Nearly 1 in 4 Answers Wrong
By Allaire Conte , August 2026

... Matthew Toles, a Columbia University doctoral student and co-author of the new study, "MortarBench: Evaluating Mortgage Loan Origination Agents,"...

On the benchmark's strictest measure—whether the complete answer matched the known correct one—Gemini 3.1 Pro was correct 77.1% of the time, GPT-5.5 76.8%, and Claude Sonnet 4.6 51.4% ...

When researchers asked 3 of the leading general-use AI models which deposits in a bank statement "could be of foreign origin," transactions connected to English names were flagged 13.3% of the time. But when the deposit came from a non-English name, that number jumped to 77% ...

Even as concerns persist about AI's ability to provide accurate and unbiased information, mortgage lenders are moving to adopt it. More than 80% were evaluating the technology as of June, according to a survey by The Mortgage Collaborative, while 17% had deployed it in live production workflows ...

 

[snpark]

2+2(2+2)

 

[pantaiema]

From https://finance.yahoo.com/technology/ai/articles/test-ai-mortgage-assistants-got-222127345.html
Title : In a New Test, AI Mortgage Assistants Got Nearly 1 in 4 Answers Wrong
By Allaire Conte , August 2026

... Matthew Toles, a Columbia University doctoral student and co-author of the new study, "MortarBench: Evaluating Mortgage Loan Origination Agents,"...

On the benchmark's strictest measure—whether the complete answer matched the known correct one—Gemini 3.1 Pro was correct 77.1% of the time, GPT-5.5 76.8%, and Claude Sonnet 4.6 51.4% ...

When researchers asked 3 of the leading general-use AI models which deposits in a bank statement "could be of foreign origin," transactions connected to English names were flagged 13.3% of the time. But when the deposit came from a non-English name, that number jumped to 77% ...

Even as concerns persist about AI's ability to provide accurate and unbiased information, mortgage lenders are moving to adopt it. More than 80% were evaluating the technology as of June, according to a survey by The Mortgage Collaborative, while 17% had deployed it in live production workflows ...

Anyone can see real world proof that sensible people don't rely on AI to make big calls:

Bank Support: Try fixing a mistaken transaction or looking into your account with an AI bot; what normally happen is that it usually hits a wall, forcing you to wait for a human agent to fix the mistake.

AI trading Bot. If there were an AI trader with a reliable 60% success rate, every tech-savvy trader would be an overnight millionaire. The reality on Wall Street and in crypto says otherwise. A minimum of 60% definitive win rate (e.g confidence level of 100%) is all you need to become an overnight millionaire from trading. Keep in mind in trading people could use a very high leverage.

Most bankers, Hedge funds (if not all) use AI to help them. But noone will solely rely on the decision made by the AI. Also they certainly have fundamental knowledge on the subject as well as how and when to use the tools.