Big Models, Bad Math: The GenAI Problem In Finance

To overcome the problem of hallucination, Yseop relies on having humans in the loop at every stage. The company’s algorithms and neural networks are co-developed by math wizards, linguistics experts, and AI developers. Their databases consist of data sourced directly from the researchers and businesses being served by the product. And the majority of their offerings are conducted via SaaS and designed to “augment” human professionals — as opposed to replacing them.

• Large Language Models (LLMs) are fundamentally transforming the financial industry, offering unprecedented capabilities in analysis, risk management, and regulatory compliance.
• By providing context to market movements, sentiment analysis powered by LLMs offers valuable insights that can inform investment decisions and strategy formulation.
• Over the last two years, LLM neural networks have been quietly expanding AI’s impact in healthcare, gaming, finance, robotics, and other fields and functions, including enterprise development of software and machine learning.

Get the TNW newsletter

A task of loan default prediction was tested on an open-source transaction dataset and achieved an accuracy of 94.5%. A task of churn rate prediction was tested on a different version of the original Prometeia dataset, and the results were compared with the real annotation of accounts closed in 2022. The prediction was very precise and better than competitors, with an accuracy of 90.8%. As part of their training, today’s LLMs ingest much of the world’s accumulated written information (e.g., Wikipedia, books, news articles). What if these models, once trained, could use all the knowledge that they have absorbed from these sources to produce new written content—and then use that content as additional training data in order to improve themselves? Building these models isn’t easy, and there are a tremendous number of details you need to get right to make them work.

Enterprise and Industry-Specific Use Cases

Tokyo-based Rinna employs LLMs to create chatbots used by millions in Japan, as well as tools to let developers build custom bots and AI-powered characters. LLMs can help enterprises codify intelligence through learned knowledge across multiple domains, says Catanzaro. Doing so helps speed innovation that expands and unlocks the value of AI in ways previously available only on supercomputers. In addition, the industry is likely to use LLMs to replace interim layers of human involvement, not remove humans from the loop entirely. Because of the existence of multiple “maker-checker” layers in most existing finance processes, such partial automation can still have dramatic (such as +50%) efficiency benefits without needing to be 100% reliable.

ChatGPT is limited to the information that is already stored inside of it, captured in its static weights. In other words, we may be well within one order of magnitude of exhausting the world’s entire supply of useful language training data. This can be loosely analogized to a human in conversation who, rather than blurting out the first thing that comes to mind on a topic, searches her memory and reflects on her beliefs before sharing a perspective. The group plans to keep adding large language models to the benchmark and to introduce opportunities for others to provide feedback, in the hopes of creating a community.

Which large language models are best for banks?

Ever since AI first started making headlines in finance, it has been a story of great promise and anticipation—and limited real-world impact. The Next Platform is part of the Situation Publishing family, which includes the enterprise and business technology publication, The Register. While there are a wide variety of LLM tools—and more are launched all the time—OpenAI, Hugging Face, and PyTorch are leaders in the AI sector. An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday. In the nearer term, though, a set of promising innovations offers to at least mitigate LLMs’ factual unreliability. These new methods will play an essential role in preparing LLMs for widespread real-world deployment.

« Our use cases are no different from the use cases that JPMorgan or another big fund management company would have, » Dayalji said. He and his team decided to make their findings public to help others get a sense of what business and finance tasks these models are good at. To test the models’ ability to extract data, the researchers feed them tables and balance sheets and ask them to pull out specific data points.

If you come across an LLM with more than 1 trillion parameters, you can safely assume that it is sparse. This includes Google’s Switch Transformer (1.6 trillion parameters), Google’s GLaM (1.2 trillion parameters) and Meta’s Mixture of Experts model (1.1 trillion parameters). LLMs’ greatest shortcoming is their unreliability, their stubborn tendency to confidently provide inaccurate information. Language models promise to reshape every sector of our economy, but they will never reach their full potential until this problem is addressed. The DeepMind researchers find that Sparrow’s citations are helpful and accurate 78% of the time—suggesting both that this research approach is promising and that the problem of LLM inaccuracy is far from solved. Examples abound of ChatGPT’s “hallucinations” (as these misstatements are referred to).

Could the advent of LLMs change that?

LLMs trained on hundreds of billions of parameters can navigate the obstacles of interacting with machines in a human-like manner. Many NLP applications are built on language representation models (LRM) designed to understand and generate human language. Examples of such models include GPT (Generative Pre-trained Transformer) models, BERT (Bidirectional Encoder Representations from Transformers), and RoBERTa. These models are pre-trained on massive text corpora and can be fine-tuned for specific tasks like text classification and language generation. Most large language models rely on transformer architecture, which is a type of neural network.

As LLMs continue to evolve, they are reshaping how financial institutions operate, make decisions, and serve their clients. If you are a Global 20,000 company and you want to build a large language model that is specifically tuned to your business, the first thing you need is a corpus of your own textual data on which to train that LLM. And the second thing you need to do is probably read a new paper by the techies at Bloomberg, the financial services and media conglomerate co-founded by Michael Bloomberg, who was also famously mayor of New York City for three terms.

While recent advances in AI models have demonstrated exciting new applications for many domains, the complexity and unique terminology of the financial domain warrant a domain-specific model. It’s not unlike other specialized domains, like medicine, which contain vocabulary you don’t see in general-purpose text. A finance-specific model will be able to improve existing financial NLP tasks, such as sentiment analysis, named entity recognition, news classification, and question answering, among others.

They’re increasingly heralded as a revolutionary disrupter of AI, including enterprise applications. As far back as 2018, the industry’s cautious approach to AI adoption was coming in for comment. The second strategy, impact assessment, includes evaluating the LLM’s potential or actual effects on social, economic, environmental, legal and human rights dimensions. Indicators, metrics and specialized auditing tools can be used to quantify and qualify how LLMs are affecting critical areas like diversity, inclusion, transparency and trust. For example, if an LLM disproportionately disadvantages certain demographic groups in its outputs, this shows a negative social impact, indicating unfairness and lack of accountability. GPT-4’s accuracy in predicting earnings changes dropped from 60% to no better than random chance, demonstrating that these models aren’t analyzing financial data meaningfully but simply matching memorized patterns.