Meta will begin training a GPT-4 level LLM in 2024.
Citing people familiar with the matter, the WSJ reported that Meta is working on a new AI system that it intends to be as advanced as the most powerful model from OpenAI (GPT-4 as of now). Link—Paywalled.
Likely to be rolled out in 2024, Meta’s new model will be several times more powerful than the current best from Meta Platforms, Llama 2. Llama 2 is (debatably) open-source and is distributed through Microsoft Azure. Currently, Llama 2-based tools compete with Google Bard and OpenAI ChatGPT.
The training is reportedly yet to begin.
Notably, GPT-4 will most probably not be the most powerful model from OpenAI in 2024.
The new model will focus on text analysis and other services that are currently offered by GPT-4-based tools such as ChatGPT Plus. The new model will be trained in-house by Meta without using Microsoft’s deployment solutions. This is one of the main reasons why Meta has also placed a large order for Nvidia’s H100 chips—Central to train LLMs of today.
The news comes when Apple is also rumored to be working on its Ajax AI model (The Verge: Apple is reportedly spending ‘millions of dollars a day’ training AI). With this, the competition increases further with each of the major tech companies including Microsoft, Apple, Meta, and Google most likely going toe-to-toe in 2024 with their own generative AI tools such as coding assistants and conversation bots.
The WSJ reports that Meta CEO Mark Zuckerberg is pushing for this upcoming model to be open-sourced (freely available for companies).
If Meta truly wishes to compete with OpenAI’s GPT-4 with this new LLM, then open sourcing it might be a bad idea. Also, it’s very likely that such a model will have significant processing requirements, making it impossible to run on user machines, for example, if they train it on 70B parameters. Meta is poised to make a remarkable splash as it already has huge data centers (while it’s developing more, according to the WSJ report for this purpose) as well as the most personal data about people in the world, arguably.
