Meta, formerly known as Facebook, has unveiled its Open Pretrained Transformer (OPT-175B), an expansive language AI model boasting over 100 billion parameters. This remarkable development represents the tech giant's effort to democratize access to state-of-the-art AI research, as outlined in their recent blog post.
Under a non-commercial license, the model will be made available primarily for research purposes. Access will be granted to academic researchers, as well as government, civil society, and industry research laboratories worldwide. However, the extent of access for researchers remains uncertain. Those interested in utilizing the model can simply fill out a request form.
Understanding Large Language Models
Large language models are advanced natural language processing systems, trained on vast amounts of text to generate creative and coherent content in various formats. Such models can produce news articles, legal summaries, movie scripts, and even serve as customer service chatbots.
OpenAI's GPT-3 is one of the industry's leading large language models, containing over 175 billion parameters for both personal and commercial use. Similarly, Meta's OPT-175B provides not only a large-scale language model, but also an accompanying codebase, and comprehensive documentation detailing the training process. Meta has also launched a series of smaller-scale baseline models with fewer parameters.
Addressing the Carbon Footprint of Deep Learning
Most innovative AI research is driven by major tech corporations like Google, Meta, Microsoft, and Nvidia. Training and operating large AI models requires an enormous amount of computational power and energy, resulting in significant expenses and a potentially large carbon footprint. A 2019 study by Roy Schwartz and Jesse Dodge revealed that deep learning computations doubled every few months, leading to an estimated 300,000x increase from 2012 to 2018 – and a significant environmental impact.
Meta claims to have addressed the carbon problem by reducing the OPT-175B's carbon footprint to one-seventh of OpenAI's GPT-3. The company says it trained the model on 992 Nvidia A100 GPUs (80 gigabytes each), with a total carbon-emissions footprint of only 75 tons, compared to GPT-3's estimated 500 tons. OpenAI has yet to comment on these claims.
Future Implications
Expect to see a surge in deep learning research innovation with Meta's recent announcement. However, alongside the technological advancements, ethical questions surrounding the responsible use of AI must be addressed globally. Meta shares the hope that the AI community – encompassing academic researchers, civil society, policymakers, and industry professionals – will join together to find answers. As more powerful tools like OPT-175B emerge, the democratization of access to AI innovation presents an opportunity for platforms like AppMaster to integrate such advances into their own no-code solutions, further empowering users to develop comprehensive applications for various industries.
