Google reveals PaLM 2, its next generation large language model

Early this week, we reported that Google is expected to announce PaLM 2, its next generation large language model at Google I/O. With no surprise, Google announced PaLM 2 on stage. PaLM 2 is Google’s next generation large language model that can perform advanced reasoning tasks, code and math, classification and question answering, translation and multilingual proficiency, and natural language generation.

According to Google’s own testing, PaLM 2 performs better than PaLM and other models developed by Google.

Google claims that PaLM 2 was evaluated rigorously for its potential harms and biases, capabilities and downstream uses in research and in-product applications. Google is already using PaLM 2 as the foundational model for Med-PaLM 2 and Sec-PaLM, and also using it for generative AI features and tools at Google, like Bard and the PaLM API. At Google I/O, Google announced over 25 new products and features powered by PaLM 2.

Here’s how PaLM 2 was built:

• Use of compute-optimal scaling: The basic idea of compute-optimal scaling is to scale the model size and the training dataset size in proportion to each other. This new technique makes PaLM 2 smaller than PaLM, but more efficient with overall better performance, including faster inference, fewer parameters to serve, and a lower serving cost.
• Improved dataset mixture: Previous LLMs, like PaLM, used pre-training datasets that were mostly English-only text. PaLM 2 improves on its corpus with a more multilingual and diverse pre-training mixture, which includes hundreds of human and programming languages, mathematical equations, scientific papers, and web pages.
• Updated model architecture and objective: PaLM 2 has an improved architecture and was trained on a variety of different tasks, all of which helps PaLM 2 learn different aspects of language.

Also, PaLM 2 is available in four sizes from smallest to largest: Gecko, Otter, Bison and Unicorn. Gecko can work on mobile devices and is fast enough for great interactive AI applications on-device without internet connectivity.

Google also revealed that Gemini will be its next model created from the ground up to be multimodal, highly efficient at tool and API integrations, and built to enable future innovations, like memory and planning. Google is still training Gemini. Internal testing at Google suggests that Gemini will offer multimodal capabilities never before seen in prior models.