Google DeepMinds Gemma 3N Unleashes New 8B Multimodal AI Powerhouse (E4B + E2B)

➡ Google DeepMind’s new model, Gemma 3n, brings advanced AI directly to mobile devices without needing an internet connection. It uses a unique architecture that allows it to contain smaller, functional sub-models, which can be customized to suit different devices. It also includes new tools for managing and benchmarking these configurations. Additionally, it introduces features that improve memory efficiency, speed up input processing, and enhance audio and video analysis, making it possible to run powerful AI models on devices with limited hardware.

A major shift in on-device artificial intelligence has just arrived with the introduction of Google DeepMind’s Gemma 3n, the newest model that’s bringing the multimodal performance of last year’s most advanced cloud-based systems directly onto mobile and edge devices, leading to a fundamental transformation of how developers and consumers can now use AI without even needing an internet connection or server backend. The secret is in Gemma 3n’s newly engineered mobile-first architecture that required rethinking every layer of the model. The foundation is Matformer, a Matrioshka transformer approach which enables a single large model to efficiently contain smaller, independently functional sub-models.

The analogy to Matrioshka dolls is apt. Within the primary 8 billion parameter model called the E4B exists yet another fully optimized 5 billion parameter sub-model called the E2B. Both are trained in tandem, providing real flexibility and elastic inference for the wide array of hardware constraints present in today’s devices. In fact, developers can download either the full capability E4B or the faster and lighter E2B sub-model. On top of this, it uses an innovative mix-and-match system to take customization even further. By adjusting the feed-forward network’s hidden dimensions and skipping layers as needed, developers can even create bespoke model sizes tailored to their specific device’s capabilities.

Plus, a new tool called Matformer Lab works alongside Gemma 3n to manage and benchmark these custom configurations using industry-standard evaluations like MMLU, giving users clear insight into performance and efficiency trade-offs of each setup. Furthermore, the Matformer design paves the way for true elastic execution, the promise that, in the near future, a single deployed model could seamlessly select between E2B and E4B inference pathways on the fly, dynamically optimizing for performance or memory usage according to real-time device conditions. But another key advancement in Gemma 3n is its use of per-layer embeddings, which is a technique that dramatically enhances memory efficiency without sacrificing quality.

By allocating the embeddings associated with each layer to the device’s central processing unit, while keeping only the core transformer weights in the more limited accelerator memory, Gemma 3n makes it possible to run powerful models on devices with tight hardware constraints. But for applications requiring understanding of long input sequences like streaming audio and video, Gemma 3n is also introducing KV cache sharing. This feature optimizes the initial pre-fill stage of input processing by sharing the middle layer’s attention keys and values across top layers, resulting in a notable two-fold improvement in pre-fill speed compared to the previous generation.

As a result, the model can now analyze and respond to lengthy prompts and multimodal data streams with much lower latency. And for audio, Gemma 3n integrates a cutting-edge encoder derived from the universal speech model. With the ability to convert every 160-millisecond segment of audio into a token, the model delivers granular sound context for both speech recognition and translation. At launch, up to 30 seconds of audio can be processed, with future updates expected to expand this window for real-time, long-form streaming use cases. This allows developers to build on-device applications for high-quality transcription and translation tasks, with especially strong results seen in English, Spanish, French, Italian, and Portuguese language pairs.

Plus, when combined with techniques such as chain of thought prompting, translation accuracy is even further improved. And to top off these audio capabilities there’s MobileNet V5 300M, Gemma 3n’s new vision encoder. This state-of-the-art module is designed to perform efficiently across a range of input resolutions for real-time image and video analysis, even on resource-constrained hardware. And this architecture is further enhanced through innovations such as universal inverted bottlenecks, mobile MQA, and a multi-scale fusion vision-language model adapter, achieving remarkable throughput of up to 60 frames per second on a Google Pixel device. Incredibly, MobileNet V5 300M is 10 times larger than its predecessor, yet due to advanced distillation and architectural refinement, it requires 46% fewer parameters and has a four-fold smaller memory footprint, all while significantly outpacing prior benchmarks with both accuracy and speed in vision-language tasks.

All the while, the introduction of RoboBrain 2.0 is being described as the most advanced open-source embodied brain model to date. In fact, when compared to RoboBrain 1.0, the new version is engineered to deliver substantial improvements in multi-agent task planning, spatial reasoning skills, and robust closed-loop execution. But the core of RoboBrain 2.0’s promise is its performance on standardized benchmarks. In head-to-head comparisons, its 32 billion parameter version has achieved state-of-the-art results across four widely recognized embodied intelligence benchmarks. On top of this, the model consistently outperformed leading open-source alternatives and, notably, even exceeded the results from several proprietary closed-source models, further highlighting its technical edge, and its architecture is a key reason for these advancements, with the model being designed to handle a range of complex visual and language inputs.

On the visual side, it supports both multi-image and long video streams at high resolutions, which are processed through a dedicated vision encoder and a multi-layer perceptron projector. Meanwhile, complex task instructions and structured scene graphs are managed on the language side. All text-based inputs are tokenized and fed into a large language model decoder, which is capable of long chain-of-thought reasoning. This allows RoboBrain 2.0 to output structured multi-step plans, spatial relationships, and even both relative and absolute coordinates for precise operation. Furthermore, RoboBrain 2.0’s task capabilities have been significantly expanded. The model can now perform interactive reasoning with long-term planning and closed-loop feedback, meaning it can dynamically adjust its actions based on real-time results.

Its spatial perception is robust enough to allow point and bounding box predictions from highly detailed instructions, while its temporal perception allows it to estimate future trajectories. Plus, its real-time structured memory enables on-the-fly scene reasoning, constructing and updating internal models of its environment as tasks progress. Examples show RoboBrain 2.0’s referential abilities in color recognition and its stability during continuous operation. Other clips highlight rapid scene adaptation, such as judging object proximity, determining orientation, and estimating distances in changing environments. The model also demonstrates real-time voice interruption adjustment, enabling it to handle spatial relationship recognition, multi-step reasoning, and responsive interaction when task parameters shift suddenly.

On top of these, RoboBrain 2.0 showcases specialized abilities like part-level orientation-related referencing, functionality-based object identification, and multi-step spatial reasoning that closely mimics human problem solving. Its structured arrangement feature allows it to understand and build patterned spatial relationships among objects, while mobile manipulation demonstrations reveal its ability to control humanoid robots for everything from tabletop tasks to indoor navigation. Other tested features include advanced object attribute recognition, differentiating between sizes and shapes, and object affordance localization, which involves identifying the most graspable part of an item, such as a mug’s handle, alongside color and distance-based object localization.

Spatial reasoning skills encompass distance perception, position awareness, and three-dimensional free space localization. [tr:trw].