Google recently announced ways to blur and replace the background in Google Meet for better focus on the person rather than the surrounding. The new features are powered by cutting-edge web machine learning (ML) technologies built with MediaPipe that work directly in the browser, without any extra steps like installing additional software. One of the main motives for developing these features was to provide real-time, in-browser performance on almost all modern devices. It accomplishes this by combining efficient on-device ML models, WebGL-based rendering, and web-based ML inference via XNNPACK and TFLite.
Overview of Web ML Solution
The solution processes each video frame by segmenting the user from their background and computes a low-resolution mask. This mask is further refined to align it with the image boundaries. It is then used to produce the output video via WebGL2, with either blurred or replaced background.
Fig: WebML Pipeline: All compute-heavy operations are implemented in C++/OpenGL and run within the browser via WebAssembly.
Thus, Google Meet introduced a new in-browser ML solution for blurring and replacing the background. With this, ML models and OpenGL shaders can run efficiently on the web. The developed features achieve real-time performance with low power consumption, even on low-power devices.
Credit: Google News