Application of Foreground and Background separation with Deep Learning

Foreground and background separation had always been a huge problem before the onset of object detection based neural networks. Techniques from image processing like color based segmentation, depth based segmentation and other traditional techniques had to be used. Today with the help of , off the shelf deep learning models, its become very easy to mask region of interest and separate foreground from background.

Applications :-

1. Portrait Mode with Deep Learning
2. Background blur on video calls while working from home

Portrait Mode with Deep Learning

Portrait mode became famous when images were captured from DSLR cameras. High end DSLR cameras have an option to reduce the focal length on the camera to make sure the camera can capture images with major focus on the small distance objects in a scene and automatically blurring the background. Such images became very famous as it automatically sharpens the subject/object infront of the camera.

This trend continued by adding two cameras to smart phones, to capture depth information. Based on depth threshold, segmenting region of interest was very easy. Portrait mode started trending on mobile phones because of dual cameras that can give depth information.

With the popularity of object detection based networks, portrait mode was possible without high end DSLR or dual cameras.

Lets learn how portrait mode works without dual camera or high end DSLR.

1. First you need a good camera and subject. Capture an image which is well focused without any distortion or evident noise.

2. Get the image and pass it through an object detection network and get the output foreground boundary from the region proposal network.

3. Separate the foreground and background as two separate images.

4. Blur the background and sharpen the foreground.

In Summary :-

Original Image, Person Detection and background blur, forground and background separation, Final Portrait Mode (Image by Author)

This way, one can create portrait mode with a single low cost camera.

The code is availbale on github.

https://github.com/sumedhvdatar/PortraitSegmentation

2. Background blur on video calls

With COVID majority of the people around the world worked from home. Working from home involved having multiple video calls. Many people did not like to show the surrounding or background where they work and wanted to hide it. Teams sharing and collaborative platforms like zoom and microsoft teams released a feature where in the user who has turned on his camera has an option to blur the background or maybe add a different background. The idea is the same as explained above.

It clearly shows how powerful neural networks can be. With the help of a simple frontal laptop camera, the same technique can be applied and its helping so many people around the world to turn on their camera and express what they want to and at the same time feel very comfortable.

Conclusion

Foreground and background subtraction was just one of the application of deep learning, similarly deep learning is having its applications in so many different avenues. All an individual has to do is bring in some creativity and apply deep learning at the right place and that creates a lasting impact on the lives of people.

Sources :

Alpha Blending using OpenCV (C++ / Python)

Automatic Portrait Segmentation for Image Stylization Xiaoyong Shen1† , Aaron Hertzmann2 , Jiaya Jia1 , Sylvain Paris2 , Brian Price2 , Eli Shechtman2 and Ian Sachs2 1The Chinese University of Hong Kong 2Adobe Research