Burst Restoration and Enhancement using Transformers

Speaker: Subrahmanyam Murala

Abstract

Burst image processing is becoming increasingly popular in recent years. Burst mode, also known as continuous shooting mode is a feature available in iPhones, and Samsung Galaxy Note3 that allows capturing multiple shots continuously in a fraction of a second. However, it is a challenging task since individual burst images undergo multiple degradations and often have mutual misalignments resulting in ghosting and zipper artefacts. Existing burst restoration methods usually do not consider the mutual correlation and non-local contextual information among burst frames, which tends to limit these approaches in challenging cases. Another key challenge lies in the robust up-sampling of burst frames. The existing up-sampling methods cannot effectively utilize the advantages of single-stage and progressive up-sampling strategies with conventional and/or recent up-samplers at the same time. To address these challenges, we come up with a novel transformer-based approach named as Gated Multi-Resolution Transfer Network (GMTNet) to reconstruct a spatially precise high-quality image from a burst of low-quality raw images. GMTNet consists of three modules optimized for burst processing tasks: Multi-scale Burst Feature Alignment (MBFA) for feature denoising and alignment, Transposed-Attention Feature Merging (TAFM) for multi-frame feature aggregation, and Resolution Transfer Feature Up-sampler (RTFU) to up-scale merged features and construct a high-quality output image. In this talk, I will be discussing burst superresolution, burst denoising, and burst low-light enhancement, with more focus on super-resolution application.

Bio

Dr. Subrahmanyam Murala is an Associate Professor in the Department of Electrical Engineering, Indian Institute of Technology Ropar, Rupnagar, Punjab, India. Dr. Murala joined the institute as an Assistant Professor in July, 2014 and received his M.Tech. and Ph.D. from the Department of Electrical Engineering, Indian Institute of Technology Roorkee, India, in 2009 and 2012, respectively. He was a Post-Doctoral Researcher in the Department of Electrical and Computer Engineering at the University of Windsor, Windsor, ON, Canada from July 01, 2012, to June 30, 2014. He is a senior member of IEEE and the recipient of the Faculty Research and Innovation Awards 2019-2020. His major fields of interests are Computer Vision, Medical Image Processing, and Object Detection. At IIT Ropar, Dr. Murala leads the Computer Vision and Pattern Recognition Laboratory (CVPR Lab) which actively focuses on deep generative models, adversarial robustness of deep neural networks, and learning-based applications like Human Action Recognition, Image Enhancement (Haze Removal), Moving Object Segmentation for videos, Image Depth Estimation, Image/Videos Inpainting, Image/Videos Super-resolution, Motion magnification, and Multi-weather Restoration. His research has resulted in many publications in several top conferences and journals including CVPR, WACV, TIP, etc. Besides contributing actively to his research group, Dr. Murala is currently handling several challenging projects in collaboration with renowned industries and the Department of Science and Technology (DST), India. Apart from this, Dr. Murala has been the Conference chair of CVIP-2021, 2022 and the Organizing member of ICVGP-2023. Being Head of the Electrical Engg. Department at IIT Ropar, Dr. Murala has also contributed to the department's making by leading on several fronts including both technical and non-technical. Further information about Dr. Murala and his research group can be found at https://www.iitrpr.ac.in/subbumurala/