Image and Video Editing via Manipulating Intermediate representations

Speaker: Kuldeep Kulkarni

Abstract

Manipulation of natural images for tasks like object insertion, out-painting or creating animations is extremely difficult if we operate purely in the pixel domain. The goal of this talk is to drive home the advantages of manipulating visual data by expressing them in intermediate representations and manipulating them instead of the pixels directly. Specifically, I will focus on two recent works with image out-painting and animating still images as target applications. I will first talk about a semantically-aware novel paradigm to perform image extrapolation that enables the addition of new object instances. Expressing the images in semantic label space allows us to complete the existing objects more effectively as well allows us to add completely new objects that otherwise is very difficult when working in pixel domain. Then I will talk about a method we developed to interactively control the animation of fluid elements that have repeating textures like water, smoke, fire in still images to generate cinemagraphs. To this end, we allow the user to provide any number of arrow directions and their associated speeds along with a mask of the regions the user wants to animate. The user-provided input arrow directions, their corresponding speed values, and the mask are then converted into a dense flow map representing a constant optical flow map. We observe that the constant flow map, obtained using simple exponential operations can closely approximate the plausible motion of elements in the image. We further show that the computed dense optical flow map can be effectively used in conjunction with generative-adversarial network (GAN) to autoregressively generate future frames.

Bio

I am a research scientist in Adobe Research, Bengaluru, India. Before that I did a post-doc stint at Carnegie Mellon University where I worked with Aswin Sankaranarayanan. I received my PhD in Electrical Engineering from Arizona State University under the supervision of Pavan Turaga. Prior to that, I received my undergraduate degree in Electrical Engineering from the National Institute of Technology Karnataka, Surathkal, India in 2009. My current research interests are in the areas of computer vision, specifically image and video synthesis