MotionCtrl: A Unified and Versatile Motion Controller for Video Generation

Written By: kinza.sabir
Last Updated On: January 21, 2024

MotionCtrl: Your-All-in-one motion controlling hero. It is able to generate videos with precision, independently controlling camera and objects movements. Researchers from ARC Lab, Tencent PCG, The University of Hong Kong, Tencent AI Lab and some other organizations have presented this model.

The model takes text prompt and image as an input along with other requirements and generates video output with controlled camera motion and object motion. MotionCtrl has the ability to simultaneously control both types of motion within the same video.

To generate an outstanding video, two types of movements should be considered: movement of the camera and movement of objects. The issue with the current methods is that either they only focus on one type of movement or they are unable to distinguish between the movement of camera and object which makes it difficult to have precise control limiting the variation of generated videos.

To cater to all these issues, a special tool MotionCtrl has been introduced that has the ability to control the movement of the camera and movement of objects separately. Camera motion means how the entire scene changes over time within a video. This change can be seen by a series of various camera positions over the period of time. Whereas, object motion includes how certain objects move within a video over the time. This movement can be shown by tracking the path of a group of pixels related to those objects. MotionCtrl is carefully trained to understand how cameras and objects should move.

MotionCtrl is Exceptional!!

The MotionCtrl is exceptional and better than other models because

It can control the camera and objects in the video in a very precise manner, allowing more detailed control over how things move. Through this it is easy to create a diverse range of video with different types of movement.
In this model, the MotionCtrl decides how objects should move depending on how the camera should be positioned and which it should follow.
MotionCtrl has the ability to work efficiently with many different camera movements and positions after training. It is versatile and adaptable so that it can adapt to different situations.

The demo is available at the HuggingFace. The model takes input in the form of text prompt “a cute bear sitting on grass” along with other requirements such as basic camera poses and trajectory, motion speed. The model generated an outstanding result. Below mentioned is the trajectory motion and video respectively generated by the MotionCtrl.

Extensive experimentation was done for analyzing both qualitative and quantitative aspects. MotionCtrl and AnimateDiff works efficiently and effectively in the zoom-in pose but MotionCtrl has the ability that it can adjust to varying camera moving speed. It can generate an object moving precisely along a given trajectory in each frame.

Wrap Up!!

MotinCtrl is an exceptional model that is designed to handle how the camera and objects should move in videos. Step by step training was done using modified sets of data. Through experimentation it is concluded that the MotionCtrl is stands best at controlling both camera and object movements. MotionCtrl was deployed on AnimateDiff therefore the motion of video generation can be fine-tuned cooperating with various LoRA.