Open-source AI tool for animating child and amateur drawings of human figures

Published on April 24, 2023

Meta AI Research, Tencent America, MIT CSAIL, and Carnegie Mellon have released Animated Drawings, a tool for creating animations from hand drawn human-like characters.

In order to achieve its goal, Animated Drawings utilizes a variety of techniques, including object detection to determine a tight bounding box surrounding the figure, image segmentation to distinguish pixels belonging to a human figure from pixels belonging to the background, and pose estimation to identify skeletal joints. In this manner, it is possible to create a digital version of a drawing that can then be animated using computer graphics techniques by applying motion data from preselected clips obtained from human actors.

In order to collect data from creators, the researchers set up a Web-based tool called Animated Drawings Demo after an initial unsuccessful attempt to construct a dataset from synthetic data. In spite of their initial goal of collecting about 10,000 drawings, the overwhelming response from users resulted in the collection of over 3 million drawings, of which approximately 200,000 were incorporated into the final dataset.

Thanks to Meta open-sourcing an implementation of the algorithm described in the paper, the results of the research, which are presented in a paper just accepted for publication in ACM Transactions on Graphics, are now available for experimental use.

Animated Drawings has been published on GitHub and has been tested on macOS Ventura 13.2.1 and Ubuntu 18.04.

To make use of additional drawings, the user must first annotate them, which can be done manually or more easily by using scripts that run a humanoid figure detector and pose estimator. Scripts are based on TorchServe and can be deployed using a Dockerfile provided. Meta notes that the annotation files may require some manual adjustment, for which the container includes a Web-based tool.

In the final step, characters can be animated using a motion clip in BVH format. There are a number of BVH animations provided by Meta, including jumping, waving hello, and others. A tool such as Rokoko can be used to create additional BVH animations from an RGB video.

The Animated Drawings system is only a first attempt at an AI-based drawing-to-animation system, and there are numerous possibilities for improvement. In particular, the researchers recommend improving segmentation by using pre-animation classification of the human figure to improve pose estimation and animation by better handling distortion, lighting effects, and shadows.