We were all once fascinated by the moving portraits in the Harry Potter saga. In the past, several companies have tried to recreate these portraits but haven’t been 100% successful, this was until Chung-Yi Weng, a researcher from the University of Washington was able to develop a system that creates magnificent 3D animations from a still picture.
The new algorithm, called “Photo Wake-Up”, simply allows characters to simply “walk out” of the static image frame – without leaving a hole in the picture behind them. The team used a program called SMPL, developed by Microsoft and the Max-Planck society. The steps followed by this algorithm are:
- Fit the image with a morphable body model;
- Estimate body label map;
- Construct a mesh part by part guided by the morphable model;
- Rig the mesh;
- Estimate mesh skinning weights;
- Rebuild textures;
- Put the subject on an inpainted background image;
- Then “walk, run, jump or sit down”.
The results obtained with this algorithm are extremely impressive. The video below shows how a girl in one of Picasso’s works can come to life:
Moreover, with the help of an AR (Augmented Reality) device, it is possible to bring a photo into real life.
The development of this innovative system was very challenging as it is difficult for a computer to recognise and generate hidden body parts such as crossed arms or legs. Thanks to the 2D mesh, the elements of the body are first formed, then the different parts such as the legs, arms and torso are precisely determined. The mesh of each of these elements is then converted into 3D. It was also important to determine the angle of the head and gaze direction in order to obtain a coherent mesh.
Additionally, the team developed an algorithm that fills the holes in the background by interpolating the missing parts. The character can “come out” of the picture without leaving a black hole behind it, which makes the movement more realistic.
However, this technique is not 100% new, since 2007, some researchers have attempted to recreate the moving portraits of Hogwarts from Harry Potter. However, the results have been far less impressive; the subject wasn’t well extracted, and the character’s textures were simply applied to a generic 3D model that did not correspond to the dimensions of the subject. Weng’s new algorithm adds a new feature: body parts including arms and legs are identified and warped individually in a way that matches the 2D cutout.
Mariem, Consultant, Leyton France
Read more on the same subject: