Parametric Reshaping of Portrait Images for Weight-change

Faces are essential to make a first impression, consciously or unconsciously. Facial appearance is also vital for communication. Beautiful faces are pleasurable to look upon.¹ Since facial shape is an important determinant of beauty, it can be desirable to modify a face to be fatter or thinner in order to be more attractive. To accomplish this, a facial weight-change simulator is needed to measure model growth and shape modification. Potential applications of this simulator are not limited to the beauty and medical industries. It also plays an important role in digital entertainment, and film and television production.

Compared to other images, processing of facial images is particularly delicate. People are also relatively good at determining the smallest differences in the appearance of a face. Photo retouching is able to present convincing adjusted faces while maintaining the natural appearance of the face (see Figure 1). However, this time-consuming work must generally be performed by a

Figure 1. Our parametric facial reshaping method automatically simulates the weight-change of a 2D portrait image and generates a fatter or thinner face as intended. (Middle) is the original input image of Albert Einstein; (left) is the result of weight-change degree -2, which indicates losing weight by 2 degrees; (right) is the result of weight-change degree +2, which implies gaining weight by 2 degrees.

skilled, talented retouching artist. Since retouching is experience-based, the result relies heavily on the users’ preference and effort. The process is also not parametric, which makes it especially difficult to control the degree of weight change.

The most related work to ours is proposed by Danino and colleagues who proposed a parametric 2D facial weight-change simulator based on 2D empirically knowledge.² This method can gener-

Figure 2. Comparison results. (a) is the input image. (b) and (d) are the results of Danino and colleagues’ method,² while (c) and (e) are our results.

ate realistic results when the input face is frontal with neutral facial expressions. However, this method does not use the semantic information of the underlying face model, and the background is simply warped without considering the contents of the image. As a result, it may introduce obvious artifacts when the weight-change is large (see Figure 2). Another related work is intro-duced by Zhou and colleagues³ who proposed an image retouching technique for realistic reshaping of human bodies in a single image. This model-based approach can create desired reshaping effects by changing the degree of reshaping which characterizes a small set of semantic attributes. However, it cannot deal with facial reshaping directly. Moreover, it relies on a large 3D whole-body morphable model which may limit its application.

Inspired by the work of Zhou and colleagues,³ we present an image-based facial reshaping method using a linear regression equation. The weight change deformation of a face is parameterized by adjusting BMI (Body Mass Index) values.⁴ We first reconstruct a 3D face model from the input 2D image using a morphable model,⁵ and label the feature points on the 3D face model. Then, we calculate the deformed positions of feature points according to the weight-change degree related to BMI. After that, we generate the deformed 3D face model using the Laplacian deformation method, and project it into 2D image as the deformed face region. With the help of content-aware image retargeting approach by Guo and colleagues,⁶ we finally blend the deformed face region and the background to obtain the reshaped 2D image.

The contributions of this work are: (i) a novel geometric weight-change simulator is presented, which is automatic, fast, and robust; (ii) parametric deformation of the face caused by varying BMI is based on a reliable face tissue depth database, which leads to a reshaped face in compliance with life experience and the repeatable reshaping process.

Related Work

Weight-change Simulator

Few approaches to weight-change simulation have been proposed during the past decades. It first appeared in the innovative work of Blanz and Vetter.⁵ The morphable 3D face model was built on hundreds of 3D face scans. Certain features, including weight, were manually labeled and mapped to the parameter space. Thus, weight-change simulation could be achieved by adjusting weight parameters. However, the simulated result is greatly affected by the constraints of the database. If the reshaping parameter is beyond the scope of the database, the reshaped face is probably unsatisfactory. Moreover, the hair part of the image is particularly problematic. Danino and colleagues presented a facial weight-change simulator for 2D images.² The face region is divided into regions characterized by different weight-change patterns. Its overall process is fast and robust, and the results are clear, sharp, and realistic. Nevertheless, the transformation between the original part and modified face parts is empirically defined without considering the semantic information of the underlying face model. In addition, the input images are limited to frontal face images with neutral facial expressions, and the involved warping method is not content-aware.

3D Face Reconstruction

Lots of facial reconstruction methods based on a single image exist. In the exceptional work of Blanz and Vetter,⁵ a morphable face model was matched to a given 2D image by optimizing the parameters for the similarity between the 2D rendering of the morphable model and the original 2D image. Similar to the morphable head model, Chai and colleagues computed around 100 principal components for a collected head model database and fitted a 3D head model to the input image.⁷ After that, a plausible high-resolution strand-based 3D hair model was developed for portrait manipulations, such as portrait pop-ups. Compared to previous 3D facial databases, FaceWarehouse by Cao and colleagues provided a much richer matching collection of expressions which can depict most human facial actions.⁸ Different from these approaches, we are interested in facial reshaping based on a face image.

In 3D craniofacial reconstruction, Greef and colleagues conducted a large-scale study on how facial soft tissue thickness changes according to sex, age, and weight.⁴ They studied 967 Caucasian subjects of both sexes, and varying ages and BMIs, and measured their facial soft tissue thickness on 52 facial feature points. For each factor and for both sexes separately, a multiple linear regression of thickness versus age and BMI was calculated. Our weight-change simulation is inspired by their regression equations.

Image Resizing and Retargeting

Many content-aware image retargeting techniques have recently been proposed. Following the insightful survey conducted by Shamir and Sorkine,⁹ the approaches fall into two categories: discrete and continuous. In discrete methods, seam carving and cropping are adopted to resize the input image. Continuous approaches optimize mapping using constraints, leading to content-aware resizing. Similar to body-aware image warping by Zhou and colleagues,³ we embed the input image into a 2D triangular mesh, which is used to drive image warping to guarantee coherent resizing effects across the background. An approach to image retargeting employing mesh parametrization is proposed by Guo and colleagues,⁶ which achieves the goals of emphasizing the important while retaining the surrounding context with minimal visual distortion. The preservation of salient object and image structure are maintained by optimizing constrained energy.

Algorithm

We divide a portrait image into two regions: face region and the remaining region. For simplicity, we call the remaining region as the background region in our paper. A reshaping algorithm of a portrait image requires several steps. Figure 3 illustrates the outline of our algorithm. A 3D face model is first reconstructed using the method developed by Blanz and Vetter.⁵ Based on forensic research results and the weight-change degree assigned by the user, deformed facial point positions are set (see 3D Face Reshaping section). Laplacian transformation is conducted afterwards (see 3D face Deformation section). Since only changing the face region is likely to introduce noticeable distortion to the background, a retargeting method is adopted (see Image Retargeting section).

Figure 3. Algorithm overview. Derived from input image (a), a 3D face model is calculated (b). Based on forensic research results and user-specified weight-change degree +3, Laplacian deformation is performed (c). Afterwards, a feature-relevant control mesh (d) is built on the original image. The target mesh (e) is produced by solving a mesh parametrization problem which preserves deformed face features with minimal visual distortion to the background. Standard texture mapping is finally used to render the target image, as shown in (f).

3D Face Reshaping

Our face reshaping algorithm is inspired by forensic research results by Greef and colleagues.⁴ This study was focused on how sex, BMI and age influence the depths of facial soft tissue. The population in their research consisted of 457 males and 510 females of varying ages and BMIs. They selected 52 feature points where 10 points located on the midline and 21 points located bilaterally. The selection of these feature points was based on the ability to reliably locate them on the face. A multiple linear regression of soft tissue thicknesses versus BMI and age was calculated for male and female separately, as tabulated in Table 1.

Our work differs from that of Greef and colleagues’⁴ in that we add two extra control points (see Figure 4 and points 53 and 54 in Table 1) and set the limit for weight-change degrees to make them more suitable for our framework. Without feature points 53 and 54, the deformed 3D faces are likely to introduce artifacts around the pterion. The linear regression equation can be expressed as follows:

$$\begin{array}{*{20}{c}}\begin{array}{c}{\bf{Y}} = {{\bf{b}}_{\bf{0}}} + {{\bf{b}}_1} \times age{\rm{ + }}{{\bf{b}}_2} \times BMI,\\{{\bf{b}}_{\bf{0}}} = (b_0^1,b_0^2, \ldots ,b_0^{54}),\\{{\bf{b}}_1} = (b_1^1,b_1^2, \ldots ,b_1^{54}),\\{{\bf{b}}_2} = (b_2^1,b_2^2, \ldots ,b_2^{54}),\end{array}&{(1)}\end{array}$$

where vector Y represents the tissue depths of the 54 feature points, BMI represents the body mass index, and b₀, b₁, b₂ represent regression coefficients, respectively.

Moreover, there should be a limit for losing weight. Even someone is emaciated, the depths of soft tissue are still above 0. Therefore, we define the limit of the weight-change degree for the i th point Ti as:

$$\begin{array}{*{20}{c}}{{T_i} = - \frac{1}{{b_2^i}}(b_0^i + b_1^i \times age + b_2^i \times BMI),i = 1,2, \ldots ,54.}&{(2}\end{array})$$

For a particular input image, the age of a person remains unchanged. Therefore, with varying BMIs, the updated feature point positions are only influenced by b₂. We assume that the variation of facial tissue depth is along the feature point normal direction:

$${{\bf{S'}}_i}\begin{array}{*{20}{c}}{ = {{\bf{S}}_i} + \frac{{db_2^i{{\bf{N}}_i}}}{{100}},i = 1,2, \ldots ,54,}&{(3}\end{array})$$

where ${{\bf{S}}_i}$ is the i th feature point position before deformation, ${{\bf{S'}}_i}$ is the deformed i th feature point position, d is the weight-change degree, and ${{\bf{N}}_i}$ is the corresponding normal of the i th point.

Figure 4. Illustration of feature points. The total number of feature points is 54, with 10 located on the midline and 22 located bilaterally.

3D Face Deformation

The 3D face model for the input portrait image is reconstructed using the method proposed by Blanz and Vetter.⁵ They collected 200 head structure data using laser scans and exploited the statistics of the dataset to derive a morphable model and a parametric description of faces. Then, a fitting algorithm is developed to match the morphable model to the input 2D face image under shape and texture constraints. After that, a 3D face model conforms to the 2D face image is reconstructed. As a preparation to our algorithm, one of the generated models needs to be labeled with feature points manually. Since the topology of the morphable model mesh remains the same, we can use the pointwise correspondence to locate the feature points on other face models automatically.

After obtaining the deformed feature point positions in the 3D Face Reshaping section, various methods are capable of calculating the displacements of the non-feature points. Noh and colleagues proposed to use Radial Basis Functions to solve this problem.¹⁰ A human face is full of abundant geometric details, and human perception is extremely sensitive to facial distortion. Therefore, we employ a Laplacian deformation method similar to that employed by Liao and colleagues,¹ which is based on the differential surface representation proposed by Sorkin and colleagues.¹¹ By utilizing Laplacian deformation, geometric details are preserved as optimally as possible.

The 54 feature points are assigned as handles, which are moved to new positions ${{\bf{S'}}_i}$ . The updated positions of feature points are calculated in 3D Face Reshaping section. A better result is obtained if the handle constraints are satisfied in a least square sense. With $c_i^x$ , $c_i^y$ , and $c_i^z$ representing the x,y,z coordinates of the new position of the i th feature point respectively, the 54 handle constraints for x -coordinates are:

$$\begin{array}{*{20}{c}}{{x_i} = c_i^x,i \in 1,2, \ldots ,54}&{(4)}\end{array}$$

Thus, all deformed face point positions ${\bf{\tilde x}}$ are obtained by solving the following quadratic minimization problem:

$$\begin{array}{*{20}{c}}{{\bf{\tilde x}} = \arg \mathop {\min }\limits_{\bf{x}} ({{\left\| {L{\bf{x}} - {\delta _x}} \right\|}^2} + \sum\limits_{i = 1}^{54} {{{\left| {{x_i} - c_i^x} \right|}^2}} ),}&{(5)}\end{array}$$

where matrix L is the topological Laplacian of the face mesh, x is the vector of the x-coordinate of all vertices, and δ is the Laplacian coordinate matrix. The y and z coordinates are calculated in the same way.

The 3D face deformation results are shown in Figure 5. The negative weight-change degree indicates the decrease of BMI, which means losing some weight. On the contrary, positive weight-change degree represents the increase of BMI.

Figure 5. 3D face deformation results. The image on the left is the original image. The following images are the 3D face deformation results of weight-change degree -4, -2, 0, +2, and +4, respectively.

Image Retargeting

Directly projecting a reshaped 3D face model into a 2D image will introduce visual artifacts. A content-aware image warping method is employed. Our method is based on the work of Guo and colleagues,⁶ which avoids the distorting the salient object and retains the surrounding background with slight distortion. In their approach, a feature consistent mesh is generated using a constrained Delaunay triangulation algorithm according to the feature points extracted from the 2D input image. Several constraints, including boundary, saliency, and structure, are defined to avoid distorting salient objects in the optimization process for retargeting. After a stretch-based mesh parametrization process, the homomorphous target mesh is calculated, and the resulting image is rendered using texture mapping.

Background Region

The control mesh should be consistent with image structure and retain uniformity of point density. The boundary of the input image is discretized first, and all of the points are set as control points. For the background part, the Canny operator is employed, and other control points are detected. Some additional points are added to keep the points well-distributed. As shown in Figure 6 (a), the blue points represent the control points in the background.

Figure 6. Control mesh comparison before and after 3D face deformation. The red dots in (a) are the control points on the contour profile as hard constraints, the green dots on the face are regarded as hard constraints, and the blue points on the background are set as soft constraints. The four points on the corners of the picture are set as fixed points. (b) is the hard constrained mesh superimposed on the source image. (c) is the comparison of deformed mesh superimposed on the source image. Since the weight-change degree is +3, the face contour expands, and the background needs to be compressed.

Face Region

The control points on face regions are selected based on the result of the Canny operator, as well. Once we obtain the 3D face model and the deformed model in 3D Face Deformation section, a pointwise correspondence is set. Consequently, the deformed face region can be achieved easily. The i th point of the original morphable model is projected into the image space and marked as ${P_i}$ . ${P_i}^\prime$ stands for the projected position of the i th point on the deformed model. The control mesh on the face region expands or shrinks with varying weight-change degrees. In Figure 6(c), the deformed constraint mesh is drawn on the source image. With +3 weight-change degree, the constrained mesh over the face expands.

Face Contour

After 3D face deformation, the locations of vertices of 3D faces will change, and also their 2D projections in 2D image. As a result, the control points originally located on the contour profile are likely to shift away from the deformed contour profile after 3D face deformation, which will lead to noticeable artifacts after the retargeting process. Therefore, the control points along the contour profile of the face must be carefully selected. Let Mc be the set of the contour points along the source image, and $P_i^c$ be the i th point in Mc , and $P_0^c,P_1^c, \ldots ,P_n^c$ are in clockwise order along the contour. With a predefined threshold l , the control points are selected by minimizing the following energy function:

min Et + λEd ,(6)

where Et is employed to distribute the control points uniformly along the contour of the face region, and Ed is employed to constrain the shifting of control points from the deformed contour. They are defined as follows:

$${E_t} = \left\{ {\begin{array}{*{20}{c}}\infty &{n = 0.}\\{{{({d_A})}^2}}&{n = 1.}\\{\sum\limits_{P_i^c \in Mc} {({{({d_{arc}}(P_i^c,P_{(i + 1)\% n}^c) - l)}^2})} }&{n > 1.}\end{array}} \right.{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} (7)$$

$${E_d} = \sum\limits_{P_i^c \in Mc} {(d{{(P_i^c;B)}^2} + d{{(P{{_i^c}^\prime };B')}^2})} ,{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} (8)$$

where B stands for the background of the source image, B⁰ is the deformed background, n is the number of points in set Mc , d_A represents the length of the contour along the face region in the source image, λ is the weight factor balancing the influence of distance threshold constraints and location energy, which is set to 10 for our results. $d(x;S)$ is the least distance of point x from set S . That is,

$$d(x;S) = \inf \left\{ {d(x,s)|s \in S} \right\}.{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} (9)$$

${d_{arc}}(P_i^c,P_{i + 1}^c)$ stands for the length of the face contour from $P_i^c$ to $P_{i + 1}^c$ in clockwise order.

Equation 6 is minimized by adding one selected control point in Mc each time, which results in the largest decrease of energy. The selected point in each iteration is located between the adjacent dots with the longest distance along the contour. Thus, the above process in general can be efficiently implemented. The final solution is reached if adding a point does not reduce energy.

Constrained Mesh Parametrization

Based on the control points selected from the background, face region and face contour, the constrained Delaunay triangulation algorithm is utilized to generate a feature-consistent mesh, as shown in Figure 3(d). Using the method proposed by Guo and colleageus,⁶ the homomorphous target mesh is achieved, as shown in Figure 3(e). The background part is rendered using texture mapping, while the face region part is rendered based on the 3D deformed model. Finally, the reshaped image is obtained, as shown in Figure 3(f).

Figure 6 shows the comparison of control mesh before and after 3D face deformation. The control mesh of the face region (see Figure 6(b)) is fattened after 3D face deformation (see Figure 6(c)). In this example, the weight-change degree (Figure 6(a)) is +3 degrees.

Figure 7. Reshaping results. The left column is the weight-change degree of -2, the middle column is the original input image, and the right column is the weight-change degree of +2.

Results and Discussion

We have implemented our algorithm on a desktop PC with Intel I7 4.0 GHz CPU and 32 G memory. The average computation time is about 0.6 seconds for images with dimensions 640×480; 1.2 seconds for dimensions 800×600; and 1.6 seconds for dimensions 1024×768. We tested our method on a variety of facial images with various backgrounds and poses. Figure 7 shows some examples. For each example, the image in the middle is the input portrait, the left and right images are the reshaping results of -2 degrees and +2 degrees, respectively.

Comparisons

One available facial reshaping method is the facial weight-change simulator proposed by Danino and colleagues.² This approach consists of the following steps. First, user marks thirteen landmarks on the portrait along the cheek and two landmarks around the neck. With user specified weight-change degree, the new locations of the landmarks are calculated based on empirically determined coefficients. After that, a thin-plate spline warping is employed to obtain the deformed facial image. To remedy the artifacts in the deformed background, a synthetic background with similar color is used to replace the actual background.

For frontal-view face images with neutral facial expressions and simple backgrounds, this method can produce realistic results. However, it may produce artifacts for non-frontal-view face images because some landmarks are hidden. Moreover, as the landmarks are mostly along the cheek regions, the nonlinear thin-plate spline warping will cause obvious distortions in other face regions (such as cheek regions as shown in Figure 8 (b) and (e). When the weight-change degrees are large, Danino and colleagues’ method² will produce obvious artifacts (see the distortions in Figure 2 (b) and (d)). For facial images with complex backgrounds, this method will generate unnatural distortions because their image warping is not content-aware. Since our approach recovers the 3D face model to simulate the weight-change of face and employs the content-aware image retargeting method, we can produce natural results with various expressions and poses.

Figure 8. Comparison results. (a) and (d) are input images. (b) and (e) are results of Danino and colleagues’ method,² while (c) and (f) are our results. Please note the differences in the regions marked by red boxes.

We also compare our reshaping results with unprocessed camera images. We collected the pictures of some celebrities who have experienced weight change from being underweight to overweight or vice versa. Figures 10 (b-d), (g-i) are camera images and (a), (e), (f), (j) are our reshaping results. These reshaping results share a close similarity with the camera images.

User Study

We have devised a user study to objectively verify the effectiveness of our facial reshaping method by measuring if the subjects can differentiate between our reshaping images and unprocessed camera images among various individuals of both sexes and varying BMIs.

Examples. We generate several reshaping images using our method described in the Algorithm section, which is called ours . We also collect various unprocessed images which contain human faces via the Internet, which is called real . The individuals shown in real have experienced significant changes of weight.

Study details. We recruit 25 subjects for this task. Each subject views 16 pairs of images of the same individual. Subjects are told to choose the most realistic image in the image pair. Two reference unprocessed images are provided in order to give users a more comprehensive impression of the person shown in the image pair. The first part of the user study is called RT . Ten of these pairs contain one real image and one reshaping image of the same person taken from different places. One example is shown in Figure 9 (a-d). (a) is our reshaping result and (b) is an unprocessed image. (c) and (d) are both unprocessed images, which are provided as references. The second part of this user study is called ST . The remaining six pairs contain one real image and the reshaping image on which it is based. One example of this is shown in Figure 9 (e-h). (f) is the original image, and (e) is the reshaping result generated from (f). (g) and (h) are provided as reference images for this pair.

Figure 9: User study examples. (a-d) are the images used in user study RT, while (e-h) are used in user study ST. (a, e) are our reshaping results, and (b, f) are unprocessed images. (c, d, g, h) are unprocessed camera images as well, which are provided as references. In RT, ours (a) are compared with other camera images taken under different circumstance (b). In ST, ours (e) are compared with the source images (f).

Figure 10: Comparison results. (b-d) and (g-i) are camera images. (a), (e), (f), (j) are our reshaping results.

These image pairs are presented in a randomly permuted order, and the placement (left or right side) of real and ours is randomized, as well.

Results . We analyze the user study data of the two cases ( RT,ST ) separately. When the objects are asked to pick which image appeared more realistic in the RT test, 49.6% of the subjects choose ours . By performing a one-sample, two-tailed t -test for these 10 examples, we find out that subjects cannot find significant differences between our results and the real images ( p -value >> 0.05). Therefore, the results of ours are as realistic as real to some extent. Regarding the ST part, fewer subjects chose ours (45.33%). Compared with the source image, subjects are able to distinguish the source image better. However, the t -test result of ST demonstrates that the difference is also not substantially obvious. Through this user study, we can conclude that our method is able to create natural reshaping results.

Limitations

For very large weight-change degrees, our approach may generate artifacts around the cheek region and introduce noticeable distortion to the background. In our current implementation, the neck region of the input image is considered as background. As a result, the artifacts near neck regions may become obvious when the weight-change degrees are large (as shown in Figure 2).

Gaining or losing some weight will influence the appearance of the face. When gaining weight, a person’s facial contours tend to expand, wrinkles seem reduced and, to some extent, a double chin emerges. When losing weight, a person’s facial contours tend to shrink, wrinkles seem increased and, to some extent, a double chin disappears. Our current approach cannot simulate such wrinkle changes and “double chin” changes. As shown in the bottom-right image of Figure 7, wrinkle artifacts around the eye regions may arise when the weight-change degree is large.

Conclusions and Future Work

We have proposed an effective image reshaping system to thin or fatten a face based on user input weight-change degree. After we gain a 3D morphable face model, forensic data are used to parameterize the reshaping process of the 3D model. We rely on the deformed 3D model to reshape the source image. We introduce a novel approach for choosing control points along the profile of the face. The effectiveness of our parametric weight-change reshaping method is proved by examples and user study.

Our system provides a real-time solution to reshaping a camera image by simply setting weight-change degree.

We are currently working on several enhancements to our reshaping system. Although the current system allows reshaping face regions, the neck region should be added to generate more visually pleasing results. In addition, there are more extensions to render the face region with the reconstructed face morphable model, such as relighting. We are also interested in extending our approach to the mobile phone platform.

Acknowledgements

We would like to thank the anonymous reviewers for their valuable suggestions and comments, which help to improve the readability of this paper. Xiaogang Jin is supported by the National Key R&D Program of China (No. 2017YFB1002600), the NSF of China (Nos. 61732015, 61472351), and the Key Research and Development Program of Zhejiang Province (No. 2017C03SA160073). Kun Zhou is supported by the NSF of China (No. 61272305), the National Program for Special Support of Eminent Professionals of China, and Lenovo’s Program for Young Scientists.