Paint3D: Generation of High-Fidelity and High-Quality 3D Images

Written By: kinza.sabir
Last Updated On: January 1, 2024

With Paint3D, now you can paint your 3D world with textures by crafting lighting-independent textures for your vision. The innovators behind this creativity are from Tencent PCG, Zhejiang University and ShanghaiTech University.

The above mention images are showing the workflow of Paint3D model. It takes two input images, first one is the reference images and second one is the input mesh. Mesh is basically a shape and structure of an object. The two images input means that the reference image has to adopt the structure of mesh input to generate the output of merger of these two images in 3D format.

Like the images input, Paint3D also works on text prompt by giving mesh image as an input with the text prompt. The input mesh image adopts the structure of text prompt to generate 3D output.

Paint3D is a flexible and innovative model that generates textured 3D models that are of high-quality and light-independent. Pre-trained 2D models are used that helps to create initial texture. Different views of the 3D objects are used to create basic texture maps. Paint3D offers highly specialized models named UV Inpainting and UVHD Diffusion for creating enhanced texture that are independent of lightening. The textured output are versatile as they are not dependent on lightening aids to easy editing and adaptable to other graphic design requirements.

Consider yourself as an artist and you have to create a detailed skin for a 3D object. You are fixing all the flaws on the skin and making it look absolutely perfect, this how Paint3D works. This model is super flexible as it can work with words and images as an initial instructions to create the textures. Lets experience how this model works with instructions and images.

Paint3D: Texture Generation based on Images

Now it is easy for you to understand that Paint3D has painted the reference image with the input image. The merger of these two images generates a high-quality texture while maintaining the consistency of the 3D objects. I have some more images to show you the consistency of this state-of-the-art model.

Through these examples, the end user can witness the flexibility and semantic consistency of this model as it significantly enhance the quality of texture maps for 3D models.

Paint3D: Texture Generation based on Texts

Here comes the most adaptable part of the Paint3D is when the input image combines with the text prompt. When the images merged with the text prompts the generated outcomes are amazing. The user can generate output according to their own desire and willingness. Lets experience some more examples of texture generation based on text.

For more interactive and amazing examples you can visit to the GitHub and if you are interested in more technical details visit Arxiv. The code of the research is not available yet.

Even though Paint3D doesn’t require lighting details in its textures but when you put light on your 3D model from various angles, the textures still produces shadows in the accurate places.

A 2D model is a flat shape just like an image whereas, 3D images are like a sculpture and shadow is the most most integral part of 3D image. That is the reason using only a 2D model to represent a 3D shape shows leaving out important details and creates weird lighting effects. Paint3D overcomes these issues and can easily generate 3D object using 2D images as shown in the clip below even there is no appropriate lighting.

Paint3D makes textures that doesn’t contain built-in information about how they should respond to light, but when you put light to your 3D stuff, the textures still create shadows in such a way that makes everything look realistic.

The model works best with the given examples by the researchers but lets suppose I will give input of truck with the mesh input of ball. The model is showing totally unrealistic output.