MIT AI Image Generator System Makes Models Like DALL-E 2 More Creative


A pattern DALLE 2 generated picture of an astronaut driving a horse in a photorealistic fashion. Credit: Open AI

A brand new technique developed by researchers makes use of a number of fashions to create extra complicated pictures with higher understanding.

With the introduction of DALL-E, the web had a collective feel-good second. This synthetic intelligence-based picture generator is impressed by artist Salvador Dali and the lovable robotic WALL-E and makes use of pure language to supply no matter mysterious and exquisite picture your coronary heart wishes. Seeing typed-out inputs similar to smiling gopher holding an ice cream cone immediately spring to life is a vivid AI-generated picture clearly resonated with the world.

It isn’t a small job to get stated smiling gopher and attributes to pop up in your display. DALL-E 2 makes use of one thing known as a diffusion mannequin, the place it tries to encode the complete textual content into one description to generate a picture. However, as soon as the textual content has much more particulars, its laborious for a single description to seize all of it. Furthermore, whereas theyre extremely versatile, diffusion fashions generally battle to grasp the composition of sure ideas, similar to complicated the attributes or relations between totally different objects.

This array of generated pictures, displaying a prepare on a bridge and a river below the bridge, was generated utilizing a brand new technique developed by MIT researchers. Credit: Image courtesy of the researchers

To generate extra complicated pictures with higher understanding, scientists from MITs Computer Science and Artificial Intelligence Laboratory (CSAIL) structured the typical model from a different angle: they added a series of models together, where they all cooperate to generate desired images capturing multiple different aspects as requested by the input text or labels. To create an image with two components, say, described by two sentences of description, each model would tackle a particular component of the image.

The seemingly magical models behind image generation work by suggesting a series of iterative refinement steps to get to the desired image. It starts with a bad picture and then gradually refines it until it becomes the selected image. By composing multiple models together, they jointly refine the appearance at each step, so the result is an image that exhibits all the attributes of each model. By having multiple models cooperate, you can get much more creative combinations in the generated images.

River Leading Into Mountains Generated Images

This array of generated images, showing a river leading into mountains and red trees on the side, was generated using a new method developed by MIT researchers. Credit: Image courtesy of the researchers

Take, for example, a red truck and a green house. When these sentences get very complicated, the model will confuse the concepts of red truck and green house. A typical generator like DALL-E 2 might swap those colors around and make a green truck and a red house. The teams approach can handle this type of binding of attributes with objects, and especially when there are multiple sets of things, it can handle each object more accurately.

The model can effectively model object positions and relational descriptions, which is challenging for existing image-generation models. For example, put an object and a cube in a certain position and a sphere in another. DALL-E 2 is good at generating natural images but has difficulty understanding object relations sometimes, says Shuang Li, MIT CSAIL PhD student and co-lead author. Beyond art and creativity, perhaps we could use our model for teaching. If you want to tell a child to put a cube on top of a sphere, and if we say this in language, it might be hard for them to understand. But our model can generate the image and show them.

Making Dali proud

Composable Diffusion the teams model uses diffusion models alongside compositional operators to combine text descriptions without further training. The teams approach more accurately captures text details than the original diffusion model, which directly encodes the words as a single long sentence. For example, given a pink sky AND a blue mountain in the horizon AND cherry blossoms in front of the mountain, the teams model was able to produce that image exactly, whereas the original diffusion model made the sky blue and everything in front of the mountains pink.

AI Generated Images Dog Sky

Researchers were able to create some surprising, surreal imagery with the text, a dog and the sky. On the left appear a dog and clouds separately, labeled dog and sky underneath, and on the right appear two images of cloud-like dogs with the label, dog AND sky, underneath. Credit: Image courtesy of the researchers

The fact that our model is composable means that you can learn different portions of the model, one at a time. You can first learn an object on top of another, then learn an object to the right of another, and then learn something left of another, says co-lead author and MIT CSAIL PhD student Yilun Du. Since we can compose these together, you can imagine that our system enables us to incrementally learn language, relations, or knowledge, which we think is a pretty interesting direction for future work.

While it showed prowess in generating complex, photorealistic images, it still faced challenges because the model was trained on a much smaller dataset than those like DALL-E 2. Therefore, there were some objects it simply couldnt capture.

Now that Composable Diffusion can work on top of generative models, such as DALL-E 2, the researchers are ready to explore continual learning as a potential next step. Given that more is usually added to object relations, they want to see if diffusion models can start to learn without forgetting previously learned knowledge to a place where the model can produce images with both the previous and new knowledge.

MIT Composable Diffusion Generated Images

This photo illustration was created using generated images from an MIT system called Composable Diffusion, and arranged in Photoshop. Phrases like diffusion model and network were used to generate the pink dots and geometric, angular images. The phrase a horse AND a yellow flower field is included at the top of the image. Generated images of a horse and yellow field appear on the left, and the combined imagery of a horse in a yellow flower field appear on the right. Credit: Jose-Luis Olivares, MIT and the researchers

This research proposes a new method for composing concepts in text-to-image generation not by concatenating them to form a prompt, but rather by computing scores with respect to each concept and composing them using conjunction and negation operators, says Mark Chen. He is the co-creator of DALL-E 2 and a research scientist at OpenAI. This is a nice idea that leverages the energy-based interpretation of diffusion models so that old ideas around compositionality using energy-based models can be applied. The approach is also able to make use of classifier-free guidance, and it is surprising to see that it outperforms the GLIDE baseline on various compositional benchmarks and can qualitatively produce very different types of image generations.

Humans can compose scenes including different elements in a myriad of ways, but this task is challenging for computers, says Bryan Russel, research scientist at Adobe Systems. This work proposes an elegant formulation that explicitly composes a set of diffusion models to generate an image given a complex natural language prompt.

Reference: Compositional Visual Generation with Composable Diffusion Models by Nan Liu, Shuang Li, Yilun Du, Antonio Torralba and Joshua B. Tenenbaum, 3 June 2022, Computer Science > Computer Vision and Pattern Recognition.

Alongside Li and Du, the papers co-lead authors are Nan Liu, a masters student in computer science at the University of Illinois at Urbana-Champaign, and MIT professors Antonio Torralba and Joshua B. Tenenbaum. They will present the work at the 2022 European Conference on Computer Vision.

The research was supported by Raytheon BBN Technologies Corp., Mitsubishi Electric Research Laboratory, and DEVCOM Army Research Laboratory.

(function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(d.getElementById(id))return;js=d.createElement(s);;js.src=”″;fjs.parentNode.insertBefore(js,fjs);}(doc,’script’,’facebook-jssdk’));


Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button