Stable Diffusion
Stable Diffusion is an advanced artificial intelligence (AI) program designed to generate images based on user-provided text prompts. Utilizing a latent diffusion model, Stable Diffusion transforms random noise into coherent images by removing this noise through multiple processing steps. Users can input simple descriptions, such as "oak tree," or more complex scenarios, like "oak tree next to a lake," and the program produces photorealistic images that closely align with the described content. Released in August 2022 and updated with Stable Diffusion 2.0 later that year, the program also features capabilities such as inpainting and outpainting, allowing users to alter existing images by adding or replacing parts and extending their dimensions.
While Stable Diffusion is open-source and available for both commercial and non-commercial use, it has notable limitations. The program often struggles with accurately rendering hands and teeth, which can reveal the AI-generated nature of the images. Additionally, copyright concerns exist, as the training process utilized existing images from the internet without the explicit permission of their creators. Despite these drawbacks, Stable Diffusion represents a significant development in AI technology, illustrating the growing intersection of creativity and computational capabilities.
On this Page
Subject Terms
Stable Diffusion
Stable Diffusion, of Stable Diffusion 2.0, is a computer program and a type of artificial intelligence (AI) that creates images from text prompts. A user enters information into the text prompt, and the program generates an image based on the prompt. Stable Diffusion can perform this task using a latent Diffusion model, which involves removing random noise from the image. Stable Diffusion also can alter existing images through processes known as inpainting and outpainting, which can remove part of the existing image or enlarge the image. Stable Diffusion was released in August 2022 and updated with a new version later that year.
Despite its unique abilities, Stable Diffusion has several drawbacks. For example, the program has difficulty with hands and teeth and the program has the potential to infringe upon copyright.
Background
Stable Diffusion is a type of AI, which is any computer hardware or program that can think on its own, make decisions, and solve problems. AI systems are generally very complex and powerful and can process huge amounts of information. AI is a constantly evolving part of the computer field.
Stable Diffusion takes text prompts from a user and generates an image based on the prompts. Essentially, a user enters a description in the text prompt of what they wish to see. The program can understand the description and proceeds to generate an image resembling the description. For example, if a user enters the word “oak tree” into the text prompt, Stable Diffusion will create an AI-generated image of an oak tree. The program also can handle more complicated text prompts, such as “oak tree next to a lake.” In this case, Stable Diffusion will generate an image resembling that description.
Stable Diffusion is open source, meaning it is available to the public to use. Furthermore, it is available for commercial and non-commercial use. The program was first released in August 2022. It was born out of the work of Robin Rombach at Stability AI, Patrick Esser at Runway ML, and a team from the CompVis Group at LMU Munich, which was led by Dr. Björn Ommer. Over the next few months, Stability AI raised more than one hundred million dollars in funding for Stable Diffusion from companies such as Coatue Management, Lightspeed Venture Partners, and O’Shaughnessy Ventures.
Overview
Stable Diffusion is one of many types of AI programs. It is particularly unique because it uses a latent Diffusion model (LDM) to produce images. In this model, random noise is produced in an image, which resembles television static. The program then runs through numerous steps to remove parts of the random noise. It keeps removing noise until the image revealed resembles the description in the text prompt. The model can do this because it has been previously trained to add noise to an image. Therefore, the model simply reverses this process.
Stable Diffusion does not create its own images. It instead relies on existing images from the Internet. Its programmers trained it using these images. Common websites where the images originated include Pinterest, Flickr, and DeviantArt. The images were accompanied by captions that include text. Stable Diffusion was trained to understand what objects look like and recognize how objects differ.
Stable Diffusion generates images that are photorealistic even though they are AI-generated. This makes it difficult to tell that an image is not a photograph of the real object, but is instead AI-generated art.
Stable Diffusion has other functions besides generating images. For example, the program can also alter existing images. It does this by adding or replacing parts of an image, which is called inpainting. Stable Diffusion also can extend an image, which makes the image larger. This process is called outpainting. To perform inpainting or outpainting, a user must upload the image they wish to alter.
In November 2022, Stable Diffusion 2.0 was released. The new version includes numerous improvements over the initial version. One of these improvements is the incorporation of new text-to-image Diffusion models. This enhances the quality of the images that are generated. Images are further enhanced by a super-resolution Upscaler Diffusion model, which produces higher resolution images. For example, Stable Diffusion 2.0 can take an image that has a low resolution and upscale it into a resolution of up to four times higher. The new version also includes a new depth-to-image Diffusion model, which allows for new creative applications and image transformations. An updated inpainting Diffusion model is featured in Stable Diffusion 2.0. This updated model enables the user to perform the process of inpainting quickly and easily.
Stable Diffusion has some drawbacks. For example, the program experiences issues with hands and teeth. If an image contains a person showing their hands or teeth, the user can likely tell that the image was AI-generated because the detail in the hands and teeth generally are poor. The reason for this is that Stable Diffusion does not understand the number of fingers on a human hand or the number of teeth in a person’s mouth. One way around this drawback is to tweak the text prompt. The user can indicate in the text prompt that the subject is in some way hiding their hands or not showing their teeth.
Another pitfall with Stable Diffusion is copyright issues with the images it uses. When the programmers of Stable Diffusion were training it, they used existing images from all over the Internet. In other words, the images were created by other people. The problem is that many of these images were used without permission from their creators. Therefore, issues with copyrights could arise.
Bibliography
Heidorn, Christian. “Stable Diffusion Release Date & Timeline.” TokenizedHQ.com, 17 Aug. 2023, tokenizedhq.com/Stable-Diffusion-release-date/. Accessed 18 Aug. 2023.
Laukkonen, Jeremy. “What Is Stable Diffusion? A Look at How an AI Model Is Reshaping the Images You See.” Lifewire, 9 May 2023, www.lifewire.com/what-is-Stable-Diffusion-7485593. Accessed 18 Aug. 2023.
Monge, Jim Clyde. “Stable Diffusion Finally Released to the Public.” Medium, 25 Aug. 2022, medium.com/codex/Stable-Diffusion-finally-released-to-the-public-db1aa417d85b. Accessed 18 Aug. 2023.
Ribeiro, Jair. “A Quick Look under the Hood of Stable Diffusion Open Source Architecture.” Medium, 30 Aug. 2022, medium.com/codex/a-quick-look-under-the-hood-of-Stable-Diffusion-open-source-architecture-2f07fc1e729. Accessed 18 Aug. 2023.
“Stable Diffusion Online.” Stable Diffusion AI, StableDiffusionai.org/. Accessed 18 Aug. 2023.
“Stable Diffusion Public Release.” Stability AI, 22 Aug. 2022, stability.ai/blog/Stable-Diffusion-public-release. Accessed 18 Aug. 2023.
“Stable Diffusion 2.0 Release.” Stability AI, 24 Nov. 2022, stability.ai/blog/Stable-Diffusion-v2-release. Accessed 18 Aug. 2023.
“What Is Artificial Intelligence (AI)?” IBM, www.ibm.com/topics/artificial-intelligence. Accessed 18 Aug. 2023.