Civitai's 100% Beginner's Guide to Generative AI Art (2024)

What is “Generative AI”, and how doesStable Diffusionfit into it? You might have heard the term Generative AI in the media – it’s huge right now; it’s on the news, it’s on the app-stores, Elon Musk is Tweeting about it – it’s beginning to pervade our lives.

Generative AI refers to the use of machine learning algorithms to generatenew datathat issimilarto the data fed into it. This technology has been used in a variety of applications, including art, music, and text generation. The goal of generative AI is to allow machines to create something new and unique, rather than simply replicating existing data.

• Stable Diffusionis one example of generative AI that has gained popularity in the art world, allowing artists to create unique and complex art pieces by entering text “prompts”.

• Chat GPT is another example of generative AI – alanguage modelthat can generate human-like text. It is capable of completing sentences, paragraphs, and even entire articles, given a short prompt. This technology is being used in a variety of applications, including chatbots, content creation, and even computer programming.I used it to write this paragraph in ~1 second. It’s a great general-purpose tool. Similar tools include Google’s Gemini, and Microsoft’s Copilot.

• GitHub Copilot is an example of a generative AI language model built for a specific purpose – to assist with programming and coding.

This guide will specifically covergenerating image/art content with Stable Diffusion, but will touch on other Generative AIart services.

In mid-2022, the art world was taken by storm with the launch of several AI-powered art services, including Midjourney, Dall-E, and Stable Diffusion. These services and tools utilize cutting-edge machine learning technology to create unique and innovative art that challenge traditional forms and blur the lines between human and machine creation.

The impact of AI art on the industry has already beensignificant. Many artists and enthusiasts are exploring the possibilities of this new medium, while many fear the repercussions for established artists’ careers. Many art portfolio websites have developed new policies that prohibit the display of AI-generated work. Some websites require artists to disclose if their work was created using AI, and others have even implemented software that can detect AI-generated art.

The Companies

There are many big-players in the AI art world – here are a few names you’ll often see mentioned;

• OpenAI– A research laboratory with both for and non-profit subsidiaries, focusing on the development of AI, in an open and responsible manner. Founded by technology investors (including Peter Thiel and Elon Musk) in 2015, OpenAI has created some highly advanced generative AI models, such as GPT-3, and the recently announced GPT-4, which are highly regarded for their language processing and generation abilities.

• Stability AI–The world’s leading open source generative AI company – the brainchild ofCEO Emad Mostaque, Stability AI is a technology start-up, focused on open source releases of tools, models, and resources. Stability AI is behind the 2022 releases of the Stable Diffusion, and Stable Diffusion 2.0 text-to-image models.

• RunwayML– One of the companies behind Stable Diffusion, RunwayML now provide a platform for artists to use machine learning tools in intuitive ways without any coding experience.

Controversies

There are already a number of lawsuits challenging various aspects of the technology. Microsoft, GitHub and OpenAI are currently facing aclass-action lawsuit, while Midjourney and Stability AI arefacing a lawsuitalleging they infringed upon the rights of artists in the creation of their products.

Whatever the outcome, Generative AI is here to stay.

That is anincrediblycomplex topic, and we’ll just touch on it very briefly here at a very very high level;

(Forward) Diffusion is the process of slowly adding random pixels (noise) to an image until it no longer resembles the original image, and is 100% noise – we’ve diffused, or diluted, the original image.

Byreversingthat process, we can reproduce something similar to the original image. There is obviously alotmore going on in the process, but that’s the general idea; we input text, the “model” processes that text, generates it from the “diffused” image, and displays an appropriate output image.

Simple!

There are a number of tools to generate AI art images, some more involved and complex to set up than others. Theeasiest methodis to use a web-based image generation service, where the code and hardware requirements are taken care of for you but there’s often a fee involved.

Alternatively, if you have the required hardware (ideallyanNVIDIAgraphics card), you can create images locally, on your own PC, with no restriction, using Stable Diffusion.

When we talk about Stable Diffusion, we’re talking about the underlying mathematical/neural network framework which actually generates the images. We need some way to interface with that framework in a user-friendly way – that’s where the following tools come in;

I don’t have a PC or a Graphics Card (GPU)! How can I make AI Generated Art!?

Don’t panic! Civitai.com has one of the best on-site Image Generator services around! Unlike all the other web-based services out there, we have direct access to the largest repository of Models and additional resources with which to build your images. Getting started with the Civitai Generator is extremely simple, and we have a detailed guide to walk you through the process!

See Also

How to Upload and Share Generative AI Images on Civitai

To run on your own PC – Local Interfaces

This guide isextremely high leveland won’t get into the deep technical aspects of installing (or using) any of these applications, but if you’d like to run Stable Diffusion on your own PC (a local install) there are many options!

Note thatto get the most outof any local installation of Stable Diffusionyou need an NVIDIA graphics card.

Images can be generated using your computer’s CPU alone, or on some AMD graphics cards, but the time it will take to generate a single image will beconsiderable.

• Automatic1111’s WebUI(Complexity factor ⭐⭐⭐⭐/5)– WebUI is the most commonly used Interface for Stable Diffusion. It is moderately complex, and has a wide range of plugins and extensions to extend the experience. There’s a great deal of community support available if you have problems.

• ComfyUI(Complexity factor ⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐/5)– ComfyUI is relatively new to the scene, and provides an exceedingly complex workflow/node based workspace which requires in-depth knowledge of the Stable Diffusion image generation process to make work. Definitely not a beginner interface, but extremely powerful for the experienced user.

• Cmdr2’s Easy Diffusion(Complexity factor ⭐⭐/5)– A great option for those starting out with a local install. Easy Diffusion has a 1-click installer for Windows, and a popular Discord server full of extremely knowledgeable people to help you get up and running. The interface itself is limited in what it can do, compared to the other Interfaces, but it remains the easiest way to get started making your own images, locally.

• Fooocus(Complexity factor ⭐⭐/5) –A very popular interface for creating Stable Diffusion images based on the SDXL model. It’s practically a one-click install on Windows and produces absolutely beautiful images. It does need a moderately powerful GPU (graphics card). Read our Fooocus Quickstart Guide here.

• InvokeAI(Complexity factor ⭐⭐⭐/5) –A popular open-source text-to-image and image-to-image interface with powerful tools, not yet as full featured as Automatic1111’s WebUI, but getting close.

• SD.Next (Complexity Factor ⭐⭐⭐⭐/5) – Also known as “Vlad” (after author, Vladmandic), SD.Next started as a “fork” of Automatic1111 WebUI, but has diverged considerably and has a wide range of advanced features.

To run on your own Mac – Local Interfaces

Mac owners can runAutomatic1111’s WebUI,InvokeAI, and also a popular, lightweight, and super simple to use Interface, DiffusionBee;

• DiffusionBee(Complexity factor ⭐/5)– DiffusionBee is an extremely lightweight MacOS interface for Stable Diffusion. It allows for basic image generation, but has a very small feature-set, to keep it as simple as possible.

• Draw Things App–(Complexity factor ?/5)– Draw Things is a popular and highly rated MacOS App. I don’t know much about it, but from anecdotal evidence it seems to have some good features!

Checkpoints, also known as “weights” or “models” are part of thebrainswhich produce our images. Each model can produce a different style of image, or a particular theme or subject. Some are “multi-use” and can produce a mix ofportrait,realistic, andanime(for example), and others are more focused, only reproducing one particular style of subject.

Models come in two file types. It’s important to know the distinction if running a local Stable Diffusion interface, as there are security implications.

Pickletensor (.ckpt extension)models may contain and execute malicious code when downloaded and used. Many websites, including Civitai, have “pickle scanners” which attempt to scan for malicious content. However, it’s safer to downloadSafetensor (.safetensor)models when available. This file type cannot contain any malicious code and is inherently safe to download.

Note that if using a Generation Service you will only be able to use the modelsthey provide.Some services provide access to some of the most popular models while others use their own custom models. It depends on the service.

Along with models there are many other files which can extend and enhance the images generated by the models, includingLoRA,Textual Inversion, andHypernetworks. We’ll look at those in a more in-depth guide

Watch our video, below, for a walkthrough of these core concepts!

Most stable diffusion interfaces come with the default Stable Diffusion models,SD1.4and/orSD1.5, possiblySD2.1orSD2.2, and maybe SDXL. These are the Stable Diffusion models from which most other custom models are derived and can produce good images, with the right prompting.

Custom models, models “trained” with new images to produce stunning styles and specific content, can be downloaded from Civitai.com!

You are here!Civitai is theleadingmodel repository for Stable Diffusion checkpoints, and other related Generative AI tools. There are tens of thousands of models to choose from, across many categories; something for everyone!

Generative AI is a huge field, with many applications. Some of the most popular and interesting tools right now are;

• ChatGPT– Mentioned above, ChatGPT is what’s known as an LLM (Large Language Model), designed to provide conversational responses to input text, understand and answer questions, provide recommendations, generate content, and more. It can solve problems, write code – it’s extremely useful, and free (with limitations). The first local models for ChatGPT like LLMs are now appearing, which can be used on your own PC.

• Magnific.ai – An out of this world “Upscaling” tool (upscaling is the process of taking a low quality/resolution image and enlarging or enhancing it to a larger size) which has wowed the internet recently with the quality of the output images;

Check out our Guide to Upscaling with Stable Diffusion, here!

• OpenAI’s Sora – Sora is a Generative AI model that can create realistic and imaginative scenes from text instructions (also known as txt2video). It’s not yet available for general use, but the outputs look phenomenal – a great example of how quickly the technology is evolving.