Sketch-to-Real and Real-to-Sketch Image Generation

Abstract

Being able to draw a sketch and then bringing it to life with realistic counterpart is a application with much util- ity. Such is the topic of our project, using Stable Diffusion as the backbone model. We also explore the inverse case of getting a sketch-styled image from a real picture. To im- prove the quality of the generations, we fine-tune the model using Dreambooth and we also experiment with Control- Net. We compare the generations qualitatively and quanti- tatively against the baseline and observe our performance differences.

Overview

In this repository you can find the part of code of Image to Sketch part of our project. Further you can find the steps necessary to reproduce the results.

Setup

1. Clone the Repository

2. Install Dependencies

pip install -r requirements.txt

Dataset

1. Downloading images and sketches

The dataset is not uploaded to the repository due to size constraints. To obtain it you need to run download.ipynb.
Ensure you have the dataset in the correct structure under 256x256/.

2. Creating dataset

You need to run sandbox.ipynb that uses dataset.py. It will create a dataset in the my_dataset/ directory.

Training the Model

To fine-tune Stable Diffusion using Dreambooth on sketches, use the following commands:

export SMALL_MODEL_NAME2="OFA-Sys/small-stable-diffusion-v0"
export INSTANCE_DIR="./my_dataset"
export OUTPUT_DIR="./dreambooth-deer-model"

accelerate launch diffusers/examples/dreambooth/train_dreambooth.py   --pretrained_model_name_or_path=$SMALL_MODEL_NAME2   --instance_data_dir=$INSTANCE_DIR   --output_dir=$OUTPUT_DIR   --instance_prompt="a deer in sks style"   --resolution=512   --train_batch_size=3   --gradient_accumulation_steps=1   --max_train_steps=1000   --learning_rate=1e-5   --lr_scheduler="constant"   --lr_warmup_steps=100   --mixed_precision="fp16" --gradient_checkpointing --enable_xformers_memory_efficient_attention --checkpointing_steps=5000000

Running Inference

For the unference you need to run run_db.ipynb for seeing the results without fine-tuning. And run run_db2.ipynb for seeing the results with fine-tuning.

Example Results

Sample results can be found in
- model_outputs/
- model_outputs2/
Especially at
- model_outputs/deer_comparison_table_*.png
- model_outputs2/deer_comparison_table_*.png

Notes

• Ensure your GPU and RAM has enough memory for Dreambooth fine-tuning and storing the models.
• If running out of storage, remove as much files as you can, don't forget to clear the cache.

Authors

• Dmitry Akulov
• Dimitri Kachler

---

This README.md provides an organized(as-much-as-possible) structure, and you can modify details as necessary.