Generating images from voice

Installation

Make sure you have virtualenv and python 3.5+ installed

bash install.sh

This will activate the virtualenv and install the proper packages.

To then launch the jupyter instance use

jupyter notebook

And you should be directed to localhost:8888.

Project structure

• Models - The resulting generated models
• Utilities - Various tools such as visualization etc.
• Scripts - The scripts used for the project
• Datasets - The datasets used for the project

Resources

Text to image

• Generative Adversarial Text to Image Synthesis

• AttnGAN

• Paragraph vectors

• StackGAN++

• CycleGAN

• Deep convolutional GAN

• Various papers with code paperswithcode

• Adversarial loss short paper

• Training GANs, Tips and Tricks

Further information

Notes

Might have to use a bag of words model or some other form of context presentation to simplify what the sentence says, look into this further.

Training GANS

Google Cloud Platform

Jupyter notebooks

To use jupyter notebooks, run this on the remote

david@torcher-vm:~/StackGAN-Pytorch$ jupyter notebook --no-browser

Then tunnel your connection through

david@fridge:~$ ssh -N -L localhost:8888:localhost:8888 david@<EXTERNAL_IP_OF_VM>

Then simply open a browser on localhost:8888 and provide it with the token that should be visible in the commandline window on the vm to connect.

Show results

The images are viewable in python notebooks and can also, be downloaded from there.

Girl holding cat stuffed animal

A little boy shows off his suitcase full of toys.

A man with a jackhammer demolishing cement.

Loss plot of training with the flickr30k dataset. 100 epochs.

• 

TODO

• Implement the correct loss function
• Integrate the text embedding into the discriminator