Tech - ESRGAN: Neural Network AI for upscaling images, and how YOU can use it

ESRGAN is a Neural Network that gets a low resolution image and upscales it, trying to rebuild details lost by the downscaling, like fur, grass, eyes, curves, etc. It doesn't simply blur everything like Bilinear or Trilinear filtering, nor does it leave everything pixelated like Point, but instead, it makes something truly.. magical. And it's quite fast, too, even on a low-end CPU like a Pentium.

Today, I'll be walking you through on how to install ESRGAN and how to get it running on your system. It's pretty easy, actually, as long as you follow these steps. If you have a Linux or MacOS operating system, I suggest you look Elsweyr; this guide's not for you, sorry.


UPDATE (18/03/2019): You can actually skip step 2a by checking a checkbox at the bottom of the Python installer that says something along the lines of "Add Python to PATH".

Step 1 - Prerequisites: To run this, you'll need to install a few things first:

• A 64-bit Operating System (Windows, Linux or MacOS)
• A 64-bit install of Python 3.5 or higher
• (Optional - GPU rendering) CUDA 9.2 or 10.0 for NVidia cards

Step 2a - PyTorch: PyTorch is another prerequisite for running ESRGAN. Before installing it, we need to:

• Set Python to the PATH variable (this is for Windows 7 64-bit, but it's easy to find how-tos for others like Win10):
• Right click on "My Computer", and click Properties;
• Left click on "Advanced System Settings" (yes, it's called A.S.S.). You can close the Computer Properties now;
• Under the "Advanced" tab, left click on "Environment Variables";
• At the bottom, in "System Variables", locate the "Path" variable, and double click it;
• In the value, add:

";C:\Users\[YOUR USER NAME HERE]\AppData\Local\Programs\Python\Python[37 for 3.7, 36 for 3.6]"*
and
";C:\[YOUR USER NAME HERE]\Usuario\AppData\Local\Programs\Python\Python[37 for 3.7, 36 for 3.6]\Scripts"*
*or wherever else you installed Python. Don't include the ""s, stupid. If you do, you might kill your OS. No joke.

• Click all the okays and applies, system should freeze up for a few miliseconds while loading the new variables.

Step 2b - PyTorch: Installing it, at last:

• Open this URL on any browser
• Stable (1.0); [your OS]; Pip; [your Python version]; [CUDA version, if installed]
• Copy the command(s) the website gave you, and open the command prompt:
• Hold the Windows Button, and press R;
• Type "cmd" into the text box;
• Press Enter.
• Right click inside the Command Prompt, and left click on "Paste". Ctrl+V does NOT work in cmd.exe.
• It'll auto-execute the first command (the first line you copied), wait a few moments. It'll be over before you know it.
• IMPORTANT: Check if it gives you any errors relating to a wheel, or .whl file. If so, make sure your Python version is 64-bit, and that you chose the correct one in the PyTorch website. It will NOT work otherwise.
• Press Enter after it finishes the first command, to execute the next one.
• When you're left with an empty, "[path]>_" line at the bottom, and pressing Enter does nothing, copy this:

pip3 install numpy opencv-python

• and paste it into cmd, same way as before. Press Enter afterwards to run the command.
• After it's done, close the Command Prompt.

Step 3 - Downloading and running ESRGAN:

• Open the GitHub page for ESRGAN
• Left click "Clone or Download", and then left click "Download ZIP".
• Extract the downloaded folder to where you want ESRGAN to be. Put it somewhere easy to access, like Documents or C:/, and open the folder.
• Download these batch files, and put them in the same folder as "test.py":

open_this_to_remaster_LR.bat
open_this_for_network_interp.bat

• (Optional - CPU rendering) If you opted not to install CUDA earlier, if you have an Intel, AMD, or very old NVidia card, open test.py with a text editor like Notepad++, Notepad or Wordpad, not with double-click, and look for the line near the top that says:

device = torch.device('cuda')

• (Optional - CPU rendering) After locating that line, do as the comment says: replace 'cuda' with 'cpu', save the .py file, and close the text editor.
• You'll notice the ESRGAN-master\models folder is empty, with a lonely readme.md. Open this Google Drive link to download the models you'll use.
• Put both models in the empty folder.

Step 4 - Running ESRGAN:

• To see if it's working, open the "open_this_to_remaster_LR.bat" file. It should start working on the two images that come with ESRGAN, a monkey and a drawing. It'll take about half a minute on a Pentium for both images, which is slower than both Bicubic and Trilinear, but the results are miles better. If it gives you an error, follow this tutorial again from scratch, and make sure you do everything correctly.
• To use the Network Interpolation mentioned in the GitHub page for ESRGAN, edit the "open_this_for_network_interp.bat" batch file and change 0.8 to any number from 0.0 to 1.0; or simply launch the .bat, if you think 0.8 is a fine value. Experiment around with it, I'm sure you'll find something you like for every image you have. Basically, to give you a run-down:
• 0.0 is lacking in details, mostly the basic shapes of everything. It may be fitting for drawings or paintings. It also has less artifacting than the ESRGAN model.
• 1.0 is too sharp, and sometimes strains the eyes. It also can look perfect on some images, for example, textures for leather things like backpacks. The artifacting in these cases ends up looking like leather, which is quite nice.
• You also have to change the number that comes after "models/interp_" to whatever number you change the decimal value above to, but without the dot. For example, interpolation of 0.6 will need a model called interp_06.

That's all you have to do to get ESRGAN up and running. Seems like a lot, and it will take an hour or two of your day, but trust me, it's worth it. Hopefully someone will take this knowledge and use it on Counter-Strike Condition Zero, Half-Life 2, Quake 1, and other old games that have low-res textures that are just detailed enough for the AI to understand what it is. It also works wonders on background images of 2D games, or sprites.

One thing to note is that ESRGAN doesn't work with alpha channels, from what I hear - at least, not 100% alpha, where most image editing programs, like Paint.NET (and I think GIMP) erase any colour data in the fully transparent areas.

Another thing to note is that the test.py application is a batch converter by default, the batch files I made just save you the trouble of having to type all that stuff into the Command Prompt every time you're going to use ESRGAN.

It's also VERY important to have a lot of memory for ESRGAN, as PyTorch tries to allocate 1gb of RAM (or VRAM, if on GPU mode with CUDA), and if it fails, it'll throw an error and give up on the file. If that happens to you, just get an even lower resolution version of that same image, or download more RAM.

Now go ahead and have fun!

UPDATE (21/01/2019): The default models that come with ESRGAN aren't very good for textures like faces and photos, so I advise you to look for new models other than the default two and interpolation levels between them. You can see a comparison between this guy's model and the default ones: