We Have AI At Home: AI Image Upscaling & Restoration

Both of these apps will be featured in my video of Highly Recommended Linux Apps

I should publish it either today or tomorrow, and I will post it here of course. Both of these apps use the same underlying stack, and they are designed to make photos larger and more high-resolution, but they run locally on your own hardware.

The AI, Machine-Learning, & Driver Combo Stack:

Real-ESRGAN: Real-world Enhanced Super-Resolution Generative Adversarial Network

NCNN: Convolutional Neural Networks

Real-ESRGAN is the primary AI-upscaling model, and NCNN is neural network processor or Machine-Learning used to train it. Those two blocks are combined with Vulkan so that your computer, tablet, phone etc. can run them on your local hardware.

Vulkan is an application programming interface(API) originally born out of AMD's Mantle API. Both Microsoft & Sony chose AMD accelerated processing units(APUs) for the newest Xbox & Playstation generations, and so Mantle was created to simplify porting from PC to the latest consoles. Eventually AMD donated the tech to Microsoft & The Khronos Group resulting in the open-source Vulkan API becoming an open-standard that functions across all of the major operating systems including Apple's. Vulkan has become central to modern Linux graphics drivers.

Upscaler

These image upscaler uses Tencent's NCNN framework with Vulkan acceleration to run the AI model ESRGAN:Enhanced Super-Resolution Generative Adversarial Networks. My first question was what does 'adversarial networks' mean?

Generative Adversarial Networks (GAN): The core AI technology used, where two neural networks (a generator and a discriminator) compete against each other to create realistic image details.

It's worth noting that this runs on your GPU, and it can take several minutes depending on the image type.

Original image: 1200x625

Upscaled: 2025x3600

Original: 256x499

Upscaled: 1024x1996

Upscaler

This app is Linux-first, and it's built with C++ & GTK4. It's extremely simple, and limited in options. You get to pick 2 models: photo or animation/art And you choose a scale. All of the examples above were done with Upscaler.

Upscayl

This is based on the same stack, but the user-interface instead is built with Electron: basically a Javascript/web-engine for building desktop apps. There are some advantages as Upscayl gives you more options for upscaling your images. You can export to jpg,png,webp formats. Includes the option to pick from multiple built-in AI models with different focuses, or load your own models.

It also lets you compare a before and after of the image you've upscaled once you've finished, and it gives you statistics on the average time it took to render on your computer. As well as a bunch of trendy color themes for the UI itself.