I have finished, or at least done enough so that my project can be seen as going to release, and not as stuck in place under the pretext of vague and incomprehensible improvements.
It is not customary to tell from the end, but here it will not work otherwise. For the past nine months, I've been working on a remaster of Deep Space 9 - ever since AI-assisted video resolution software began to hit the market. After seeing what quality can be squeezed out of old MKV files, I decided to start from the very beginning using the original and higher quality source - DVD. Nine months later, I achieved my goal of creating a deep space 9 remaster and upscaling method that did not rely on manually combing through scenes and fine-tuning deinterlacing algorithms in an attempt to minimize the compromises in picture quality. And in this article I will demonstrate the results of this work.
I created a progressive version of Deep Space 9 at a frame rate of 23.976 using a model codenamed Rio Grande. I haven’t tested this method on the entire series yet - however, it worked well in selected episodes from seasons 2, 4, 5, and 6. I want to thank Cyril Niderprim for finding the ridiculously simple solution below. And while Rio Grande can throw errors - I'll give examples of them below - if it spoils the episode so badly that you can't accept it, there is an alternative for it.
I also managed to convert to 59.94 frames per second with identical quality using a model codenamed "Orinoco". Orinoco retains movement correctly in all scenes, and is attached to the project for insurance purposes. In terms of quality, it is comparable to the Rio Grande, but has 2.5 times more frames, which take 2.5 times longer to process. The Rio Grande should work - but if it doesn't work, then Orinoco will. If both methods are not enough for a particular episode, I have come up with a way to tweak the system manually.
Set all YouTube videos included in the article in 4K for the best playback quality - lower resolutions do not transmit picture quality.
I made two titles to demonstrate the two different resolution scaling options Topaz offers: Theia Detail (default settings) and Gaia-CG. Here is the output from Gaia-CG:
This is not a step-by-step guide describing the process of processing video in a way that anyone can handle. Writing such an instruction would be a separate project. However, I will give out enough information so that someone with a superficial knowledge of AviSynth can repeat both approaches.
The result of the work of Theia Detail:
Part of the frame disappears for a few seconds as the Bajoran freighter goes around the station. This is a unique error, I have not seen it anywhere else in the show, and it occurs quite often, regardless of the filters used. It always appears on the credits, and almost does not spoil the picture - there is no need to worry that random frames from the video will lose blocks.
Most of the videos are captured in at least two different ways. If you don't like Theia's output, try Gaia-CG. If Gaia-CG doesn't suit you, you might like Gaia-HQ - I've included only one example of how it works in this article.
Maximum warp
Rio Grande and Orinoco try to make as few compromises as possible when it comes to picture quality. Each stage of the process is thought out in such a way as to save all the details at the expense of storage space until the last coding, after which you are free to choose the degree of loss in quality.
All the clips shown here were encoded in 0.0 H.264, and then the resolution was increased frame by frame and without loss. I encoded these clips in H.265 format with a CRF of 6, which turned out to be too much. Live and learn. There is a subtle difference between CRF values of 14 and 20 in freeze frames, and 25 looks pretty ugly already.
Increase Resolution Models: Avoid Artemis and Increase Resolution by 200%
I have recorded at least one example for each of Topaz's upscaling modes, with the exception of Artemis-LQ and Artemis-HQ. These algorithms are completely inappropriate for Deep Space 9 - not that they give out some kind of perfect garbage, but they react poorly to the contents of this series. Consider that they are not suitable for the task. The result of Gaia-HQ's work did not impress me much, although I posted one example. There are two models left, Theia and Gaia-CG.
I've always liked Gaia-CG better, so most of it is done with this model, although I also demonstrated the Theia family of algorithms. They have configurable settings for noise, clarity, and detail restoration - all of which can be very helpful when trying to remove noise from an ugly spot on a wall or carpet.
Increasing the resolution by 200% should be avoided. It generates errors where there are none in the source, and the quality of the picture is significantly inferior to that obtained at 400% magnification. Please note that the restoration process has a peculiarity - it makes a seam on the right or left side of the frame. It can then be easily cut off, but this must be taken into account.
Topaz offers options for lossless upscaling through images, or through lossy MP4. All videos that you see are made through lossless conversion to PNG.
How many additional parts have been remanufactured?
The crazy idea that got me going down this route was that I can extract additional visual information by accessing the original DVD sources instead of the old backup files. Here is an example of my best attempt at increasing resolution from January:
This is how it was in the past.
While some improvements are made possible by low-level improvements to AI programs to increase the resolution, the program needs to get enough information to increase the resolution of the picture without turning it into a jumble. Imagine a shot where everything is rendered in DVD resolution and Odo's hair is in 4K. It would look very strange. The more details the program is able to extract, the more chances it has to produce something beautiful. Returning to DVD gave the program more material to work with.
This is how it looks today.
To give an idea of the progress, here are three frames of "Define" - what it looks like on DVD, the best magnification I could achieve in February, and the best result to date:
"Define" on DVD. Better than Netflix.
February 2020: better, but I thought it could be done better.
That's much better. Encoder Orinoco.
Here's my favorite "Design" shot from the entire series. In April, I was very proud of what came out of it. And it still looks good.
I still like it, but we can still do better. The
help of a properly tuned AI model is obvious. Improved handling may result in the model being cleaned up a little differently, however some body parts have changed shape.
The benefits are clear. Every time I compare an upscaled video made from old MKV files to a video made from DVD, the latter wins.
So how good is the result?
Careful processing and good resolution upscaling, in my opinion, can give Deep Space 9 the crispness of what is commonly referred to as HD content - although the picture is still limited by the NTSC color gamut, compared to later standards like Rec. 709 ". At the very least, deviating from the ideal, it looks better than any DVD you've ever seen. And in the best cases - and I consider the frame with Sisko one of the best - at least in the resulting picture, the detailing reaches the HD level.
The steps I took during the recovery have fixed some of the source's flaws, but not all. In the episode "The Warrior's Path" there is one sequence of frames that looks like something was spilled on the tape - how to fix that, I don't know. All coding results below were taken on the Rio Grande.
Warrior's Path: It Is Not an Illusion
The episode features two of the best battle scenes from Season 4. In the first, Sisko uses Defynt to rescue a Cardassian ship being fired upon by the Klingons. It can be seen that the special effects specialists experimented with re-showing the death of characters from an outside perspective - and this is not the case in "The Next Generation". Obviously, the special effects guys had a lot of fun with "Defynt".
The second large battle scene is shown below. It has a beautiful nod to the show's pilot where Kira bluffs to make a squadron of Cardassian ships believe the station is far better armed than it actually is.
But this time everything is done differently. First, the video is upscaled with Theia Fidelity with default settings, and then processed in Gaia-CG. The last one I previously used to take screenshots:
I've included characters' communication scenes in the review to demonstrate that these processing technologies work for more than just combat in space or spaceships. Deep Space 9 isn't just about battles filmed on a computer, but the show has some great examples of these battles, especially by 90s standards. Below is Gaia-CG:
Lost heaven
The Defiant fights the Lakota in this battle, while part of the Starfleet is attempting a coup. The clip specifically shows what happens when the Rio Grande does not quite correctly process the scene. For the most part, it works, but at the end it twitches noticeably. And I made sure that the case here in the Rio Grande - Orinoco handles everything perfectly smoothly. If you are not worried about a little twitching, then use the Rio Grande. If worried, try Orinoco.
Far beyond the stars
One of the benefits of science fiction is that show creators can easily get by with the issues of time travel. In this episode, Benjamin Sisko is abandoned in the 1950s - at least from his point of view - and he faces the racial injustices of that era. Resolution increased in Gaia-CG:
In the pale moonlight
Deep Space 9's greatest episode ends with a chilling monologue. Resolution increased in Gaia-CG:
The same scene rendered in Theia Detail with 10% sharpening, 20% noise removal and 20% detail restoration.
Sacrifice of angels
You knew we were going to return to this episode? In the episode "First Fleet" (below, rendered in Gaia-CG), the federation decides to fight the superior combined military forces of the Cardassians and Dominions. I like the maneuvers of Galaxy-class ships. From certain angles, they often appear thick and awkward, while from others they look very good. In "Deep Space 9" the situation with this is much better than in "The Next Generation".
The first attack prevents Defynt from making the gap he needs. Starfleet decides to try to break through the enemy front, although the enemy's forces are twice as large. I decided to render this scene in different ways. First - Gaia-CG:
Then Gaia-CG with CRF = 20. Here the quality loss turned out to be small, but I do not recommend raising this value more. The problem with using DVD as a source is that the quality starts to suffer after a small amount of detail is lost, and the information obtained from the interpolated and enlarged picture is not particularly good anyway. With a gradual decrease in quality, at some point, it drops very sharply. CRF = 25 turned out to be too much.
Gaia-HQ:
Theia Detail, default settings:
Finally, Theia Fidelity with 20% sharpness, 30% detail restoration, 20% noise reduction:
Although I've concentrated on the later seasons of the show, this coding and upscaling works in the early seasons. Just be aware that the early seasons source material available to us is not as good quality.
The demonstration is finished. Let's talk about implementation.
How to code the Rio Grande
For reasons not clear to me yet, the files that StaxRip creates using the above methods do not play normally in Windows Media Player or Apple Quicktime. But they play fine in VLC or MPC-HC. If the scripts below return an error, make sure the editor does not replace the quotes.
For Rio Grande encoding, you need to rip the DVD with DVD Decrypter and create a D2V index file using DGIndex or another similar program. Then fire up your favorite AviSynth interface, if you are using one. I personally use StaxRip.
The script for AviSynth is given below:
TFM()
TDecimate()
QTGMC2 = QTGMC(Preset="Very Slow", SourceMatch=3, InputType=2, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset="Very Slow", MatchPreset2="Very Slow")
QTGMC3 = QTGMC(preset="Very Slow", inputType=3, prevGlobals="Reuse")
Repair(QTGMC2, QTGMC3, 9)An alternative to the Rio Grande and Orinoco models, which I, however, did not test for long:
TFM()
TDecimate()
QTGMC2 = QTGMC(Preset="Very Slow", SourceMatch=3, TR2=5, InputType=2, Lossless=2, noiserestore=0.1, NoiseDeint="Generate", grainrestore=0.1, MatchEnhance=0.75, Sharpness=0.5, MatchPreset="Very Slow", MatchPreset2="Very Slow")
QTGMC3 = QTGMC(preset="Very Slow", SourceMatch=3, Lossless=2, InputType=3, TR2=5)
Repair(QTGMC2, QTGMC3, 9)Such a script seems to give out a little better quality than Orinoco Standard in terms of ripples.
If your picture is distorted, try rolling back to TR2 = 4. The same applies to Orinoco, only it will need to be removed
InputType=2 InputType=3from the Rio Grande.
That's all. You will also be able to insert noise back into the video while these scripts are running. The noise added to one of the QTGMC passes will remain in the picture after recovery. The Rio Grande does not give a 100% perfect picture, yet at 29.976 fps it works almost perfectly on all scenes that I tested except one. If he has more problems than I expected, that's exactly why I made Orinoco. Technically, it will be possible to render a specific scene at 59.94 fps, and then include it in the 23.976 fps version by assembling a VFR file during compilation of the resulting images into video.
QTGMC runs are required to improve the picture, and although I'm open to suggestions, so far nothing I've tried has come close to the quality this script produces. It can be tweaked in terms of the depth of impact on the picture and the processing time, but you will notice that when you go from Very Slow to Slow or Medium, the quality deteriorates quite noticeably. If you love noise in the picture and don't mind spending time looking for artifacts (or know how to get rid of them in another way), then Slow is your choice.
The only drawback of this way of running QTGMC is that sometimes it produces ripples in some part of the scene. It's easier for me to ignore it, and all the solutions to this problem that I tried to find, in the end, harm even more. And although I would like to get rid of this, for the time being it belongs to the category of a tolerable "feature".
The sequences are
InputType=2 InputType=3very important. Running a single copy of QTGMC will not have the same effect. Running two copies with InputType=2will not give the same effect. There are nine possible ways to combine input types when running image recovery, and I've tried all of them. The 2,3 combination works best - from the episode "The Emissary" to "What You Left Behind."
I also tested all 25 Repair functional modes that AviSynth supports. You need a mode, 9. If for some reason it doesn't suit you, try it, 1. Most of the others have practically no effect on the result, and some of them even break the picture in an interesting way. I've checked all of them many times just to see how the final result changes depending on the mode I choose.
How to encode Orinoco
The Orinoco is not easy to code, but it does a lot better than I could have expected.
It all starts with the fact that instead of one file, we create two. Code the first version of the sequence using the following script:
TDeint(mode=1, type=2, tryweave=true, mtnmode=3, full=false, ap=10, aptype=2, slow=2)It instructs TDeint to issue a doubled set of frames, interpolate the kernel [kernel interpolation], try to recover the frame using weaving, if the result is less artifacts than with deinterlacing, and apply deinterlacing to those frames where there is interlacing. This mode preserves progressive frames included in the NTSC source. The second best option after Type = 2 was Type = 5, but it posed more problems than it solved.
Now let's put aside the resulting clip and turn to another. It needs to be run through the following script:
QTGMC2 = QTGMC(Preset="Very Slow", SourceMatch=3, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset="Very Slow", MatchPreset2="Very Slow")
QTGMC3 = QTGMC(preset="Very Slow", prevGlobals=”Reuse”)
Repair(QTGMC2, QTGMC3, 9)The script is similar but not identical to what we used for the Rio Grande. That QTGMC implementation runs in progressive mode. In the case of Orinoco, we need to double the frame rate, from the base 29.97 fps to 59.94 fps. After the frame rates of QTGMC and TDeint match, we start recovery on them as well. If you run two clips with different frame rates in Repair mode, you will immediately see it - a strange moving seam will be visible on the screen.
This is, of course, as I understand it, an upside-down approach to video editing - but my madness makes sense. The reason people don't usually insert a bunch of video frames in order to bump up the VFR from 23.976 / 29.97 to 59.94 fps is because cards interpolated in this way that increase the frequency can introduce visual errors and other problems. There is catch-22: if you do not include them, the output file will turn out to be jerky. If enabled, the change of frame boundaries will sometimes look very strange, or different parts of the frame will merge together. The situation is so-so.
But there is a way to fix it.
clip1=FFVideoSource(“C:\DS9S6D2\Sacrifice-TDeint.mkv”)
clip2=FFVideoSource(“C:\DS9S6D2\Sacrifice-QTGMC-ToPairWithTDeint.mkv”)
Repair(clip1, clip2, 9)We create an interpolated QTGMC file to make it a crisp and clean source for the Sacrifice-TDeint. As I said, I use QTGMC because I have not found another such method for getting clean video. Use the Very Slow setting to reduce noise in the picture.
On the resulting video, you can optionally run QTGMC again, but then the video will look too processed. Theia models are able to somewhat compensate for this effect thanks to a fairly effective noise reduction algorithm.
In terms of quality, Orinoco and Rio Grande give approximately the same result, and I often cannot even notice the difference when comparing them frame by frame (apart from interpolated frames, of course). I prefer the Rio Grande as it is simpler and faster, but I can't guarantee it will work on all 176 episodes of the show. If you come across an episode in which the movement is shaking or jumping, try to drive it through the Orinoco instead of the Rio Grande.
Disadvantages and problems
Orinoco and Rio Grande are not perfect. I am still looking for a method to eliminate ripples. This is a small problem, but practically the only one to be solved. By far the best solution would be to use MCTemporalDenoise, however this method seems to be completely single-threaded and also gives me other quality issues.
If you want to experiment with MCTemporalDenoise, I recommend setting the number of pixels to 7 and the effect strength not exceeding 2-3. The ripples do not completely disappear, but in some places they get better. If you don't like the resulting quality, get around Theia's resolution upscaling models and try applying 10-20% noise reduction.
My goal was to create a universal template for the entire project, but I will be recording any changes that need to be made to the models in order to cope with problematic episodes.
Important Notice About Reading Season 1 DVD
StaxRip behaves very strangely when loading a D2V file containing 98-99% of the video. He changed the frame rate for me more than once - from automatic 29.97 to 23.976. And in such cases, it overwrites the file generated by DGIndex. If you apply the Rio Grande model to a 23.976 file, you get 19fps content that looks really bad.
I'm still trying to figure out why this is happening. On episodes with 98.5-99.5% of video, you can usually assume that only titles are going with a frequency of 29.97, and encode these episodes by omitting the TFM / TDecimate segments.
Next time ... on the Deep Space 9 resolution enhancement project
After finishing the project, I can start ripping and encode episodes from discs. I'm going to write a training kit with instructions on how to do this, do a speed test for workstations, and solve problems with specific episodes. I came up with a completely free method for increasing the resolution, which, although it does not work as well as Topaz, obviously improves the picture. I also want to deal with PAL.
ViacomCBS doesn't consider Deep Space 9 worthy of remastering. I think they are wrong. The quality that can be squeezed out of DVD is not ideal, so I do not call the results of my work 4K or UHD, although in such projects people often throw such terms. You can stretch Deep Space 9 to 4K, but you can't squeeze that quality out of a DVD source. The solution is not perfect, but it is many light years ahead of all previous options. If it turns out that the Rio Grande often makes mistakes with a frequency of 23.976, we will find a solution for these epiodes. There is every reason to believe that the new Nvidia GeForce RTX 3080 card will seriously accelerate this increase in resolution, which will facilitate the conversion to a frame rate of 59.94.
No other Star Trek fanatic will have to wait for ViacomCBS to find the financial value of investing in its own archive. DVDs and software will cost you money, and you will have to spend a lot of computing time on the project. On an RTX 2080 card, it takes about 8-11 hours to increase the resolution of one episode at a frame rate of 23.976. A frequency of 59.94 takes proportionally more time, although encoding can be paused and continued as needed. To date, the best rate has been 10-14 hours per episode. This is about 2464 hours, or about 103 days for the entire series. The project is approximately on the border of what is possible for an individual.
Again, you can simply run the DVD through AviSynth to dramatically improve the quality and resolution of individual episodes. I, for example, thought about skipping the 1st and 2nd seasons altogether. I also expect Ampere to be able to significantly improve the efficiency of its AI engine.
Will most people do this? No. Most people will watch the series on Netflix, where the quality will be much lower than on DVD.
But from today, no one is obliged to do this. Now you can see the work of visual effects specialists and actors as they intended. Orinoco and Rio Grande are not perfect, but they show how much quality you can squeeze out of DVDs from the late 90s. GPUs are getting faster. AI0 resolution boosters are getting smarter. Even if your current computer or GPU cannot cope with such a project, the next one may be on the shoulder.