As the launch of the next generation of consoles approaches, Insomniac Games has begun posting more material revealing details about Marvel's Spider-Man and Marvel's Spider-Man: Miles Morales. Both projects use ray tracing technology. In the opinion of Alex Battaglia from Digital Foundry, she looks very dignified, if not excellent. Not bad for a start.
However, while reading comments on the Internet, he encountered a considerable amount of criticism in relation to the implementation of the technology, as well as to the frame rate and resolution. So in a new video from Digital Foundry, he decided to introduce some sort of tutorial on ray tracing on next-gen consoles and explain why it looks like that in Marvel's Spider-Man.
So, first things first. Let's discuss exactly what ray tracing is for in Marvel's Spider-Man on PS5.
At 30FPS at 4K resolution, ray tracing looks great. At the same time, as is customary in Insomniac games, dynamic resolution may or may not be used. This is where ray tracing is applied to in-game reflections like those seen on car doors.
Or in this screenshot, where the hero and the city in the distance are reflected in the building window.
Ray tracing is also used for background shading (Ambient Occlusion, AO), according to the press release. It is more difficult to identify in screenshots or videos. Typically, AO simulates shadows from indirect lighting in areas where light reflected around the scene is more difficult to reach.
As far as comments on social media and forums go, there is a disappointment in the quality of ray-traced reflections, which, at first glance, look only at a quarter of the basic resolution. But they clearly have filters on them, so it's hard to say for sure.
Reflections are shown in 1080p, while the rest of the image is shown in 4K. Commentators complain, for example, that the leaves in the reflection are more rare or even absent than those that can be seen on the "real" tree. Or that some in-game objects, such as cars or pedestrians, may not show up in reflections. Or the lack of shadows. Or neglecting small details like the Spider-Man suit.
In fact, all this is not surprising and refers to the forced measures, it would seem, taken for granted. This is due to the fact that ray tracing is performed in several steps, each of which affects the overall performance, which has a very limited budget.
Let's pretend that we are the very developers who need to create the most realistic reflections within a given budget.
Let's divide the ray tracing process into four steps. Let's say we need to meet 8ms to create reflections for a 30 FPS game at 4K resolution.
Let's move on to the first step. For ray modeling, the GPU must have an understanding of the scene and geometric objects on and off the screen. In the first step, we create an easy-to-read version of the game scene. The more objects it contains and the more dynamic they are, the longer this step will take.
In the second step, we fire rays into the structure created in the first.
To see if the beam is hitting anywhere, surround all objects with invisible hitboxes. Thus, a ray that hits the scene moves through it until it registers a hit into the geometry. The more rays we fire, the further they go, or the more hitboxes we hit, the longer it will take for this step. This part of the ray tracing pipeline can be hardware accelerated on consoles and PCs.
The next step is hit shading or pixel color selection.
The more hits are shaded or the more difficult the shading is, the longer it will take for this step.
The third step gives us the colors and shades we want, but they can have poor transitions or be noisy. This problem is solved at the fourth stage. As a rule, the more accurately the result of the third step is cleared in the fourth, the longer the process takes.
The entire pipeline needs to be laid in a total of eight milliseconds, and at the beginning we are far from that goal. We need to tweak the steps to keep it time-consuming and also to make sure our reflections still look good and match the game design. This can lead to a decrease in the quality of reflections in some areas and an increase in others.
One of the first optimizations we'll use is reducing the resolution for ray tracing. This will limit the number of rays coming out and reduce the time required for the second and third steps.
As you can see in the screenshot below, the reflection looks almost a quarter of the original resolution. This optimization is performed not only for console GPUs, but for all GPUs in general, since ray tracing is incredibly resource intensive.
Let's take a look at a screenshot from The Ghost Runner demo with 4K ray tracing enabled on an RTX2060 Super graphics card. Here we see some differences compared to Spider-Man. Unreal Engine 4, on which the game is created, uses a ray tracing method that is slightly more expensive in time, but physically more accurate than in Spider-Man.
The RTX 2060 Super works in a similar way to the RTX 2070, has the same memory bus bandwidth as the PS5 and the same 8GB storage. The RTX 2060 Super is likely to be a great starting point for comparison with the PlayStation 5 GPU.
When the scene is rendered in 4K along with reflections, and the amount of reflections on the screen is as small as in the case of the static puddle above, the GPU barely fits into 30 FPS. If there were more dynamics in the scene, the frame rate would definitely be lower. Reducing the ray tracing axial resolution by 50% and thus the total resolution to 1080p increases performance by about 27%, leaving free resources in the GPU to increase the frame rate up to 30 FPS.
More dramatic results can be seen in other scenes. Like, for example, here. The reflective glass surface occupies almost the entire screen. In this case, reflections at native 4K resolution result in a frame rate of 21 FPS. At the same time, lowering the resolution of reflections to 1080p increases the frame rate by 58% to 33 FPS with the required 30 FPS.
So, on a GPU with the same performance as the PS5, you need to use lower resolution ray-traced reflections to be able to maintain a consistent 30 FPS frame rate at an overall 4K resolution. It then becomes clear why Spider-Man will only use reflections at 1080p to reduce the cost of ray tracing.
Let's go back to our calculation example. We've reduced the resolution of the reflections so that the second and third stages are now less time consuming. But we still do not fit into the given budget for the duration of the frame.
Let's try to shorten the time of the first step by decreasing the dynamism or the number of objects in relation to which ray tracing will be performed. So, in the screenshots shown earlier, some dynamic objects may have been missing in reflections. Or here, in the Miles Morales gameplay video, in crowd scenes you can sometimes notice that some of the NPCs are not reflected on surfaces.
We know from experience of other developers that drawing many characters on the screen can be very resource intensive. For example, some scenes in Metro Exodus with a lot of characters on the RTX 2080 Ti take over 4ms to render. In our case, this will be half of the allocated budget.
So, reducing the number of dynamic objects in the first stage of ray tracing speeds up the process, and then we fit into the allotted 8 ms.
However, the result may not be as impressive as we would like.
For example, take a look at these screenshots of Battlefield V.
As we can see above, buildings appear and disappear in reflections. This is not the most desirable effect, so we will increase the distance that the rays can travel at the second step, as well as the distance at which objects are included in the structure at the first.
Returning to our frame from Spider-Man, we can see that the reflections show objects that are very far away. It would probably be very unpleasant if objects on the horizon were absent in reflections or constantly blinking, appearing and disappearing.
Therefore, taking into account the buildings located far from the source of reflection in the reflections, and increasing the distance that the rays travel, we will have to sacrifice something else to pay off.
Let's go back to our calculation. Deciding to take into account distant objects in reflections, we again found ourselves outside the limits of our budget. Therefore, we will once again review the pipeline for possible simplifications. Perhaps the second step will bring us back to the desired 8ms.
Let's look at our city screenshot again. In the reflection on the window, you can see fewer or even missing leaves compared to how they appear on the original tree model. Leaves are classified as transparent here, and for transparent objects, the second step of the pipeline takes longer. We know this from experience with Battlefield V, where particles and reflections make certain scenes extremely expensive.
For Spider-Man, we can reduce the number of leaves on the trees in the reflections before the second step. We will also make it so that the particle effects are not processed by ray tracing, as here, where the fog in the reflection is not displayed.
Now we almost fit into the frame budget, but still slightly exceed it.
The last optimization can be done in the third step, where we use shading.
Let's go back to our screenshot. Have you noticed the lack of small details in the reflection of the Spider-Man suit? Which are visible on his real suit? If we simplify the material in reflection, it will take less time to draw it.
There is nothing wrong with that, as many games that use ray tracing, such as Wolfenstein: Youngblood, use the same technique.
As a result, we fit into the allotted 8 ms. So, we improved the quality of reflections in some aspects, for example, adding reflections of distant objects, but at the same time resorted to certain simplifications, such as lowering the resolution of reflections.
Hopefully this example has helped us get a sense of the dilemmas game developers face when dealing with limited performance budgets and costly technologies like ray tracing.