Killzone Shadow Fall is one of the Playstation 4’s most visually impressive titles, serving as a great inspiration and showing to what Sony’s new console is capable of. Long before the game came out, the developers released a series of PDF’s and technical documentations explaining their thought processes and creation of the game. Some of these documents for the public and press served multiple purposes, for example demonstrating the number of the PS4’s CPU cores available for games developers (six AMD Jaguar cores of the eight total).
Killzone Shadow Fall in single player runs at a native resolution of 1080P, 30FPS. It is important to remember that based on the documentation Guerrilla games released that their MRT (Multi Render Target) was around 800MB. For the sake of online, the team decided to do something a little different and instead tried out a new technology known as Temporal Reprojection.
Temporal Reprojection is only used in the multi-player mode of Killzone Shadow Fall. In the teams own words, it combines the pixels and motion vectors from lower resolution frames, combining them to create a full 1080P image. Immediately, internet forums were a frenzy with gamer’s claiming that this isn’t true native 1080P.
Back In November the 4th, Michiel v.d. Ludwig spoke about TSSAA (Temporal Super Sample Anti-Aliasing). “We use FXAA + TMAA now, will dig deeper in a future publication, but it does complement each other”. We covered this back in our Killzone Shadow Fall vs Killzone 3 technology article, so for more info on this click here. Suffice to say, he’d previously spoken about the usage of the previous frame of animation to create the Temporal Super Sample Anti Aliasing.
What is resolution scaling?
Although it’s likely many already understand what scaling is, it’s important that we establish the basics. If you take an image and scale it bigger in both X/Y, then essentially you’re stretching the image. So, yes, you can take a 1280x720p image and stretch to 1920×1080, but you’re not inventing new detail. Essentially speaking the pixels are ‘guessed’ and therefore the blown up image becomes much blurrier than the original. You can test this yourself by taking a random image into photoshop (or your image editing application of choice) and making the image bigger. All you’re doing is stretching the image.
In this case, for games the console or PC’s GPU will scale the image up to stretch it for the TV. Or in the case of a movie, the TV will usually do the stretching for you – which is why old games consoles look worse on modern HDTV’s than the older CRT’s. If you’re PC gaming, you could however force the GPU to simply use a 1:1 mapping. So if you’re using a 1280x720P image on a say a 1080P monitor, you’ll get borders around the image as you’re telling it to only map directly to pixels from the source, not make new ones up by scaling it.
Temporal Reprojection is very different to scaling, and uses a lot of calculations and prediction to understand what will happen in the future based on previous frames of animation. This is extremely important, as using the previous frames as reference points the software can make a good accurate guess at what will happen next.
Let’s use Thief on the Playstation 4 as an example to show off frames of animation. Here we have a fairly early part of the game, and you’ll notice that I’ve labeled the frames of animation as ‘one’ and ‘two’. This simply means that the second frame is quite literally the frame after. You’ll notice that the two frames of animation are pretty damn similar to each other. There’s not huge differences in lighting, position of characters and so on.
In the case of games, they run a variety of different frame rates – but in this example case, Thief runs at 30FPS on the PS4. This means that for every single second, providing that you don’t get any slow down, effectively the GPU will traditionally render 30 of these still images per second. This means that in the case of a 30FPS game, each frame will be on screen (in theory) for about 33.3ms. This is assuming you’re using a traditional rendering technique like forward rendering. Now we have a basic understanding of how graphics work, let’s move onto the Temporal Reprojection.
For a more in-depth and far more detailed explanation on frame rate, resolution scaling, and image quality checkout our guide here.
What is Temporal Reprojection?
Guerrilla Games have given a little bit of information regarding Temporal Reprojection on their official blog so we’ll go over that firstly.
“So, in a bit more detail, this is what we need for this technique:
We keep track of three images of “history pixels” sized 960×1080
The current frame
The past frame
And the past-past frame
For each pixel we store its color and its motion vector – i.e. the direction of the pixel on-screen
We also store a full 1080p, “previous frame” which we use to improve anti-aliasing
Then we have to reconstruct every odd pixel in the frame:
We track every pixel back to the previous frame and two frames ago, by using its motion vectors
By looking at how this pixel moved in the past, we determine its “predictability”
Most pixels are very predictable, so we use reconstruction from a past frame to serve as the odd pixel
If the pixel is not very predictable, we pick the best value from neighbors in the current frame”
So eagle eyed readers will notice this is in many ways a logical extension to the TSSAA mentioned above where the previous frame was used to act as an anti-aliasing. This is an extremely effective rendering technique in terms of performance vs the fidelity that it creates. It it works and the guess is correct, you’d be hard pressed to find a difference between it and a regular 1080P natively rendered image. If it is “wrong” however you’ll get a vertical line because the two images don’t quite match up.
By combining the two 960 width frames (which is half the width of the full 1920 width) the game can combine these two images and create a full 1920x1080P image. So in other words, it simply keeps a copy of these images its memory buffer (storing that part of the scene) for reference with the next image.
The reason for all of this is simply performance. I’d not personally be surprised, particularly as the generation goes on if more games begin to employ this technique as in effect it means that you’ll get more out of the console. This isn’t a new technique in the sense of it being created for Killzone, and has been used and known about for shadow mapping (wiki for shadow maps) for awhile now.
The big question this starts to raise is how will studios deal with such attention from the public. Many studios are already feeling the pressure, and I do wonder how healthy it is for gamer’s to focus so much on the rendering techniques and the resolution. While the more technically advanced gamer may understand the techniques studios employ, and the reason for the graphical sacrifices there are also a great many that simply won’t.
Sources:
http://levelofdetail.wordpress.com/2008/11/06/pixel-correct-shadow-maps-with-temporal-reprojection/
http://www.cg.tuwien.ac.at/research/publications/2007/Scherzer-2007-PCS/
