ExtraSS Part 3: Frame Extrapolation
So this is all about Intel's frame generation, right? That's what is the headline from the paper (which I missed when it was first released last year, the demo videos are giving it the second rounds, thanks for flagging
@Dakhil). So why all the talk about the upscaler improvements?
Because one of Intel's goals is to unify the upscaler and frame generator. FSR and DLSS do frame generation as a second pass, potentially with it's own inputs. In the case of FSR and DLSS, they both need optical flow data. FSR uses the GPU and async compute to do it (which all modern GPUs have). Nvidia uses their specialized optical flow hardware (which leaves the GPU free to do other stuff). It's unclear if DLSS's AI model is used for frame generation (it probably is), the overall algorithm is separate.
Intel wants to feed the same data into both frame gen and upscaling, ideally as the same algorithm. Essentially, Intel wants frame gen to be upscaling from nothing... except it's not nothing. It's from the G-buffer. That's the brilliance
and the cheat. Let's start with the brilliance
The G-buffer is an early step in the rendering pipeline. Not only does it provide the upscaler with basic 3D information, it's the core of normal rendering anyway. AMD and Nvidia are taking complete frames, which are fully rendered, but just 2D images, and then trying to generate a third 2D image between them. Intel wants instead to take the G-buffer, and early step in the rendering pipeline
and use past rendering to draw the colors over the shapes.
Intel's upscaler already understands how to use the G-buffer to correctly move the past frame's pixels over the actual objects moving in scene. WIth a new G-buffer, they can just do that without any pixel data at all, exclusively using last frames colors. And where FSR and DLSS need to do optical flow for the whole frame to do generation, the G-buffer means that optical flow only needs to be done for the parts of the screen that have things like shadows in them. And the
shading refinement model already handles that, at low resolution even.
Which brings us to the cheat. The cheat is... you still have to
make the G-buffer.
Hold on, let's go back a second. DLSS frame generation was, in some ways, a response to game developers having trouble with CPUs. CPUs are getting more and more cores, but game development is still locked into single core technologies. Meanwhile, GPUs are really good at using lots of cores and have continued to get more and more powerful. 4k is as far as any sane person wants to go for resolution, so gamers would love to push frame rates up to get more
smoothness, more
perceived detail.
But the games are being CPU limited, leaving the GPU with horsepower sitting there untouched. Along comes frame generation, which uses up the extra power in the GPU to give you smoothness even when the CPU is busy.
Intel frame extrapolation is solving a different problem. ExtraSS isn't giving you extra frames using excess GPU power. Instead, ExtraSS depends on the
CPU being ahead of the GPU. It still needs the CPU to generate all the stuff it normally does for a frame, but lets the GPU skip all the shaders and jump right to a completed frame.
It really is upscaling from 0, or at least close to 0 - in that the CPU still has to actually run at the high frame rate, and initiate the frame, but then the upscaler is like "you know what, don't even bother with color or lighting or shading or textures or any of that shit, just draw me a freakin' pencil sketch and I'll do the rest." And pencil sketch isnt too far off. Intel is explicit that for "extrapolated frames" you generate the G-buffer at really low resolutions.
The advantage of this technique isn't just "new frames without added latency." Its "new frames, plus all the reduced latency you'd expect from high frame rates." Since the CPU runs at the full frame rate, the CPU can sample the controller at the full frame rate as well!
And it opens up the possibility of only doing
some of the CPU work every frame. Think about animations - you already see animations running at lower frame rates in the background, in order to reduce CPU load. On these "extrapolated" frames, games could shave down just animations really near the camera. Only run physics on "real" frames.
Frame generation - even frame extrapolation - is a bad name for this technique. Intel isn't generating frames that were never rendered. And their not extrapolating frames that haven't been rendered either! The frames are rendered all the way up to geometry. It's really
neural shading. Intel is using AI to take the first step of the rendering pipeline and then
estimate the shading that would have happened by reusing shading from past frames, the same way that temporal upscaling reuses past pixels.
It's a clever technique that will requires substantially more engine integration, and will benefit a totally different class of titles than the ones that benefit from current frame generation techniques.
If I had to bet, the long term advantages of this technique are much higher than Nvidia and AMD's approach. Currently games are CPU limited not because they're out of CPU power, but because multi-threading is hard, and retrofitting it into existing engines is even harder. But intuitively, it seems like games ought to be able to take advantage of more cores - like, every enemy should be able to run their AI and animations and even basic physics for non-colliding bodies on separate threads.
But GPU growth is slowing down because the node shrinks are slowing. Assuming that games will be CPU limited might prove to be short sighted as engines catch up to multi-core designs, while GPUs put more and more power into things other than shader performance (like AI). Intel making every other frame free on the GPU side, as long as the CPU does the work, might be the better bet.
The fact that it improves their upscaler as well is just a nice addition.
gdcvault.com
