Frame Generation, an explainer
What the hell is Frame Generation? Don't games already generate frames?
Well, yes, sorta. Games
draw frames based on
game logic that is currently executing, and the
geometry, textures, and
effects that the GPU is processing.
Frame generation is the process creating additional frames
without any of that. No game logic, no geometry, no game shaders, nothing.
Wait, what? How?
I'm going to tell you, but first, a word from our sponsor,
Hello Fresh Frequently Asked Questions
Is this DLSS? FSR?
DLSS is a suite of tools. One of those tools is Frame Generation, which Nvidia calls DLSS-FG. This tool was added in DLSS 3, so it's sometimes
called DLSS 3, but that can get real confusing real fast.
FSR is a similar set of tools, and one of those tools is also Frame Generation, which AMD named FSR-FMF, or
Fluid Motion Frames. It was also added in version 3 of FSR, and is sometimes called FSR3. Again, confusing.
"I saw that FSR frame generation, in Quality mode-"
Lemme stop you right there, hoss. FSR-FMF doesn't have a "Quality" mode. Like I said, FSR is a suite of tools. One of them is Super Resolution, which
does have a Quality mode. Remember what I said about the naming being confusing?
Same with DLSS. Frame generation is either "on" or "off", no other settings apply.
How it works, the short(ish) version
Normally, when a game draws a frame, it sends it to your screen, then starts the next frame.
With frame gen, the frame doesn't go to your screen immediately. It gets saved for later. Meanwhile, your game gets on to drawing frame #2. That doesn't get sent to the screen
either. It also gets buffered.
Now your game is starting on frame #3. That's when frame generation gets to work. Frame gen takes frame #1 and frame #2, and tries to figure out a frame that would go
between those two frames.
No game uses 100% of the GPU 100% of the time. That's part of frame gen's secret. By only using the old frames, and these idle GPU resources, frame gen (hopefully) gets you new frames effectively for free. While the CPU is busy working on frame #3, frame gen quickly assembles a frame #1.5.
Frame #3 gets buffered. Frame #1.5 also gets buffered. And finally, the original frame #1 goes out to the screen. Game engine goes on to working on frame #4, while frame gen gets to work on frame #2.5, using the buffered copy of frame #2 and the frame #3 it just got.
So far, we've done a lot of work, and we've still only put one frame on the screen. But this is the moment when the magic happens. Frame gen stops what it's doing for a second, and puts frame #1.5 on the screen,
before frame #4 is done. Then when frame #4 comes in, frame #2 goes out.
When all works
exactly right, you get double your frame rate. Imagine a 120Hz display. 120 times a second, tick-tock-tick-tock, the display refreshes.
Without frame gen, say you are running a 60fps game. Every
tick you get a new image on screen
, every
tock the image just stays there. Turn on frame gen, now you get the same frames you always got on the
tick, but new generated frame every
tock, hugely improving how smooth the image looks.
That wasn't short at all
Believe me, it could get longer. In fact, it will!
Also, I can immediately see some problems
Turns out, you're right about all of them!
The Problem(s) of Frame Generation
Ghosting/Artifacting
How good are these generated frames? The answer is, pretty good but not perfect.
Here is an example from Spider-Man. The circled area is a big blurry glob around Spidey's leg. You can see smaller but similar artifacting around his hands.
Latency
Latency is just the measure of the time between pressing a button and something happening on screen. Frame generation depends on holding frames so you can make new frames in between then. Obviously that increases latency significantly.
Stutter/Pacing
Sometimes, frame gen won't finish in time, before the screen refreshes. When that happens, frame gen has to decide when to send out the frames it's holding - which means it might need to throw out work it's done on generated frames and start the process over again - or continue to hold the existing frames, causing a frame drop, and increased latency. Both these things will be experienced by the player as microstutter, and frame rate instability.
The Solution (singular) for Frame Pacing
Frame gen offers one solution to
all these problems - only use frame generation
when frame rates are already high.
Ghosting/Artifacting
When you start from 20fps, objects can move a
lot in a single frame. That makes it harder for frame gen to guess the right thing between the two frames, causing more visible artifacts. And with a lower frame rate, those artifacts are on screen longer, making them easier to see.
But if you start from 60fps, objects can't move as far each frame, so frame gen can make smarter choices about the generated frames. And because frames are only on screen for 10ms or less, the artifacts don't stick around long before being replaced by a clean, natively rendered frame.
Latency
Frame gen needs to buffer two frames, so you get 1-2 frames of latency, at the native frame rate. If you start with a 20fps game, and add frame gen, thats
100ms of latency. Massive.
But if you start from a 60fps game, that's only 33ms of latency. The higher the base frame rate, the lower the latency that frame gen adds.
Stutter/Pacing
Once you get past 90+fps, frame pacing problems are almost impossible to notice. 120fps means a frame is only on screen for 8ms. 90fps is 11ms. That difference is so tiny, that you don't detect the stutter. It's just smoothness.
The Unsolvable Problem of Frame Generation
What if there are no spare GPU cycles for frame gen to run. What then?
Well... nothing. Frame gen needs spare GPU power to run. This is (one of) the key things to understand about frame generation.
Frame gen doesn't exist to help small, overloaded GPUs. Frame gen exists for big, underused GPUs.
The speed at which GPUs are improving is much much faster than CPUs are improving. And where CPUs are improving is by adding more cores. Game engines are exceptionally bad at using more cores.
Nvidia and AMD would
really really like you to buy their big new GPU. But if you're not able to use all that GPU power, why would you? Enter frame generation. FG could slurp up all the unused power inside the GPU and give improvements even if the games themselves couldn't keep up.
Why FG isn't good for Switch NG
Hopefully by now it should be obvious. I'm not talking about what's technologically
possible. I'm talking about what's good for the system.
It wasn't designed to solve the sub 30fps problem, and is pretty bad at it.
See "problems" above.
It doesn't work well with small GPUs
A medium sized GPU running at 95% efficiency might have 1 spare TFLOP of performance in it's occasional idle moments. But that tiny bit of spare power that frame gen is trying to slurp up is like the entire performance of the Switch NG's GPU! Even if Switch NG is idle for significant portions of time, it's an open question of how much frame generation it would be able to do.
It doesn't work well in consoles
This is kind of a subtle point, but bear with me. A PC game isn't optimized for exactly your combination of GPU and CPU, and its exact load is going to depend on the settings you put into it. There is plenty of opportunity to create PC settings which leave room for frame gen.
But with a console, you've got a limited performance budget, dedicated hardware, and fewer tweakable settings. Poorly optimized games won't be able to use frame gen as a "crutch" because there won't be idle GPU power lying around, and well optimized games will be pushing the hardware to its limit.
You might be able to imagine the sort of graphically simple game that is running at 60fps, on Switch, but has the CPU pegged to the wall, while leaving plenty of GPU power idle. But does that sort of simulation heavy, low graphics game really going to play so much better at 120fps?
Why FG probably won't be on the Switch NG, Part 1
Now we are talking about the technical hurdles. And to do that, we need to do a deeper dive on Frame Gen itself.
We understand how frame gen works at a high level, but the question we skipped ironically, is - how does it decide what to generate? We know what it's inputs are - it's starting with two natively rendered frames. but how does it figure out what goes between?
Optical Flow
Imagine an Instagram filter for a second. One that gives you a pair of cute little faerie horns, say. When you tilt your head to the side, the horns follow. When you lean back, the horns tilt up, when you twist your head one horn gets larger the closer it gets to the camera, the other gets smaller.
How does it do that? How does it know how your head is moving in 3D space when it only has 2D pictures of your face?
The answer is pretty simple, it’s watching your eyes. Eyes get bigger, you’re closer to the camera, eyes get smaller, you’re further away. One eye gets larger, the other smaller, you’re twisting your head, the shape of your eyes seems to get wider at the bottom, you’re tilting your head up.
This is a simple example of something called
optical flow. Take two images and try to figure out the 3D motion between them.
When frame gen starts working on making a generated frame, it’s first step is to perform an optical flow analysis of the two frames to figure out how to objects in the scene are moving.
Optical Flow? Sounds familiar
If you’ve heard folks talk about Nvidia’s OFA, or
optical flow accelerator. Nvidia has custom hardware for these sorts of operations. In the RTX 40 cards, Nvidia increased the size of that OFA in order to improve frame generation. The reason should be pretty obvious by now - it's about being able to do optical flow at 60fps. The old, RTX 30 OFA is likely fast enough for 30fps games, but frame gen doesn't like low frame rates.
AMD doesn't
have optical flow hardware, so instead it uses the existing GPU to do this analysis.
Interpolation
With two native frames, and optical flow data, it's time to actually generate the "fake" middle frame.
FSR does this with hand-written code that runs on the GPU. DLSS uses AI. Both are trying to do the same thing - to shift pixels so they are halfway between the first frame and the second frame, but do so in a way that doesn't make objects appear to break up into a smear of pixels. You want Spidey's hand to stay looking like a hand. Otherwise instead of
more frames what your eyes actually see is
lots of motion blur.
Why it won't be on Switch NG, Part 2
Performance
It seems that Switch NG won't have the larger OFA that Nvidia introduced in their newer cards. That leaves frame gen having to use the GPU for optical flow, like FSR does. And Frame Gen doesn't like low frame rates.
For high frame rate frame gen, we're asking the GPU to not only render a game at 60fps, but to leave enough resources on the GPU to do optical flow at 60fps, and to interpolate at 60fps.
What about lower frame rates? What if I don't care about quality?
Well, I would suggest that most developers, who are going to control whether or not frame gen is available,
do care about quality.
But also, why not... lower the settings? There are other ways to increase frame rates while sacrificing image quality. Remember, frame gen
only increased smoothness. It doesn't do anything else that extra frames do in video games. So if the resulting product is
less smooth then why use it? Here is a quick screenshot of the main character from
Forspoken on the ROG Ally, with frame generation off
Now, if you've got a good eye, you can tell that this is a YouTube clip, and that there is video compression here, and you would be right. You might be about to argue that any comparisons might not be valid, because YouTube artifacts are getting in the way. Hold your horses. This is what happens when you turn frame gen on.
I guarantee that compression artifacts aren't why
she doesn't have the top of her head. With low frame rates as inputs to frame gen, it can't easily tell where the edges of objects are, and so a little head move by Freya and frame gen just erases her whole skull.
Which is not to say no games will ship with some form of frame generation. I'm sure someone will do it. But will it be
common or
halfway decent? Probably not!
Appendix: Anti-lag/Reflex
Nvidia's marketing has especially pushed this "lag free" narrative around frame generation, so I wanted to take a second and talk about that.
Frame gen creates 2 frames of lag. Period. It always, always, always does.
However, both AMD and Nvidia have introduced new lag reduction technologies, and both require that these lag reduction tools be on when using frame gen. The idea is that these tools can reduce latency in other places, to compensate for the latency added by frame-gen.
Sometimes this works. Sometimes it doesn't. Most of the time it does
okay. But these latency technologies can be turned on without using frame gen, and they only work if the game
has latency to fix in the first place.
Nvidia's Reflex solution - the only one available on Switch NG - involves forcing the GPU to be slightly underutilized all the time. You can see that this is part of Nvidia's move to take advantage of the lots of extra power that their new GPUs offer, even if games can't use it directly. You can also see how that might be a bad fit for a small device which doesn't want to waste a drop of performance.
