GRAAAAAA NEW DOCTRE DROPPED
We already new about the job listing tho : P
I don't think this is related to DLSS. We know from the Nvidia hack that DLSS was already integrated via the NVN2 API on their side, and Nvidia's existing DLSS implementation is likely already extremely well optimised for the Ampere architecture. There's not a whole lot for Nintendo to do there other than test and provide feedback.
And no, before anyone asks, it's got nothing to do with ChatGPT either. Models like ChatGPT are
Large Language Models (LLMs), and are far too big to run on a portable device like Switch 2, even if they were of any use in a videogame, which is debatable. Machine learning is a lot more than just DLSS and ChatGPT.
Nintendo are about to release the first gaming system with dedicated machine learning acceleration hardware. This means that developers, both inside and outside Nintendo, are going to have access to functionality they haven't had before, and are going to need assistance to effectively use it. Game developers are good at optimising for hardware with limited performance and memory, but most don't have any expertise in machine learning. Meanwhile, ML experts know how to design and train models, but aren't typically experts in optimising for hardware like games consoles. That's where the people applying for the job below would come in.
Here's the job description:
We at Nintendo are looking for a Data Engineer to help with integration of machine learning technologies on low-power embedded platforms. You will be working at the intersection of machine learning inference engines and embedded systems, facing challenges that stem from processing and memory constraints and a power budget. Tasks include, but are not limited to, porting of machine learning frameworks to embedded platforms, evaluation and benchmarking of machine learning hardware solutions, selection and optimization of machine learning models to fit power, memory, and CPU budgets.
DESCRIPTION OF DUTIES:
- Design, develop and maintain a formal framework for validating and benchmarking machine learning solutions.
- Perform evaluations of machine learning hardware.
- Research, evaluate, analyze and optimize machine learning models.
Let's say you're a game developer, and you've got a colleague with a background in machine learning, and together you think that you've come up with a neat ML-based solution for, say, character animation, and you want to run it on Switch 2. Your first problem will be just physically getting it to run on the hardware at all. You're most likely creating and training the model using a framework like PyTorch, using Python. You wouldn't want to actually run Python on a games console, and besides, PyTorch doesn't exactly list "Nintendo's unreleased game console" as compatible hardware.
So, first you need a way to take a model that's been created and trained in PyTorch and run it on the Switch hardware. This means compiling the model for the Ampere GPU, and it's not entirely different than writing code in a language like C++ and compiling it for an x86 or ARM CPU. Except, unlike compiling C++ code, where you can easily find and use any number of compilers (although Nintendo likely provides an official one), there aren't exactly a range of easy to use high-performance ML compilers around. In fact, part of Nvidia's dominance in the AI space is a result (and/or cause) of this problem, as many AI implementations are dependent on Nvidia's libraries like
cuDNN which are pre-compiled for Nvidia's GPUs.
There has been some movement in the last couple of years to open-source, hardware-agnostic machine learning frameworks. OpenAI's
Triton is the one which seems to be gaining the most traction, and it includes a language for defining the execution of ML models, an intermediate representation (based on LLVM-IR, for those curious) and a compiler. PyTorch now utilises Triton as a backend for
their compiler module when used on GPUs. Unfortunately, these still don't fully solve the problem for a device like the Switch, as they're built around JIT (just-in-time) compilation, where code is compiled on the device immediately before it runs. This isn't a problem for big models, where compilation is far quicker than the execution of the model itself, but on a games console you want to save every bit of frame-time you can get, so ahead-of-time compilation is a must.
Nintendo needs a solution to this, both for their own developers, and to provide to third parties. When the job description says "porting of machine learning frameworks to embedded platforms", this is the kind of thing they're talking about; taking a popular machine learning framework like PyTorch and providing a way to take models from it and compile them for the Switch 2's GPU. I wouldn't be surprised if they're using open-source software like Triton for this and updating it for their own purposes, like pre-compilation and optimisation for Switch 2's hardware.
Even once you've got your model compiled and running on the hardware, that's only the first step. You then need to get it running
well. Creating ML models is an iterative process, which generally involves repeatedly tweaking and re-training the model until you get the accuracy or performance you need, or just can't squeeze any more out of it. For an embedded system like a games console, performance is vital, and at each step you need to test performance on the embedded hardware itself (ie your Switch 2 dev kit), not just the GPU on the workstation or server you're using. This means you need a way to actually measure and profile the performance of your ML code on the hardware, or as the job description puts it a "framework for validating and benchmarking machine learning solutions".
TLDR: The Switch 2 has ML accelerators (tensor cores). For developers to actually use them effectively, Nintendo needs people like the applicants for this job to make sure their ML models both run and perform well on the hardware.
As a separate note, as DLSS and ChatGPT seem to be almost the only things brought up regarding uses for machine learning, I feel I should point people to some of the research out there on the many other potential uses for ML in video games. Both Ubisoft (
here) and EA (
here) have done quite a bit of research and published several papers in the area. Some promising avenues include animation (
here and
here), physics (
here) and pathfinding (
here). These are just a few examples from a single team of researchers, there's a lot more out there. There have also been several games which have already shipped with some use of ML, even without dedicated hardware, for instance cloth physics in
Madden 21 and texture upscaling in
God of War Ragnarok. (The GoW presentation also illustrates how difficult it is to optimise an ML model for a platform when you don't have a good compiler).
In general, anything that can be computed can be implemented via machine learning. In many (or most) cases the existing approach is going to be superior in terms of performance and/or quality, but ML is often pretty well suited as a "good enough" solution, where it's not strictly as accurate as an existing method, but can run quicker and produces results which are close enough that people generally can't tell the difference. Video games are built on "good enough" solutions, where any time you talk about a game being well optimised or looking good for its hardware, it's because the developers have come up with the cheapest possible way to do things that still looks "good enough" to the player. I don't expect ML to completely overtake traditional techniques in gaming, but I do think there are going to be a lot of little places where a fast ML implementation can be "good enough" and be a better choice than existing non-ML techniques. As Switch 2 will be the first hardware with dedicated acceleration for these tasks, it will be at the forefront of developers implementing these new techniques.