Monday, November 16, 2026

Improving the Wii U's Wii Mode

One feature of the Wii U is the backwards compatibility with Wii games through Wii Mode, often referred to as vWii. In theory, this is a really nice way to enjoy Wii games on a modern TV, since the Wii U has HDMI output. Unfortunately, Wii Mode does not look great. Ironically, the output of an original Wii can look better.
The two most prominent issues are the force-cropped output and color bleed issues.

Scaled up image of the chroma bleed issue in Super Mario Sunshine and The Legend of Zelda: Skyward Sword.
You can see the green tint on the left side of the characters. (Image credit: Rulumi)


Reports of these issues go back to 2013[1], shortly after the Wii U was released, but a fix was never provided.

In this blog post we'll take a closer look at the technical architecture of Wii Mode graphics, how two simple typos in the firmware caused these issues, and what modern AMD GPUs have in common with the Wii U.

The DMCU

So who can we blame for these issues? One candidate is the DMCU. To get a quick overview of the DMCU, I'm quoting fail0verflow's description from their 2014 Wii U blog post[2]:

You may have noticed that good old VI (the display controller / Video Interface of ye olde GameCube and Wii) is gone. Huh? How does that work? Turns out they’re emulating that in software and translating its configuration to the R7xx’s CRTC registers. But where does this emulation software run? Sneaky: they added a special microcontroller to the Latte just for this purpose. The DMCU is a 68HC11 compatible 8-bit CPU whose sole purpose is to perform VI emulation. It has a frontend hardware shim that looks like VI to the Wii software, and behind the scenes it translates those registers to the Radeon’s, including upscaling configuration. cafe2wii loads the DMCU’s firmware (the DMCU doesn’t seem to be active in Wii U mode). It has its own dedicated RAM and access to both the faux-VI back side and the Radeon’s register area. [...]

To get to the bottom of these issues, I decided to reverse-engineer the firmware of the DMCU. The DMCU firmware is located in the Wii U's storage and is loaded into the DMCU's RAM before launching cafe2wii. Unfortunately, my favorite reverse-engineering tool Ghidra does not support the HC11, so in 2024 I ended up writing my own processor specification for it[3]. Luckily, Ghidra already supported the HCS08 and HCS12, which I could base my specification on.

I then spent several days reverse-engineering the DMCU firmware. The firmware itself is relatively straightforward. The DMCU has an interface to access the 32-bit registers of the display controller. The faux-VI hardware shim triggers interrupts in the DMCU firmware, which then configures the display controller accordingly.

Reverse engineering the DMCU quickly reveals the reason for the cropped image. The DMCU configures the display controller's viewport to simulate overscan. This crops off up to 16 pixels, which does not look great in several titles. Luckily, this is an easy fix; I can simply patch the bytes of the firmware directly to change the values written to the viewport registers.

AMD's DMCU

What fail0verflow didn't know at the time, or at least didn't mention, is that the DMCU is not a Wii U-specific microcontroller, but is found in other AMD GPUs too. A patch submitted by an AMD employee in 2018 to the amd-gfx mailing list[4] mentions the following:

DMCU (Display Microcontroller Unit) is a GPU chip involved in eDP features like Adaptive Backlight Modulation and Panel Self Refresh.

At least now we know what DMCU stands for. Could this really be the same chip design as used by the Wii U? The linux-firmware repository has several DMCU binary blobs for different GPUs[5]. After downloading raven_dmcu.bin and navi12_dmcu.bin, I wrote a quick tool to parse the headers of the firmware blobs and extract the part that actually gets loaded into ERAM[6].

After loading the AMD Raven and Navi 12 DMCU firmware into Ghidra, I was able to confirm that the newer AMD GPUs still use an M68HC11-based DMCU. Even the display controller interface is almost identical to the one used on the Wii U.

Decompiling the DMCU

Since I grew tired of having to apply byte patches to the existing DMCU firmware, I decided to create a C decompilation of the firmware. This would also allow me to get a deeper understanding of how everything works.

At first, I needed a proper toolchain to compile my project. GCC used to have upstream support for the M68HC11 and M68HC12[7]. The last upstream release with support is GCC 3.3.5 from 2005. Later versions broke it before it was removed entirely. I eventually stumbled across releases from 2006 on the personal website of Stéphane Carrez[8], the author of the original port. This leaves me with GCC 3.3.6 and binutils 2.15, which will have to do for now.

To my surprise, I was able to byte-match entire functions by hand from the original Wii U DMCU firmware using this compiler. To help with matching I created a modified version of decomp.me with support for the previously mentioned compiler.

For example, the following function (compiled with gcc3.3.6-m68hc1x-20060122 -Os -m68hc11 -fomit-frame-pointer -mshort) results in an exact byte-match in the Wii U DMCU firmware.

void FUN_0e3b(FVIVideoMode videoMode, FVIState* state)
{
uint32_t postBlanking = 0;
uint32_t preBlanking = 0;
uint32_t uVar2 = 0;
uint32_t uVar3 = 0;

if (videoMode == FVI_VIDEO_MODE_288P_312 ||
videoMode == FVI_VIDEO_MODE_240P_262) {
preBlanking = state->preBlanking;
postBlanking = state->postBlanking + 1;
uVar2 = state->unk0x0C - 1;
uVar3 = state->unk0x14 + 2;
} else if (videoMode == FVI_VIDEO_MODE_288P_313 ||
videoMode == FVI_VIDEO_MODE_240P_263) {
preBlanking = state->preBlanking;
postBlanking = state->postBlanking - 1;
uVar2 = state->unk0x0C + 1;
uVar3 = state->unk0x14 - 2;
}

DEINT_Lock();

FUN_1292();

DC_Write32(0x10A6, postBlanking | (preBlanking << 16) | 0x80008000ul);
DC_Write32(0x10A7, uVar3 | (uVar2 << 16) | 0x80008000ul);

DAT_0120 = 1;

FUN_133f();

DEINT_Unlock();

DEBUG_Write0x6(0x40);
}

Looks like AMD or Nintendo (I'm not sure who was responsible for writing the Wii U's DMCU firmware) might have used Carrez's GCC port from 2006 when they were writing the Wii U's DMCU firmware. Are they still using it? Unfortunately, I wasn't able to get an exact match in the Raven or Navi 12 DMCU. They might have used some convoluted compiler flags that I couldn't figure out, or a different compiler entirely.

One interesting thing I noticed during reverse engineering is that the nn_cmpt.rpl library responsible for loading the DMCU firmware ends up writing the current display resolution into the middle of the DMCU's ERAM. These resolutions are unused by the DMCU firmware, but end up overwriting instructions in the middle of the firmware. By sheer luck, the overwritten functions happen to be unused, resulting in no issues being caused by this. Since I'm loading my custom firmware in the same way as the original one, this required some special attention in my decompilation. I had to create a carveout section for the resolutions and place the text section before these resolutions, manually placing larger functions into a section behind this carveout.

After manually decompiling the entire DMCU firmware to C, I had a functionally equivalent firmware. Not every function is byte-identical, but it seems good enough for me. I ended up publishing this firmware under the name OpenDMCUFirm in 2025, in the hope that others could help with taking a closer look.

The Cause of Color Bleed

Several months after releasing OpenDMCUFirm, I noticed two bugs in the DMCU firmware, which led to the color bleed issues in Wii mode.

The DMCU firmware processes events in the main event loop. These events are triggered based on interrupts from the faux-VI hardware shim. In the IRQ handler, status bits are mapped from an FVI register to events, which are then processed in the main loop. The code looks roughly like this[9]:

struct {
uint32_t inStatus;
uint16_t outEvent;
} irqEventMasks[4] = {
{ 0x00000003, DMCU_EVENT_0, },
{ 0x08007B1C, DMCU_EVENT_1, },
{ 0x07C084E0, DMCU_EVENT_2, },
{ 0x003C0000, DMCU_EVENT_3, },
};

uint32_t irqStatus = DC_Read32(0x1061);
uint32_t eventFlags = gDmcuState.eventFlags;

for (uint32_t i = 0; i < 4; i++) {
if (!(irqStatus & irqEventMasks[i].inStatus)) {
continue;
}

handledFlags |= irqStatus & irqEventMasks[i].inStatus;

flags = irqStatus >> DMCU_EVENT_ACTIVE_SHIFT;
if ((flags & irqEventMasks[i].outEvent & 0xf) != 0) {
eventFlags |= irqEventMasks[i].outEvent << DMCU_EVENT_RETRIGGER_SHIFT;
} else {
eventFlags |= irqEventMasks[i].outEvent;
}
}

gDmcuState.eventFlags = eventFlags;

If an event is currently already being handled by the main loop, it is marked as "active". The IRQ handler checks for that and marks it to be "retriggered" instead. This causes the event loop to immediately run the event handler again after the handler returns[10]:

getEventHandler(&eventHandler, &eventMask, gDmcuState.eventFlags & 0xF);

// ...

IRQ_Lock();
gDmcuState.eventFlags = (gDmcuState.eventFlags | (eventMask << DMCU_EVENT_ACTIVE_SHIFT)) & ~eventMask;
IRQ_Unlock();

do {
eventHandler(&gDmcuState);

if (((gDmcuState.eventFlags >> DMCU_EVENT_RERUN_SHIFT) & eventMask) != 0) {
gDmcuState.eventFlags &= ~(eventMask << DMCU_EVENT_RERUN_SHIFT);
} else if ((gDmcuState.eventFlags >> DMCU_EVENT_RETRIGGER_SHIFT) & eventMask & 0xf) {
gDmcuState.eventFlags &= ~(eventMask << DMCU_EVENT_RETRIGGER_SHIFT);
} else {
IRQ_Lock();
gDmcuState.eventFlags &= ~(eventMask << DMCU_EVENT_ACTIVE_SHIFT);
IRQ_Unlock();
goto loop_start;
}
} while (1);

Looks like someone made a typo in the IRQ handler, though. Instead of checking if the event active bit is set in eventFlags, they're checking if the bit is set in irqStatus, which is the hardware status register. If the bit for the active event happens to be set in the IRQ status register, the event is now "retriggered" and never marked as pending. This means the event is never actually processed by the main event loop. The fix for this is really simple:

-flags = irqStatus >> DMCU_EVENT_ACTIVE_SHIFT;
+flags = eventFlags >> DMCU_EVENT_ACTIVE_SHIFT;

I tested this on hardware and confirmed that this occurs for events 2 and 3. They are executed once and are then always stuck in the "retrigger" state and are never processed again. All because of a simple typo.

So what are events 2 and 3 used for? Event handler 2 configures a bit in the DEINT register block called CBCR_SWAP, based on the horizontal origin of the image[11]:

// This register consists of (HBS << 16) | HBE
// See YAGCD 5.3 VI - Video Interface: HTR1 - Horizontal Timing 1
DC_PerformRead(0x1082);
uint8_t hbe = DC_Get8(0);

DEINT_Lock();

if ((hbe & 0x1) == 0) {
DEINT_EnableCBCRSwap();
*outEnabled = TRUE;
} else {
DEINT_DisableCBCRSwap();
*outEnabled = FALSE;
}

DEINT_Unlock();

There's actually another typo in this function, which was probably never noticed since this event handler is never called: CBCR_SWAP should be enabled for odd origins, not even origins. The fix for this is again really simple:

-if ((hbe & 0x1) == 0) {
+if ((hbe & 0x1) != 0) {

Edit 2026-11-17: It turns out this part actually used to be correct, but Nintendo had updated the DMCU firmware before, where they inverted the CBCR_SWAP logic. I assume they knew something was wrong, but couldn't figure out the root cause in the IRQ handler, so they tried inverting the logic here. Credits to Rairii for noticing this.

Making these two changes fixes the color bleed issues.

I'm not entirely sure yet what event handler 3 is used for. There don't seem to be any visual differences without it. It appears to be similar to event handler 1, so maybe they moved event handler 3's functionality there after noticing it wasn't working.

Try it out now!

Scaled up before and after image in Super Mario Sunshine. (Image credit: Rulumi)

Scaled up before and after image in The Legend of Zelda: Skyward Sword. (Image credit: Rulumi)

These fixes are part of the custom branch in OpenDMCUFirm. You can load this version of OpenDMCUFirm with the latest release of evWii. Let me know how it looks!

You can also help me pay for my Wiimote batteries here: https://ko-fi.com/garyodernichts


  1. https://gamefaqs.gamespot.com/boards/631516-wii-u/66227322↩︎

  2. https://fail0verflow.com/blog/2014/console-hacking-2013-omake/↩︎

  3. https://github.com/GaryOderNichts/ghidra-hc11-lang↩︎

  4. https://lists.freedesktop.org/archives/amd-gfx/2018-September/026550.html↩︎

  5. https://gitlab.com/kernel-firmware/linux-firmware/-/blob/main/amdgpu/raven_dmcu.bin↩︎

  6. https://gist.github.com/GaryOderNichts/ab38b340ea077daae8c4d58cd367c87b↩︎

  7. https://www.gnu.org/software/m68hc11/m68hc11_gcc.html↩︎

  8. https://web.archive.org/web/20240530035034/http://gnu-m68hc11.vacs.fr/blog/↩︎

  9. https://github.com/GaryOderNichts/OpenDMCUFirm/blob/ee11a04c536825de35e19996ea030d065afcae8e/source/dmcu.c#L91-L111↩︎

  10. https://github.com/GaryOderNichts/OpenDMCUFirm/blob/ee11a04c536825de35e19996ea030d065afcae8e/source/dmcu.c#L848-L865↩︎

  11. https://github.com/GaryOderNichts/OpenDMCUFirm/blob/ee11a04c536825de35e19996ea030d065afcae8e/source/deint.c#L350-L365↩︎


Thursday, November 13, 2025

Privileged Arbitrary Code Execution on STM32H73XXX microcontrollers

This is somewhat of a follow-up to the Nintendo Alarmo blog post from last year. This time the blog post is about the security of the STM32H730 microcontroller series itself, which is also used by the Alarmo. The STM32H730 contains an Arm Cortex-M7 core and 128 Kbytes of internal flash memory. In this blog post we'll take a look at a vulnerability that allows arbitrary code execution in the secure bootloader on STM32H73xxx and STM32H7Bxxx microcontrollers.

 

STM32H730 Security Overview

Like many other microcontrollers, the STM32H730 supports a feature called readout protection (RDP) to protect the contents of the internal flash from being accessed when a debugger is connected. RDP offers three different levels of protection, which can be set through a configuration register. At RDP level 0 the readout protection is disabled and the contents of the internal flash can be read out using a connected debugger. At RDP level 1 the contents of the flash are made inaccessible as soon as a debugger is connected to the system. At RDP level 2 establishing a connection to a debugger is no longer possible at all. RDP can only be regressed from level 1 back to level 0, doing so will completely erase the internal flash. Once the RDP level is set to 2, it can never be changed back.

In addition to the readout protection, the STM32H730 features a global "secure access mode". When this mode is enabled, a special secure-only area can be defined in the user accessible part of the internal flash, which can only be accessed while the CPU is still running secure code at boot. Attaching a debugger while running such secure code is not possible. This allows separating secure user code from non-secure code, for example to run startup code that handles any sort of secret data, like cryptographic keys, which should not be retrieved after system startup. On the Nintendo Alarmo, the code within the secure-only area sets up the cryptographic engine with a secret key stored in the secure area, then loads a second stage bootloader from external eMMC, verifies its signature, and decrypts it. After jumping to this second stage bootloader using a special function stored in system memory, the secure-only area is no longer accessible.

In order to enable the secure access mode, the SECURITY bit is set in the option bytes using a configuration register. This can be done from unprivileged user code. To clear the SECURITY bit after configuring the secure-only area, a RDP regression is required. This will completely erase the internal flash, including any secure-only areas, allowing the SECURITY bit to be cleared again.

The Secure Bootloader

Once the secure access mode is enabled, the system always boots into a piece of code located in protected system memory, called the secure bootloader. This secure bootloader should not be confused with STMicroelectronics' non-secure bootloader, which allows programming the flash over regular communication ports like USB or UART.
The purpose of the secure bootloader is to securely boot into a secure-only user area which might have been set and/or to run services on startup called root secure services (RSS) (these services will be described in more detail later).

Attaching a debugger is not possible while the CPU is executing the secure bootloader, even with RDP disabled (level 0). The secure bootloader code is located in protected system memory and gets locked out before jumping to regular user code, making it inaccessible.

Because the secure-bootloader's internal behavior is mostly undocumented, I decided to attempt dumping it. Since I didn't want to risk bricking an Alarmo with my experiments, I started with a development board containing a similar STM chip with security protections.

One of the first things I tried was reading the secure bootloader while executing code from a secure-only user area, since both are regarded as "secure code" by the reference manual. Unfortunately, it wasn't this easy. Any attempts to read this code returned only zeroes and sets the read secure error bit RDSERR.
Notably, the secure bootloader doesn't clear its stack (located in DTCM RAM) before branching to the user code, allowing the stack of the secure bootloader to be dumped from user code as soon as it runs. This reveals some return addresses and variables left on the stack. This confirms that the secure bootloader is actually running, but does not allow for much else.

Secure bootloader stack values dumped starting at stack top (0x20000800).
Stack offset Value Description
-0x00 0xFFFFFFFF
-0x04 0x00000000
-0x08 0x00000008
-0x0C 0x00000008
-0x10 0x1FF06D03 Return address in system memory
-0x14 0xCC000007
-0x18 0xAA0038A0
-0x1C 0x20000800
-0x20 0xE000EDA0
-0x24 0x00000000
-0x28 0x20004000
-0x2C 0x00000000
-0x30 0x1FF04209 Return address in system memory
... ... ...

Root Secure Services (RSS)

In order to perform some security-critical tasks, like setting up on-the-fly decryption (OTFD) or configuring secure-only areas, the root secure services (RSS) are used. To access RSS functionality, the secure access mode needs to be enabled, since these services are performed by the secure bootloader. A small RSS caller library is located at the end of the system memory. It can be accessed by non-secure user code and provides several functions to access the secure services. The function addresses are stored at hardcoded addresses in ROM to allow for easily calling them. The RSS code itself then runs from within the secure bootloader at startup. In order to run a service, a service ID together with all parameters are written to DTCM RAM and a system reset is performed. After reset, the secure bootloader runs with highest possible privileges and executes the secure service specified by its ID in RAM.

On-the-Fly Decryption (OTFD) Service

One of these RSS services is the on-the-fly decryption (OTFD) service resetAndEncrypt. OTFD is another security feature of the STM32H730 which allows directly mapping an encrypted external SPI-flash in memory space, performing decryption on-the-fly when accessing this memory. The flash can be split into up to four different encrypted regions. The data is encrypted using AES-128 in counter mode, with a key and nonce specified by the user.
In order to encrypt the data to be stored on the flash, the RSS library function resetAndEncrypt is called with pointers to a plaintext buffer, key and nonce stored in RAM. The function then performs a system reset, causing the secure service to encrypt the buffer in-place with the specified key and nonce, before jumping back to the user code after completion. The user code can then write the encrypted data to the external flash and enable on-the-fly decryption.

Gaining code execution in RSS

I have to admit, an anonymous person managed to glitch the chip before I was able to find a way to dump the secure bootloader. This means I actually got access to a dump of the entire protected system memory beforehand, allowing me to take a look at the implementation of these secure services, revealing a vulnerability in one of them.
The vulnerability itself was actually easy to spot and is rather straight forward: The OTFD secure service doesn't check the address of the buffer which will be in-place encrypted by RSS, allowing encryption of a buffer anywhere in memory with a chosen key and nonce. By choosing the right key and nonce pair, the RSS code can encrypt a return address on its stack, transforming it into a valid address in SRAM.
The actual AES128-CTR encryption happens in the CRYPT engine of the microcontroller, using an undocumented set of registers. Luckily, the process itself is described in detail in Reference Manual RM0468 and Application Note AN5281. Half of the 128-bit IV is made up of the 64-bit user-controlled nonce, while the other half contains the region firmware version, region ID, and the current read address.

Nonce
Unused
Region FW
Version
[127:64]
[63:48]
[47:32]
Region ID
[31:28]
Read address
[27:0]
Structure of the 128-bit IV used for OTFD.


With the structure of the final IV revealed, it's time to find some key and nonce candidates which produce a valid address in SRAM when encrypting one of the return addresses before jumping to user code. For this I wrote a brute-forcing tool in C which simply encrypts the return address with random key and nonce pairs and checks if the resulting value is a nicely aligned address in SRAM. The source for this tool can be found here.

$ ./bruteforcer  
Candidate: 24012000 Key: 1839323a 50458bf1 abd34402 124dad4d Nonce: 591164ac 70f2bb52  
Candidate: 24016000 Key: ff0f6ec9 10950afb 929f6a00 da3d42d5 Nonce: 5016ce89 6d800495  
Candidate: 24004000 Key: b95154df 05bf505f 08ef0476 b50ae336 Nonce: 43a44da3 92261edc

The tool quickly found some key and nonce candidates, from which I chose the one at located at 0x24004000 which is a nicely aligned address in SRAM.

Dumping the RSS Code

In order to make sure that this actually works, I needed another device with an STM32H730. While I already had the Alarmo, I didn't want to fully erase it. Unfortunately STMicroelectronics doesn't have any development boards with an STM32H730. The closest match would be a discovery kit with an STM32H735, but that costs over a hundred bucks and doesn't even have the right chip. The two options I had were buying a pin-compatible development board like the NUCLEO-H723ZG and resoldering it with an STM32H730 or buying a cheap no-name development board from China. I ended up doing both, having the board from China as a backup in case something didn't end up working after the chip transplant.
All exception handlers in the secure bootloader set a flag to clear the stack and perform a system reset. This nicely allows for testing if an exception has been raised while in the secure bootloader: if the unused area of the stack has been cleared, an exception was raised. And indeed, simply encrypting the return address right after the command handler with a zero key and nonce results in a cleared stack, proving that an exception was triggered while running the secure bootloader.
As a payload to load into SRAM, I wrote a small program which prints "Hello world" over UART. Using the key and nonce found by the bruteforcer, the resetAndEncrypt service can be called, resulting in the message being printed over UART, proving successful code execution.

Output of "Hello world" payload.



With successful code execution from within the context of the RSS code, I wrote a payload which dumps the secure bootloader code itself over UART. The full dumper code can be found on this GitHub repository, allowing anyone to dump the RSS code from vulnerable devices.

Can this bypass RDP or secure-only areas?

Short answer: No.
Secure-only areas cannot be dumped since the root secure services are disabled once a secure-only area has been set. There is one exception to this: if the first reserved entry in the secure-only area's vector table (offset 0x1C) is set to the magic value 52 53 53 45 ("RSSE") (0x45535352 as a little-endian word), RSS services remain accessible.
While I was initially under the assumption that it's possible to use this to bypass the readout protection, I stumbled across two things which prevented me from dumping the internal flash while readout protection was at level 1.
The first issue is that an external reset (via NRESET) does not clear the RDP status: if RDP was tripped, even the secure-bootloader cannot access the internal flash after an external reset. The only way to reset the readout protection seems to be by performing a full power-cycle, which will also erase any RSS commands in DTCM. A simple external reset keeps the readout protection tripped.

The other issue is due to the OTFD engine itself. After RDP has been tripped, the engine will only return zeroes upon attempting to read from an OTFD region. This also affects the encryption in the RSS code, which is now only writing zeroes over the specified buffer. Additionally it seems to now only write 8-byte aligned buffers and sizes, which still allows overwriting the return address on the stack with zeroes. This would allow jumping to a payload in ITCM RAM, which is mapped at 0x0, but without having the least significant bit of the address (T-bit) set, the Thumb-only CPU will simply abort upon jumping there.

Unless there is a way to perform a full power cycle while preserving the contents of DTCM RAM (where the RSS commands are stored), this technique cannot bypass readout protection.

What now?

Even though this didn't break the security as much as I initially expected, I reported this vulnerability to ST's PSIRT team. I also asked for further comment on the behavior of OTFD encryption under RDP regression, which they declined to comment on. They published a security advisory SA0053 in late September, recommending enabling RDP level 2 on affected devices. I'm not sure if this is really necessary, but maybe ST knows something I don't :)

Disclosure Timeline

06 April 2025: Technical details about the vulnerability and example code provided to ST PSIRT.
12 May 2025: Response from PSIRT acknowledging the vulnerability and asking for further details about potential disclosure. Reply from me, mentioning that all details are planned to be disclosed.
17 June 2025: Follow-up email from me asking for further details and status of the vulnerability.
18 June 2025: Response from ST needing more time to investigate.
29 September 2025: Security advisory published by ST.
13 November 2025: Publication of this blog post with technical details and example code on GitHub.




Improving the Wii U's Wii Mode

One feature of the Wii U is the backwards compatibility with Wii games through Wii Mode, often referred to as vWii. In theory, this is a rea...