DELL VROC Stack Overflow

tl;dr The DELL VROC Stack Overflow results from creating a RAID-1 Volume that corrupted a doubly linked list (_LIST_ENTRY).

>> Home » Blog » DELL VROC Stack Overflow

Estimated reading time: 4 minutes

Author: David Hoyt | dhoyt@hoyt.net | @h02332

Date: October 14, 2024

OS_VERSION: 10.0.26100.1

Additionally, Memory corruption was detected by !chkimg in vds.exe

7ff74a5db770  74  6f  81  fa  4f *5b *81 *72 *2f *57 *62 *fc *36 *b2 *06 *e6

PoV

Dell is poorly equipped to handle such a product defect report.

Layman Summary

I purchased a Dell VROC Chip for a Dell 7820 Tower, with an Intel Silver 4216, running ASUS Hyper M.2 x16 PCIE Card provisioned with Samsung 980PRO SSD’s for Fuzzing.

I have a reliable reproduction of the DELL VROC Stack Overflow that should have been found via Product testing.

Product Defect Report: KERNEL_SECURITY_CHECK_FAILURE (0x139) Due to Memory Corruption in iaVROC.sys

Overview for iaVROC.sys

This report details a system crash (BSOD) with Kernel Dump from a Dell 7820 Tower caused by a memory corruption issue in the Dell distributed iaVROC.sys driver. The system encountered a KERNEL_SECURITY_CHECK_FAILURE with BugCheck code 0x139, which occurred due to corruption in the kernel’s _LIST_ENTRY structure, likely related to improper memory handling by the iaVROC.sys RAID driver.

Summary for iaVROC.sys

BugCheck Code: 0x139 (KERNEL_SECURITY_CHECK_FAILURE)
Faulting Driver: iaVROC.sys (Intel Virtual RAID on CPU – VROC driver)
Crash Cause: Memory corruption, specifically in a doubly linked list (_LIST_ENTRY) structure
Key Indicators: Corrupt pointers (Flink and Blink), invalid memory access, and corrupted pool allocation.
Security Consideration: Memory corruption in a kernel driver may lead to privilege escalation or system instability.

Crash Details for iaVROC.sys

Instruction and Context of iaVROC.sys

The crash occurred when the system attempted to execute the following instruction:

fffff800`97687cc0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffe104`714cea20=0000000000000139

This instruction is part of the bug check dispatch routine, preparing to pass the BugCheck Code (0x139) to KeBugCheckEx. The value 0x139 indicates a KERNEL_SECURITY_CHECK_FAILURE due to memory corruption.

Memory Corruption in `_LIST_ENTRY`

A corrupted doubly linked list (_LIST_ENTRY) was detected, which caused the kernel to fail with a fast fail mechanism. Details of the corrupted linked list are as follows:

Flink: 0xffffffff00000002 (Invalid pointer)
Blink: 0x0 (Null pointer, which should not occur in normal operation)

These invalid pointers suggest either:

Use-after-free: Memory was freed and accessed again.
Memory corruption: The linked list was improperly manipulated, possibly due to buffer overflow.

Pool Allocation Analysis for iaVROC.sys

Memory analysis of pool allocations showed several unknown pool tags, likely associated with the iaVROC.sys driver. A key finding included a large page allocation with the tag Cont:

Pool page ffff8106d7b5c000 region is Nonpaged pool
*ffff8106d7b5c000 : large page allocation, tag is Cont, size is 0xa000 bytes
Pooltag Cont : Contiguous physical memory allocations for device drivers

In contrast, memory at address 0x9800 showed invalid or corrupted data:

Pool page 0000000000009800 region is Unknown
*0000000000000000 : large page allocation, tag is ...., size is 0xfffff8009761d1b0 bytes
Owning component: Unknown (corrupted or freed memory)

Driver Details for iaVROC.sys

Driver Name: iaVROC.sys
Driver Version:
Module Path: \SystemRoot\System32\drivers\iaVROC.sys
Timestamp:

Relevant Pool Tag Information

Several unknown pool tags were identified, likely related to the iaVROC.sys driver or other third-party drivers. These tags show significant memory allocations, which may correlate with the memory corruption issue:

IWD0, IWE0, IWB0, IWC0: Unidentified pool tags potentially associated with the RAID driver.
ClfB: CLFS Log base file lookaside list (related to clfs.sys).
Ipcr, Ipur, TTcb: Tags associated with tcpip.sys, used for network connections and TCP/IP processing.
RaSr, RaDr: Tags related to storage operations (storport.sys), which may be interacting with the RAID driver.

Unknown Pool Samples

 mM           1          720          0            0    UNKNOWN pooltag 'mM  ', please update pooltag.txt
 MPic        15         8640          0            0    UNKNOWN pooltag 'MPic', please update pooltag.txt
 Wfra         3         8688          0            0    UNKNOWN pooltag 'Wfra', please update pooltag.txt
 IWTJ         1         2560          0            0    UNKNOWN pooltag 'IWTJ', please update pooltag.txt
 IWU0        32         1536          0            0    UNKNOWN pooltag 'IWU0', please update pooltag.txt
 KMDL         1          112          0            0    UNKNOWN pooltag 'KMDL', please update pooltag.txt
 IWL0         1          288          0            0    UNKNOWN pooltag 'IWL0', please update pooltag.txt
 IWAJ         1         1088          0            0    UNKNOWN pooltag 'IWAJ', please update pooltag.txt
 IWR0        32        21504          0            0    UNKNOWN pooltag 'IWR0', please update pooltag.txt
 IWQJ         1         2048          0            0    UNKNOWN pooltag 'IWQJ', please update pooltag.txt
 IWJJ        17        28704          0            0    UNKNOWN pooltag 'IWJJ', please update pooltag.txt

windbg

kd> k  ; For a regular stack trace
 # Child-SP          RetAddr               Call Site
00 ffffe104714cea18 fffff8009786f4e9     nt!KeBugCheckEx
01 ffffe104714cea20 fffff8009786faf2     nt!KiBugCheckDispatch+0x69
02 ffffe104714ceb60 fffff8009786d728     nt!KiFastFailDispatch+0xb2
03 ffffe104714ced40 fffff8009754eb93     nt!KiRaiseSecurityCheckFailure+0x368
04 ffffe104714ceed0 fffff8002a1cdcce     nt!ExInterlockedInsertTailList+0x33
05 ffffe104714cef00 fffff8002a1c41f3     iaVROC+0xddcce
06 ffffe104714cef40 fffff8002a1c469c     iaVROC+0xd41f3
07 ffffe104714ceff0 fffff8002a1be49f     iaVROC+0xd469c
08 ffffe104714cf020 fffff8009747577c     iaVROC+0xce49f
09 ffffe104714cf070 fffff800974d72ca     nt!KiExecuteAllDpcs+0x3dc
0a ffffe104714cf2a0 fffff8009785c53e     nt!KiRetireDpcList+0x28a
0b ffffe104714cf500 0000000000000000     nt!KiIdleLoop+0x9e
Address expression missing from 'or a regular stack trace'

21: kd> kp ; Stack trace with parameters
 # Child-SP          RetAddr               Call Site
00 ffffe104714cea18 fffff8009786f4e9     nt!KeBugCheckEx
01 ffffe104714cea20 fffff8009786faf2     nt!KiBugCheckDispatch+0x69
02 ffffe104714ceb60 fffff8009786d728     nt!KiFastFailDispatch+0xb2
03 ffffe104714ced40 fffff8009754eb93     nt!KiRaiseSecurityCheckFailure+0x368
04 ffffe104714ceed0 fffff8002a1cdcce     nt!ExInterlockedInsertTailList+0x33
05 ffffe104714cef00 fffff8002a1c41f3     iaVROC+0xddcce
06 ffffe104714cef40 fffff8002a1c469c     iaVROC+0xd41f3
07 ffffe104714ceff0 fffff8002a1be49f     iaVROC+0xd469c
08 ffffe104714cf020 fffff8009747577c     iaVROC+0xce49f
09 ffffe104714cf070 fffff800974d72ca     nt!KiExecuteAllDpcs+0x3dc
0a ffffe104714cf2a0 fffff8009785c53e     nt!KiRetireDpcList+0x28a
0b ffffe104714cf500 0000000000000000     nt!KiIdleLoop+0x9e
Couldn't resolve error at 'tack trace with parameters'

21: kd> !thread ; Displays information about the thread that caused the crash
THREAD ffff8106c7cee280  Cid 0000.0000  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 21
Not impersonating
DeviceMap                 ffffba8505446050
Owning Process            fffff800981cdf80       Image:         Idle
Attached Process          ffff8106c7ad2040       Image:         System
Wait Start TickCount      130085         Ticks: 0
Context Switch Count      313033         IdealProcessor: 21             
UserTime                  00:00:00.000
KernelTime                00:32:58.343
Win32 Start Address nt!KiIdleLoop (0xfffff8009785c4a0)
Stack Init ffffe104714cf530 Current ffffe104714cf4c0
Base ffffe104714d0000 Limit ffffe104714c9000 Call 0000000000000000
Priority 0  BasePriority 0  IoPriority 0  PagePriority 0
Child-SP          RetAddr               : Args to Child                                                           : Call Site
ffffe104714cea18 fffff8009786f4e9     : 0000000000000139 0000000000000003 ffffe104714ced40 ffffe104714cec98 : nt!KeBugCheckEx
ffffe104714cea20 fffff8009786faf2     : 0000000000000000 fffff8009768e2c0 0000000000000000 0000000000000000 : nt!KiBugCheckDispatch+0x69
ffffe104714ceb60 fffff8009786d728     : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiFastFailDispatch+0xb2
ffffe104714ced40 fffff8009754eb93     : ffff8106d7b058f0 ffff8106d7b50048 ffff8106d7b63488 0000000000000001 : nt!KiRaiseSecurityCheckFailure+0x368 (TrapFrame @ ffffe104714ced40)
ffffe104714ceed0 fffff8002a1cdcce     : ffff8106d7b058f0 ffffe104714cefa8 ffff8106d7b4f870 fffff80097543d88 : nt!ExInterlockedInsertTailList+0x33
ffffe104714cef00 fffff8002a1c41f3     : ffffe104714cf000 000003f4ba3b0000 000000000001fc25 fffff80029f100a7 : iaVROC+0xddcce
ffffe104714cef40 fffff8002a1c469c     : 0000000000000000 fffff8002a1dc201 0000000000000013 ffff8106e68fc4a0 : iaVROC+0xd41f3
ffffe104714ceff0 fffff8002a1be49f     : ffff8106d7b31980 ffffe104714cf3a0 000003f4ba3b6375 fffff800974f2c3b : iaVROC+0xd469c
ffffe104714cf020 fffff8009747577c     : ffff94004f7a6d30 0000000000000000 00000004bb86b6cf 0000000000000000 : iaVROC+0xce49f
ffffe104714cf070 fffff800974d72ca     : ffff9400509ea180 ffff8106c7cee280 0000000000000000 0000000000000000 : nt!KiExecuteAllDpcs+0x3dc
ffffe104714cf2a0 fffff8009785c53e     : ffff9400509ea180 ffff9400509ea180 ffff8106c7cee280 ffff8107077f3080 : nt!KiRetireDpcList+0x28a
ffffe104714cf500 0000000000000000     : ffffe104714d0000 ffffe104714c9000 0000000000000000 0000000000000000 : nt!KiIdleLoop+0x9e

Couldn't resolve error at 'isplays information about the thread that caused the crash'
21: kd> !irql
Debugger saved IRQL for processor 0x15 -- 2 (DISPATCH_LEVEL)
21: kd> lmvm iaVROC
Browse full module list
start             end                 module name
fffff8002a0f0000 fffff80030a66000   iaVROC     (no symbols)           
    Loaded symbol image file: iaVROC.sys
    Image path: \SystemRoot\System32\drivers\iaVROC.sys
    Image name: iaVROC.sys
    Browse all global symbols  functions  data  Symbol Reload
    Timestamp:        Thu Dec  8 11:45:21 2022 (639214A1)
    CheckSum:         0012793C
    ImageSize:        06976000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Security Implications

While the issue requires administrative privileges to trigger, the memory corruption within iaVROC.sys poses a security risk. Memory corruption in kernel-mode drivers can lead to privilege escalation, code execution, or denial of service (DoS) attacks. This vulnerability class that requires developer attention.

Take Away

This crash is caused by memory corruption in the iaVROC.sys driver, which manages RAID operations. The corruption leads to a KERNEL_SECURITY_CHECK_FAILURE (0x139), requiring a driver update and further investigation to resolve. While the crash currently requires administrative access, it poses an to system stability and should be addressed promptly.

vds.exe Crash Details

1. Overview

Application: vds.exe
Crash: Illegal instruction (0xc000001d) at 00007ff74a5db7f5
OS Version: 10.0.26100.1
Occurred 14 seconds after process start, 494 seconds after system boot.

2. Exception Details

Faulting Instruction: 0xD6 (Invalid opcode)
Exception Address: vds!CVdsService::Initialize+0x5
Registers at crash:
RIP: 00007ff74a5db7f5
RAX: 00000ffee94bb6e5
RBX: 0000000000000002
(Full register state provided)

3. Memory Corruption Evidence

255 discrepancies found in vds.exe image:
Example: Expected 74 6f 81 fa 4f, found 5b 81 72 2f
Analysis of affected memory shows corrupted instructions.

4. Call Stack and Code Analysis

Stack trace (from thread 0):
vds!CVdsService::Initialize+0x5
memory_corruption!vds.exe+0x0
Disassembly of faulting instruction:
00007ff74a5db7f5: d6 ??? (illegal instruction)

Takeaway

If you are from Dell or Intel and read this Post, I have a 32G MEMORY.DMP for you to download from xss.cx