Determining The Interrupt Line For A Particular PCI-E Slot
Hi debuggers, this is Graham McIntyre again. These days I’m working more closely with hardware so I thought I’d share some hardware related debugging tips. I recently debugged an issue where a PCI-E storage device failed to work after hot swapping it from one slot to another slot on the system without rebooting. We determined the issue was due to the device not receiving interrupts once it was moved. So in the process I learned how line based interrupts are routed to a particular PCI slot. Interrupt routing is quite a hefty subject, but here’s one example of how to determine what the expected interrupt line is for a particular PCI-E slot using a live kernel debug.
There are two ways the routing can be defined in the ACPI tables:
- Static routing (most common for APIC systems)
- Link Node routing (most common for PIC systems)
Since APIC is much more common, I am focusing on method 1 for static routing. Though, it is legal to use Link Node routing with IOAPICs, it’s not common, so I am omitting how to parse that. This is also specifically for devices that use physical line based interrupts (LBI), not Message Signaled Interrupts (MSI).
Here is the general method for determining the static routing IRQ for a particular device:
- Locate the devstack for the device, and determine its parent devices in the PCI hierarchy. (!pcitree)
- Determine the interrupt pin which the device uses
- Walk the parent devices to find the closest PCI Routing Table (_PRT) which will describe the mapping of interrupt pin to IRQ.
- If the parent device does not have a _PRT, then swizzle the pin, since the pin number can change when moving to the upstream side of the PCI bridge (you may end up swizzling the pin several times). We will discuss how to swizzle the pin number later in this article.
- If the parent device has a _PRT, then move to the next step
- Convert the IntPin number from PCI to ACPI numbering
- Parse the _PRT method to find the static routing table
- Find the routing entry which represents our IntPin
Here’s the in-depth steps, along with an example:
Step 1: Locate the devstack for the device, and determine its parent devices in the PCI hierarchy.
To determine this, use !pcitree to dump the PCI hierarchy. Then locate your device by ven/dev ID. You could also use !devnode to dump the hierarchy.
The way !pcitree shows the hierarchy may be a little confusing. When it encounters a PCI bridge, it dumps the child buses under the bridge. The indenting tells you what bus a device is on. A device is always indented one level from the entry of the parent bus. In my case, I know the device I'm interested in is VEN FEFE DEV 1550.
kd> !pcitree
Bus 0x0 (FDO Ext fffffa80053efe00)
(d=0, f=0) 80863406 devext 0xfffffa80054d51b0 devstack 0xfffffa80054d5060 0600 Bridge/HOST to PCI
(d=1, f=0) 80863408 devext 0xfffffa80054d9b70 devstack 0xfffffa80054d9a20 0604 Bridge/PCI to PCI
Bus 0x1 (FDO Ext fffffa80054e8680)
(d=0, f=0) 14e41639 devext 0xfffffa80051b91b0 devstack 0xfffffa80051b9060 0200 Network Controller/Ethernet
(d=0, f=1) 14e41639 devext 0xfffffa80051ba1b0 devstack 0xfffffa80051ba060 0200 Network Controller/Ethernet
(d=3, f=0) 8086340a devext 0xfffffa80054dab70 devstack 0xfffffa80054daa20 0604 Bridge/PCI to PCI
Bus 0x2 (FDO Ext fffffa80054e9460)
(d=0, f=0) 14e41639 devext 0xfffffa80051bcb70 devstack 0xfffffa80051bca20 0200 Network Controller/Ethernet
(d=0, f=1) 14e41639 devext 0xfffffa80051cab70 devstack 0xfffffa80051caa20 0200 Network Controller/Ethernet
(d=4, f=0) 8086340b devext 0xfffffa80054dbb70 devstack 0xfffffa80054dba20 0604 Bridge/PCI to PCI
Bus 0x3 (FDO Ext fffffa80054ec190)
(d=0, f=0) 10000079 devext 0xfffffa80051cd1b0 devstack 0xfffffa80051cd060 0104 Mass Storage Controller/RAID
(d=5, f=0) 8086340c devext 0xfffffa80054dcb70 devstack 0xfffffa80054dca20 0604 Bridge/PCI to PCI
Bus 0x4 (FDO Ext fffffa80054ede00)
No devices have been enumerated on this bus.
(d=6, f=0) 8086340d devext 0xfffffa80054ddb70 devstack 0xfffffa80054dda20 0604 Bridge/PCI to PCI
Bus 0x5 (FDO Ext fffffa80054ee9c0)
No devices have been enumerated on this bus.
(d=7, f=0) 8086340e devext 0xfffffa80054deb70 devstack 0xfffffa80054dea20 0604 Bridge/PCI to PCI << Root Port
Bus 0x6 (FDO Ext fffffa80054f1190)
(d=0, f=0) abcd8632 devext 0xfffffa80051d91b0 devstack 0xfffffa80051d9060 0604 Bridge/PCI to PCI << Upstream switch port
Bus 0x7 (FDO Ext fffffa80051cd850)
(d=4, f=0) abcd8632 devext 0xfffffa80051d71b0 devstack 0xfffffa80051d7060 0604 Bridge/PCI to PCI
Bus 0x8 (FDO Ext fffffa8006f44ac0)
No devices have been enumerated on this bus.
(d=5, f=0) abcd8632 devext 0xfffffa80058d6a10 devstack 0xfffffa80058d68c0 0604 Bridge/PCI to PCI
Bus 0x9 (FDO Ext fffffa80051ba850)
No devices have been enumerated on this bus.
(d=6, f=0) abcd8632 devext 0xfffffa8007075b70 devstack 0xfffffa8007075a20 0604 Bridge/PCI to PCI << Parent PDO (Downstream Switch Port)
Bus 0xa (FDO Ext fffffa8007312b60)
(d=0, f=0) fefe1550 devext 0xfffffa8006f67b70 devstack 0xfffffa8006f67a20 0180 Mass Storage Controller/'Other' << Device
(d=7, f=0) abcd8632 devext 0xfffffa80051e5b70 devstack 0xfffffa80051e5a20 0604 Bridge/PCI to PCI
Bus 0xb (FDO Ext fffffa80052d2e00)
No devices have been enumerated on this bus.
(d=14, f=0) 8086342e devext 0xfffffa80054dfb70 devstack 0xfffffa80054dfa20 0800 Base System Device/Interrupt Controller
(d=14, f=1) 80863422 devext 0xfffffa80054e0b70 devstack 0xfffffa80054e0a20 0800 Base System Device/Interrupt Controller
(d=14, f=2) 80863423 devext 0xfffffa80054e1b70 devstack 0xfffffa80054e1a20 0800 Base System Device/Interrupt Controller
(d=1a, f=0) 80862937 devext 0xfffffa80054e2b70 devstack 0xfffffa80054e2a20 0c03 Serial Bus Controller/USB
(d=1a, f=1) 80862938 devext 0xfffffa80054e31b0 devstack 0xfffffa80054e3060 0c03 Serial Bus Controller/USB
(d=1a, f=7) 8086293c devext 0xfffffa80054e3b70 devstack 0xfffffa80054e3a20 0c03 Serial Bus Controller/USB
(d=1d, f=0) 80862934 devext 0xfffffa80054e41b0 devstack 0xfffffa80054e4060 0c03 Serial Bus Controller/USB
(d=1d, f=1) 80862935 devext 0xfffffa80054e4b70 devstack 0xfffffa80054e4a20 0c03 Serial Bus Controller/USB
(d=1d, f=7) 8086293a devext 0xfffffa80054e51b0 devstack 0xfffffa80054e5060 0c03 Serial Bus Controller/USB
(d=1e, f=0) 8086244e devext 0xfffffa80054e5b70 devstack 0xfffffa80054e5a20 0604 Bridge/PCI to PCI
Bus 0xc (FDO Ext fffffa80054f2e00)
(d=3, f=0) 102b0532 devext 0xfffffa80051d51b0 devstack 0xfffffa80051d5060 0300 Display Controller/VGA
(d=1f, f=0) 80862918 devext 0xfffffa80054e61b0 devstack 0xfffffa80054e6060 0601 Bridge/PCI to ISA
(d=1f, f=2) 80862921 devext 0xfffffa80054e6b70 devstack 0xfffffa80054e6a20 0101 Mass Storage Controller/IDE
Total PCI Root busses processed = 1
Total PCI Segments processed = 1
To recap the devices in the tree (Bus,Device,Function):
(0,7,0) : Root Port, PCI-PCI Bridge (devstack 0xfffffa80054dea20)
(6,0,0) : Upstream Switch Port (devstack 0xfffffa80051d9060)
(7,6,0) : Downstream Switch Port (the PDO for the slot) (devstack 0xfffffa8007075a20)
(a,0,0) : Device (devstack 0xfffffa8006f67a20)
I scanned the output looking for my ven/dev ID, and found it at Bus A, Device 0, Function 0.
Step 2: Determine which interrupt pin the device uses.
For this step, you can use !pci to dump the PCI config space for the device. The output will show you the interrupt pin the device uses, labeled as IntPin.
!pci 1 a 0 0
PCI Bus 10
00:0 FEFE:1550.01 Cmd[0007:imb...] Sts[0018:c....] Device SubID:1344:1008 Other mass storage controller
cf8:800a0000 IntPin:1 IntLine:2e Rom:0 cis:0 cap:40
MEM[2]:df5fd000 MEM[3]:df5fc000 IO[4]:cff1 MEM[5]:df5fe000
So our IntPin is 1.
Step 3: Walk the parent devices to find the closest PCI Routing Table (_PRT) which will describe the mapping of interrupt pin to IRQ.
Now, we will traverse the parent PCI devnodes until we find a PCI bridge which has an associated ACPI object with a _PRT method. This may be the root port, or an integrated bridge.
- Start by running !devstack on the parent. We can determine the parent device using the indentations of the !pcitree output.
- If the devstack shows an ACPI filter driver, then dump the filter using !acpikd.acpiext to find the associated AcpiObject
- Dump the ACPI object and its children to see if it has a _PRT method defined
- If it does not have a _PRT, then you need to swizzle the Interrupt Pin to find what the pin number will be on the upstream side of the bridge
- We have to use a method called “swizzling” because the pin may become a different pin on the upstream side of the bridge. The way to calculate the pin is:
- IntPin = ((((IntPin -1) + DeviceNumber) % 4) +1)
- Where IntPin is the current IntPin value, and DeviceNumber the device number of the device you’re swizzling.
- You will start with the IntPin value from !pci output of the device itself. If you need to swizzle multiple times, you take the result of the previous swizzle as the input to the next swizzle
- The device number for the first time will be the device number of the target device, and subsequent times will be the device number of the parent device you’re swizzling.
- We have to use a method called “swizzling” because the pin may become a different pin on the upstream side of the bridge. The way to calculate the pin is:
- If it does have a _PRT, then move onto Step 4.
- If it does not have a _PRT, then you need to swizzle the Interrupt Pin to find what the pin number will be on the upstream side of the bridge
Example:
First, we’ll swizzle the pin of the device itself (a,0,0). The IntPin is 1 so:
IntPin = ((((1-1)+0) % 4) +1) << The Swizzled Pin is still IntPin 1
Next, I dumped the parent device (7,6,0), !devstack 0xfffffa8007075a20. It didn’t have an ACPI filter driver on the stack. So I need to swizzle the pin.
IntPin = ((((1-1)+6) % 4) +1) << The Swizzled Pin is now 3
I now dump the next parent up, (6,0,0), !devstack 0xfffffa80051d9060. It also didn’t have an ACPI filter driver on the stack so I need to swizzle the pin again.
IntPin = ((((3-1)+0) % 4) +1) << The Swizzled Pin is still 3
I am now at the root port. The first devstack which has a _PRT method in my case is the root port.
kd> !devstack 0xfffffa80054dea20
!DevObj !DrvObj !DevExt ObjectName
fffffa80054f1040 \Driver\pci fffffa80054f1190
fffffa80054e5800 \Driver\ACPI fffffa80051c1510 << Has an ACPI filter driver in the devstack
> fffffa80054dea20 \Driver\pci fffffa80054deb70 NTPNP_PCI0006
!DevNode fffffa80054e1750 :
DeviceInst is "PCI\VEN_8086&DEV_340E&SUBSYS_02351028&REV_13\3&33fd14ca&0&38"
ServiceName is "pci"
kd> !acpikd.acpiext fffffa80051c1510
ACPI!DEVICE_EXTENSION fffffa80051c1510 - 70000
DevObject fffffa80054e5800 PhysObject fffffa80054dea20 NextObject fffffa80054dea20
AcpiObject fffffa80052e3890 ParentExt fffffa80051c07d0
PnpState Started OldPnpState Stopped
Dispatch fffff880011cbb50
RefCounts 4-Device 1-Irp 0-Hiber 0-Wake
State D0
SxD Table S0->D0 S4->D3 S5->D3
Flags 0540100002000240
Types Filter Enumerated ValidPnP
Caps PCIBus
Props HasAddress Enabled AcpiPower
Dump the namespace object. Use /s to display the subtree under this object and look for a _PRT method.
kd> !amli dns /s fffffa80052e3890
ACPI Name Space: \_SB.PCI0.PEX7 (fffffa80052e3890)
Device(PEX7)
| Integer(_ADR:Value=0x0000000000070000[458752])
| Integer(_STA:Value=0x000000000000000f[15])
| Method(_PRT:Flags=0x0,CodeBuff=fffffa80052e3aa9,Len=144) << A _PRT method exists for this object
Now, we have a swizzled IntPin value of 3, and a pointer to the _PRT method. We can move on to the next step.
Step 4: Convert the pin number from PCI to ACPI numbering
The !pci or !devext output, and subsequent swizzling will show pin numbering in PCI format where 1 = INTA. But the ACPI table uses 0 for INTA. So you need to subtract one from the PCI pin number to get the ACPI pin number.
|
PCI pin number |
ACPI pin number |
INTA |
1 |
0 |
INTB |
2 |
1 |
INTC |
3 |
2 |
INTD |
4 |
3 |
Once you’ve converted to ACPI pin numbering, you have to dump the _PRT method to find the package which maps to that pin number.
For my example since the PCI IntPin value is 3, which corresponds to INTC, the ACPI pin number is 2
Step 5: Parse the _PRT method to find the static routing table
Now that we located the correct _PRT entry, we need to use the AMLI debugger extension to parse the method and find the static routing table. The command !amli u will unassemble an ACPI method
kd> !amli u \_SB.PCI0.PEX7._PRT
AMLI_DBGERR: Failed to get address of ACPI!gDebugger
fffffa80052e3aa9 : If(LNot(PICF))
fffffa80052e3ab1 : {
fffffa80052e3ab1 : | Name(P10B, Package(0x4)
fffffa80052e3ab9 : | {
fffffa80052e3ab9 : | | Package(0x4)
fffffa80052e3abc : | | {
fffffa80052e3abc : | | | 0xffff,
fffffa80052e3abf : | | | 0x0,
fffffa80052e3ac1 : | | | LK00,
fffffa80052e3ac5 : | | | 0x0
fffffa80052e3ac7 : | | },
fffffa80052e3ac7 : | | Package(0x4)
fffffa80052e3aca : | | {
fffffa80052e3aca : | | | 0xffff,
fffffa80052e3acd : | | | 0x1,
fffffa80052e3acf : | | | LK01,
fffffa80052e3ad3 : | | | 0x0
fffffa80052e3ad5 : | | },
fffffa80052e3ad5 : | | Package(0x4)
fffffa80052e3ad8 : | | {
fffffa80052e3ad8 : | | | 0xffff,
fffffa80052e3adb : | | | 0x2,
fffffa80052e3add : | | | LK02,
fffffa80052e3ae1 : | | | 0x0
fffffa80052e3ae3 : | | },
fffffa80052e3ae3 : | | Package(0x4)
fffffa80052e3ae6 : | | {
fffffa80052e3ae6 : | | | 0xffff,
fffffa80052e3ae9 : | | | 0x3,
fffffa80052e3aeb : | | | LK03,
fffffa80052e3aef : | | | 0x0
fffffa80052e3af1 : | | }
fffffa80052e3af1 : | })
fffffa80052e3af1 : | Store(P10B, Local0)
fffffa80052e3af7 : }
fffffa80052e3af7 : Else
fffffa80052e3af9 : {
fffffa80052e3af9 : | Name(A10B, Package(0x4)
fffffa80052e3b01 : | {
fffffa80052e3b01 : | | Package(0x4)
fffffa80052e3b04 : | | {
fffffa80052e3b04 : | | | 0xffff,
fffffa80052e3b07 : | | | 0x0,
fffffa80052e3b09 : | | | 0x0,
fffffa80052e3b0b : | | | 0x26
fffffa80052e3b0d : | | },
fffffa80052e3b0d : | | Package(0x4)
fffffa80052e3b10 : | | {
fffffa80052e3b10 : | | | 0xffff,
fffffa80052e3b13 : | | | 0x1,
fffffa80052e3b15 : | | | 0x0,
fffffa80052e3b17 : | | | 0x2d
fffffa80052e3b19 : | | },
fffffa80052e3b19 : | | Package(0x4)
fffffa80052e3b1c : | | {
fffffa80052e3b1c : | | | 0xffff,
fffffa80052e3b1f : | | | 0x2,
fffffa80052e3b21 : | | | 0x0,
fffffa80052e3b23 : | | | 0x2f
fffffa80052e3b25 : | | },
fffffa80052e3b25 : | | Package(0x4)
fffffa80052e3b28 : | | {
fffffa80052e3b28 : | | | 0xffff,
fffffa80052e3b2b : | | | 0x3,
fffffa80052e3b2d : | | | 0x0,
fffffa80052e3b2f : | | | 0x2e
fffffa80052e3b31 : | | }
fffffa80052e3b31 : | })
fffffa80052e3b31 : | Store(A10B, Local0)
fffffa80052e3b37 : }
fffffa80052e3b37 : Return(Local0)
fffffa80052e3b39 : Zero
fffffa80052e3b3a : Zero
fffffa80052e3b3b : Zero
fffffa80052e3b3c : Zero
fffffa80052e3b3d : Zero
fffffa80052e3b3e : Zero
fffffa80052e3b3f : Zero
fffffa80052e3b40 : HNSO
fffffa80052e3b44 : Not(Zero, )
fffffa80052e3b47 : Zero
fffffa80052e3b48 : Zero
fffffa80052e3b49 : AMLI_DBGERR: UnAsmOpcode: invalid opcode class 0
AMLI_DBGERR: Failed to unassemble scope at 382d4a0 (size=4096)
There are 2 different _PRT tables here, each with 4 packages (think of it as 2 arrays, each containing 4 structures). The first is using link nodes, the second is using static interrupts. The first list is used if we are in PIC mode, the second if we are in APIC mode.
We can check the value of PICF to determine the mode. (I expect it to be APIC but let’s check)
kd> !amli dns \PICF
ACPI Name Space: \PICF (fffffa80052ded18)
Integer(PICF:Value=0x0000000000000001[1])
So we’re in APIC mode (PICF != 0), we use the static routing mode. So we will use the 2nd table. What does each package represent? Each is a PCI Routing Table. From ACPI spec section 6.2.12, which describes the _PRT:
Table 6-14 Mapping Fields
Field |
Type |
Description |
|---|---|---|
Address |
DWORD |
The address of the device (uses the same format as _ADR). |
Pin |
BYTE |
The PCI pin number of the device (0–INTA, 1–INTB, 2–INTC, 3–INTD). |
Source |
NamePath Or BYTE |
Name of the device that allocates the interrupt to which the above pin is connected. The name can be a fully qualified path, a relative path, or a simple name segment that utilizes the namespace search rules. Note: This field is a NamePath and not a String literal, meaning that it should not be surrounded by quotes. If this field is the integer constant Zero (or a BYTE value of 0), then the interrupt is allocated from the global interrupt pool. |
Source Index |
DWORD |
Index that indicates which resource descriptor in the resource template of the device pointed to in the Source field this interrupt is allocated from. If the Source field is the BYTE value zero, then this field is the global system interrupt number to which the pin is connected. |
fffffa80052e3af9 : | Name(A10B, Package(0x4)
fffffa80052e3b01 : | {
fffffa80052e3b01 : | | Package(0x4)
fffffa80052e3b04 : | | {
fffffa80052e3b04 : | | | 0xffff,
fffffa80052e3b07 : | | | 0x0, << INTA
fffffa80052e3b09 : | | | 0x0,
fffffa80052e3b0b : | | | 0x26 << Interrupt line
fffffa80052e3b0d : | | },
fffffa80052e3b0d : | | Package(0x4)
fffffa80052e3b10 : | | {
fffffa80052e3b10 : | | | 0xffff,
fffffa80052e3b13 : | | | 0x1, << INTB
fffffa80052e3b15 : | | | 0x0,
fffffa80052e3b17 : | | | 0x2d
fffffa80052e3b19 : | | },
fffffa80052e3b19 : | | Package(0x4)
fffffa80052e3b1c : | | {
fffffa80052e3b1c : | | | 0xffff,
fffffa80052e3b1f : | | | 0x2, << INTC
fffffa80052e3b21 : | | | 0x0,
fffffa80052e3b23 : | | | 0x2f
fffffa80052e3b25 : | | },
fffffa80052e3b25 : | | Package(0x4)
fffffa80052e3b28 : | | {
fffffa80052e3b28 : | | | 0xffff,
fffffa80052e3b2b : | | | 0x3, << INTD
fffffa80052e3b2d : | | | 0x0,
fffffa80052e3b2f : | | | 0x2e
fffffa80052e3b31 : | | }
fffffa80052e3b31 : | })
Step 6 - Find the routing entry which represents our IntPin
Now, we just have to locate the entry in the routing table with a IntPin value of 2.
fffffa80052e3b19 : | | Package(0x4)
fffffa80052e3b1c : | | {
fffffa80052e3b1c : | | | 0xffff,
fffffa80052e3b1f : | | | 0x2, <<< IntPin 2 (INTC)
fffffa80052e3b21 : | | | 0x0,
fffffa80052e3b23 : | | | 0x2f << IRQ is 0x2f
So the device should be assigned IRQ 0x2F. However, you may have noticed from the !pci output above that in this case the device was actually assigned IntLine (IRQ) 0x2e! Since the wrong interrupt line was assigned after the device changed slots in the system, the device did not receive interrupts and hence was not functional.
I hope this was useful to help understand how interrupts are assigned to LBI devices.
More reading / references:
PCI IRQ Routing on a Multiprocessor ACPI System:
https://msdn.microsoft.com/en-us/windows/hardware/gg454523.aspx#EDD
ACPI 4.0 spec
https://www.acpi.info/DOWNLOADS/ACPIspec40a.doc
Comments
Anonymous
September 01, 2011
Nice writeup!. who assign the wrong int. 2e to that device? [ACPI.SYS did. We have plans to correct this in the future. We got the cart before the horse when we published this article before a solution was available.]Anonymous
September 07, 2011
The comment has been removedAnonymous
September 20, 2011
How the problem has been solved in this case?Anonymous
September 22, 2011
Why ACPI.SYS changing that as it should only read tables from BIOS? Can you elaborate in what situation ACPI.sys will map to other intr.? [Typically we provide this information in the KB article that documents the fix. I would rather not comment on the exact cause until the solution is finalized and published. As I said earlier, we got the cart before the horse on this one; although the purpose of the article was to show how PCIe addressing works and not necessarily provide a deep dive on this problem. I'll update the article when the fix is available.]Anonymous
September 25, 2011
Not a Issue. By the way Great analysis. Thanks Graham. Really good step by step discription.