1

Here is my problem:

I'm unable to get BAR addressing from prefetchable end-point devices behind a pci-e bridge. Can anyone help me about that?

Here is what I did until now:

1- I'm able to get address for a prefetchable pci-e device without the bridge. But I cannot read/write to this space.

2- I'm able to get address for a non-prefetchable pci-e device without the bridge. I can read/write to this space successfully.

3- I'm unable to get address for a single prefetchable device behind the bridge.

4- I'm able to get addresses for multiple non-prefetchable devices behind the bridge. I also can read/write to them.

5- I'm unable to get address for one prefetchable and one non-prefetchable device behind the bridge. In this position, I'm also unable to get non-prefetchable one that can be used in the above trial.

What my suspicion is:

I've validated these tests with two different root-complex devices to achieve these results, both are kernel version 4.9, so I'm pretty sure this issue is related to the kernel.

Thanks for your time in advance.

Edit:

When I use lspci, I can get the vendor IDs, device IDs and device types of the all devices I have used above. When I meant that I cannot get the BAR addresses, I'm simply pointing out that the virtual memory mappings do not appear in lspci -vv outputs.

For my reads and writes, I'm using memtool which is a simple C program that can read and write bytes to a given memory.

The files that need to appear at /sys/bus/pci_express/devices related to the devices I mentioned are not appearing in the failing cases.

When I checked dmesg for any related messages here are the related ones:

[    2.496007] pci 0000:03:00.0: [104c:b005] type 00 class 0x048000
[    2.496065] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x007fffff]
[    2.496099] pci 0000:03:00.0: reg 0x14: [mem 0x00000000-0x007fffff pref]
[    2.496268] pci 0000:03:00.0: Max Payload Size set to 128 (was 256, max 256)
[    2.496605] iommu: Adding device 0000:03:00.0 to group 59
[    2.496610] arm-smmu: forcing sodev map for 0000:03:00.0
[    2.510188] pci 0000:03:00.0: BAR 0: no space for [mem size 0x00800000]
[    2.510192] pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00800000]
[    2.510196] pci 0000:03:00.0: BAR 1: no space for [mem size 0x00800000 pref]
[    2.510200] pci 0000:03:00.0: BAR 1: failed to assign [mem size 0x00800000 pref]
[    2.510942] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt

[    2.497054] pci 0000:04:00.0: [1172:e001] type 00 class 0xff0000
[    2.497114] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
[    2.497691] iommu: Adding device 0000:04:00.0 to group 60
[    2.497696] arm-smmu: forcing sodev map for 0000:04:00.0
[    2.510254] pci 0000:04:00.0: BAR 0: no space for [mem size 0x04000000]
[    2.510259] pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x04000000]
[    2.510949] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt

The device 0000:03:00.0 is the prefetchable one, and the 04 is the non-prefetchable one. In the dmesg, it says that I don't have space for them, but I know the card I'm using (an Nvidia Jetson TX2) have 127 MB of space for non-prefetchable devices, and 896 MB space for prefetchable. The devices I try have 8 MB + 8 MB prefetchable and 64 MB non-prefetchable memory.

By the way, thanks for the edit suggestion, I realized that I had so little detail while I'm editing the question.

Here are the dmesgs for my pcie switches: 01:00.0 is my upstream port, while the other two is downstreams.

[    2.472535] pci 0000:01:00.0: [111d:808c] type 01 class 0x060400
[    2.472856] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    2.473055] iommu: Adding device 0000:01:00.0 to group 56
[    2.473060] arm-smmu: forcing sodev map for 0000:01:00.0
[    2.481784] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.486074] pci 0000:01:00.0: BAR 14: no space for [mem size 0x08000000]
[    2.486078] pci 0000:01:00.0: BAR 14: failed to assign [mem size 0x08000000]
[    2.486221] pci 0000:01:00.0: PCI bridge to [bus 02-04]
[    2.486589] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

[    2.482057] pci 0000:02:08.0: [111d:808c] type 01 class 0x060400
[    2.482419] pci 0000:02:08.0: PME# supported from D0 D3hot D3cold
[    2.482798] iommu: Adding device 0000:02:08.0 to group 57
[    2.482807] arm-smmu: forcing sodev map for 0000:02:08.0
[    2.483622] pci 0000:02:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.486090] pci 0000:02:08.0: BAR 14: no space for [mem size 0x01000000]
[    2.486093] pci 0000:02:08.0: BAR 14: failed to assign [mem size 0x01000000]
[    2.486114] pci 0000:02:08.0: PCI bridge to [bus 03]
[    2.486592] pci 0000:02:08.0: Signaling PME through PCIe PME interrupt

[    2.482957] pci 0000:02:10.0: [111d:808c] type 01 class 0x060400
[    2.483284] pci 0000:02:10.0: PME# supported from D0 D3hot D3cold
[    2.483475] iommu: Adding device 0000:02:10.0 to group 58
[    2.483480] arm-smmu: forcing sodev map for 0000:02:10.0
[    2.483650] pci 0000:02:10.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.486083] pci 0000:02:10.0: BAR 14: no space for [mem size 0x04000000]
[    2.486087] pci 0000:02:10.0: BAR 14: failed to assign [mem size 0x04000000]
[    2.486170] pci 0000:02:10.0: PCI bridge to [bus 04]
[    2.486598] pci 0000:02:10.0: Signaling PME through PCIe PME interrupt

I see a configuration invalid message there, but that invalid configuration message still appears when I switch my setup to experiment 2 again, I'm not sure this is the issue.

iamonur
  • 13
  • 5
  • Please edit your question and indicate *how* you get the address, and how you attempt to read/write. What does `lspci` show? Do the BARs show up under `/sys`? Any error messages in `dmesg`? (Basically your question doesn't contain any concrete information that would allow to debug the problem). – dirkt May 27 '20 at 19:52
  • @dirkt thanks for the suggestion, I think adding more detail definitely improved the question. – iamonur May 28 '20 at 08:48
  • So now the question is why there is no space for the BARs, which probably has to do with how the PCI bridge is configured. Can you include the relevant lines from `dmesg` for that? My suspicion is that the bridge doesn't get enough memory for the BARs behind them, for whatever reason. – dirkt May 28 '20 at 17:24
  • Hey, for sure! I was suspecting from my pcie bridge too. The bridge I use is a programmable one made by my colleagues, and digging into its program is on my TODO list from now. If you suspect anything from the dmesg, please tell me and I will look into the source for that. Thanks for your time, you are a great help! – iamonur May 29 '20 at 06:07

1 Answers1

1

I mean, bridge configuration invalid and BAR 14 (instead of BAR 1 or some sensible number), and no space for it already sounds like the bridge is completely utterly broken, or at least the configuration it presents to the OS is garbage.

But if the bridge is programmable, it's probably possible to fix that...

dirkt
  • 16,421
  • 3
  • 31
  • 37
  • Yeah, probably it is the switch. I'm on it, will note you and mark this answer if I can solve it. Thanks! – iamonur Jun 01 '20 at 04:38
  • 1
    The thing was the disabled bars on the endpoint device. They still exist somehow, but also disabled. What happens is kernel tries to address them, but cannot learn how large they are. Thus, whole addressing collapses. – iamonur Jun 05 '20 at 05:35