Path: ...!weretis.net!feeder9.news.weretis.net!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: mitchalsup@aol.com (MitchAlsup1)
Newsgroups: comp.arch
Subject: Re: PCIe MSI-X interrupts
Date: Sat, 22 Jun 2024 19:31:20 +0000
Organization: Rocksolid Light
Message-ID: <6caaf9cb208191a70900c68ac4ce0562@www.novabbs.org>
References: <bb16865f7675526d4e2b87283e28c2c5@www.novabbs.org> <sKmdO.62321$G9_a.28048@fx13.iad> <0faeb2831a76d32cd6fb8cff7b546807@www.novabbs.org> <EmBdO.18027$Gurd.13722@fx34.iad>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
	logging-data="842243"; mail-complaints-to="usenet@i2pn2.org";
	posting-account="65wTazMNTleAJDh/pRqmKE7ADni/0wesT78+pyiDW8A";
User-Agent: Rocksolid Light
X-Spam-Checker-Version: SpamAssassin 4.0.0
X-Rslight-Site: $2y$10$97KPBWo4RrbY4dP2AWwXv.6Z8IRx/UpUXUkbGbwvOeApvzfal6ax2
X-Rslight-Posting-User: ac58ceb75ea22753186dae54d967fed894c3dce8
Bytes: 4171
Lines: 86

Scott Lurndal wrote:

Again allow me to express my gratitute on the quality of your posts !

A couple of dumb questions to illustrate how much more I need to
learn::

> mitchalsup@aol.com (MitchAlsup1) writes:
>>Scott Lurndal wrote:

>>> The PCI address (aka Stream ID) is passed to the interrupt
>>> controller and IOMMU and used as an index to determine the
>>> page table root pointer.
>>
>>> The stream id format is
>>
>>>     <2:0>  PCI function number
>>>     <7:3>  PCI device number
>>>     <15:8> PCI bus number
>>>     <xx:16> PCI segment (root complex) number.
>>
>>I use ChipID for the last field in case each chip has its own
>>PCIe tree. {Except that the bits are placed elsewhere in the 
>>address.}

> Each root complex needs to be an unique segment.  A single
> SRIOV endpoint can consume the entire 8-bit bus space and
> the 8-bit dev/function space.  In this context, a root complex
> can be considered a PCI express controller with one or more
> root ports.  Each root port should be considered an unique
> 'segment'.

> This is for device discovery, which uses the PCI express
> "Extended Configuration Access Method" (aka ECAM) to scan
> the PCI configuration spaces of all PCI ports.

Within a 'Chip' there are k cores, 1 last level cache, and
1 HostBridge with (say) 256 pins at its disposal. Said
pins can be handed out in powers of 2 of 4-pins each
so multiple PCIe trees of differing widths emanate from
the 256-PCIe-pins.

I guess you are calling each point of emanation a root.
I just bundle them under 1 HostBridge, and consider how
the "handing out" is done to be a HostBridge problem.
But As seen on the on-chip interconnect there is one
HostBridge which accesses all devices attached to this
Chip. Basically, I see on-chip-interconnect with one 
HostBridge knowing that the pins will be allocated
"efficiently" for the attached devices.

Thanks for the ECAM pointer, that clears up a raft of
questions.


>>
>>But (now with the new CXL) instead of allocating 200+ pins
>>to DRAM those pins can be allocated to PCIe links; making any
>>chip much less dependent on which DRAM technology, which chip-
>>to-chip repeaters,... So, the thought is all I/O is PCIe + CXL;
>>and about the only other pins chip gets are RESET and ClockIn.

> Note that bridging to PCI signalling will increase latency
> somewhat, even with PCIe gen 6.

Unavoidable.

>>
>>Bunches of these pins can be 'configured' into standard width
>>PCIe links (at least until one runs out of pins.)
>>
>>Given that one has a PCIe root complex with around 256-pins
>>available, does one need multiple roots of such a wide tree ?

> You basically need a root per device to accommodate SRIOV
> devices (like enterprise grade network adapters, high-end
> NVMe devices, etc).

As noted above: I knew more bits than B:D,F were needed,
but not which and where. And if a single SR-IOV device
consumes a whole B:D,F space sobeit. ECAM alignment 
identifies those bits and the routings.


I guess reading m post backwards I did not pose any questions.

My thanks again.