Path: ...!weretis.net!feeder9.news.weretis.net!panix!.POSTED.spitfire.i.gajendra.net!not-for-mail From: cross@spitfire.i.gajendra.net (Dan Cross) Newsgroups: comp.arch Subject: Re: DMA is obsolete Date: Wed, 21 May 2025 12:36:12 -0000 (UTC) Organization: PANIX Public Access Internet and UNIX, NYC Message-ID: <100khbs$elq$1@reader1.panix.com> References: <2025May2.073450@mips.complang.tuwien.ac.at> Injection-Date: Wed, 21 May 2025 12:36:12 -0000 (UTC) Injection-Info: reader1.panix.com; posting-host="spitfire.i.gajendra.net:166.84.136.80"; logging-data="15034"; mail-complaints-to="abuse@panix.com" X-Newsreader: trn 4.0-test77 (Sep 1, 2010) Originator: cross@spitfire.i.gajendra.net (Dan Cross) Bytes: 7936 Lines: 149 [Apologies for the excessively long time to respond; it's been a very busy few weeks!] In article , Scott Lurndal wrote: >cross@spitfire.i.gajendra.net (Dan Cross) writes: >>[snip] >>These are issues solveable with the software architecture and >>build system for the host OS. The important characteristic is >>that the software coupling makes architectural sense, and that >>simply does not require using the same ISA across IPs. > >I think there are good reasons to have specialized (or low cost, >e.g. riscv) ancilliary cores in a processor package. Having >been on both sides of the keep them proprietary vs. fully document >them for the OS folks argument, I remain ambivilent. > >There are good reasons for both positions. The same reasons behind >the MP1.5 spec and UEFI apply in many cases - widening the >ecosystem and 'you-fix-it' capabilities. On the other hand, >there may be trade secrets, or system security implications that >might preclude full disclosure. Once a capability is documented >in the PC world, it tends to live forever good or bad (ISA anyone?) >which may limit future choices in the product line (or discommode >customers). The idea that one introduces interfaces to facilitate change across multiple independent dimensions is a good one. But the interfaces that we have are awful. >>At work, our service processor (granted, outside of the SoC but >>tightly coupled at the board level) is a Cortex-M7, but we wrote >>the OS for that, > >What, not Zephyr? lol nope. https://hubris.oxide.computer >> and we control the host OS that runs on x86, >>so the SP and big CPUs can be mutually aware. Our hardware RoT >>is a smaller Cortex-M. We don't have a BMC on our boards; >>everything that it does is either done by the SP or built into >>the host OS, both of which are measured by the RoT. >> >>The problem is when such service cores are hidden (as they are >>in the case of the PSP, SMU, MPIO, and similar components, to >>use AMD as the example) and treated like black boxes by >>software. It's really cool that I can configure the IO crossbar >>in useful way tailored to specific configurations, but it's much >>less cool that I have to do what amounts to an RPC over the SMN >>to some totally undocumented entity somewhere in the SoC to do >>it. Bluntly, as an OS person, I do not want random bits of code >>running anywhere on my machine that I am not at least aware of >>(yes, this includes firmware blobs on devices). > >As a hardware (and long-time OS) person (not necessarily in that order), >I sympathize, but, yet, see above. > >>And if I already have to modify or configure the OS to >>accommodate the existence of these things in the first place, >>then accommodating an ISA difference really isn't that much >>extra work. The critical observation is that a typical SMP view >>of the world no longer makes sense for the system architecture, >>and trying to shoehorn that model onto the hardware reality is >>just going to cause frustration. Better to acknowledge that the > >Most of the hardware should be standardized through ACPI calls, >allowing the underlying implementation to vary over time. I strongly disagree. ACPI is a disaster; it may be the diaster we know, but it's a disaster nonetheless. Plus, with its tight entanglement with UEFI, it forces you into that world. Taking a step back, the real desideratea here might be some well-defined interface that creates a seam between the hardware and the OS, but UEFI+ACPI ain't it. Granted, we have the massive luxury of developing the hardware and software together and in concert at my job. I recognize that that is not the common case. > > >>Also, on AMD machines, again considering EPYC, it's up to system >>software running on x86 to direct either the SMU or MPIO to >>configure DXIO and the rest of the fabric before PCIe link >>training even begins (releasing PCIe from PERST is done by >>either the SMU or MPIO, depending on the specific >>microarchitecture). Where are these cores, again? If they're >>close to the devices, are they in the root complex or on the far >>side of a bridge? Can they even talk to the rest of the board? > >It's not the core that's proprietary in this case, it's the >intimate knowledge of the mainboard that is required for that >operation - consider address routing - once a function BAR >is programmed, the SoC fabric needs to be configured to route >that range of addresses to the correct PCI controller to >be converted to PCIe TLPs to the target function. Fortunately, we designed and built the board ourselves. :-D But this is an issue that the OS must contend with anyway; consider the case of PCIe hotplug: a device newly inserted in a system must be programmed with a useful BAR, which largely means that the OS must be capable of allocating physical address space to that device and setting up those routes. One might argue that, perhaps, the OS ought to use some sort of existing interface (like an AML flow or whatever) to do so, but that doesn't change the fact that ultimately the responsibility belongs to the OS. I suppose another argument might be that system firwmare just sets up all the root bridge ports with a hunk of address space and routes to those, and then all the OS has to do is plop a number into a BAR when a new device shows up, but you're already half way there, and again, the OS has to contend with understanding the routing that firmware has set up, which in turn introduces non-trivial complexity. In any event, we found that setting up the routes isn't that onerous; we defined some structs that make up effectively a DSL that lets us specify these in a pretty declarative way; comparing with e.g. AGESA, or OpenSIL, they do things in a pretty similar way. In either case, it's up to the board manufacturers to provide all of that data. >That routing can be incredibly complicated, requiring >very substantial and rather tricky configuration >steps in several related IP blocks as well as the >inter-cpu routing fabric/mesh. Getting it right >is difficult, and there's really no reason for the >OS level to be aware of it (and given it is highly >mainboard/SoC dependent, just complicates the >operating software when not behind a standard >configuration mechanism like ACPI.) Again, I disagree. What is the OS there for, if not to control and configurethe hardware and provide useful abstractions to application software? Hiding the actual hardware configuration from the OS means that the OS is not, in fact, the final entity in charge of the hardware. Having a second operating system, in parallel, gives me all of the same problems I had with BIOSes. This opens all sorts of issues with respect to safety and security, provenance of software, fault management, and ultimately, control. It's my machine, I want to run it as I see fit. - Dan C.