Path: ...!weretis.net!feeder9.news.weretis.net!panix!.POSTED.spitfire.i.gajendra.net!not-for-mail
From: cross@spitfire.i.gajendra.net (Dan Cross)
Newsgroups: comp.arch
Subject: Re: MSI interrupts
Date: Tue, 25 Mar 2025 17:44:25 -0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Message-ID: <vruq1p$cv2$1@reader1.panix.com>
References: <vqto79$335c6$1@dont-email.me> <kVgEP.1277108$_N6e.605199@fx17.iad> <vrsea2$siu$1@reader1.panix.com> <8uzEP.115550$Xq5f.66883@fx38.iad>
Injection-Date: Tue, 25 Mar 2025 17:44:25 -0000 (UTC)
Injection-Info: reader1.panix.com; posting-host="spitfire.i.gajendra.net:166.84.136.80";
	logging-data="13282"; mail-complaints-to="abuse@panix.com"
X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
Originator: cross@spitfire.i.gajendra.net (Dan Cross)
Bytes: 15500
Lines: 320

In article <8uzEP.115550$Xq5f.66883@fx38.iad>,
EricP  <ThatWouldBeTelling@thevillage.com> wrote:
>Dan Cross wrote:
>> In article <kVgEP.1277108$_N6e.605199@fx17.iad>,
>> EricP  <ThatWouldBeTelling@thevillage.com> wrote:
>>> Dan Cross wrote:
>>>> In article <fe9715fa347144df1e584463375107cf@www.novabbs.org>,
>>>> MitchAlsup1 <mitchalsup@aol.com> wrote:
>>>>> On Thu, 20 Mar 2025 12:44:08 +0000, Dan Cross wrote:
>>>>>> Sometimes you really don't want to be interrupted.
>>>>> And sometimes you don't want to be interrupted unless the
>>>>> "house is on fire"; I cannot see a time when "the house is
>>>>> on fire" that you would not want to take the interrupt.
>>>>>
>>>>> Is there one ?!?
>>>> Consider a thread that takes a spinlock; suppose some
>>>> high-priority interrupt comes in while the thread is holding
>>>> that lock.  In response to the interrupt, software decides to
>>>> suspend the thread and switch some other thread; that thread
>>>> wants to lock the spin lock that the now-descheduled thread is
>>>> holding: a classic deadlock scenario.
>>>
>>> Terminology: mutexes coordinate mutual exclusion between threads,
>>> spinlocks coordinate mutual exclusion between cpu cores.
>>> Windows "critical sections" are mutexes with a fast path.
>> 
>> A spin lock is simply any lock where you spin trying to acquire
>> the lock, as opposed to a blocking synchronization protocol.
>> Here I'm using the terminology of Herlihy and Shavit [Her08].
>
>I'm using terminology in common use on multiprocessor systems like
>VMS since 1980's and later WinNT.
>
>Each OS data structure has different reentrancy requirements.
>- Ones accessed by threads are guarded by mutexes.
>   Example on Windows would be the page tables as paging is done by threads.
>- Ones accessed by OS software interrupt level are guarded by spinlocks.
>   Example on Windows is the Scheduler tables.
>- Ones accessed by HW interrupts are guarded by interrupt spinlocks.
>   Example on Windows is device interrupt service routines.

I'm not sure what you're saying here, or rather, how it is
relevant.

Your earlier statement was that "mutexes coordinate mutual
exclusion between threads, spinlocks coordinate mutual exclusion
between cpu cores."  But this is not the case.

Taking VMS as an example, from [Gol94], sec 9.5 (page 255):

|A thread of execution running on one processor acquires a
|spinlock to serialize access to the data with threads of
|execution running on other processors.  Before acquiring
|the spinlock, the thread of executions raises IPL to block
|accesses by other threads of execution running on the same
|processor.  The IPL value is determined by the spinlock being
|locked.

Goldberg and Saravanan seems quite explicit that threads can
acquire and hold spinlocks on OpenVMS, which seems to be
something that you were objecting to?

Turning to Windows, [Rus12] says about spin locks on page 179:

|Testing and acquiring the lock in one instruction prevents a
|second thread from grabbing the lock between the time the first
|thread tests the variable and the time it acquires the lock.

Note that, again, the language here is thread-centric.

By the way, regarding terminology, on page 177 [Rus12] also
says,

|Sections of code that access a nonshareable resource are called
|_critical_ sections.  To ensure correct code, only one thread
|at a time can execute in a critical section.

This is the generic definition of a critical section and note
that this is the context in which we are using the term
throughout this thread.

What you referred to earlier as a "critical section" in Windows
is a _userspace_ synchronization primitive with a kernel assist
and mostly unrelated to concerns around interruptibility.  From
the description on page 201 of [Rus12], they seem similar to
futexes on Linux, though somewhat more limited: Russinovich et
al states that they cannot be used between processes, while a
Linux futex can if it's in a region of memory shared between
cooperating processes.

>> Traditional Unix "sleep" and "wakeup" are an example of a
>> blocking protocol, where a thread may "sleep" on some "channel",
>> yielding the locking thread while the lock cannot be acquired,
>> presumably scheduling something else until the thread is later
>> marked runnable by virtual of something else calling "wakeup" on
>> the sleep channel.
>> 
>> But note that I am deliberately simplifying in order to
>> construct a particular scenario in which a light-weight
>> synchronization primitive is being used for mutual exclusion
>> between concurrent entities, hence mentioning spinlocks
>> specifically.
>> 
>> Suggesting that spin locks iare only applicable to
>> multiprocessing scenarios is flatly incorrect.
>
>I didn't.

Your exacty words were, "spinlocks coordinate mutual exclusion
between cpu cores."  See the text quoted above.  This is
incorrect in that it ignores that threads are, in contxt, the
software primitive for the computational entities that are doing
the acquiring, and while the "between cpu core" part is the
usual case, it is not the only case.

>I am saying the OS by design guards *ITS* different data structures
>with different mechanisms, and those mechanism must only be used in
>a very controlled manner. And if you want to do something that requires
>access to one of *ITS* data structures then you must obey *ITS* rules.

This is axiomatic.

But I fail to see how it's relevant to the topic at hand.  In
context we're discussing scenarios where a system may want to
disable interrupts entirely, and whether a particular aspect of
a hardware design is sufficient to implement critical sections
generally.

>In Windows and VMS and from what I understand of *nix spinlocks guard most
>of the core OS data structures with spinlocks acquired at the software
>interrupt level. If you are not at that level then the spinlock will not
>provide the necessary synchronization for access to that data.

Precisely.

>> Software may use
>> the technique to spin on a "busy" bit on some IP in a device for
>> example, even on a uniprocessor system.
>> 
>>>> A valid response here might be, "don't context switch from the
>>>> interrupt handler; use a DPC instead".  That may be valid, but
>>>> it puts a constraint on the software designer that may be onerus
>>>> in practice: suppose the interrupt happens in the scheduler,
>>>> while examining a run queue or something.  A DPC object must be
>>>> available, etc.
>>> That is exactly what DPC/SoftIrq are design to do - allow the bulk of
>>> the kernel, like the scheduler, non-paged heap management, IO pre and
>>> post processing, to be interrupted without causing reentrancy.
>>> The higher level device interrupt enqueues a request to the lower software
>>> interrupt level, which is processed when it will not cause reentrancy.
>>>
>>> That constraint is by design and any good OS will crash if violated.
>> 
>> Yes, I'm aware of the technique.  That wasn't the point.
>> 
>>>> Further, software must now consider the complexity of
>>>> potentially interruptable critical sections.  From the
>>>> standpoint of reasoning about already-complex concurrency issues
>>>> it's simpler to be able to assert that (almost) all interrupt
>>>> delivery can be cheaply disabled entirely, save for very
>>>> special, specific, scenarios like NMIs.  Potentially switching
>>>> away from a thread holding a spinlock sort of defeats the
>>>> purpose of a spinlock in the first place, which is a mutex
>>>> primitive designed to avoid the overhead of switching.

(It occurs to me that perhaps this is why you are going on about
how VMS and windows uses spin locks, and how they interact with
interrupt priority levels.)

>>> This is why I mentioned the terminology thing: threads do not hold
>>> spinlocks, they hold mutexes.
>> 
>> See above.  Threads can certainly "hold" a spin lock, as they
>> can hold any kind of lock.  To quote from sec 7.6.1 of [Val96],
>> page 202:
>
>When a particular OS defined spinlock is used to guard a particular OS
>defined data structure, exactly WHEN the OS allows this decides how
>the OS avoids all the nasty interrupt pitfalls you identified.

How is this relevant to what you wrote earlier and my response,
quoted just above?  My statement was that threads can hold spin
locks.  This is obviously true.

========== REMAINDER OF ARTICLE TRUNCATED ==========