Path: ...!weretis.net!feeder9.news.weretis.net!panix!.POSTED.spitfire.i.gajendra.net!not-for-mail
From: cross@spitfire.i.gajendra.net (Dan Cross)
Newsgroups: comp.arch
Subject: Re: MSI interrupts
Date: Wed, 26 Mar 2025 04:08:57 -0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Message-ID: <vrvukp$n29$1@reader1.panix.com>
References: <vqto79$335c6$1@dont-email.me> <e4ecfb53ef4f33f10b6d2548d5930159@www.novabbs.org> <vrva4r$9ka$1@reader1.panix.com> <b1c74762d01f71cc1b8ac838dcf6d4fa@www.novabbs.org>
Injection-Date: Wed, 26 Mar 2025 04:08:57 -0000 (UTC)
Injection-Info: reader1.panix.com; posting-host="spitfire.i.gajendra.net:166.84.136.80";
	logging-data="23625"; mail-complaints-to="abuse@panix.com"
X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
Originator: cross@spitfire.i.gajendra.net (Dan Cross)
Bytes: 10697
Lines: 250

In article <b1c74762d01f71cc1b8ac838dcf6d4fa@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:
>On Tue, 25 Mar 2025 22:19:07 +0000, Dan Cross wrote:
>
>> In article <e4ecfb53ef4f33f10b6d2548d5930159@www.novabbs.org>,
>> MitchAlsup1 <mitchalsup@aol.com> wrote:
>>
>>>> Your stuff sounds fine for getting into the critsec; but it
>>>> doesn't help you once you're in it, and it seems like it may
>>>> hurt you if some higher-priority thing comes along and yanks
>>>> your lock out from under you.
>>>
>>>I conceded that point at least 2 days ago.
>>
>> Ok, now we're back to the question then: what purpose does it
>> serve?
>
>You (well Chris) can write critical sections where a low priority
>holder can give way to the higher priority requestor such that
>the low priority holder reverts back to before it entered CS,
>and does not see the lock being stolen.

You still have not defined what you mean by, "the lock being
stolen."

Tell you what.  Earlier, you posted some code for a critical
section that inserts a node into a linked list.  That code is
written in terms of some primitives (presumably intrinsics,
though you call them macros); `esmLOCKload`, `esmLOCKprefetch`,
`esmINTERFERENCE` and `esmLOCKstore`.  Here's that code, slightly
reformatted:

// Removes the element `from` from a doubly linked list, and
// inserts it into a (possibly different) doubly linked list,
// after the element pointed to be `to`.
// Assumes <stdbool.h>
bool
MoveElement(Element *from, Element *to)
{
	assert(from != NULL);
	assert(to != NULL);

	Element *from_next = esmLOCKload(from->next);
	Element *from_prev = esmLOCKload(from->prev);
	Element *to_next = esmLOCKload(to->next);
	esmLOCKprefetch(from_next);
	esmLOCKprefetch(from_prev);
	esmLOCKprefetch(to_next);
	if (!esmINTERFERENCE()) {
		// Unlink the "from" node from its list.
		from_prev->next = from_next;  // XXX: `from_prev` may be nil?
		from_next->prev = from_prev;  // XXX: `from_next` may be nil?
		// Insert the node after "to".
		to->next = from;
		to_next->prev = from;         // XXX: `to_net` may be nil?
		from->prev = to;
		esmLOCKstore(from->next, to_next);
		return true;
	}
	return false;
}

Leaving aside the unchecked boundary conditions I highlighted
above, there are other questions that this raises.  You
mentioned that the ESM stuff can use up to 8 cache lines in a
single atomic "event"; let's count how many this code may
encounter:

1. `from` may point to a cache line
2. `to` may point to a different cache line.
3. Does your architecture use a stack?  What sort of alignment
   criteria do you impose?   At first blush, it seems to me that
   the pointers `from_next`, `from_prev`, and `to_next` could be
   on the stack and if so, will be on at least one cache line,
   and possibly 2, if the call frame for `MoveElement` spans
   more than one.  Of course, it's impossible to say when this
   is compiled, but maybe you require that the stack is always
   aligned to a cache line boundary or something.  Or maybe
   these are all in registers and it's fine.
4. Depending on the size and alignment criteria of the the
   `Element` structure and the placement of the `next` and
   `prev` elements in the struct (unknowable from this example),
   `to->next` may be on another cache line.
5. Similarly, `from->next`
6. And `from->prev`
7. And `from_prev->next`
8. And `from_next->prev`
9. And `to_next->prev`.

So potentially this code _could_ touch 9 or 10 cache lines, more
than you said are supported for a single atomic event.  What
happens in that case?  Does the code just always return false?
The programmer best make sure the optimizer is keeping those
temporary pointers off the stack; good luck with a debug build.

Anyway, let's assume that's not the case, and we don't hit the
pathological case with cache lines, and we're not dealing with
edge cases at the front or end of the lists (so various next or
prev pointers are not NULL).  Why don't I try to explain what I
think this code is doing, and you tell me whether I'm right or
wrong.

If I understand this correctly, the `esmLOCKload`s are simply
loads, but they make sure that the compiler emits instructions
that tie into your atomic support framework.  Earlier you wrote,
"esmINTERFERENCE is a query on the state of th event and whether
a 2nd party has performed a memory action that kills the event."
Ok, I take this to mean _at the time of the store_, which
appears later in the program.  If it were simply a query at the
time it was written in the code, then it seems like it opens a
TOCTOU bug (what if something interfers after the check, but
before the the locked store at the end?).  Or perhaps it signals
to the architecture that this is the point to which the
processor should rollback, if anything subsequently fails; sort
of a `setjmp` kind of thing that conceptually returns twice.

Assuming nothing has poisoned the event, the various pointer
stores inside the conditional are conditionally performed; the
`esmLOCKstore` to `from->next` commits the event, at which
point all side effects are exposed and durable.

Is that correct?  Let's suppose that it is.

But later you wrote,

>So, if you want the property whereby the lock disappears on any
>control transfer out of the event {exception, interrupt, SVC, SVR, ...};
>then you want to use my ATOMIC stuff; otherwise, you can use the
>normal ATOMIC primitives everyone and his brother provide.

Well, what precisely do you mean here when you say, "if you want
the property whereby the lock disappears on any control transfer
out of the event"?

What I _hope_ you mean is that if you transfer "out of the
event" before the `esmLOCKstore` then anything that's been done
since the "event" started is rolled back, but NOT if control
transfer happens _after_ the `esmLOCKstore`.  If THAT were the
case, then this entire mechanism is useless.

Perhaps you would clarify?

But for the time being, let's assume that `esmLOCKstore` is
conceptually a commit that ends the atomic event, and the store
is durable and observable after that.  Let's see if we can
construct a simple spinlock from these primitives.

typedef struct Spinlock Spinlock;
struct Spinlock {
	uint32_t lock;
};

uint32_t
xcas(volatile Spinlock *lk, uint32_t expected, uint32_t val)
{
	uint32_t v = esmLOCKload(&lk->lock);
	if (v == expected && !esmINTERFERENCE()) {
		esmLOCKstore(&lk->lock, val)
	}
	return v;
}

void
spin_acquire(volatile Spinlock *lk)
{
	splhi();
	while (xcas(lk, 0, 1) != 0)
		relax();	// Spin loop intrinsic.
}

void
spin_release(volatile Spinlock *lk)
{
	xcas(lk, 1, 0);
	// Maybe we should `spllo` here, maybe not.  Either choice
	// reflects lots of assumptions about whether or not we might
	// take and hold multiple locks concurrently, etc.
	spllo();
}

Does this seem correct?  It is not immediately clear to me
whether `esmINTERFERENCE` is actually needed here, but
avoiding the store seemed fine.  One could imagine shorting
`xcas` to simply:
========== REMAINDER OF ARTICLE TRUNCATED ==========