Path: ...!eternal-september.org!feeder3.eternal-september.org!panix!.POSTED.spitfire.i.gajendra.net!not-for-mail
From: cross@spitfire.i.gajendra.net (Dan Cross)
Newsgroups: comp.arch
Subject: Re: MSI interrupts
Date: Fri, 28 Mar 2025 02:53:57 -0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Message-ID: <vs5305$rd4$1@reader1.panix.com>
References: <vqto79$335c6$1@dont-email.me> <7a093bbb356e3bda3782c15ca27e98a7@www.novabbs.org> <vs41ap$n43$1@reader1.panix.com> <34434320650f5844b18b1c0b684acf43@www.novabbs.org>
Injection-Date: Fri, 28 Mar 2025 02:53:57 -0000 (UTC)
Injection-Info: reader1.panix.com; posting-host="spitfire.i.gajendra.net:166.84.136.80";
	logging-data="28068"; mail-complaints-to="abuse@panix.com"
X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
Originator: cross@spitfire.i.gajendra.net (Dan Cross)
Bytes: 7667
Lines: 225

In article <34434320650f5844b18b1c0b684acf43@www.novabbs.org>,
MitchAlsup1 <mitchalsup@aol.com> wrote:
>On Thu, 27 Mar 2025 17:19:21 +0000, Dan Cross wrote:
>> In article <7a093bbb356e3bda3782c15ca27e98a7@www.novabbs.org>,
>> MitchAlsup1 <mitchalsup@aol.com> wrote:
>-------------------
>> Or maybe even just swapping places between two elements in a
>> linked list.  For example:
>>
>> void
>> swap_places(Node **head, Node *a, Node *b)
>> {
>>         Node *hp, *an, *ap, *bn, *bp;
>>
>>         assert(head != NULL);
>>         assert(a != NULL);
>>         assert(b != NULL);
>>
>>         if (a == b)
>>                 return;
>>
>> 	  esmLOCKprefetch(*head);
>
>This should be a load not prefetch--you want the value of *head

I wondered about that.  I ASSumed that `esmLOCKprefetch` was
enough to set up a cache line address monitor on `*head`, but if
you tell me I need `esmLOCKload`, then I need `esmLOCKload`:

	hp = esmLOCKload(*head);
	if (hp == a)
		*head = b;  // Note I need to set whatever head points to, not hp.
	else if (hp == b)
		*head = a;

(You can see a vestige of me trying that earlier by the `hp`
temporary that I left in when I posted that initially, but I
didn't like how it looked, so I changed it to what was posted.)

>>         if (*head == a) 	// see !
>>                 *head = b;
>>         else if (*head == b)
>>                 *head = a;
>
>There is a ESM rule that states:: all participating cache lines
>must be touched before any participating cache lines can be
>modified.

Are these rules in a reference somewhere I can see?  I kinda
reckoned `esmLOCKprefetch` would be enough to "touch" the line,
but I fully admit I was just guessing.  Looks like `prefetch` is
mostly for the case where you'll store, but don't need to read?

>Also note: Participating cache lines are checked for write permission
>at touch time, and on cache miss, read with intent to modify.

Ok.

>>         an = esmLOCKload(a->next);
>>         ap = esmLOCKload(a->prev);
>>         bn = esmLOCKload(b->next);
>>         bp = esmLOCKload(b->prev);
>>
>>         b->next = an;
>>         if (an != NULL) {
>>                 esmLOCKprefetch(an->prev);
>>                 an->prev = b;
>> 		}
>>         b->prev = ap;
>>         if (ap != NULL) {
>>                 esmLOCKprefetch(ap->next);
>>                 ap->next = b;
>> 		}
>>
>>         a->next = bn;
>>         if (bn != NULL) {
>> 				esmLOCKprefetch(bn->prev);
>>                 bn->prev = a;
>> 		}
>>         if (bp != NULL) {
>> 		        esmLOCKprefetch(bp->next);
>>                 bp->next = a;
>> 		}
>>         esmLOCKstore(a->prev, bp);
>> }
>
>What I think you want:: (ignoring the 9 participants limit)

You mean 8 participants limit?  That counting thing again.  ;-}

What I really wanted was an example that exceeded that limit as
an expository vehicle for understanding what happens when the
limit is exceeded.  What does the hardware do in that case?

>> void
>> swap_places(Node **head, Node *a, Node *b)
>> {
>>         Node *hp, *an, *ap, *bn, *bp;
>>
>>         assert(head != NULL);
>>         assert(a != NULL);
>>         assert(b != NULL);
>>
>>         if (a == b)
>>                 return;
>
>top_of_ATOMIC_event:
>
>// this is the recovery point is you don't use esmINTERFERENCE()
>// the very next instruction begins the event.

Yup.  Tracking.

>>         esmLOCKprefetch( an = esmLOCKload(a->next) );
>>         esmLOCKprefetch( ap = esmLOCKload(a->prev) );
>>         esmLOCKprefetch( bn = esmLOCKload(b->next) );
>>         esmLOCKprefetch( bp = esmLOCKload(b->prev) );

What exactly does this do?  This "touches" whatever lines the
things that `an`, `ap`, `bn`, and `bp` are all in, as well as
the lines that the `a` and `b` prev and next pointers point
into as well, right?

> 	  Node *Ehead = esmLOCKload(*head);

You shoulda just used the `hp` local I left for you for exactly
this.  :-D

>// by placing all the the touching before any manifestation, you put
>// all the touch latency* in the code before it has tried to damage any
>// participating memory location. (*) and TLB latency and 2nd party
>// observation of your event.
>
>// this would be the point where you would insert if( esmINTERFERENCE(
>))
>// if you wanted control at a known failure point rather than at the
>// top of the event on failure.
>
>>         if (Ehead == a)
>>                 *head = b;
>>         else if (Ehead == b)
>>                 *head = a;
>>
>>         b->next = an;
>>         if (an != NULL) {
>>                 an->prev = b;
>> 		}
>>         b->prev = ap;
>>         if (ap != NULL) {
>>                 ap->next = b;
>> 		}
>>
>>         a->next = bn;
>>         if (bn != NULL) {
>>                 bn->prev = a;
>> 		}
>>         if (bp != NULL) {
>>                 bp->next = a;
>> 		}
>>         esmLOCKstore(a->prev, bp);
>> }
>
>// now manifestation has lowest possible latency (as seen by this core
>alone)

Doesn't this code now assume that `an->prev` is in the same
cache line as `an`, and similarly that `bn` is in the same line
as `bn->prev`? Absent an alignment constraint on the `Node`
type, that's not guaranteed given the definition posted earlier.

That may be an odd and ill-advised thing for a programmer to do
if they want their list type to work with atomic events, but it
is possible.

The node pointers and their respective `next` pointers are okay,
so I wonder if perhaps this might have been written as:

void
swap_places(Node **head, Node *a, Node *b)
{
        Node *hp, *an, *ap, *bn, *bp;

        assert(head != NULL);
        assert(a != NULL);
========== REMAINDER OF ARTICLE TRUNCATED ==========