Path: ...!news.mixmin.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: "Randy Brukardt" <randy@rrsoftware.com>
Newsgroups: comp.lang.ada
Subject: Re: "Red" And The DoD Language Competition
Date: Sat, 14 Sep 2024 01:27:22 -0500
Organization: A noiseless patient Spider
Lines: 32
Message-ID: <vc3acd$1b4vk$1@dont-email.me>
References: <vbdndk$lt27$1@dont-email.me> <vbhvre$1eabl$1@dont-email.me><vbimad$1j26j$6@dont-email.me> <vbtsbq$38je$1@dont-email.me><vbvpta$esm6$8@dont-email.me><0001HW.2C93AE0500F5F854306B4D38F@news.individual.net><vc058k$h5a9$1@dont-email.me> <87seu42t7x.fsf@nightsong.com> <vc091i$ljiq$2@dont-email.me>
Injection-Date: Sat, 14 Sep 2024 08:27:26 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="252dd07fce3ba32f52749428b1b0da73";
	logging-data="1414132"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX19MeyAzG6b1XAFIr2Xe8/ZvfSRevrWXATU="
Cancel-Lock: sha1:HgdW2KXZ4LqEqPhmFLoVWgvWpG4=
X-Newsreader: Microsoft Outlook Express 6.00.2900.5931
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.7246
X-RFC2646: Format=Flowed; Original
X-MSMail-Priority: Normal
X-Priority: 3
Bytes: 2740

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message 
news:vc091i$ljiq$2@dont-email.me...
> On Thu, 12 Sep 2024 19:16:50 -0700, Paul Rubin wrote:
>
>> I run into sites all the time that block the wget user agent, but that I
>> can retrieve with curl.
>
> And I run into sites all the time that block the default wget user agent,
> but that I can retrieve with wget.

You're confused. The attackers aren't using Wget, but they are *claiming* to 
be WGet. As you point out, real WGet users tend to claim to be other things. 
So blocking WGet would be more likely to block the attackers than real 
users. (As you state, real users know how to get around the blocks, so the 
inconvinience for them is minor. Usually, the attackers don't change their 
attacks often, there's plenty of sites that don't protect themselves at all. 
So they are more effective against attackers.)

And anyone that thinks that ad revenue is important is probably blocking all 
grabbers, and probably throttling everything else so that grabbing multiple 
pages is very slow (at human reading speeds). (At least 90% of the browser 
hits I see are obviously fake, and if I cared enough I would block all of 
them - it would just take a bit of programming to check if the behavior is 
similar to that of a live human. But I only block when something is causing 
performance problems, and generally by IP.)

                     Randy.


                               Randy.