Path: ...!news.mixmin.net!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: Chris Angelico <rosuav@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
 GB) from Kenna API
Date: Thu, 3 Oct 2024 08:51:01 +1000
Lines: 20
Message-ID: <mailman.29.1727909476.3018.python-list@python.org>
References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org>
 <9dfcd123-c31d-4207-869c-d5466487cba4@tompassin.net>
 <CAJQBtgkLVyNK+vw4u3bFCFEQDH8T3rpyTL+ERyyYHZJskQR6PQ@mail.gmail.com>
 <CAJQBtgnpNkpg-mF2yFCS4P4GYAYsKQ9nEw3Xygja=SE3-=N2Dw@mail.gmail.com>
 <mailman.19.1727796506.3018.python-list@python.org>
 <lm391bFu38hU1@mid.individual.net>
 <CAJQBtgmZehSeBu0y73ALdVq00LHi-R_KKS893FwJkEjkLnsXtA@mail.gmail.com>
 <CAPTjJmq6QUcBgkNcn50VzyyHoDAEE1JLPgPU+segiEykcieVSw@mail.gmail.com>
 <CAJQBtgkWcDH-7c8xTF84bxfbkvOURTBd80A6JBkEKn-f6Xvnew@mail.gmail.com>
 <CAPTjJmoWHrKCmktm=4bzCS2dekbR6=u9PD6gc=LZfo+4dq=7zQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de xAsiMeYieY0cUWzUxdSmFAnDzrSSU4w3QiGsdq5v2cwg==
Cancel-Lock: sha1:GZ7I7TkJSPpm/0g9GFiwK2CN2uY= sha256:84br6gQ4iJoQ4J1fcsn22s84uv5nws7fXwvizXriHvk=
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
 reason="2048-bit key; unprotected key"
 header.d=gmail.com header.i=@gmail.com header.b=MojOcFHN;
 dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.013
X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'subject:API': 0.07;
 'cc:addr:python-list': 0.09; 'language,': 0.09; 'cc:no real
 name:2**0': 0.14; '2024': 0.16; 'alphabet': 0.16; 'chrisa': 0.16;
 'examples,': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris
 angelico': 0.16; 'length.': 0.16; 'packets': 0.16; 'wrote:': 0.16;
 'subject:Help': 0.17; "can't": 0.17; 'thu,': 0.19;
 'cc:addr:python.org': 0.20; 'language': 0.21; 'saying': 0.25;
 'cc:2**0': 0.25; 'seems': 0.26; 'bit': 0.27; 'language.': 0.32;
 'validate': 0.32; 'message-id:@mail.gmail.com': 0.32;
 'subject:for': 0.33; 'hold': 0.33; 'header:In-Reply-To:1': 0.34;
 'received:google.com': 0.34; 'words': 0.35; 'from:addr:gmail.com':
 0.35; 'subject:from': 0.37; 'way': 0.38; 'single': 0.39;
 'between': 0.63; 'your': 0.64; 'right': 0.68; 'playing': 0.69;
 'subject:Data': 0.71; 'little': 0.73; 'left': 0.83; 'subject: \n
 ': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1727909474; x=1728514274; darn=python.org;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:from:to:cc:subject:date:message-id:reply-to;
 bh=bbnsjas95O8EixlyTTDCuuiKnWMVbSlcVLCCl0x2g3Y=;
 b=MojOcFHN3SErb/1VPGLwm0+3EEESHeflV0ra/Fn70RCs4uDrhOis/s6GWPUsMTLcdo
 tGEBk7/L4sYTwCwtmyy7k1rtBgiR4lsq6ZtG2AL8JSj9h6MPdwtUFIYPKg5+xQxcq+12
 4223QQUDVxNJajBF3kkFKCsssN6uYyXw2GAcG5/7LkZM1btgFQ5D3mb6t0v6E03cPWIR
 qxWHGMKVZIAcaeg3sVK/yQPGGjvn/qRTaHxw7Kh3DPaBD9WAvAb2zNGIiR905N2m9HnD
 xEr3waYMePq3pP1gu7dLfCRbQqJZ5J2KCtckL24IWHI6SULty/W8bfdM6MapLEM5Y4tz
 +/eA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1727909474; x=1728514274;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=bbnsjas95O8EixlyTTDCuuiKnWMVbSlcVLCCl0x2g3Y=;
 b=CPBS5IASWDLEsY01Is5HRte9p/H+pGsEcMm6Wa6qo07g+PJX1noAjTDiWhqWSuNSoM
 5cVG5sl3hOoNRc/Jb3cEb3HwSNy0ncrlnp1qyjnIMhJDGeKzmXAUhJfpPAkl+1iGIlfF
 BwCzcjOF3hjAt+gXMzEFRTtR+XvXhEYGQvdxcoQY424jMSaG5rOS/DK7qMucnhLjDxmS
 Y6uSjW7Cgcmo81RZ6jRs6Hp4ne0S8/d4bO+PCgokw+kVqHoE9SwkPDfmETdQ1bZtk4Cg
 lsHoBHkF+soaR+RREvBeVc+dEDqz0iIshHbEl9bj0s3xDO9WrUK32oSXsvWEbsCQz8Dr
 eOiA==
X-Gm-Message-State: AOJu0YzIIROQYt6be/WcERugPsbzIs1VmbEVjTpficQBYCBqFKXBQ/zf
 499XjXCBPXmtTUOeiQG2xPrr/i1jGUxF0yUsFLFFhgZJimToRcxiTWAbG3n7gqrjILAaFxdKYCu
 WKZQmIhKR8GlYMIb/pOMPydSZHAo=
X-Google-Smtp-Source: AGHT+IEdC46gfXUPSgEJaDMOMx5SQJuJ4Mc7Q61CsVpFB27q/Z00KOJzZ1I3e52777hxE77glzgPFEPOa82gH4u283A=
X-Received: by 2002:a05:6512:3d8d:b0:539:530e:9de5 with SMTP id
 2adb3069b0e04-539a07a89f2mr2872604e87.56.1727909473384; Wed, 02 Oct 2024
 15:51:13 -0700 (PDT)
In-Reply-To: <CAJQBtgkWcDH-7c8xTF84bxfbkvOURTBd80A6JBkEKn-f6Xvnew@mail.gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
 <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
 <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
 <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmoWHrKCmktm=4bzCS2dekbR6=u9PD6gc=LZfo+4dq=7zQ@mail.gmail.com>
X-Mailman-Original-References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org>
 <9dfcd123-c31d-4207-869c-d5466487cba4@tompassin.net>
 <CAJQBtgkLVyNK+vw4u3bFCFEQDH8T3rpyTL+ERyyYHZJskQR6PQ@mail.gmail.com>
 <CAJQBtgnpNkpg-mF2yFCS4P4GYAYsKQ9nEw3Xygja=SE3-=N2Dw@mail.gmail.com>
 <mailman.19.1727796506.3018.python-list@python.org>
 <lm391bFu38hU1@mid.individual.net>
 <CAJQBtgmZehSeBu0y73ALdVq00LHi-R_KKS893FwJkEjkLnsXtA@mail.gmail.com>
 <CAPTjJmq6QUcBgkNcn50VzyyHoDAEE1JLPgPU+segiEykcieVSw@mail.gmail.com>
 <CAJQBtgkWcDH-7c8xTF84bxfbkvOURTBd80A6JBkEKn-f6Xvnew@mail.gmail.com>
Bytes: 6733

On Thu, 3 Oct 2024 at 08:48, Left Right <olegsivokon@gmail.com> wrote:
>
> > You can't validate an IP packet without having all of it. Your notion
> > of "streaming" is nonsensical.
>
> Whoa, whoa, hold your horses! "nonsensical" needs a little bit of
> justification :)
>
> It seems you don't understand the difference between words and
> languages! In my examples, IP _protocol_ is the language, sequences of
> IP packets are the words in the language. A language is amenable to
> streaming if the words of the language are repetition of sequences of
> symbols of the alphabet of fixed length.  This is, essentially, like
> saying that the words themselves are regular.

One single IP packet is all you can parse. You're playing shenanigans
with words the way Humpty Dumpty does. IP packets are not sequences,
they are individuals.

ChrisA