Deutsch English Français Italiano |
<mailman.31.1727920575.3018.python-list@python.org> View for Bookmarking (what is this?) Look up another Usenet article |
Path: ...!eternal-september.org!feeder3.eternal-september.org!fu-berlin.de!uni-berlin.de!not-for-mail From: Left Right <olegsivokon@gmail.com> Newsgroups: comp.lang.python Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API Date: Thu, 3 Oct 2024 00:56:36 +0200 Lines: 36 Message-ID: <mailman.31.1727920575.3018.python-list@python.org> References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com> <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org> <9dfcd123-c31d-4207-869c-d5466487cba4@tompassin.net> <CAJQBtgkLVyNK+vw4u3bFCFEQDH8T3rpyTL+ERyyYHZJskQR6PQ@mail.gmail.com> <CAJQBtgnpNkpg-mF2yFCS4P4GYAYsKQ9nEw3Xygja=SE3-=N2Dw@mail.gmail.com> <mailman.19.1727796506.3018.python-list@python.org> <lm391bFu38hU1@mid.individual.net> <CAJQBtgmZehSeBu0y73ALdVq00LHi-R_KKS893FwJkEjkLnsXtA@mail.gmail.com> <CAPTjJmq6QUcBgkNcn50VzyyHoDAEE1JLPgPU+segiEykcieVSw@mail.gmail.com> <CAJQBtgkWcDH-7c8xTF84bxfbkvOURTBd80A6JBkEKn-f6Xvnew@mail.gmail.com> <CAPTjJmoWHrKCmktm=4bzCS2dekbR6=u9PD6gc=LZfo+4dq=7zQ@mail.gmail.com> <CAJQBtgmCyaYSN44U5rZttxdGgbsWFmKbiArZxMf+jMc6pfqobA@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Trace: news.uni-berlin.de XO4Fy/wOi76DBWZFMHG+0gPN1EjEeaFABaH0mepa94Uw== Cancel-Lock: sha1:MLHY1Rvn3qEhFYMJjtklsalwSfo= sha256:v4zbvVzwOMmbRtx/vfc0PBRDr+qrmAMPtmVB77Xtg08= Return-Path: <olegsivokon@gmail.com> X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org Authentication-Results: mail.python.org; dkim=pass reason="2048-bit key; unprotected key" header.d=gmail.com header.i=@gmail.com header.b=DQ9WmjF4; dkim-adsp=pass; dkim-atps=neutral X-Spam-Status: OK 0.024 X-Spam-Evidence: '*H*': 0.95; '*S*': 0.00; 'subject:API': 0.07; 'angelico': 0.09; 'cc:addr:python-list': 0.09; 'general,': 0.09; 'language,': 0.09; 'like,': 0.09; 'cc:no real name:2**0': 0.14; '2024': 0.16; 'alphabet': 0.16; 'chrisa': 0.16; 'examples,': 0.16; 'length.': 0.16; 'mount': 0.16; 'packets': 0.16; 'rude': 0.16; 'words.': 0.16; 'wrote:': 0.16; 'subject:Help': 0.17; "can't": 0.17; 'thu,': 0.19; 'cc:addr:python.org': 0.20; 'language': 0.21; 'languages': 0.22; 'idea': 0.24; 'saying': 0.25; 'cc:2**0': 0.25; 'seems': 0.26; 'bit': 0.27; 'chris': 0.28; 'language.': 0.32; 'validate': 0.32; 'message-id:@mail.gmail.com': 0.32; 'subject:for': 0.33; 'hold': 0.33; 'someone': 0.34; 'header:In- Reply-To:1': 0.34; 'received:google.com': 0.34; 'trying': 0.35; 'words': 0.35; 'from:addr:gmail.com': 0.35; 'subject:from': 0.37; 'way': 0.38; 'single': 0.39; 'wrote': 0.39; 'tell': 0.60; 'between': 0.63; 'your': 0.64; 'company': 0.64; 'look': 0.65; 'let': 0.66; 'worked': 0.67; 'right': 0.68; 'playing': 0.69; 'subject:Data': 0.71; 'little': 0.73; 'left': 0.83; 'distinction': 0.84; 'ridiculous.': 0.84; 'subject: \n ': 0.84; 'you:': 0.84; 'pleasure': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727909807; x=1728514607; darn=python.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=iYf6e0HYfyqnGUXTmZEASpTHJaMTDquMBfhINep1P44=; b=DQ9WmjF4jQpmo2PqduyHIo2/qlkejO0uhxleDC1tDm5aBjrcy/Ojcvkn93XckY4B1X 88BfVy/XRzAtOj9EoSUyru/sHMyWUoOSikMnyrCSCrUF2HCygt066lzRLch+sDffcdSG zyfyWIVLXFzUTtFyE18m5w46oJIuQjuYNizRJCbiUBQbd3N3Mvf7g0qmxusCwjtexu6I 95JUW0lH+puksIfP61IN+onwGXYpvfGCS5uu9UtYgq9t0iZwHfWLrK8SUDQmxeKi8GVj xgu/xccnymJHcugeEfXvSsiwsuAv15ce+cLSou+ZPwMvCQtKOitSApJlc2Ys9jv+lEgN xZcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727909807; x=1728514607; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iYf6e0HYfyqnGUXTmZEASpTHJaMTDquMBfhINep1P44=; b=tGCTVljbCZXzz8fmYXLcm2X/CyfRUKoTpuwESxhofM7/x26wvYnlYVKyJdfp/vtsFf lukziwiZMf5EZwDUz48gIn7MUoilfyD2+3oQie+EgseqM8+JbK7CS4EhcxaH/rUfm4Cd 0dccuul4wI/hZBONBJvMSnMkSXlA3azoXPVR6Ty2zSzNXUUijUpvHhPQaANjZYPc1QBu 2CBLl0PyCwb78Bt7wdF+OEBh1IZrhoG5lFPf7nK4FbXMF73ttx70BXsBAnexhSmwwm6b SCZdjij9uvbDqmbjsXJHNLAbKEsR9Gw0bX8ofJnk9Sc1Em0cBpK7uxOjzlQYnsjqrOTq 6wgg== X-Gm-Message-State: AOJu0YwRJD7kZo0L8ehPfBcuQFKRX2w4WUCNNNwbudCAG6DzWwbGID/z qMaO19D48qWPVkOo0v1Liw0O1Vp+94cQFnPNEQHIDHTeP+CSiWvR6SXjUQulWUGJs2kf1NYAFi/ gIMceeOv1VFBo4KGI4hiFsNSXtMI= X-Google-Smtp-Source: AGHT+IG/iHQEVyXUcA97HYj03IvibKTZgGCDcdaZaS2B5oRUs2kZZP93jgqV8aGJ2InulAFyIWpMycO5HC7tsRiPmG8= X-Received: by 2002:a05:6214:5b02:b0:6c7:c645:ddf2 with SMTP id 6a1803df08f44-6cb81a36e5amr63636246d6.30.1727909807290; Wed, 02 Oct 2024 15:56:47 -0700 (PDT) In-Reply-To: <CAPTjJmoWHrKCmktm=4bzCS2dekbR6=u9PD6gc=LZfo+4dq=7zQ@mail.gmail.com> X-Mailman-Approved-At: Wed, 02 Oct 2024 21:56:13 -0400 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: General discussion list for the Python programming language <python-list.python.org> List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe> List-Archive: <https://mail.python.org/pipermail/python-list/> List-Post: <mailto:python-list@python.org> List-Help: <mailto:python-list-request@python.org?subject=help> List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe> X-Mailman-Original-Message-ID: <CAJQBtgmCyaYSN44U5rZttxdGgbsWFmKbiArZxMf+jMc6pfqobA@mail.gmail.com> X-Mailman-Original-References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com> <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org> <9dfcd123-c31d-4207-869c-d5466487cba4@tompassin.net> <CAJQBtgkLVyNK+vw4u3bFCFEQDH8T3rpyTL+ERyyYHZJskQR6PQ@mail.gmail.com> <CAJQBtgnpNkpg-mF2yFCS4P4GYAYsKQ9nEw3Xygja=SE3-=N2Dw@mail.gmail.com> <mailman.19.1727796506.3018.python-list@python.org> <lm391bFu38hU1@mid.individual.net> <CAJQBtgmZehSeBu0y73ALdVq00LHi-R_KKS893FwJkEjkLnsXtA@mail.gmail.com> <CAPTjJmq6QUcBgkNcn50VzyyHoDAEE1JLPgPU+segiEykcieVSw@mail.gmail.com> <CAJQBtgkWcDH-7c8xTF84bxfbkvOURTBd80A6JBkEKn-f6Xvnew@mail.gmail.com> <CAPTjJmoWHrKCmktm=4bzCS2dekbR6=u9PD6gc=LZfo+4dq=7zQ@mail.gmail.com> Bytes: 8088 > One single IP packet is all you can parse. I worked for an undisclosed company which manufactures h/w for ISPs (4- and 8-unit boxes you mount on a rack in a datacenter). Essentially, big-big routers. So, I had the pleasure of writing software that parses IP _protocol_, and let me tell you: you have no idea what you just wrote. But, like I wrote earlier: you don't understand the distinction between languages and words. And in general, are just being stubborn and rude because you are trying to prove a point to someone you don't like, but, in reality, you just look more and more ridiculous. On Thu, Oct 3, 2024 at 12:51=E2=80=AFAM Chris Angelico <rosuav@gmail.com> w= rote: > > On Thu, 3 Oct 2024 at 08:48, Left Right <olegsivokon@gmail.com> wrote: > > > > > You can't validate an IP packet without having all of it. Your notion > > > of "streaming" is nonsensical. > > > > Whoa, whoa, hold your horses! "nonsensical" needs a little bit of > > justification :) > > > > It seems you don't understand the difference between words and > > languages! In my examples, IP _protocol_ is the language, sequences of > > IP packets are the words in the language. A language is amenable to > > streaming if the words of the language are repetition of sequences of > > symbols of the alphabet of fixed length. This is, essentially, like > > saying that the words themselves are regular. > > One single IP packet is all you can parse. You're playing shenanigans > with words the way Humpty Dumpty does. IP packets are not sequences, > they are individuals. > > ChrisA