Path: ...!feeds.phibee-telecom.net!2.eu.feeder.erje.net!feeder.erje.net!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: Greg Ewing <greg.ewing@canterbury.ac.nz>
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
 GB) from Kenna API
Date: Wed, 2 Oct 2024 18:27:54 +1300
Lines: 17
Message-ID: <lm43usF3fl1U1@mid.individual.net>
References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org>
 <9dfcd123-c31d-4207-869c-d5466487cba4@tompassin.net>
 <CAJQBtgkLVyNK+vw4u3bFCFEQDH8T3rpyTL+ERyyYHZJskQR6PQ@mail.gmail.com>
 <CAJQBtgnpNkpg-mF2yFCS4P4GYAYsKQ9nEw3Xygja=SE3-=N2Dw@mail.gmail.com>
 <mailman.19.1727796506.3018.python-list@python.org>
 <lm391bFu38hU1@mid.individual.net>
 <020101db1459$65b0c4d0$31124e70$@gmail.com>
 <mailman.24.1727825216.3018.python-list@python.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: individual.net vafC4e745QwBHchUd864oga9JEw8opcXUh5GjyTcN8RbnsM9Pq
Cancel-Lock: sha1:WQKbVzh6QOaQSUNzorDoy8f8bxk= sha256:rUZbI3LzG/sfqNz7KEjPnhrWppqBPrbEsYvnHsK7R4c=
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:91.0)
 Gecko/20100101 Thunderbird/91.3.2
Content-Language: en-US
In-Reply-To: <mailman.24.1727825216.3018.python-list@python.org>
Bytes: 2117

On 2/10/24 12:26 pm, avi.e.gross@gmail.com wrote:
> The real problem is how the JSON is set up. If you take umpteen data
> structures and wrap them all in something like a list, then it may be a tad
> hard to stream as you may not necessarily be examining the contents till the
> list finishes gigabytes later.

Yes, if you want to process the items as they come in, you might
be better off sending a series of separate JSON strings, rather than
one JSON string containing a list.

Or, use a specialised JSON parser that processes each item of the
list as soon as it's finished parsing it, instead of collecting the
whole list first.

-- 
Greg