Deutsch English Français Italiano |
<vmcvoo$3umjn$1@dont-email.me> View for Bookmarking (what is this?) Look up another Usenet article |
Path: ...!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!eternal-september.org!.POSTED!not-for-mail From: Paul <nospam@needed.invalid> Newsgroups: comp.os.linux.advocacy,alt.comp.os.windows-10 Subject: Re: Cult of Unix Date: Fri, 17 Jan 2025 02:10:16 -0500 Organization: A noiseless patient Spider Lines: 106 Message-ID: <vmcvoo$3umjn$1@dont-email.me> References: <1VcgP.54962$XfF8.39289@fx04.iad> <vm1itg$1f1ma$3@dont-email.me> <vm40cl$21e8l$2@dont-email.me> <6h1bojt7kdp4d5euq0f78rtuvqpg7edc3e@4ax.com> <HHghP.135123$5c34.129668@fx47.iad> <la6bojl7t4686ll2teomlj0ig70ma8o8c8@4ax.com> <Q3hhP.45732$nlJ1.37298@fx41.iad> <ru7bojpl0j6ot182uuhhvrakcflqsadi30@4ax.com> <vm48v6$23a1f$2@dont-email.me> <luodsdF6geaU1@mid.individual.net> <vm755l$2lqjk$1@dont-email.me> <vma3u7$3cdmt$1@dont-email.me> <vmb8t8$3id01$1@dont-email.me> <lusp13Frb66U1@mid.individual.net> <vmbrbq$3ln7i$1@dont-email.me> <vmc3cq$3n232$1@dont-email.me> <vmcgfd$3osq8$2@dont-email.me> <luu2eeF3kf9U1@mid.individual.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Injection-Date: Fri, 17 Jan 2025 08:10:18 +0100 (CET) Injection-Info: dont-email.me; posting-host="d5b086e1984efeff9d44e00ff8712524"; logging-data="4151927"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/Kcss3tYzXBI+5qQwqF28qDXa32IhLIbY=" User-Agent: Ratcatcher/2.0.0.25 (Windows/20130802) Cancel-Lock: sha1:CN0t75lvcaaDItMDULS+jPs1iiw= Content-Language: en-US In-Reply-To: <luu2eeF3kf9U1@mid.individual.net> Bytes: 5933 On Thu, 1/16/2025 10:51 PM, vallor wrote: > On Fri, 17 Jan 2025 02:49:17 -0000 (UTC), Lawrence D'Oliveiro > <ldo@nz.invalid> wrote in <vmcgfd$3osq8$2@dont-email.me>: > >> On Thu, 16 Jan 2025 18:06:00 -0500, Paul wrote: >> >>> You can freeze a copy of C: for example, and run a Robocopy over it. >> >> Until you hit the limitations of Windows and Robocopy, and have to give >> up on it and use Linux instead. >> >> <https://www.theregister.com/2010/09/24/sysadmin_file_tools/> > > I suspect things may have changed in the last *14 years*. > I think we discussed this one already. That's *not* how you transfer sixty million files. You image the partition with a cluster level tool, and by comparison to file level, it screams. It runs at disk speed. Then at the other machine, you restore at the cluster level. Now, all sixty million files are there, and you've saved enough time to take a vacation. I have samples of fragmented volumes here, stored as .mrimg . If I want to show an example of a fragmented volume, I can load one of those up in about ten minutes or so. The top one, is 64 million files stored in a single directory. The second one is 64 million files in a tree. writeflat-64million-Ddrive-736391-00-00.mrimg 2,323,353,194 bytes writeidx4-64million-Ddrive-736391-00-00.mrimg 2,381,377,130 bytes It's the same with using 7ZIP for archiving. 7ZIP can run pretty fast, on solid block files of a good size. However, if you feed it a tree structure full of small files, it doesn't matter how fast your CPU is, the program "starves" for want of responses from the file system. You might get 10MB/sec from it. That's because the disk heads are flying around. Even if you use my RAMDrive for this, it's *still* slow. That doesn't help either, as just the file system stack takes too long per file handled. But working at the cluster level works a treat, a backup and restore and you're done. And it isn't even lunch hour yet. It does take time, for the backup tool to "crawl" the tree and make the index they like to use. That's still an expensive part of it. If you attack that problem file-by-file, it's the heat death of the universe bad. dir in command prompt took four minutes. Dumped to list.txt $ tail list.txt 05/26/2024 06:09 AM 7 3FFFFF8.txt 05/26/2024 06:09 AM 7 3FFFFF9.txt 05/26/2024 06:09 AM 7 3FFFFFA.txt 05/26/2024 06:09 AM 7 3FFFFFB.txt 05/26/2024 06:09 AM 7 3FFFFFC.txt 05/26/2024 06:09 AM 7 3FFFFFD.txt 05/26/2024 06:09 AM 7 3FFFFFE.txt 05/26/2024 06:09 AM 7 3FFFFFF.txt 67108864 File(s) 469,762,048 bytes 1 Dir(s) 7,497,859,072 bytes free $ head list.txt Volume in drive F is RAMDrive Volume Serial Number is 50A8-F4E5 Directory of F:\out 05/26/2024 06:09 AM <DIR> . 05/26/2024 04:47 AM 7 0000000.txt 05/26/2024 04:47 AM 7 0000001.txt 05/26/2024 04:47 AM 7 0000002.txt 05/26/2024 04:47 AM 7 0000003.txt $ wc -l list.txt 67108872 list.txt Windows has a copy of tar.exe in System32 now, and I can try that, but it processes about 10MB/sec. A wild guess is that this might take two hours, but I can't really be sure. (You can't afford to turn on Verbose, or the terminal might become the limiting factor.) I'm trying to avoid explorer.exe getting a "whiff" of the folder, because once that happens, explorer rails on one core as it counts files and so on. The way to stop explorer.exe from working on that, is to rip the file system away, and once the "pointer" to the file system is not available any more, it stops. You would not want explorer.exe doing the copy. tar.exe is looking less efficient than a backup and restore (the restore of 64 million files took about 3 minutes 40 seconds). You can also kill explorer.exe in Task Manager and start another one, if you like. Robocopy should not be counting files like explorer.exe does when it prepares to do a copy, but I would expect it's going to be one of those two hour things. It's going to be in a different ballpark. Paul