[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Wget-dev] wget2 | Add proper WARC API + support by wget2 (#65)

From: Chillar Anand
Subject: Re: [Wget-dev] wget2 | Add proper WARC API + support by wget2 (#65)
Date: Fri, 09 Nov 2018 16:32:12 +0000

Wget has warc support and archiving millions of urls is simple & quite fast 
compared to many other tools. 

To make this process faster, we have to use multiprocessing and spawning 1000+ 
process takes lot of RAM(~10GB). Instead if we have multi threading support, 
this processing will become much simpler.

Currently wget2/aria2 or any other multi threading tools doesn't have warc 

Reply to this email directly or view it on GitLab: 
You're receiving this email because of your account on gitlab.com.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]