Bug 385711

Summary: download smaller packages/patches parallely when downloading bigger
Product: [openSUSE] openSUSE 10.3 Reporter: Jakub Rusinek <kontakt>
Component: YaST2Assignee: E-mail List <yast2-maintainers>
Status: RESOLVED DUPLICATE QA Contact: Jiri Srain <jsrain>
Severity: Enhancement    
Priority: P5 - None CC: ma
Version: Final   
Target Milestone: ---   
Hardware: i686   
OS: openSUSE 10.3   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Jakub Rusinek 2008-05-01 18:57:14 UTC
to improve time of downloading packages/patches, smaller files should be downloaded parallely to bigger.

for example, if you are downloading 100MB file, 10kB will not hurt, but will decrease list of items awaiting download.
Comment 1 Peter Poeml 2008-05-02 07:20:19 UTC
I'm almost certain that this won't speed anything up.

It makes most sense (in terms of overhead and bandwidth usage) to keep a
Keepalive HTTP connection saturated by requesting one file after the
other. [This will give the fastest download possible, with the least
overhead on client _and_ server. Keeping in mind that load caused on the
server will (in the end) affect that same client also, is important.]
Whether this is what libzypp is doing is unknown to me. I expect it isn't,
because it after downloading one file it usually installs it first,
before downloading the next, and Keepalive timeouts are commonly very
short on busy servers (like 2-5 seconds). 
Yum does it that way (downloading all packages over a single connection
and installing them after that all at once).


I'm not counting methods to open several connections to the same server,
which is what you suggest. You basically (could) suggest that the client
opens 10 connections for the same 100MB file and requesting it in
segments. That _can_ apparently speed up things for the client _if_
server-side bandwidth or server load is the bottleneck, and the client
has bandwidth free. However, this goes on the cost of all other clients,
and will eventually hurt the client itself. If everybody does it,
everybody would have a disadvantage from it (much higher server load,
much higher latencies). And we would need to install protection against it.

What you _may_ consider is parallel downloads (in segments) from
_different_ servers. Which leads to my proposal here
http://en.opensuse.org/Libzypp/Failover
It is okay (and can be useful) to retrieve several segments of the same
big file from several servers. (This must be done in a way that the
client bandwidth is not exceeded, because otherwise it would occupy
slots on the server longer than necessary.) 

Metalinks (http://metalinker.org) support this transparently. You could
talk to the zypp folks to integrate this in libzypp.
Comment 2 Jakub Rusinek 2008-05-02 08:38:01 UTC
I thought small files downloaded in background will not hurt, but I see you proposed better solution. I had problems with downloading packages, because packages existed virtually (they were not available on servers).

You can propose Metalinks too :) .
Comment 3 Peter Poeml 2008-05-02 08:43:58 UTC
I'll leave this for consideration by the yast/zypp team.
Comment 4 Michael Andres 2008-05-08 09:16:29 UTC
Also covered by FATE request #120340 - Run download and install in parallel.

*** This bug has been marked as a duplicate of bug 370054 ***