Bug 370054

very good idea, slow downloads have been the biggest annoyance for me in updating any Linux distros i have tried...

I think this enhancement is the same as I suggested in Bug 370457.

You could even make this configurable via yast. Wither to enable it or not. How many parallel downloads is wanted, with 5 being the default. If you want zypp/YaST to be fast, then this is a must. 

*** This bug has been marked as a duplicate of bug 370457 ***

No dupe.

please do it! my connection is fast but the servers are not.

*** Bug 385711 has been marked as a duplicate of this bug. ***

Segmented downloading with aria2 in parallel would make most sense,
especially as download.opensuse.org supports metalinks transparently.
See http://en.opensuse.org/Metalink for more information.

At the same time, it would fix numerous problems with download failures
(connection errors, timeouts, broken replies, ...).
Refer also to http://en.opensuse.org/Libzypp/Failover)

*** Bug 425031 has been marked as a duplicate of this bug. ***

What is the status of this for 11.1 or zypper 1.0?

I would like to put my oar in here :D

The most useful "parallel download" solution is to prefetch packages in the background - not to "maximize bandwidth" by fetching many packages simultaneously, but to get the next package while you are installing the previous.

I have seen these effects on Gentoo Portage, and to be honest, a lot of workloads do not produce an appreciable effect. What alleviates it in Gentoo is compiling packages takes a long, long time, far longer in MOST cases than simply extracting/installing them.

So here goes;

~

* Imagine you have some packages which are going to be installed. The first few packages are very very small (deltas or just small updates) - these can be downloaded in parallel (as in multiple HTTP downloads simultaneously) very fast and be ready as soon as zypper needs them. It usually takes far longer to extract and update the RPM DB than to download them. This is the ideal scenario.

* Imagine that the 5th package in the list is huge - kernel-source or something of that ilk. You may be able to download the first 4 packages while the first is installing, and then take a fast path to installing 3 cached packages, but will still be waiting on kernel-source to download by the time they're installed. This scenario does not offer an appreciable speedup.

* What happens if the next packages after kernel-source are also rather large, and the next set of TINY packages are too far away from the "parallel download limit"? Then, this does not offer any appreciable speedup.

* It may even slow down the installation as downloading many large packages in parallel (as in many HTTP requests at once), on a moderate speed connection (~3Mbit/s) will actually increase the wait time while packages are downloaded. There really is not a safe default here as connections vary both in quality AND user behaviour.

* And also when the packages to install are large but relatively easy to install (source rpms especially) then they can complete far quicker, leaving zypper with a wait for a package anyway.

~

Perhaps the most prudent solution is;

 - Implement a package prefetch cache which can download packages in the background during installation of others ("background downloading").

 - Limits on prefetch cache - an example might be a maximum of 100MB and 5 files in the cache, and simultaneous downloading of 2 packages plus one prioritized foreground download (total 3).

 - Limits can be temporarily relieved on the basis of need - if you have kernel-source and can opportunistically fetch 30 smaller packages while it is installing, then the 5 file limit can be discarded in favour of the 100MB limit. In the event that these 30 packages can be fetched far less time than the installation, the limit on simultaneous downloads may be increased to increase the number of cached items.

 - Where the package sizes are small these can be favoured over larger packages, if only to reduce the latency of initiating a download task through curl, fetching it, and then installing it. Where this hits a wall by stumbling on a large package which has not even attempted to be prefetched yet, it will simply download as before, and not affect performance (no improvement, but no slowdown either).

 - Limits must be scaled back at the first, best opportunity to free up resources. Hard maximum limits should be enforced.

 - Prefetching should be based on the solver results such that packages which are to be installed could their major dependencies prefetched first - for instance there are two paths to walk in installing both KDE and GNOME components at the same time. In this scenario GNOME packages and KDE packages may prefetch at the same time and zypper can opportunistically switch between the two paths depending on availability in the cache, rather than a fixed route. Zypper should endeavour to install dependencies of both paths first so as to make way for parallel downloads and switching of dependency paths to the final packages.


Thoughts?

We agree with that. We want to make installing and fetching in parallel. The multiple connection stuff we get it for free with the aria2c backend.

Sadly this feature shows up all the time as high priority together with a bug bunch of other features with high priority. I hope we can do it for 11.2 with the planned ZYpp transaction rework.

I look forward to it if you get to it :)

Now also https://features.opensuse.org/307862 exists about this. Don't track in Bugzilla anymore?

openFate #307862