Bug 238095

Summary: libzypp (zypper/YaST) can't add certain repos anymore (mirrors out of sync)
Product: [openSUSE] openSUSE 10.2 Reporter: Wolfgang Rosenauer <wolfgang>
Component: libzyppAssignee: Peter Poeml <poeml>
Status: RESOLVED FIXED QA Contact: Klaus Kämpf <kkaempf>
Severity: Major    
Priority: P5 - None CC: aj, casualprogrammer, crrodriguez, dmacvicar, forgotten_t-K-SYtr-p, novell, trango
Version: Final   
Target Milestone: ---   
Hardware: All   
OS: Other   
Whiteboard:
Found By: Other Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 239483    
Attachments: zypper.log

Description Wolfgang Rosenauer 2007-01-24 10:01:28 UTC
Yesterday I wanted to run YOU but it only was hanging at some point (on two different machines).
I tried zypper afterwards which stopped with "Failed to restore ..."

When I ran yast2 inst_source two of my buildservice repositories were reported as not accessible and were removed from my repo-list.
Afterwards YOU and zypper was working again.

Now the problem is, that I can't add those repos again:

Hygiea:~ # zypper sa http://software.opensuse.org/download/openSUSE:/Tools/openSUSE_10.2 bs-tools
Determining http://software.opensuse.org/download/openSUSE:/Tools/openSUSE_10.2 source type...
http://software.opensuse.org/download/openSUSE:/Tools/openSUSE_10.2 is type YUM
Unknown Error: Downloading metadata failed (is YUM source?) or user did not accept remote source. Aborting refresh.

YaST fails with the same error.

I'll attach zypper.log.
Comment 1 Wolfgang Rosenauer 2007-01-24 10:02:01 UTC
Created attachment 114590 [details]
zypper.log
Comment 2 Wolfgang Rosenauer 2007-01-24 10:26:35 UTC
It's really interesting since the above happens on two different machines with 10.2 and the same repos added. So something not local to the machines triggered an error in libzypp and caused some corruption.
Comment 3 Wolfgang Rosenauer 2007-01-29 07:12:36 UTC
Hmm, someone has any suggestion how to it to work again?
Maybe zypp only gets confused very seldom and so this issue is not of high priority but for me it is. So any idea how to fix the current zypp state at least as a workaround?
Comment 4 Wolfgang Rosenauer 2007-01-29 12:48:22 UTC
As it just came up on IRC: I never ever had ZMD/rug installed on these systems.
Comment 5 Martin Vidner 2007-01-29 13:19:31 UTC
2007-01-24 11:00:34 <5> Hygiea(8180) [base] Exception.cc(log):94 SourceImpl.cc(getPossiblyCachedMetadataFile):400 THROW:    SourceImpl.cc(getPossiblyCachedMetadataFile):400: http://software.opensuse.org/download/openSUSE:/Tools/openSUSE_10.2./repodata/filelists.xml.gz  fails checksum verification.

But the checksum seems fine nevertheless. Puzzling.
Comment 6 Harald Mueller-Ney 2007-01-29 13:23:45 UTC
Christoph, could it be related to the redirector of openSUSE? Getting files from various hosts which are not 100% in sync?
Comment 7 Martin Vidner 2007-01-29 13:26:37 UTC
Aha, it has indeed succeeded on the third try for me.
Comment 8 Wolfgang Rosenauer 2007-01-29 13:39:44 UTC
I've now used ftp://ftp4.gwdg.de/pub/opensuse/repositories/openSUSE:/Tools/openSUSE_10.2
and it worked. So yes, it seems to be a redirector problem.
Interesting is that it happened only for the above two repositories for me. (I have tried others and they worked the first time).
Strange behaviour at least.
Comment 9 Peter Poeml 2007-01-29 17:01:22 UTC
It is not directly related to the redirector. It is related to the fact
that we don't set sensible cache controlling headers. (Assuming that the
mirrors have exact copies.) 

We now set cache-control: no-cache, must-revalidate headers on repo-md
files, on repos.opensuse.org. However, mirrors that don't do that could
still serve files which are cached even though they are outdated. Thus,
we need to find a way to cooperate with our mirrors so they also set
those header (or don't mirror those files so they are always served by
us...).
Comment 10 Christoph Thiel 2007-01-29 19:15:08 UTC
Harald, the redirector implements some kind of "session handling" to make sure to redirector the user to only one mirror for those kind of metadata files. However, as Peter already elaborated, we seem to be hitting some nasty caching-related problems right now.
Comment 11 Peter Poeml 2007-01-31 14:57:27 UTC
Meanwhile I'm pretty sure that most issues referenced here are not
caused by inconsistent mirrors, or the fact that we redirect things, but
rather to caching of repo metadata by web caches in between.

Meanwhile, we set an appropriate Cache-Control header on those repo
metadata which are frequently re-created by the buildservice. Today, I
added Expires header also, just in case that cache doesn't implement
Cache-Control, (or some client software like zypper, yum, smart) makes
their own caching decision and doesn't have a complete implementation.

The fact that we redirect comes _only_ into play because if metadata is
pulled from a mirror, because on mirrors we have no influence on the
HTTP headers, and can't fix them as described above. Thus, since
2007-01-29 18:20:53 +0100 we are no longer redirecting metadata requests
to mirrors at all.

For packages it should not matter that much, whether they get stale or
mirrors are inconsistent, because the filename is altered at each
rebuild.

A squid with default configuration will deliver HTTP content from cache
without revalidation up to 72 hours, so please allow some time for our
latest changes to become effective.

By the way, all this just displays how well the integrity checking of
repositories is done :)

I'm regarding the bug fixed; please reopen if the problem persists.
Comment 12 Peter Poeml 2007-01-31 18:54:32 UTC
*** Bug 239483 has been marked as a duplicate of this bug. ***
Comment 13 Jan Kupec 2007-05-10 18:57:53 UTC
*** Bug 251275 has been marked as a duplicate of this bug. ***
Comment 14 Jan Kupec 2007-05-10 19:00:39 UTC
(In reply to comment #11)
> The fact that we redirect comes _only_ into play because if metadata is
> pulled from a mirror, because on mirrors we have no influence on the
> HTTP headers, and can't fix them as described above. Thus, since
> 2007-01-29 18:20:53 +0100 we are no longer redirecting metadata requests
> to mirrors at all.

According to bug 251275, this seems not to be true. Problems persist.
Comment 15 Jan Kupec 2007-05-10 19:07:30 UTC
Peter, reassigning the bug to you, please reassign as/if needed.
Comment 16 Jan Kupec 2007-05-10 19:13:44 UTC
*** Bug 239625 has been marked as a duplicate of this bug. ***
Comment 17 Jan Kupec 2007-05-10 19:16:54 UTC
Why do we do these redirects, anyway? To pick up the best mirror for each file
accessed? What about redirect just once per session (to just one mirror)?
Comment 18 Casual J. Programmer 2007-05-10 19:24:03 UTC
The idea as such seems brilliant, you only need to register one source, then depending on load or whatever, you distribute the requests to all available servers subsequently. Problem is, how is synchronicity checked and used as selection criterion.
Comment 19 Peter Poeml 2007-05-10 22:39:44 UTC
Thanks for reopening the bug. Testing with the media file referenced in
bug 251275, I found that I did a mistake. In my last attempt to fix
this, I added a ZrkadloExcludeFileMask for the Factory tree, matching on
files which do not contain digits. However, the expression was
erroneously written to match on the entire request URI, instead of the
last path component. 

-        ZrkadloExcludeFileMask "^[^0-9]*$"
+        ZrkadloExcludeFileMask "/[^0-9]*$"

Sorry, my bad. 

It didn't become apparent in my testing, because I didn't test with URIs
which t contain digits, like it is the case for the the media.1
directory...