|
Bugzilla – Full Text Bug Listing |
| Summary: | The Redirector on OpenSuSE.org needs to work every time or it needs to go. | ||
|---|---|---|---|
| Product: | [openSUSE] openSUSE.org | Reporter: | Jacob Caudill <novell> |
| Component: | BuildService | Assignee: | Christoph Thiel <cthiel> |
| Status: | RESOLVED DUPLICATE | QA Contact: | Adrian Schröter <adrian.schroeter> |
| Severity: | Critical | ||
| Priority: | P5 - None | CC: | adrian.schroeter, forgotten_7L3tOtZIov, francis, marcio.ferreira, martin.schlander, mrueckert, tschmidt |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | Other | ||
| Whiteboard: | |||
| Found By: | Other | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
| Bug Depends on: | 238095 | ||
| Bug Blocks: | |||
| Attachments: | y2log from user having redirect issues | ||
There's certainly plenty room for improvement.. I hear about a lot of problems in Danish forums and IRC too.. and from my father.. Whenever I've used the redirects personally I've had problems from time to time too. Occasionally I have too. Can you please report exactly when you have tried to use which file and/or repository ? When you have tried with YaST, please send the output of /var/log/YaST2/y2log or any other appropriate log file of your client. There is a known issue with yum-style (rpm-md) metadata being cached by web cached even they are stale. This is caused by inappropriate headers we send for these objects. This affects mainly software/repos.opensuse.org. If you have other problems, please provide details if possible, or describe the exact symptoms. Thanks. I dont have the lgos either, you will have to wait for a "lucky" moment of desynced mirrors... But I can tell what happens. I believe that whent he mirror is out of sync, or there is some missing file, or the situation from comment #4 happens, yast outputs a message that "the source is not YUM or yast type", something like that. It really happens a lot. By the way the comment I want to upt in here is that this problem happens a lot and a lot and a lot. When I told that to AJ and he said he wasnt aware I was really surprised, I think I see this happening almost everyday in the irc channel. Hope someone get some logs quick. I will try too. It could also help to see program output with an respective error message, provided it is documented with exact timestamp and IP address. Also, with which sort of repositories is it observed? With 10.1/10.2 install source? With build service repos? Debuginfo? Christoph, does the scanner notice if a repomd.xml (or similar) copy on a mirror is stale, and drop it from the cache? IOW, does it look on the timestamps? Sorry, this may be stupid question. Other qestion: I know that we push buildservice repos to mirrors; do we also have mirrors that poll? We should automatically purge the latter from our mirror cache if a repo is updatd. (Both questions refer to buildservice repos only) Created attachment 115963 [details]
y2log from user having redirect issues
this user is in the Malaysian time zone which is GMT+8
his ip is 60.53.2.171
# grep "\<60\.53\.2\.171" software.opensuse.org-access_log 60.53.2.171 - - [29/Jan/2007:14:38:17 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:15:13:12 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:15:14:13 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:16:22:48 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:17:02:28 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:17:30:24 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:17:54:44 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:17:56:35 +0100] "GET /download/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:11:01 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/media.1/products HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:11:04 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:11:05 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:11:06 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.key HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:12:07 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:13:08 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz HTTP/1.1" 302 - "-" "Novell ZYPP Installer" 60.53.2.171 - - [29/Jan/2007:18:20:40 +0100] "GET /download/Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz HTTP/1.1" 302 - "-" "Novell ZYPP Installer" # grep "\<60\.53\.2\.171" /srv/www-local/software/redirect.log Mon, 29 Jan 2007 14:38:18 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp5.gwdg.de/pub/opensuse/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 15:13:13 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp.iasi.roedu.net/mirrors/opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 15:14:13 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp.iasi.roedu.net/mirrors/opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 16:22:48 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp1.opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 17:02:28 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp.iasi.roedu.net/mirrors/opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 17:30:24 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp5.gwdg.de/pub/opensuse/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 17:54:44 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp1.opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 17:56:35 +0100 60.53.2.171 home:/mrdocs/openSUSE_10.2/repodata/repomd.xml => http://ftp1.opensuse.org/repositories/home:/mrdocs/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 18:11:01 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/media.1/products => http://repos.opensuse.org/Emulators:/Wine/openSUSE_10.2/media.1/products Mon, 29 Jan 2007 18:11:04 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml => http://ftp1.opensuse.org/repositories/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 18:11:05 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml => http://ftp1.opensuse.org/repositories/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml Mon, 29 Jan 2007 18:11:06 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.key => http://ftp.halifax.rwth-aachen.de/opensuse/repositories/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.key Mon, 29 Jan 2007 18:12:07 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc => http://ftp.halifax.rwth-aachen.de/opensuse/repositories/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc Mon, 29 Jan 2007 18:13:08 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz => http://ftp.halifax.rwth-aachen.de/opensuse/repositories/Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz Mon, 29 Jan 2007 18:20:40 +0100 60.53.2.171 Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz => http://ftp.halifax.rwth-aachen.de/opensuse/repositories/Emulators:/Wine/openSUSE_10.2/repodata/filelists.xml.gz (In reply to comment #7) > Christoph, does the scanner notice if a repomd.xml (or similar) copy on > a mirror is stale, and drop it from the cache? IOW, does it look on the > timestamps? Sorry, this may be stupid question. > Other qestion: I know that we push buildservice repos to mirrors; do we > also have mirrors that poll? We should automatically purge the latter > from our mirror cache if a repo is updatd. > (Both questions refer to buildservice repos only) My implementation of the scanner compares the local filesize with the remote. I'm not sure about the new implementation of jw/mploster. In addition to that, the PHP-based redirector tries to (by its session magic) redirect requestions to the same mirror, to make sure the user gets a coherent state. Peter, what's your finding from the above pasted log excerpts? the ftp and rsync backends of the scanner can record exact filesize. The http backend cannot. So comparing size is difficult sometimes. [[[ $ curl -sSI http://suse.inode.at/opensuse/distribution/10.2/iso/dvd/openSUSE-10.2-GM-DVD-ppc.iso ... ETag: "-2064790363" Content-Length: 3957841920 Last-Modified: Fri, 01 Dec 2006 14:15:17 GMT ... ]]] this should be enough. the etag should be the same for multiple requests but it differs per mirror. but you can at least utilize it to notice changes on the server. lastmodified can differ per mirror aswell. content-length should be the same on all mirrors though. Regarding the y2log, I cannot recognize much in the 33000 lines; in the lines I find (e.g. 10035) I cannot conclude much, because the log doesn't tell me whether it logs the followed redirected URL, or the initial one. Anyway, the type of error which yast logs, but does _not_ show to the user, but which would be more helpful to show, is: 2007-01-29 12:10:34 <5> linux-houston(6321) [base] Exception.cc(log):94 SourceImpl.cc(getPossiblyCachedMetadataFile):400 THROW: SourceImpl.cc(getPossiblyCachedMetadataFile):400: http://software.opensuse.org/download/Emulators%3a/Wine/openSUSE_10.2//repodata/filelists.xml.gz fails checksum verification. I cannot correlate the logs together, but I've got something here: In this special case, the problem is secondary to a timeout trying to reach software.opensuse.org. May be due to a loaded server, or to network outage. poeml@batavia510 /tmp % grep "fails checksum verification" y2log.txt 2007-01-29 12:10:34 <5> linux-houston(6321) [base] Exception.cc(log):94 SourceImpl.cc(getPossiblyCachedMetadataFile):400 THROW: SourceImpl.cc(getPossiblyCachedMetadataFile):400: http://software.opensuse.org/download/Emulators%3a/Wine/openSUSE_10.2//repodata/filelists.xml.gz fails checksum verification. 2007-01-29 12:37:35 <5> linux-houston(6853) [base] Exception.cc(log):94 SourceImpl.cc(getPossiblyCachedMetadataFile):400 THROW: SourceImpl.cc(getPossiblyCachedMetadataFile):400: http://software.opensuse.org/download/Emulators%3a/Wine/openSUSE_10.2//repodata/filelists.xml.gz fails checksum verification. 2007-01-29 12:57:46 <5> linux-houston(7137) [base] Exception.cc(log):94 SourceImpl.cc(getPossiblyCachedMetadataFile):400 THROW: SourceImpl.cc(getPossiblyCachedMetadataFile):400: http://software.opensuse.org/download/Emulators%3a/Wine/openSUSE_10.2//repodata/filelists.xml.gz fails checksum verification. poeml@batavia510 /tmp % grep "Emulators.*xml.asc" y2log.txt 2007-01-29 12:10:21 <0> linux-houston(6321) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-29 12:11:02 <0> linux-houston(6321) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-29 12:37:11 <0> linux-houston(6853) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-29 12:38:17 <0> linux-houston(6853) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-29 12:57:33 <0> linux-houston(7137) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-30 01:16:05 <0> linux-houston(5954) [media] MediaCurl.cc(doGetFileCopy):882 URL: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc 2007-01-30 01:17:06 <5> linux-houston(5954) [base] Exception.cc(log):94 MediaCurl.cc(doGetFileCopy):1049 THROW: MediaCurl.cc(doGetFileCopy):1049: Curl error for: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc: Error code: Unrecognized error Error message: connect() timed out! 2007-01-30 01:17:06 <5> linux-houston(5954) [base] Exception.cc(log):94 MediaCurl.cc(doGetFileCopy):1054 RETHROW: MediaCurl.cc(doGetFileCopy):1049: Curl error for: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc: Error code: Unrecognized error Error message: connect() timed out! 2007-01-30 01:17:06 <5> linux-houston(5954) [base] Exception.cc(log):94 MediaCurl.cc(getFileCopy):728 RETHROW: MediaCurl.cc(doGetFileCopy):1054: Curl error for: http://software.opensuse.org/download/Emulators:/Wine/openSUSE_10.2/repodata/repomd.xml.asc: Error code: Unrecognized error Error message: connect() timed out! So it is not due to the problem we were suspecting, in this single case. *** Bug 240337 has been marked as a duplicate of this bug. *** I regard this bug fixed as duplicate of bug 238095. See my last comment there. https://bugzilla.novell.com/show_bug.cgi?id=238095#c11 *** This bug has been marked as a duplicate of bug 238095 *** |
on the irc channel ever since opensuse.org started using that screwy mirror redirector for {software,repos,download}.opensuse.org people have had problems with it. I've have very limited success adding them as sources myself. When it did add successfully, it would later give errors when refreshing or trying to download a package. Here lately, more and more people come into #suse asking what is wrong with the repos. I am sure there are MANY more people having this problem than just those that come to #suse. I would also bet we have lost users because they judge SuSE by the fact that they cant add mirrors and download software easily. I don't know how this is setup, but it either needs to work every time for all users, or it needs to go all together.