Bug 1192433

Summary: zypper ipv6 problem
Product: [openSUSE] openSUSE Tumbleweed Reporter: Dmitry Markov <werwolf131313>
Component: libzyppAssignee: E-mail List <zypp-maintainers>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: andrii.nikitin, werwolf131313
Version: Current   
Target Milestone: Current   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: Curl error 47 Maximum (6) redirects followed

Description Dmitry Markov 2021-11-08 05:36:48 UTC
if the system contains ipv6 (on my machine is hurricane electrics tunnel) zypper gets errors often (to many redirects or content size), but if i
```
export ZYPP_MEDIA_CURL_IPRESOLVE=4
```
before run zypper then there is no problem

P.S.: sorry for my english
Comment 1 Michael Andres 2021-11-08 10:21:05 UTC
Dimitry, please 

   export ZYPP_MEDIA_CURL_DEBUG=2

Do _not_ set ZYPP_MEDIA_CURL_IPRESOLVE and let a few errors occur. Then attach the /var/log/zypper.log here. We'll check the log then. Maybe they reveal soemthing, but it may also be an issue in your IPV6 environment.
Comment 2 Dmitry Markov 2021-11-08 17:32:27 UTC
I tried for half a day, but the only thing I get is this:


Warning: Digest verification failed for file 'wine-gecko-2.47.2-65.17.noarch.rpm'


I will return when the problem recurs, it is floating and does not happen regularly, but it will definitely repeat itself within a week, maybe two.
Comment 3 Dmitry Markov 2021-11-09 15:55:30 UTC
Created attachment 853636 [details]
Curl error 47 Maximum (6) redirects followed
Comment 4 Michael Andres 2021-11-09 17:31:43 UTC
> grep -B 1 'redirects' zypper.log

Shows a typical redirection sequence like this:

> URL: https://download.opensuse.org/repositories/network/openSUSE_Tumbleweed/x86_64/nextcloud-desktop-dolphin-3.3.6-52.4.x86_64.rpm
> redirecting to location: https://mirrorcache.opensuse.org/repositories/network/openSUSE_Tumbleweed/x86_64/nextcloud-desktop-dolphin-3.3.6-52>
> redirecting to location: https://mirrorcache-us.opensuse.org/repositories/network/openSUSE_Tumbleweed/x86_64/nextcloud-desktop-dolphin-3.3.6>
> redirecting to location: https://provo-mirror.opensuse.org/repositories/network/openSUSE_Tumbleweed/x86_64/nextcloud-desktop-dolphin-3.3.6-5>

And sometimes all of a sudden it starts looping:

> URL: https://download.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm
> redirecting to location: https://mirrorcache.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.>
> redirecting to location: https://mirrorcache-us.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0->
> redirecting to location: https://mirrorcache.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.>
> redirecting to location: https://mirrorcache-us.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0->
> redirecting to location: https://mirrorcache.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.>
> redirecting to location: https://mirrorcache-us.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0->
> redirecting to location: https://mirrorcache.opensuse.org/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.>
> curl error: 47: Maximum (6) redirects followed, temp file size 0 bytes.

This is IMO nothing zypp/libcurl can influence. It almost looks like mirrorcache-us.opensuse.org sometimes redirects back to mirrorcache.opensuse.org and then it loops...
Comment 5 Dmitry Markov 2021-11-09 17:49:53 UTC
While I have been able to catch only this problem, there are others. But let me remind you that they stop if I disable ipv6. I wrote in he, so far, they are sure that it's not about them.
Comment 6 Andrii Nikitin 2021-11-09 22:22:20 UTC
Currently download.opensuse.org uses two different redirectors for https and http requests:
http - old one MirrorBrain
https - new one MirrorCache
Somehow your IP is resolved differently on different mirrorcache instances, this should be possible to fix on Thursday.
Just in case you can compare country resolved using commands below - which ones are incorrect?:

curl -6 http://mirrorcache.opensuse.org/rest/myip
curl -6 http://mirrorcache-us.opensuse.org/rest/myip
curl -4 http://mirrorcache.opensuse.org/rest/myip
curl -4 http://mirrorcache-us.opensuse.org/rest/myip

There are two immediate workarounds:
1. use http instead of https - old redirector shouldn't have this issue.
2. Add ?COUNTRY=us at the and of baseurl in *.repo files, e.g.
baseurl=https://download.opensuse.org/repositories/network/openSUSE_Tumbleweed/?COUNTRY=us

Let me know if you need assistance with 1 or 2.

You can also report similar problems at https://github.com/openSUSE/MirrorCache/issues
Comment 7 Dmitry Markov 2021-11-10 02:15:59 UTC
>[werwolf@home] ~  
>❯ curl -6 http://mirrorcache.opensuse.org/rest/myip
>{"country":"US","ip":"XXX:XXX:XXX:XXX","region":"NA"}
>[werwolf@home] ~  
>❯ curl -6 http://mirrorcache-us.opensuse.org/rest/myip
>{"country":"US","ip":"XXX:XXX:XXX:XXX","region":"NA"}
>[werwolf@home] ~  
>❯ curl -4 http://mirrorcache.opensuse.org/rest/myip
>{"country":"RU","ip":"YYY.YYY.YYY.YYY","region":"EU"}
>[werwolf@home] ~  
>❯ curl -4 http://mirrorcache-us.opensuse.org/rest/myip
>{"country":"RU","ip":"YYY.YYY.YYY.YYY","region":"EU"}

maybe the problem is that "region":"NA" in ipv6 answer?
Comment 8 Andrii Nikitin 2021-11-10 07:17:32 UTC
Exactly, if country and region are correct when you try your IPv6 at https://www.maxmind.com/en/geoip-demo or https://www.maxmind.com/en/locate-my-ip-address - then it is just an outdated Geo database at mirrorcache. (Which is likely and could be fixed tomorrow).
Otherwise you should report the problem at https://support.maxmind.com/geoip-data-correction-request/ , so you are not redirected to the NA instance.
In any case the redirection loop shouldn't happen, it should be fixed tomorrow as well.
Comment 9 Andrii Nikitin 2021-11-10 11:44:58 UTC
So looking at the MirrorCache logs - it looks zypper connected to mirrorcache.o.o using IPv6 , so the country was detected as US and the request was redirected to mirrorcache-us.o.o . But then for some reasons zypper connected there using IPv4 (see ipv4 column below), so country was detected as RU and the request was sent back.

```
mirrorcache> select path, country, dt, ipv4 from stat where dt between '2021-11-09 15:27:00' and '2021-11-09 15:27:59' and path like '/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16%' order by id desc;
                                               path                                               | country |         dt          | ipv4 
--------------------------------------------------------------------------------------------------+---------+---------------------+------
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | us      | 2021-11-09 15:27:12 | f
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | us      | 2021-11-09 15:27:13 | f
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | us      | 2021-11-09 15:27:13 | f
```
```
mirrorcache-us> select path, country, dt, ipv4 from stat where dt between '2021-11-09 15:27:00' and '2021-11-09 15:27:59' and path like '/repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16%' order by id desc;
                                               path                                               | country |         dt          | ipv4 
--------------------------------------------------------------------------------------------------+---------+---------------------+------
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | ru      | 2021-11-09 15:27:13 | t
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | ru      | 2021-11-09 15:27:13 | t
 /repositories/home:/Dead_Mozay/openSUSE_Tumbleweed/x86_64/gnome-firmware-3.36.0-16.49.x86_64.rpm | ru      | 2021-11-09 15:27:13 | t
```

I agree that the loop shouldn't occur even when such strange behavior happens. I will change the behavior, so only mirrorcache.o.o may redirect to regional instance and never in opposite direction.
Comment 10 Michael Andres 2021-12-17 11:09:03 UTC
Closing it. Please re-open if the issue persists.