Bug 1092493

Summary: aws CLI tool fails to run: ModuleNotFoundError: No module named 'botocore.vendored'
Product: [openSUSE] openSUSE Tumbleweed Reporter: Robert Munteanu <rombert>
Component: OtherAssignee: Robert Schweikert <rjschwei>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: cyberkiller8, rainer.klier, rombert, tchvatal
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Robert Munteanu 2018-05-09 08:06:46 UTC
With an up-to-date TW installation I am no longer able to run the `aws` tool:

$ aws
/usr/lib/python3.6/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.16) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "/usr/bin/aws", line 19, in <module>
    import awscli.clidriver
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 35, in <module>
    from awscli.help import ProviderHelpCommand
  File "/usr/lib/python3.6/site-packages/awscli/help.py", line 27, in <module>
    from awscli.clidocs import ProviderDocumentEventHandler
  File "/usr/lib/python3.6/site-packages/awscli/clidocs.py", line 21, in <module>
    from awscli.argprocess import ParamShorthandDocGen
  File "/usr/lib/python3.6/site-packages/awscli/argprocess.py", line 21, in <module>
    from awscli.paramfile import get_paramfile, ResourceLoadingError
  File "/usr/lib/python3.6/site-packages/awscli/paramfile.py", line 16, in <module>
    from botocore.vendored import requests
ModuleNotFoundError: No module named 'botocore.vendored'

There is a related upstream issue reported at https://github.com/aws/aws-cli/issues/3092 . The initial report was for Windows but there are other two confirmations from TW users.
Comment 1 Tomáš Chvátal 2018-05-10 08:59:51 UTC
Thanks for the report.
We remove the bundled code from botocore and some other aws packages expect it to be there and it needs to be patched out.

I enabled testsuite to make sure this problem is gone, for now grab version from:
devel:languages:python:aws/aws-cli
Comment 2 Robert Munteanu 2018-05-10 13:21:36 UTC
(In reply to Tomáš Chvátal from comment #1)
> Thanks for the report.
> We remove the bundled code from botocore and some other aws packages expect
> it to be there and it needs to be patched out.
> 
> I enabled testsuite to make sure this problem is gone, for now grab version
> from:
> devel:languages:python:aws/aws-cli


Thanks for the quick reply. I've installed python3-aws-cli-1.15.17-4.1.noarch.rpm python3-botocore-1.10.17-4.1.noarch.rpm from https://download.opensuse.org/repositories/devel:/languages:/python:/aws/openSUSE_Tumbleweed/noarch/ but the error still appears (in a different place though)

$ aws
Traceback (most recent call last):
  File "/usr/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 58, in main
    driver = create_clidriver()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 69, in create_clidriver
    event_hooks=emitter)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/lib/python3.6/site-packages/awscli/handlers.py", line 27, in <module>
    from awscli.customizations.cloudformation import initialize as cloudformation_init
  File "/usr/lib/python3.6/site-packages/awscli/customizations/cloudformation/__init__.py", line 13, in <module>
    from awscli.customizations.cloudformation.package import PackageCommand
  File "/usr/lib/python3.6/site-packages/awscli/customizations/cloudformation/package.py", line 26, in <module>
    from awscli.customizations.s3uploader import S3Uploader
  File "/usr/lib/python3.6/site-packages/awscli/customizations/s3uploader.py", line 22, in <module>
    from s3transfer.manager import TransferManager
  File "/usr/lib/python3.6/site-packages/s3transfer/manager.py", line 34, in <module>
    from s3transfer.download import DownloadSubmissionTask
  File "/usr/lib/python3.6/site-packages/s3transfer/download.py", line 23, in <module>
    from botocore.vendored.requests.packages.urllib3.exceptions import \
ModuleNotFoundError: No module named 'botocore.vendored.requests.packages.urllib3'; 'botocore.vendored.requests.packages' is not a package
Comment 3 Tomáš Chvátal 2018-05-10 13:27:53 UTC
You need to have them all from this repo, as they will now work only together, there is also the s3transfer botocore and boto3 that will make it work.

Alternatively you have to wait for the integration to be finished for all of them in tumbleweed.
Comment 4 Robert Munteanu 2018-05-10 13:35:48 UTC
Understood, thanks.
Comment 5 Tomáš Chvátal 2018-05-14 11:45:27 UTC
Since robert wants it maintained in Cloud:Tools he has to fix it in that repo.
Comment 6 Robert Schweikert 2018-08-22 14:40:34 UTC
Fixed with: https://build.opensuse.org/request/show/629037
Comment 7 Rainer Klier 2018-08-30 14:33:39 UTC
i just switched system packages to https://download.opensuse.org/repositories/Cloud:/Tools/openSUSE_Tumbleweed and "aws --help" results in:

Traceback (most recent call last):
  File "/usr/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 58, in main
    driver = create_clidriver()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 69, in create_clidriver
    event_hooks=emitter)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/lib/python3.6/site-packages/awscli/handlers.py", line 20, in <module>
    from awscli.paramfile import register_uri_param_handler
  File "/usr/lib/python3.6/site-packages/awscli/paramfile.py", line 17, in <module>
    from botocore.vendored import requests
ModuleNotFoundError: No module named 'botocore.vendored'

i have the following packages installed from https://download.opensuse.org/repositories/Cloud:/Tools/openSUSE_Tumbleweed:
aws-cli-1.15.76-47.1.noarch
python3-boto-2.48.0-2.2.noarch
python3-boto3-1.7.75-12.1.noarch
python2-boto-2.48.0-2.2.noarch
python2-boto3-1.7.75-12.1.noarch
python2-botocore-1.10.75-16.1.noarch
python3-botocore-1.10.75-16.1.noarch
python3-jmespath-0.9.3-33.1.noarch
python2-jmespath-0.9.3-33.1.noarch
python2-s3transfer-0.1.13-7.1.noarch
python3-s3transfer-0.1.13-7.1.noarch
python-rpm-macros-2017.12.22.d9968ab-52.1.noarch
Comment 8 Robert Schweikert 2018-08-30 21:05:45 UTC
Hmm, interesting

> aws --help
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: the following arguments are required: command

> zypper info aws-cli
Loading repository data...
Reading installed packages...


Information for package aws-cli:
--------------------------------
Repository     : cloud_tools                                 
Name           : aws-cli                                     
Version        : 1.15.76-47.1                                
Arch           : noarch                                      
Vendor         : obs://build.opensuse.org/Cloud              
Installed Size : 3.8 MiB                                     
Installed      : Yes                                         
Status         : out-of-date (version 1.15.76-46.1 installed)
Source package : aws-cli-1.15.76-47.1.src                    
Summary        : Amazon Web Services Command Line Interface  
Description    :                                             
    The AWS Command Line Interface (CLI) is a unified tool to manage AWS
    services. With this tool, multiple AWS services can be controlled
    from the command line and automated through scripts.

-> zypper info python3-botocore
Loading repository data...
Reading installed packages...


Information for package python3-botocore:
-----------------------------------------
Repository     : cloud_tools                                    
Name           : python3-botocore                               
Version        : 1.10.75-16.1                                   
Arch           : noarch                                         
Vendor         : obs://build.opensuse.org/devel:languages:python
Installed Size : 28.7 MiB                                       
Installed      : Yes                                            
Status         : out-of-date (version 1.10.75-15.1 installed)   
Source package : python-botocore-1.10.75-16.1.src               
Summary        : Python interface for AWS                       
Description    :                                                
    A low-level interface to a growing number of Amazon Web Services.


Looks the same tome me.

But I can confirm there is still a reference of the darn vendored 

-> grep vendored /usr/lib/python3.6/site-packages/awscli/paramfile.py
from botocore.vendored import requests
Comment 9 Robert Munteanu 2018-08-31 15:43:13 UTC
$ aws --help

aws help
/usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.16) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "/usr/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 58, in main
    driver = create_clidriver()
  File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 69, in create_clidriver
    event_hooks=emitter)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/lib/python3.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/lib/python3.6/site-packages/awscli/handlers.py", line 20, in <module>
    from awscli.paramfile import register_uri_param_handler
  File "/usr/lib/python3.6/site-packages/awscli/paramfile.py", line 17, in <module>
    from botocore.vendored import requests
ModuleNotFoundError: No module named 'botocore.vendored'


Information for package aws-cli:
--------------------------------
Repository     : openSUSE-Tumbleweed-Oss                   
Name           : aws-cli                                   
Version        : 1.15.76-1.1                               
Arch           : noarch                                    
Vendor         : openSUSE                                  
Installed Size : 3.8 MiB                                   
Installed      : Yes                                       
Status         : up-to-date                                
Source package : aws-cli-1.15.76-1.1.src                   
Summary        : Amazon Web Services Command Line Interface
Description    :                                           
    The AWS Command Line Interface (CLI) is a unified tool to manage AWS
    services. With this tool, multiple AWS services can be controlled
    from the command line and automated through scripts.
Comment 10 Cyber Killer 2018-09-07 12:24:20 UTC
Affects also 42.3 with package from both the main repo and Cloud:Tools:

:~> aws
Traceback (most recent call last):
  File "/usr/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 58, in main
    driver = create_clidriver()
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 69, in create_clidriver
    event_hooks=emitter)
  File "/usr/lib/python2.7/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/lib/python2.7/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/lib/python2.7/site-packages/awscli/handlers.py", line 20, in <module>
    from awscli.paramfile import register_uri_param_handler
  File "/usr/lib/python2.7/site-packages/awscli/paramfile.py", line 17, in <module>
    from botocore.vendored import requests
ImportError: No module named vendored

:~> zypper info aws-cli

pakiet aws-cli — informacje:
----------------------------
Repozytorium              : Cloud:Tools                               
Nazwa                     : aws-cli                                   
Wersja                    : 1.15.76-47.1                              
Architektura              : noarch                                    
Dostawca                  : obs://build.opensuse.org/Cloud            
Rozmiar po zainstalowaniu : 4,0 MiB                                   
Zainstalowano             : Tak                                       
Status                    : aktualny                                  
Pakiet źródłowy           : aws-cli-1.15.76-47.1.src                  
Podsumowanie              : Amazon Web Services Command Line Interface
Comment 11 Robert Schweikert 2018-09-17 19:44:10 UTC
Fixed now in CLoud:Tools
Comment 13 Swamp Workflow Management 2018-12-12 08:42:32 UTC
SUSE-RU-2018:4074-1: An update that solves one vulnerability and has 5 fixes is now available.

Category: recommended (moderate)
Bug References: 1088310,1092493,1098125,1105988,1118021,1118027
CVE References: CVE-2018-15869
Sources used:
SUSE Linux Enterprise Module for Public Cloud 15 (src):    aws-cli-1.16.61-4.7.1
SUSE Linux Enterprise Module for Packagehub Subpackages 15 (src):    python-boto3-1.9.57-3.5.1, python-botocore-1.12.57-3.5.1, python-s3transfer-0.1.13-3.3.6
SUSE Linux Enterprise Module for Open Buildservice Development Tools 15 (src):    python-boto3-1.9.57-3.5.1, python-botocore-1.12.57-3.5.1, python-s3transfer-0.1.13-3.3.6
SUSE Linux Enterprise Module for Basesystem 15 (src):    python-boto3-1.9.57-3.5.1, python-botocore-1.12.57-3.5.1, python-s3transfer-0.1.13-3.3.6
Comment 14 Swamp Workflow Management 2018-12-13 11:12:01 UTC
openSUSE-RU-2018:4114-1: An update that solves one vulnerability and has 5 fixes is now available.

Category: recommended (moderate)
Bug References: 1088310,1092493,1098125,1105988,1118021,1118027
CVE References: CVE-2018-15869
Sources used:
openSUSE Leap 15.0 (src):    aws-cli-1.16.61-lp150.3.3.1, python-boto3-1.9.57-lp150.2.3.1, python-botocore-1.12.57-lp150.2.3.1, python-s3transfer-0.1.13-lp150.2.3.1
Comment 16 Swamp Workflow Management 2020-01-28 20:11:20 UTC
SUSE-SU-2020:0251-1: An update that solves one vulnerability and has four fixes is now available.

Category: security (moderate)
Bug References: 1092493,1105988,1118021,1118024,1118099
CVE References: CVE-2018-15869
Sources used:
SUSE OpenStack Cloud Crowbar 8 (src):    aws-cli-1.16.297-22.11.1
SUSE OpenStack Cloud 8 (src):    aws-cli-1.16.297-22.11.1
SUSE Linux Enterprise Module for Public Cloud 12 (src):    aws-cli-1.16.297-22.11.1
HPE Helion Openstack 8 (src):    aws-cli-1.16.297-22.11.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.