Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Instead of checking against a file that represents a distribution that hasn't
existed for years, fetch package names for Clear Linux instead.

[ YOCTO #10601 ]

Signed-off-by: Ross Burton <ross.burton@intel.com>

This library suffered as part of the Python 2 to Python 3 migration and stopped
working entirely.

Fix all the migration problems such as files being treated as strings but opened
in binary mode, insufficient use of with on files, and so on.

Rewrite large amounts to be Pythonic instead of C-in-Python.

Update OpenSuse and Fedora URLs.

Fedora now splits the archive alphabetically so handle that.

[ YOCTO #10562 ]

Signed-off-by: Ross Burton <ross.burton@intel.com>

create_socket: Use urllib because urllib2 is now urllib in python3
and proxies as argument are deprecated so export them in the environ
instead.

get_links_from_url: Change usage of sgmllib for parsing HTML because
is deprecated in python 3, use instead bs4 that is already imported
in the bitbake tree.

[YOCTO #9744]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

If the proxies dictionary has a proxy set to None urllib will throw an exception
instead of not using a proxy (abridged stack):

File: '/home/ross/Yocto/poky/meta/lib/oe/distro_check.py', lineno: 43, function: get_links_from_url
 *** 0043:    with create_socket(url,d) as sock:
     0044:        webpage = sock.read()
File: '/home/ross/Yocto/poky/meta/lib/oe/distro_check.py', lineno: 5, function: create_socket
     0003:def create_socket(url, d):
     0004:    import urllib
 *** 0005:    socket = urllib.urlopen(url, proxies=get_proxies(d))
File: '/usr/lib/python2.7/urllib.py', lineno: 87, function: urlopen
     0086:    if data is None:
 *** 0087:        return opener.open(url)
File: '/usr/lib/python2.7/urllib.py', lineno: 203, function: open
     0201:        else:
     0202:            proxy = None
 *** 0203:        name = 'open_' + urltype
     0204:        self.type = urltype
     0205:        name = name.replace('-', '_')
Exception: TypeError: cannot concatenate 'str' and 'NoneType' objects

Filter out unset values so that the dictionary only has valid assignments in.

Signed-off-by: Ross Burton <ross.burton@intel.com>

There are two ways to pass the proxies values into urllib.open: exporting
the environment variables *_proxy and passing as dictionary into the
urllib.open call. The latter is currenty used and values should not have
the '_proxy' string, so removing it. Also, ignore uppercase proxies, these
are not taken into account by the library.

Tested on a network with proxy, adding 'inherit +="distrodata"' on local.conf
with the following commands

    for distrotask in distrodataall distro_checkall checklicenseall; do
        bitbake universe -c $distrotask
    done

[YOCTO #7567]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>