RFC: CI limitations and using 'six' module for python2/3 compatability

[Noel Power via samba-technical]

Hi team,

I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://pythonhosted.org/six/

Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?

What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?

For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows

  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote

Thoughts?

Noel

[1] Note: not really looking for a review of this patch but rather the
issue of the missing 'six' However this is part of 
https://github.com/samba-team/samba/pull/161 :-)

[jim via samba-technical]

Noel,

Why not add 'six' to third party and make it available from there?

Jim

Post by Noel Power via samba-technical
Hi team,
I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpythonhosted.org%2Fsix%2F&data=02%7C01%7Cjim.brown%40miami.edu%7C117363128f53458d342508d5a3b36833%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636594913303852541&sdata=IsjrP8pgzHLBdqwAPw55G1nZaZE%2FVWN8eYyFWhVxH3E%3D&reserved=0
Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?
What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?
For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows
  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote
Thoughts?
Noel
[1] Note: not really looking for a review of this patch but rather the
issue of the missing 'six' However this is part of
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsamba-team%2Fsamba%2Fpull%2F161&data=02%7C01%7Cjim.brown%40miami.edu%7C117363128f53458d342508d5a3b36833%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636594913303852541&sdata=Fq67adH8dI45NcFl%2FrKoBMt329NBARUXEZTLbQm%2BOL0%3D&reserved=0 :-)

[Alexander Bokovoy via samba-technical]

Post by Noel Power via samba-technical
Hi team,
I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://pythonhosted.org/six/
Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?
What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?
For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows
  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote
Thoughts?

python-six is a good thing to use and it should already be available on
all platforms we test against, either as a native package (python-six on
Ubuntu/RHEL/CentOS, python{2,3}-six on Fedora) or can be installed with
pip otherwise.

I certainly would prefer using python-six to inventing our own
compatibility code for those places where we have an overlap as
python-six is quite widely used in Python community already.

Metze, could we get it installed in autobuild?

[Stefan Metzmacher via samba-technical]

Post by Alexander Bokovoy via samba-technical

Post by Noel Power via samba-technical
Hi team,
I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://pythonhosted.org/six/
Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?
What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?
For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows
  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote
Thoughts?

python-six is a good thing to use and it should already be available on
all platforms we test against, either as a native package (python-six on
Ubuntu/RHEL/CentOS, python{2,3}-six on Fedora) or can be installed with
pip otherwise.
I certainly would prefer using python-six to inventing our own
compatibility code for those places where we have an overlap as
python-six is quite widely used in Python community already.
Metze, could we get it installed in autobuild?

python-six was already installed and I installed python3-six now.

We'd have to adjust .travis.yml too.
I'm not sure what's needed for gitlab.

metze

[Andrew Bartlett via samba-technical]

On Mon, 2018-04-16 at 22:25 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical

Post by Alexander Bokovoy via samba-technical
Metze, could we get it installed in autobuild?

python-six was already installed and I installed python3-six now.
We'd have to adjust .travis.yml too.
I'm not sure what's needed for gitlab.

The docker image just needs to be updated. I can do that, and we
should probably merge the scripts for that into a team-controlled repo
(they are currently in Catalyst's samba-cloud-autobuild repo).

The docker registry it hosted on gitlab.com

I'm happy to share admin access to the samba-team gitlab organisation
around to other team members BTW, I don't want to 'own' that in
perpetuity.

Andrew Bartlett

[Garming Sam via samba-technical]

Hi,

I would have some reservations about using six. The last time I talked
to Andrew, he felt quite strongly about the matter too. The overall
summary of the concern is that we would be left at a half-way point
which is neither purely-standard Python 2 nor Python 3 (as well as
gaining the additional dependency on six). If the distros are as serious
about dropping support for Python 2 as they claim, then having pure
Python 3 would be preferable. The last thing I would want is a secondary
effort required just to remove six as a dependency and rewrite it in
Python 3 anyways.

If the consensus really is to use six, at the very least I would propose
that we use it in as constrained a way as possible. The straightforward
renames and the example with urllib seem reasonable enough. On the other
hand, using pretty much any of the six object model (or even syntax
compatibility) does not seem like a particularly great idea in my view.

(In regards to Gitlab, it needs to be added to the list of dependencies
and then an image rebuilt).

Cheers,

Garming

Post by Noel Power via samba-technical
Hi team,
I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://pythonhosted.org/six/
Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?
What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?
For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows
  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote
Thoughts?
Noel
[1] Note: not really looking for a review of this patch but rather the
issue of the missing 'six' However this is part of 
https://github.com/samba-team/samba/pull/161 :-)

[Alexander Bokovoy via samba-technical]

Post by Garming Sam via samba-technical
Hi,
I would have some reservations about using six. The last time I talked
to Andrew, he felt quite strongly about the matter too. The overall
summary of the concern is that we would be left at a half-way point
which is neither purely-standard Python 2 nor Python 3 (as well as
gaining the additional dependency on six). If the distros are as serious
about dropping support for Python 2 as they claim, then having pure
Python 3 would be preferable. The last thing I would want is a secondary
effort required just to remove six as a dependency and rewrite it in
Python 3 anyways.

Six helped us a lot in FreeIPA ports to Python3. We have to maintain
Python2/Python3 versions with a single code base and six has reduced the
amount of maintenance work dramatically.

$ cd freeipa
$ git grep -E '(import|from) six'|wc -l
390

FreeIPA's use of six is around following areas:
- string and integer types (six.text_type, six.string_types,
six.integer_types)
- Python version detection (six.PY2 and six.PY3)
- iterators (six.itervalues, six.iteritems, six.moves.zip)
- addressing renames of modules (six.moves...)
- metaclass programming (six.with_metaclass)

That's all. It allows us to have a version that works in RHEL and
Fedora (and Debian/Ubuntu once new 4.7 release is finally out).

Post by Garming Sam via samba-technical
If the consensus really is to use six, at the very least I would propose
that we use it in as constrained a way as possible. The straightforward
renames and the example with urllib seem reasonable enough. On the other
hand, using pretty much any of the six object model (or even syntax
compatibility) does not seem like a particularly great idea in my view.

As you can see, apart from metaclass abstraction it is basically the
same for FreeIPA, just that we had already complex Python code base.
Samba Python code is not very Python idiomatic so in many cases it is
not about porting to Py2/Py3 compatible code but rather rewriting it in
a more Pythonic way which can reduce number of borrowed features from
six too.

Post by Garming Sam via samba-technical
(In regards to Gitlab, it needs to be added to the list of dependencies
and then an image rebuilt).
Cheers,
Garming

Post by Noel Power via samba-technical
Hi team,
I like to start a discussion about the use of 'six' (a python2/3
compatability layer api) https://pythonhosted.org/six/
Recently I have code across a couple of instances where I wanted to use
the 'six' library to port some samba python code to be python2/python3
compatible. Currently it is not possible to use 'six' as the CI machine
does not have it installed. I guess it is likely (I don't know) that the
autobuild machines also don't have it, does someone know ?
What is the possibility of getting the 'six' library in place it the CI
(and/or) autobuild machines?
For the moment my approach is to use the existing python/samba/compat.py
(currently not used) see attached patch [1] But.. I fear this is going
to become poor relation 'six' clone, my hope is using and extending this
is a temporary measure until we can kill it and replace it with six. The
transition from compat to six should be trivial e.g. and example of code
using the the existing content of the compat.py could be changed as follows
  -from samba.compat import urllib_quote
  +from six.moves.urllib.parse import urllib_quote
Thoughts?
Noel
[1] Note: not really looking for a review of this patch but rather the
issue of the missing 'six' However this is part of 
https://github.com/samba-team/samba/pull/161 :-)

[Andrew Bartlett via samba-technical]

On Tue, 2018-04-17 at 12:20 +1200, Garming Sam via samba-technical

Post by Garming Sam via samba-technical
Hi,
I would have some reservations about using six. The last time I talked
to Andrew, he felt quite strongly about the matter too. The overall
summary of the concern is that we would be left at a half-way point
which is neither purely-standard Python 2 nor Python 3 (as well as
gaining the additional dependency on six). If the distros are as serious
about dropping support for Python 2 as they claim, then having pure
Python 3 would be preferable. The last thing I would want is a secondary
effort required just to remove six as a dependency and rewrite it in
Python 3 anyways.
If the consensus really is to use six, at the very least I would propose
that we use it in as constrained a way as possible. The straightforward
renames and the example with urllib seem reasonable enough. On the other
hand, using pretty much any of the six object model (or even syntax
compatibility) does not seem like a particularly great idea in my view.
(In regards to Gitlab, it needs to be added to the list of dependencies
and then an image rebuilt).
Cheers,

Thanks Garming,

This captures my views on that matter very well.

Thanks,

Andrew Bartlett