I'm embarrassed to admit it

Let's file this under "embarrassing misses which I hope make me a better engineer": I've been beating up getting the Solaris Userland build of Python Cryptography updated from the somewhat-old v1.7.2 to the current v2.1.4. The first thing I realised is that we needed a new version of cffi. Pretty easy to update, though that mean I had to update xattr too (all updates, like bugs, have friends).

After migrating cffi from building with Studio to gcc and mucking around with patches I finally had something could install in a crash-n-burn kz in our cloud. Yay, or so I thought.

When I tried to run pkg verify in that kz with the new packages, the packaging system died in a heap with an error that I distilled down to this:

$ python2.7   
Python 2.7.14 (default, Jan 31 2018, 05:35:05) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from cryptography.hazmat.bindings._openssl import ffi, lib
>>> from cryptography.x509 import certificate_transparency
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/vendor-packages/cryptography/x509/__init__.py", line 7, in <module>
    from cryptography.x509 import certificate_transparency
ImportError: cannot import name certificate_transparency

What on earth does that mean? A week or so of googling and delving deep into the code left me no wiser. I came across one "solution" on stackoverflow which said to "merely" update pip from v7 to v9 and "that solved everything". That's not what I think of as a solution. It might be sufficient for an end-user, but I'm not in that position; I'm the bloke who's doing the release engineering aspect to make sure that the end-user doesn't have a problem.

After finally realising that there's a Python Cryptography channel (#cryptography-dev) on irc.freenode.net I wandered in and asked for help. Alex asked if I got an exception from

from cryptography.hazmat.bindings._openssl import ffi, lib

which I didn't, but a subsequent from cryptography.x509 import certificate_transparency got me the same ImportError I noted above. THAT got me wondering if cffi was built correctly. As it happens, it wasn't (all my fault, yes):

>>> import cffi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/vendor-packages/cffi/__init__.py", line 4, in <module>
from .api import FFI
  File "/usr/lib/python2.7/vendor-packages/cffi/api.py", line 3, in <module>
    from .error import CDefError
ImportError: No module named error

In short order this lead me to copy the bits from my build system's proto area directly to the crash-n-burn kz for both cffi and cryptography, and boom I could import certificate_transparency.

You can see where this is going - a quick check of the proto areas for files which weren't included in each package manifest, updating the p5m (along with pkgfmt) and a gmake publish:

$ cd build/prototype/`uname -p`
$  for f in `find usr  -name \*py -o -name \*h -o -name \*so | \
   sed -e"s,python[23].[745],python\$\(PYVER),g" |sort |uniq | \
   grep -v  cpython`; do \
        grep -q "$f" ../../../$P5M || \
        echo "file path=$f" >> ../../../$P5M; 
$ cd ../../..
$ pkgfmt $P5M

I'm kicking myself that I forgot to include that tiny step in my efforts over the last week, but I won't in future.