I’ve been having problems since I installed my own build of OpenSSL - I wanted to be able to use SSL2/3 for testing and some weird ciphers, and the build on Ubuntu wasn’t playing nice. wget and ansible have been giving me grief about SSL trust, and I finally got annoyed enough to sort it out.
I knew wget --ca-certificate=/etc/ssl/certs/ca-certificate.crt https://yaleman.org worked, so it wasn’t the certificate store itself.
Searching the internet for “wget ssl trust” or “openssl ca trust” is just a waste of time - you might as well search for “cat pictures”.
After trying to run lsof quickly enough to trace things, I searched with the right magic incantations and found this stackoverflow thread about strace which led me to the fix 😀
$ strace -e trace=open,openat wget https://www.openssl.org/source/openssl-1.1.0i.tar.gz
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libidn2.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/local/lib/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/local/lib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libpsl.so.5", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libunistring.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/wgetrc", O_RDONLY) = 3
openat(AT_FDCWD, "/home/yaleman/.wget-hsts", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
--2018-09-06 18:26:29-- https://www.openssl.org/source/openssl-1.1.0i.tar.gz
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/en_AU/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_AU/LC_MESSAGES/libc.mo", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/ssl/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
openat(AT_FDCWD, "/usr/local/ssl/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/local/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_AU/LC_MESSAGES/wget.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/wget.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_AU/LC_MESSAGES/wget.mo", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/wget.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
Resolving www.openssl.org (www.openssl.org)... openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3
23.12.75.221, 2600:1415:8:283::c1e, 2600:1415:8:289::c1e
Connecting to www.openssl.org (www.openssl.org)|23.12.75.221|:443... connected.
openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
ERROR: cannot verify www.openssl.org's certificate, issued by 'CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US’:
Unable to locally verify the issuer's authority.
To connect to www.openssl.org insecurely, use `--no-check-certificate'.
+++ exited with 5 +++
The key was this line:
openat(AT_FDCWD, "/usr/local/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
It wasn’t looking in /etc like I was expecting, it was looking in /usr/local - which makes sense, because that’s probably more standard for a non-standard build.
The fix was the following command:
$ sudo ln -s /etc/ssl/certs/ca-certificates.crt /usr/local/ssl/cert.pem