Want to be a better developer? Ask better questions.
Experienced engineers know that Knowledge teaches us that there are screws that need turning. Experience teaches which screws to turn and which ones to ignore.
Recently, I had a simple goal: Connect to someone’s API. This required an initial step of trivially getting authorisation via OAuth2.
In my head, this sounded like a 15-30 minute task. I expected the flow to be the following:
Read the provider’s API doc
Create the ClientID and Client Secret
Create a custom URL
Create a Golang router and call the URL
Get the authID
Eat 🍰
Use the authID/Token to connect to the API to get things done.
What happened instead?
A Step 4 things became interesting:
I got an unexpected "
x509: certificate signed by unknown authority
".Scratch head (cause I had no clue what happened here, and it means that cake is going to be delayed).
I checked the API documentation again. It read: “Next, you will have to call the Authorization URI - https://accounts.[url].com/oauth/v2/auth”
On reflection, things really had begun to fall apart here. My interpretation of that one line was the source of so many hours of grief. 🥴😭😣.
Initially, when I saw that line (...you will have to call…), I naturally used Go’s http.NewRequest(...) API and the http.DefaultClient.Do(...)
to connect to the resulting URL from my server.
On the path to a solution, I did the following things:
Got some insights here to understand what was happening. Thanks to: https://newrelic.zendesk.com/hc/en-us/articles/360051503373-Go-Getting-issue-x509-certificate-signed-by-unknown-authority-in-golang-newrelic-agent
Checked who their certificate issuer was on google
Verified that the issuers weren't in the local trust store with this:
awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crtDownloaded their certs to my server with:
openssl s_client -showcerts -connect www.[url].com:443
Configuring a Certificate Authority on Ubuntu so that I could add their root Certificate Authority to digital oceans local trust store, thanks to:
https://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntuThen added the certs with: sudo update-ca-certificates
Surprise surprise, it still didn’t work. 🤦🏽♂️🤬
The code still ran and returned the same exact error: "x509: certificate signed by unknown authority". Only this time. It seemed like it was yelling at me.
Interlude to success. Interrogate the problem and my understanding of it.
I have a simple rule of thumb. Whenever I get stuck for upwards of 4 hrs on any problem, then this typically means that there is something wrong with my understanding of the problem. It also means that I’m not asking the right questions.
The solution then requires shifting mental models.
Knowing this, I asked:
What if the code isn’t the problem?
What else could be causing the problem?
Missteps
I had mistakenly believed that my code was magically reading from the server’s store which was at /etc/ssl/certs/ca-certificates.crt. I eventually realised that Docker has its own local trust store (/etc/ssl/cert.pem for alpine containers). 🤦🏽♂️ 😮💨
The obvious fix was to map the volumes. Simple enough (I thought).
For the Dockerfile we just:
FROM scratch
ADD /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
This failed.
Maybe from the docker-compose.yml file?
volumes:
- /etc/ssl/certs/:/etc/ssl/certs/
This also failed. 😯😏
Rabbit holes
A few more hours. Then the familiar thinking technique: Am I asking the right question?
The same code runs without the X509 error on my local machine. The same docker container runs without producing the X509 error when it runs on my mac locally. However, when deployed to the remote Ubuntu server, the code fails although in a docker container.
Reading more, I found that docker doesn’t always play well with symbolic links. So I placed a copy of the ca-certifcates.crt file in the same folder of the docker-compose.yml file and changed the volumes to the following:
volumes:
- ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
What do you know! The X509 error goes away. 😱🥳
Almost 🍰🧁 time...
All installations of the code were operating in the same manner.
The code calls the Authorisation end-point… and then… gives me a fabulous “cookies aren’t enabled on the browser” error. 😅😕🤷🏽♂️
Initially, I assumed I got this error (locally) because I was calling their Authorisation URL from my local machine and they were blocking the call due to my IP-address.
Getting unstuck
After a few more hours digging around. I returned to my favourite thinking technique. I asked myself: What if the code isn’t the problem? What questions am I not asking?
This forced me to yet again, revisit the vendors’ API documentation. The word “call” struck me. What if I miss-understood their intent for invocation?
5 minutes later, I crafted the code to provide a link that I could navigate to their API end-point from an unlisted page on the site. I clicked the link and 🍰🧁!
I had the solution all along.
The failure was in my mental model and how I interpreted the use of the word call. API’s author’s use of the word had a different meaning from me. For them a “call” was equivalent to a navigate (in my world).
Takeaways
Interrogating your mental model of the world is important.
The quality of your code isn’t a reflection of your self worth. It's more a reflection of your cumulative experiences and how you think about problems. So, when stuck, interrogate the problem differently.
Knowledge teaches us that there are screws that need turning, experience teaches us which screw to turn and which ones to ignore.
Eat 🍰🧁 responsibly.