Finding nasty bugs is never pleasant. Finding nasty bugs that others left and that you’re pretty much powerless to solve is even worse. Trying to solve them with the wrong tools is mission impossible.
A couple days ago I encountered a particularly interesting bug regarding Cloud Run and environment variables. To be more precise, a value ending in a space () character:
When setting said ENV var via the UI*, everything would show up as you would expect. The value had the space at the end and it all looked good. Yet the app was crashing. Logs pointed us to that ENV var which, at runtime, was somehow trimmed (we obviously weren’t doing that via our code). This left me baffled for a good while. I checked, rechecked. “But it all looks good!!” – I said to myself. And therein lies the issue.
The bug came from Google’s Cloud Run service, sure, but I was not using the right tools at my disposal to find it.
That’s the main takeaway I wanted to share. I quickly opened up my terminal and updated the variable via gcloud, using double quotes to ensure the value was set correctly. Lo and behold, problem fixed. Somehow setting it via the UI didn’t play nice, but I was banging my head against the wall (figuratively) by using the UI to try to solve it.
Fortunately it only took about 10 minutes for me to realise this, but I could’ve spent a good couple hours – hell, even tried to convince my team to change that value for something else altogether! – had I not stopped for a second to think.
TL;DR: Don’t forget the other tools in your toolbelt. Using a wrench to nail something might work, but not be ideal. Save yourself time.
Note: I’ve already reported this issue to Google, and it’s the second one I’ve encountered regarding ENV vars with their services. We’re all human, even Google Engineers. Contribute to OSS however you can!
* I know, I know. We’re in the process of migrating to IaC with OpenToFu, but we’re a small startup and other more pressing matters take precedence T.T
