Why I added Go to my Salesforce toolkit - and what broke first

Every Salesforce Solution Architect eventually hits a wall where platform-native tools aren't enough. To truly master the "System" in System Architect, you need a low-level language in your tool belt. Because even Salesforce has its own platform language and tools, real implementations always have edges the platform can't reach — integration load testing, data migration at volume, backup and archiving — all of these live in the gaps between Salesforce and everything else.

The Node.js years

Salesforce platform is more comfortable with supportive tool-stack, as a Solution Architect, I had to design many integrations, planned, supported and conducted countless data migration processes.

Almost every project demands data migration, and almost none of the data migration processes finish uneventfully. Given my background, my go-to language was Javascript with node.js (I love Java but man, it is NOT lightweight at all!!). While it was a good option for data transformation and migration including its API capabilities for direct load via Bulk API - Integration testing was a different topic all together.

Node.js is good for functional testing and can be used with load testing, however there are better options available with more concurrency power.. like Go.

Why Go?

The incident

It all started with a project assessment that has a Go backend. Raw performance Go offers vs the nostalgic syntax of it took me to my university years where I was a freshman, learning C. I didn't give it a lot of thought except "this looks cool" until I faced a real problem. It was a requirement, not a world changing as per se - but it did change my world.

One of the largest scale projects I have ever worked on was live for a few months and everything was going well, until it wasn't. One big loyalty campaign and within seconds, thousands of concurrent requests started triggering alarms. With the multi layer/system projects - it is not that easy to pinpoint the issue immediately, especially on a live system flooding with millions of requests per day.

The diagnosis

At first, the issue was identified as the external middleware system causing the throttle and requests weren't reaching Salesforce, but once that side was solved - I saw the same pattern in Salesforce with a system level throttle to keep global infrastructure safe. Once I started capturing the logs, and information from middleware side - I saw that issue is not about server availability but "too many concurrent callouts".

I started looking around for the critical paths, trying to identify the endpoints/processes causing the issue and tried to understand the volume. It took some hours that felt like months - but the answer was there, root cause was the communication timing. More than 6 million customers received a communication about campaign ending all around the world and many simultaneously clicked the notification to see details which narrowed down the suspicious endpoints into a single one. Salesforce was very fast, responded the case very quickly to help me validate the findings and confirmed that their internal logs captured the same endpoint causing the throttling. It was a lucky moment for every party in the system that, it happened on the last day of the campaign and outage was not more than a few minutes also thanks to Salesforce product team increasing the available resources but the incident exposed that we had no way to simulate that level of concurrency-traffic. The recovery was lucky. Next time I wanted to find the wall before customers do.

The realization

This was a wake up call on my side, I immediately started planning concurrency test and how I can execute it as soon as possible. I have started thinking about an architecture based on Node.js for simulating the load. At that moment, I realized that while Node.js is great for many things, its single-threaded nature was a bottleneck for the sheer volume of concurrent 'noise' I needed to simulate. Then it hit me! Could Go be the answer of my trouble of simulating this level of traffic? A bit of mind-search turned up the answer: Goroutines. Lightweight threads managed by Go, which is arguably gold standard for the job.

I designed the toolkit architecture myself; Gemini was a sparring partner for the Go syntax. A few hours later the PoC was running and the initial results landed — there wasn't a single bottleneck, but three endpoints causing similar issues at different points in the process. The results were eye-opening; we'd been blind to two of the three problems. With that visibility, I could design the optimization solution. After this, I continued working on the project (and still continuing the work for humoring my inner perfectionist) and created a stable version to be able to use it on every other integration project.

If you're curious what the architecture looks like — or want to use it as a starting point for your own integration testing — I open-sourced the toolkit at github.com/aliozdemir13/clockwork-piranha. It's a WIP working MVP, honest README about what's done and what's next.

I am not saying every Salesforce Solution Architect should use Go as their supportive language, but I am saying every Salesforce Solution Architect should have a toolkit in their tool belt for the proper integration testing, and Go is one of the best languages to do the job.

What I'd do differently

Integration testing is vital, but load testing alone isn't enough. Load testing answers "can the system handle X requests per second over time?" Concurrency testing answers "can it handle X requests in the same second?" For B2C projects — where a single notification can put millions of customers on the same endpoint within seconds — that second question is the one that breaks production. The first one was already in my plan. The second one wasn't