This series of posts explores the intriguing challenges I’ve faced in my career. Today, I’ll discuss a significant hurdle: transforming Splunk’s single-tenant architecture to support multi-tenancy, a feature our competitors leveraged to share resources cost-effectively.
The initial problem was clear: running Splunk docker containers in the cloud was prohibitively expensive for smaller customer workloads. Given that Splunk clusters are inherently large, creating a small cluster was still financially burdensome for our customers. This limitation raised doubts about our ability to offer a competitive, cost-effective solution.
Splunk was traditionally designed to handle one customer workload at a time, making the concept of multi-tenancy seem nearly impossible. The challenge was compounded by our organizational structure, which was divided into siloes overseen by various VPs with different priorities. Even securing the necessary resources was a challenge due to differing team skills and languages.
To tackle this, we formed a cross-functional virtual team that met weekly. We divided the project into smaller, manageable tasks and celebrated each milestone, which helped build camaraderie. Our team bonding activities, like board game nights, lunches, and off-site events, played a crucial role in our project’s success.
Our efforts paid off spectacularly. We managed to set up independent clusters for indexing, ingesting, and searching that could serve multiple customers simultaneously. We introduced controls to adjust multi-tenancy levels, demonstrating our engineering team’s capabilities and gaining significant momentum within the company.