<img src="https://ws.zoominfo.com/pixel/6169bf9791429100154fc0a2" width="1" height="1" style="display: none;">
Curious about how StrongDM works? 🤔 Learn more here!
Close icon
Search bar icon

How Splunk Built A Practical Approach to DevSecOps At Scale

StrongDM manages and audits access to infrastructure.
  • Role-based, attribute-based, & just-in-time access to infrastructure
  • Connect any person or service to any infrastructure, anywhere
  • Logging like you've never seen

What Splunk Does

Joel Fulton is the Chief Information Security Officer for Splunk. At Splunk, they've put effort into transforming their organization from a waterfall approach to agile, to now a DevSecOps approach.

In case you're not familiar, Splunk is a software development company focused on machine data aggregation. They collect your data on to your on-prem and they count on you to manage and protect that. Splunk relies on your perimeter, your moat, your infrastructure, your detection, your response. Splunk's licensing model guarantees nobody puts innocuous data into Splunk. They charge on ingestion, so you only put the most important pieces of your data into Splunk. When that situation is then delivered through the cloud, you're delivered you not only your Splunk as a service but all of the vulnerabilities that are part of that Splunk code as well as the infrastructure in which it's hosted. So now it's a different risk problem for Splunk because the customer assumes, correctly, and demands that the security they've got in place meets or exceeds customer requirements. That times ten thousand customers.

Joel Fulton CISO Splunk Speaks at StrongDM's conference

Splunk's Development Process

How Splunk got there is they backed into this process. Splunk started with waterfall. Splunk was built under a waterfall methodology. They quickly moved to agile which for them meant waterfall without the rules, and that creates lots of chaos. Splunk's  process of getting into cloud was to take their on prem software and host it in an AMI in Amazon. So that's really on prem hosted. So all of the problems that originated in their waterfall style: product checks, secure development lifecycle, code reviews, before release to production, the assessment of the amalgamation of vulnerabilities, that was all done at a checkpoint at a moment before making it live for the customer. Because it wasn't built for the cloud, they lacked things like orchestration so they could provision systems with rapidity and consistency, but then the customers would each demand snowflakes. They wanted to tweak this way they wanted this bits added to it. And that was a necessary request because Splunk was on prem hosted in the cloud at the time.

So then Splunk gets drift in their systems and all of these hosts serving, transacting, storing, and processing customer data are now starting to drift in their configuration. Joel likens the process to an El Camino. You know what an El Camino is? It's like a truck and a sedan put together; you've got all the carrying capacity of a sedan and all the passenger capacity of a truck - the worst or both worlds.

So it's this agile machine within a waterfall framework because security was the problem. And the problem with this is the difference between agile and DevSecOps, is when do you put security into the process? When do you understand the cumulative impacts of vulnerabilities that you have. And that was their real problem because they can assess source code as it goes through but what Splunk couldn't assess in the agile process is what is the concomitant result of the amalgamated vulnerabilities when it goes into GA?

If the customer would come to Splunk and say "hey why were we exploited? Why was there a problem?" The customer doesn't care that it wasn't the 'security' team, it was the 'product' team. It's the amalgamation of vulnerabilities that Joel and his team and then Splunk at large cares about. The difficulty that Splunk had was in the planning phase.  There wasn't a security member in the daily standup, seeing the user stories. And so they don't know what's coming. Therefore Joel couldn't predict what the result of the amalgamation of these changes in the environment in the milieu in which they offered this to customers was going to be. That bit of blindness, that bit of lack of visibility, is what stopped Joel and his team from being part of their process.

How Splunk Went from a Waterfall Approach to Agile, to Now a DevSecOps Approach

So, to integrate into what was maturing agile and maturing DevOps process, here's what Joel did:

First, Splunk has members of the security team that are part of engineering and development organization involved in the daily standups and involved in taking bug reports, triaging, doing the selection of features - the user stories that include security as an emphasis. Joel has a security team that are directly under him and they're separate. And the reason they've separated this is they are conforming to the org that the product team have. People think Splunk is a security company, but they're not, they're a software development company. The distinction and the difference between those two is significant in terms of persuasion priorities and what features are released. So the product team have segregated the application from the infrastructure and the hosting environment. And so security is meeting them on their grounds and they're in their daily standups. But security are the ones that take a step back and put them together to understand the consequences of the differing vulnerabilities when they come together for the customer.

The next step, and where Splunk is today is moving the organization further into a security mindset. How do we hold them accountable and reward them. The features that matter to engineers are not the features that matter to security necessarily. It is less cool to go back and fix something and get really no credit except nothing bad happens so there's a negative credit as opposed to new feature new functionality that people will create and trumpet about. When security has a good day it's a boring day. That's not what most people seek out of their career. And so being able to motivate, being able to train, doing threat modeling and sharing the information: This has been a massive advantage to Splunk to persuade a change in the engineering environment.

The focus what on what matters and how to align those things. For engineers, they're all in the result of that code bears on them personally. It bears on their relationship with their peers; they take quality as an emblem of pride. The production matters to them. They viewed Joel and his team as kind of having an interest but not skin in the game. Getting that skin in the game, having Joel's team embedded in the engineer's team - that enabled security to meet engineering where they were. And that's what moved the line for Splunk in getting a better outcome and getting Sec in the DevSecOps.

About the Author

, Chairman of the Board, began working with startups as one of the first employees at Cross Commerce Media. Since then, he has worked at the venture capital firms DFJ Gotham and High Peaks Venture Partners. He is also the host of Founders@Fail and author of Inc.com's "Failing Forward" column, where he interviews veteran entrepreneurs about the bumps, bruises, and reality of life in the startup trenches. His leadership philosophy: be humble enough to realize you don’t know everything and curious enough to want to learn more. He holds a B.A. and M.B.A. from Columbia University. To contact Schuyler, visit him on LinkedIn.

StrongDM logo
💙 this post?
Then get all that StrongDM goodness, right in your inbox.

You May Also Like

How Fixing Access Improves Security & Productivity Webinar
How Fixing Access Improves Security & Productivity Webinar
Breaking the Cycle: How access, security, and productivity create a vicious cycle, how this manifests in the real world, and importantly, how to break it.
Devising Cloud Strategies and Solutions
Event: Devising Cloud Strategies and Solutions
Attend this MegaCast for a ton of new ideas on how to use the cloud to advance your long-range IT goals. What are the ways that software-as-a-service could replace some on-premises infrastructure? How could a switch to modern applications supercharge an area of your business? Where could an expansion of infrastructure-as-a-service usage reduce your overall costs or dramatically improve your capacity and capability?
DevSecOps: The Core Curriculum Opening Remarks
DevSecOps: The Core Curriculum Opening Remarks
Listen to CEO Liz Zalman give opening remarks at the 2019 DevSecOps conference!
Why ASICS Digital Builds 12-Factor Apps
Why ASICS Digital Builds 12-Factor Apps with a Focus on Infrastructure
John Noss is a Senior Site Reliability Engineer at ASICS Digital, formerly Run Keeper. In this talk, he shares how ASICS Digital builds 12-Factor apps with an emphasis on infrastructure.
How Hearst Eliminates DevOps Complexity
How Hearst Eliminates DevOps Complexity -- An Architecture Review
In this talk, Jim Mortko (responsible for leading all Internet-based engineering and digital production efforts) and DevOps Engineer Manuel Maldonado, they discuss how Hearst eliminated DevOps complexity through automation and tooling decisions. Listen as they walk through their services and application architecture and download the slides now.