DevOps is redefining the way organizations handle software development. But it’s also challenging security professionals in their efforts to manage digital risk. With that said, there are security teams need to be strategic about how they approach DevOps security. Here are some expert recommendations on what to do and what to avoid when implementing security in the DevOps lifecycle.
What are some tips that folks should do when implementing security in the DevOps lifecycle?
Kim Crawley, Cyber Security Writer | @kim_crawley
Things won't be looked after without responsibility, and no one can take responsibility without accountability. At the beginning of a software development project or product, make sure that all roles and responsibilities are clearly and concisely delegated to each individual who is involved. Per Beta News, nearly half of people who work in DevOps say that establishing clear ownership and responsibility is a challenge when it comes to implementing security. And security must be integrated into each and every stage of the DevOps lifecycle.
Leigh Honeywell, CEO at Tall Poppy | @hypatiadotca
Bring security to where people are already working. Make it easy to kick off security processes within the tools your engineers are already using. We did this in my previous role at Slack, and it was essential to scaling up.
Anthony Israel-Davis, Senior Manager, R&D at Tripwire | @anthony_id
The great thing about DevOps is that it automates and streamlines a lot of the traditional stage gates, but those stage gates are still critical controls from both a security and compliance standpoint. Separation of duties is still important, and the pipeline can take what used to be manual hand-off points and build those into an automated process. Triggering a deployment, functional and quality testing, and deployment should all be access-controlled where appropriate and be gated by workflows to mitigate risk. For instance, a container should be scanned for vulnerabilities and secure configurations before deployment. If it passes, move to the next stage if not stop the deployment. Also, be sure to maintain evidence of the stage gates to make it easier for your auditors.
Eric Johnson, Principal Security Engineer at Puma Security | @emjohn20
In my experience, I have worked with many organizations and discussed DevSecOps programs with thousands of information security students in the classroom. One of the first topics we discuss is their DevSecOps culture. Traditional security cultures are always ready to say NO, fail to share information across the organization, and do not tolerate failure. This directly contradicts the DevOps culture, which creates a diverse working environment, empowers teams, enables collaboration and problem-solving, fails fast, and continuously improves. Building a successful DevSecOps program requires security teams to embrace this culture. Security must understand the engineering process and tools that enable DevOps teams to move quickly before contributing. Many security teams fail because they do not understand the tools, jump in too quickly, and disrupt the engineering workflow. To ensure success, slowly introduce one security control at a time and ensure the results are valuable to the team. Join the DevOps culture by continuously monitoring and improving security processes to minimize disruptions. This often includes evaluating results, fine-tuning scanners, and working directly with the engineering teams to minimize false positives. Failing to make the culture transition is the primary reason that DevSecOps fails.
Adrian Sanabria, VP of Strategy and Product Marketing at NopSec | @sawaba
I think some of the most innovative and original security ideas I've seen in the last 10 years have come from DevOps. One that I'd recommend is to task one person in each DevOps group with security responsibility. It might not even be that person's primary role, but having one person who knows they're responsible for security ensures it doesn't get left out or forgotten. Another is to identify security and compliance requirements beforehand and automate as much of it as possible. I've seen some organizations where audit logs go straight into a shared read-only folder for the auditor in real-time. The auditor never even has to ask for them.
Justin Sherman, Co-Founder & Vice President at Ethical Tech | LinkedIn
Implementing security into the software design process is absolutely essential; however, it is by no means an absolute fix and must be implemented carefully. For instance, the majority of software developers do not have adequate training in security. They cannot be reasonably expected to suddenly write secure software when that hasn't been the practice before, but it might be tempting to set this expectation anyway. Thus, organizations must carefully train their software developers on best security practices and invest significant time, money, and resources into ensuring they can do those tasks well. Without this investment and education, efforts at integrating security into DevOps will fail before they even begin.
Lane Thames, Senior Security Researcher at Tripwire | @Lane_Thames
Fortunately, the computing industry has learned that bolting on security to software and systems after they have been designed and deployed does not work well. As a result, the idea of “building security in” has gained traction, and this appears to be the case with emerging DevOps technologies. My belief is that organizations building software and using Agile and DevOps methodologies should DO the following: adopt a holistic security approach and culture. This holistic approach should consider security in the pre-pipe, in-pipe, and post-pipe stages of the continuous integration, continuous deployment, and continuous delivery pipeline. Pre-pipe approaches should consider software security during the planning and coding stage. I would love to see something like ‘Secure Test-Driven Development’ enter the mainstream where we create security-based tests alongside our normal functional tests during the planning stage. In an agile world, automation is key, and we are already defining tests to ensure that our software functions correctly in order to ensure high quality. Software security is a fundamental aspect of software quality, and it should be considered early with security-based tests developed during the early stages. This ultimately leads, over time, to a large library of tests that get executed over and over again during the in-pipe stage where we automate builds and associated tests. Quality gates for these security related tests can stop potentially vulnerable code from moving on into production. Next, we must integrate new technologies that check for security while software works its way through the in-pipe process. This includes adding quality gates within the pipeline based on automated security assessments such as automated, in-pipe vulnerability assessment of the underlying operating system, third-party libraries, and installed applications. Also, this should include quality gates for automated, in-pipe application-level vulnerability assessment using static and/or dynamic analysis for the actual code being pulled into the pipe. Lastly, we must focus on post-pipe security. Automation is a must for Agile and DevOps methodologies, but it will never be a solve-all approach. This is largely where traditional security comes into play such as pentesting, configuration compliance, online vulnerability assessment, red and blue teaming, and overall foundational controls.
Matt Williams, Technical Product Manager at Tripwire | @everydaysec
Threat modeling: Take time to visualize how data flows through your system, when it crosses boundaries, and how it may be misused. Do this for both your product and your CI/CD pipeline itself. This will improve not only security but also your architecture in general. Trust, but verify, public cloud infrastructure: Proper use of public cloud infrastructure can enhance security in addition to development velocity. With the right processes and tools in place, “public” doesn’t have to mean “less secure.”
Stuart Winter-Tear, Secure Design Analyst at Continuum Security | @StegoPax
For me, the single most important *do* in implementing security within the DevOps lifecycle is communication. This may sound obvious, but for far too long, we’ve had the situation in which services are thrown over the wall to a siloed security team at the end of the development process. This has led to a culture in which the security team is viewed as the bottleneck “department of no” presiding over mystical tribal knowledge. The first step in communicating involves learning other languages, and by that I mean security teams must learn to speak “DevOps” and communicate via the tools they work in. Communication facilitates the dissemination of knowledge, and the obvious benefit is the spreading of security know-how to our cross-discipline partners - most notably, those responsible for building and deploying our applications. Communication is a two-way street, however, and security teams must provide a space for pushback, for asking questions and for fostering a collaborative environment. This necessitates “active listening” for building trust, rapport and, most importantly, learning from our DevOps colleagues. From my experience, leveraging threat-modeling and establishing cross-functional “security champions” are the most effective and powerful mechanisms for establishing a DevSecOps communication process.
Pete Zerger, CEO and Managing Partner at Lumagate | @pzerger
Identify cybersecurity champions in Dev and Ops teams. Incorporating focus on security (DevSecOps) into your DevOps process will be difficult without someone who understands security. An Ops-focused security pro will likely focus more on infrastructure-as-code and endpoint security, whereas a Dev-focused security pro will be conversant in security in code (unencrypted connection strings, etc.), though I’ve seen resources from the Ops side catch code-level concerns a developer missed. Identifying these trusted resources early in your DevOps/DevSecOps transition will ensure security awareness is baked into your process with every member of the team. While you’re at it, guide your cybersecurity champions on mutually respectful communication in coaching the Dev and Ops teams on security matters that arise in the process. In the end, you want open communication and a tone that fosters healthy discussion and drives incremental improvement of your Dev team rather than a blame culture that raises anxieties across the team. In the end, this is a journey, a learning experience Dev and Ops are taking together, and mistakes will be made. Make sure you set the tone so these mistakes are a catalyst for maturation of both your DevOps process and team members.
What are some tips that folks should avoid when implementing security in the DevOps lifecycle?
Lori MacVittie, Principal Technical Evangelist at F5 Networks | @lmacvittie
Don’t bring traditional security practices with you. Traditional security solutions are often bolted onto the front of application architectures after the fact and tightly coupled to the network. Modern applications need more flexibility and may not have the luxury of control in their ultimate deployment destination over the network architecture needed to adopt traditional security measures. Consider what you’re trying to secure – the app, its protocol stack, the platform, and the data – and what controls, policies, and application services can provide the security necessary and integrate into the DevOps pipeline.
Clint Malson, Manager of IT DevOps, Tripwire
I think the best advice I can give is to avoid rushing in and forgetting foundational controls. We have been on a journey at Tripwire to build a DevOps environment that meets our developer’s ever-growing and changing needs. There have been countless times when we wanted to charge onward to embrace DevOps to its fullest. The reality is that it takes time. Start slowly and build your controls; once they are in place, expanding becomes a breeze. Don’t think just because you’re not doing DevOps today that you have to rush in unprepared.
Andrew Storms, VP of Security Services at New Context | @st0rmz
A common place to implement security into the DevOps pipeline is at the integration stage. This may take the form of using a tool like Jenkins to run integration tests and also kick off common security tools that perform checks such as static analysis or vulnerability scanners. The unexpected outcome of implementing all these great tools happens when the more security-minded folks crank up the knob to level 11 for all those various security tools. The net outcome might be better insight of your code’s lack of security, but now the integration tests went from taking one minute to one hour. The words of experience are as follows: ease into implementing security and don’t try to go from zero to hero overnight.
Carolyn Taisey Lear, Senior Cyber Security Leader at GE Digital | LinkedIn
Don’t leave the DevOps team out of important tooling decisions. Make them your partner! It’s critical to include the DevOps team in a POC of any security tooling or scanner to be integrated into a CI/CD pipeline.
- Obtain requirements from the DevOps team to evaluate. Example: Scalability, OS integration requirements and management such as patching and updates.
- Include them in technical reviews of the tools with the vendors being accessed. This cuts down critical evaluation time.
- Make sure someone from the DevOps team is a required reviewer of the vendor contract. They know how teams build and push code, thereby helping to ensure the correct license approach.
Always treat the DevOps team as a partner to ensure greater success and strategic direction. As each project or effort is completed, this synergy becomes the norm!
Bob Thomas, Software Engineer at Tripwire | LinkedIn You can fail at DevOps by trying to just hire a “DevOps team” and keep the rest of your process the same. Just like you can’t keep your old culture and process the same and be successful at DevOps, you can’t add security to DevOps by following the same process as traditional IT security. Your DevSecOps team needs to be trained in secure development and operations practices so security is built in to the entire process – not an approval gate before release.
Sunny Wear, Information Security Architect, Web App Pen Tester and Developer | @SunnyWear
As companies attempt to embed security into their DevOps pipeline, organizational leaders look to traditional, educational settings to train secure coding techniques to their programmers. However, this setting is not as effective as hoped. Part of the problem is that secure programming is contextual in nature, germane to a language, framework, or version. For secure code training to be successful, the learning must be natural to the way developers normally do their daily jobs. The use of interactive static code assistant plugins is an alternative form of learning. An interactive static code assistant is a plugin available in the most popular Integrated Development Environments (IDE). The plugin pops up instant code warnings, detailed explanations of risk, and auto-generated code fixes, specific to the offending line of code. In a 2017 study, Tabassum, Watson, & Lipford found programmers have a heightened security awareness when using the IDE plugin while coding. The plugin provides the opportunity to write secure code as well as learn secure programming practices. Companies should avoid looking to traditional classroom-style settings as the only form of secure code learning. Instead, organizations should also consider the use of interactive static code plugins such as OWASP ASIDE/ESIDE, Puma, and SonarQube to achieve better programmer knowledge of secure coding techniques.
To learn how Tripwire’s solutions can help organizations take a smarter approach to DevOps security, click here.
Mastering Security Configuration Management
Master Security Configuration Management with Tripwire's guide on best practices. This resource explores SCM's role in modern cybersecurity, reducing the attack surface, and achieving compliance with regulations. Gain practical insights for using SCM effectively in various environments.
