Break In Case Of Consultancy: DevOps Pipelines

Break In Case Of Consultancy is a series of articles designed to serve as emergency primers on a range of subjects that you'll likely be forced to brush up on at the last second - usually before a meeting or an engagement starts.

This one particular is everything you need to know to get through an engagement on DevOps and DevOps Pipelines.

What Is DevOps And How Did We Get Here?

Ask three different people in tech what DevOps actually means in the real world, and you'll likely get three different answers.

Ask a stressed-out front-end dev and you'll hear something along the lines of:

“So what do you do?” “I’m lean,” “What?”, “I’m agile,” “What?”, “Fine. I make websites.” - @iamdevloper

Ask a technical leader or senior product manager and you'll hear something like:

“To successfully implement continuous delivery, you need to change the culture of how an entire organization views software development efforts.” – Tommy Tynjä

The closest answer that most agree on as at least being half-accurate, though is this:

“Currently, DevOps is more like a philosophical movement, not yet a precise collection of practices, descriptive or prescriptive” - Unknown

If you've never heard of DevOps before in your life, what people are referring to when they say "DevOps" is a set of practices, tools and techniques that combine both:

Software development, where code and software gets written (the Dev portion), and
IT operations, where stressed-out people strung out on coffee keep everything running (the Ops portion).

You don't install DevOps or do DevOps - think of it more as the framework under which your organization thinks about software, creates software and maintains/updates software.

Don't Go Implementing Waterfall, Stick To The Rivers And The Lakes That You're Used To

The rise of DevOps as a software development philosophy marks a big departure from the classical 'Waterfall' model that was in use for decades.

Waterfall is a method of breaking down all the activity that make up successful software development into a series of linear, sequential steps.

One step's work and output feeds directly into the next step, flowing downwards like a waterfall. Roughly the steps of the Waterfall approach are as follows:

System and software requirements: What does the client want, exactly?
Analysis: What does the thing we're building need to do, exactly?
Design: How should the thing we're building look and feel, and how will it work?
Coding: Time to build the thing!
Testing: Does the thing we're building work how it's supposed to?
Operations: The thing is built now, time to hand it over to the people who'll keep it running!

This all makes sense when you look at like this, written down on paper.

Anyone who's spent a nanosecond in software development, project management or consulting though, will happily tell you that your client will never, ever, ever have a laid-out, concrete set of requirements that won't change after.

Waterfall depends on the work in one step being finished almost completely before moving onto the next step.

When your client inevitably reads something on LinkedIn and wants to implement it 4 months in, or decides they want to use a different type of data source entirely, you have to abandon your current progress and go back up the waterfall to re-do your work.

This slows down the software development lifecycle (SDLC) considerably, and just generally gets on everyone's tits.

I Don't Have A Complete Agile Joke Yet, But Check Back After Next Sprint?

After Waterfall came Agile.

The butt of a billion jokes about Scrum Masters and Jira tickets, Agile software development is a method of software development that came to popularity in the 1990s/2000s.

Agile software development is all about breaking down large projects into a series of smaller chunks, approached in teams at great pace.

Agile refers to these bursts of heavy activity as "sprints" and any Agile project will consist of several sprints.

Each sprint will be run by an (ideally) skilled project manager who facilitates daily/near-daily status meetings and discussions called "standups" or "scrums".

The feature, target or "thing" the sprint is aimed at producing or finishing is referred to as a "deliverable", of which there will be many over an entire Agile project lifecycle.

At the end of a sprint, the project manager gets the team together to discuss what went well, what didn't and what should be learned for the next sprint in a meeting called a "retrospective".

Agile is governed by 12 major principles, of which some of the most important are shown below:

Customer satisfaction by early and continuous delivery of valuable software. (Referred to as CD)
Welcome changing requirements, even in late development. (This is the 'agile' part!)
Deliver working software frequently (weeks rather than months). (Sprints!)
Close, daily cooperation between business people and developers. (Agile standups/Scrums!)
Working software is the primary measure of progress. (Deliverables!)
Sustainable development, able to maintain a constant pace.
Simplicity — the art of maximizing the amount of work not done — is essential.(More 'agile'!)
Regularly, the team reflects on how to become more effective, and adjusts accordingly. (Retrospectives!)

Agile is all about breaking down tasks and making the SDLC more efficient and productive, and allowing teams to adapt to the inevitability of changing customer requirements.

Despite all its benefits, Agile isn't perfect - as anyone who's been stuck in their third standup of the morning in a row can tell you.

The administrative overhead of tracking sprints and running standups requires a skillset in shepherding people, resources and an ability to navigate consistent chaos.

Sometimes that skillset doesn't come as naturally to those who've spent their careers so far, you know, developing software.

Those individuals with the communications, organization and management skills to do that job often built that skillset at the expense of deep technical expertise in the work they're shepherding to completion.

This often leads to tension (and sometimes outright conflict) between project managers/Scrum Masters and the developers on their teams.

Scrum became the dominant form of Agile development in the early 2000s and is still an extremely popular form of Agile in many corporate teams.

Due to the rise of automation technology and Scrum's omission of a lot of the engineering practices that developers/engineers liked, groups of individuals who embraced automation in the development process eventually splintered off into what would become....

Modern DevOps

We now arrive at modern DevOps, which focuses on the continous integration, building, testing, delivery and deployment of software.

It doesn't much matter anymore whether this software was written under Waterfall, Agile or any other SDLC philosophy - DevOps principles can be applied

The main differences to take note of regarding DevOps as a development philosophy are:

DevOps is primarily used to build software in what we call a microservices architecture, rather than the classical monolithic architecture from decades past.
Microservices will undoubtedly get their own BICOC article soon, but for now just imagine breaking one big piece of software into its constituent parts and treating each separately.
DevOps represents a big cultural change from times before, with operations, developers, project managers and testers working a lot closer together than they ever did before.
DevOps is geared towards the use of automation wherever possible, to ensure consistency, reliability and enhanced speed of software delivery.
Entire teams of developers will work on the same codebase, sometimes even the same files with version control being massively important in tracking changes, avoiding conflicts and making sure builds and deployments occur reliably.

The DevOps Cycle & Associated Concepts

The DevOps Cycle refers to a rough set of steps that any project done under this approach will go through:

Plan: Define the business case, requirements, metrics, release plan, and timing.
Create: Design the software and code it until you have a reliable release candidate.
Verify: Testing the software to ensure it meets the agreed requirements.
Package: Transforming the software into a releaseable state and moving it from a development environment into a pre-production or 'staging' environment, whilst the software gains the required approval(s) to release into production.
Release: Scheduling, orchestrating and eventually releasing the software into the production environment, whilst planning for failure and rollback if issues occur.
Configure: Into operations DevOps now, provisioning and configuring any required storage, access, compute or networking requirements the software needs to operate effectively.
Monitor: Monitor software performance, error rate, user experience and gather feedback to feed into next Plan phase.

When we say software here, we're not talking about entire working programs.

DevOps is all about producing workable, usable features for an overall software project and continously integrating code, continously delivering working software and continuously deploying that software into production.

We call this continous process CI/CD.

Continous Integration (CI)

Continuous Integration is a pivotal part of DevOps software development, and refers to the practice of integrating all the working developer copies of various source code files into one shared mainline codebase, often multiple times a day.

Here's an issue though: if there are dozens of other developers working and submitting their own changes, will the copy of the codebase the developer downloaded even be accurate?

What if in the meantime, new code and new libraries have been added?

What if the changes one developer made conflict with the changes another developer made?

Merging and integrating code without CI can lead to a situation where merging/integrating new code takes longer than writing the new code itself - a situation called merge hell.

That's the problem CI aims to solve!

As the diagram above shows, DevOps teams can consist of dozens or hundreds of developers - each working on their own unique thing.

When developers go off to make a change to a file, they download a copy of the current codebase to work on.

They make their changes and eventually they've finished the "thing" they're working on. Now what?

The developer will likely then run a series of unit tests on their own local machine to look for major flaws and issues.

All of these tests should pass without issue before the developer attempts to commit their code to the shared mainline codebase.

Ideally multiple times a day, developers will then submit their source code in the form of a commit to a shared code repository.

A build server (normally on the same platform, but doesn't have to be) then takes the collective source code and builds a version of the application.

Integration tests, security scans and several other tests/scans then occur to ensure that the changes the developer made do not break the application and stop it from compiling.

If the build is successful and the tests all pass, then the test results and proposed code to be merged in gets sent to an approver, who then actions the integration of the new code into the mainline codebase, to be downloaded alongside all the other code when the next developer takes their copy.

This submission, testing, building and integrating should be happening all the time, with a minimum time between build and commit.

The process should be more or less continous, hence continous integration!

Note: For a breakdown of Git, repositories, version control, branching, pull requests and commits - see this Break In Case Of Consultancy article.

Examples of CI tools you might see in the wild include:

Bamboo
TravisCI
CircleCI
GitLab CI
TeamCity

Continous Delivery and Deployment (CD)

Continous Delivery (CD), you'll be happy to know, is a bit simpler a concept to grasp than a lot of the other stuff we've looked at in this article!

CD is an approach to software development that slightly deviates from the pure DevOps approach, but still aims to speed up the process through producing software in short cycles.

The idea behind CD is that software written in short cycles can be automatically moved through a "production-like" environment and go through a series of automated tests (as seen in the above diagram).

Eventually, what comes out the other end (after all the tests pass) is software that can be reliably relased into production at the press of a button.

Crucially, the final release step into production is manual in a continous delivery workflow.

This differs from continous deployment (also abbreviated to CD), where that final push into production is also automated.

Examples of CD tools you might see in the wild include:

AWS CodeDeploy
GitLab
DeployBot
Jenkins
Bamboo Deploy
Octopus Deploy

DevOps & CI/CD Pipelines

Almost all organizations that have a DevOps culture will organize their tools, services and processes into a (somewhat) clear sequence of steps, actions, tests and acceptances that take code from an idea, to a developer's laptop, all the way through into production.

We call this sequence of steps a DevOps pipeline or a CI/CD pipeline.

If you're a consultant, recruiter or salesperson working with any large company that writes their own tools or develops their own code, don't be surprised to see a DevOps or CI/CD pipeline being used.

Hell, I even use a very simple CI/CD pipeline to create/update the website you're reading right now!

I write JavaScript code and Markdown (MDX) articles on my local machine in the Visual Studio Code (VSCode) IDE.
My VSCode is integrated with my GitHub account and the repository this site's code sits in.
That GitHub repository is linked into Amazon Web Services (AWS), my cloud provider of choice.
Everytime I commit a change to GitHub from VSCode, my application gets built in AWS Amplify.
AWS Amplify launches a Docker container to hold my application and compiles the collected code.
Once done, the compiled application is pushed into "production" (my AWS environment associated with the codebloodedcyber.com domain)
Any issues are returned to me as error logs and failed builds, which I fix and then re-attempt this process.

In the corporate world, however, these DevOps or CI/CD pipelines can get significantly more complicated:

Credit to https://www.basistechnologies.com/what-is-ci-cd-for-sap/ for the example pipeline diagram

You should understand a lot more of this diagram after reading all of this, but you can see how this example corporate-grade DevOps pipeline is signficantly more complex than mine.

Overall though, even though some of the steps have different names, you can see that the fundamental DevOps cycle is being followed.

TL;DR / In Conclusion

Writing software has changed a LOT over the last decade or so - from monoliths to microservices, from Waterfall through to DevOps.

What used to take months of planning, coding, testing and scheduling to release, now happens continously.

Teams work quickly and near-constantly to update features and get them shipped out to customers in production.

There's still a process, though: just one that is now heavily reliant on automation and collaboration between the development teams writing software and those responsible for the continued operation of> that software.

Things to note as security consultants or interested parties:

If you're consulting on or auditing a DevOps team or workflow, be very considerate of time.
These people are constantly operating at speed, every walkthrough you need to "get up to speed" is another hour they're falling behind.
As always, study up on your clients beforehand as a mark of respect for their time, and to make your life easier.
Due to the workloads and deadlines these teams are often under, auditors and consultants should look for documentation and written processes, first and foremost.
After that, look at records of code respository commits, unit test results, build artefacts and merges to the master mainline codebase.
These should be coupled ideally with documented change records/requests in some sort of ticketing system like Jira or ServiceNow.
Auditors and consultants need some sort of proof that all of these steps are being followed and that artefacts are being kept.
Other areas to look at from a security perspective should be identity & access management, privileged access provisioning and vulnerability scanning/management/remediation.

If you enjoyed this article, check out the other articles in the Break In Case Of Consultancy series:

Break In Case Of Consultancy: Git, Version Control & Branching
Break In Case Of Consultancy: Microservices & Containerization
Break In Case Of Consultancy: Guide To Cybersecurity Assessments