The Jenkins pipeline is a collection of Jenkins jobs that are interlinked with one another in a sequence. A Jenkins Job is a project or unit of work where tasks are arranged and executed in sequence. In Jenkins, there are several ways of creating a job including a Freestyle job and a Pipeline Job. In this blog, we are going to discuss everything about the Jenkins pipeline, and how it automates and orchestrates the entire software delivery process.
Table of Content
What is Jenkins Pipeline?
A Jenkins pipeline is a set of plug-ins to create automated, recurring workflows that constitute CI/CD pipelines ensuring that the changes in deployment are automatically reflected whenever changes are made in source code.
Imagine a scenario where a DevOps team is working on publishing an application. This process will include three stages:
- Building Stage: In this stage, the source code is compiled, dependencies are resolved, and artifacts are generated.
- Testing Stage: The testing stage involves executing various tests like unit tests, integration tests, and possibly end-to-end tests, to check the functionality and quality of the code.
- Deployment Stage: This stage focuses on deploying the application finally, ensuring a smooth process.
If the above stages are conducted in using traditional way, it will require manual intervention again and again. This can lead to inefficiencies, inconsistencies, and delays in the SDLC.
However, with a Jenkins pipeline, each process can be automated and connected, with conditions such as the next phase building automatically if the previous job is successful. You can even access other tools like GitHub, Maven, Docker, Kubernetes, TeamCity, Selenium, and many more by simply installing plugins(extensions providing additional functionality to Jenkins) under ‘Manage Jenkins’ and defining them correctly in Jenkins Job. This provides efficiency, consistency, and faster deployment without manual intervention again and again.
What is a Jenkins Job?
A job in Jenkins is a set of tasks, typically divided into sequential steps. For example, a job has to fetch the source code from a Git repository, compile it, run it, test it for any errors, and send an email notification to the user. One Jenkins job may contain several stages and steps.
Types of Jenkins Job
There are various types of Jobs available in Jenkins like Freestyle projects, Pipeline, Multi-Configuration projects, and many more. However, let us define the most frequently used job types below:
Freestyle Project: A freestyle project in Jenkins is a GUI-based project configuration allowing simple build tasks.
Pipeline: In this method, can define all the stages through a script defining a set of instructions and conditions under each stage. To declare the python, two types of syntax are available.
- Declarative: Pipelines written using this syntax are sometimes referred to as Declarative Jenkins Pipeline.
- Scripted: Pipelines written using this syntax are sometimes referred to as Scripted Jenkins Pipeline.
Declarative Pipeline vs Scripted Pipeline Job
| Feature | Declarative Pipeline | Scripted Pipeline |
| Syntax | Structured and Simplified | Groovy-based Scripting |
| Readability | Easier to Read and Write | Requires Understanding of Groovy language, More Code-Like Syntax, Can Be Less Readable |
| Flexibility | Limited Flexibility | Higher Flexibility |
| Use Cases | Straightforward CI/CD Requirements | Complex Build and Deployment Scenarios |
| Who should use | Beginners and Experienced Users | One who is skilled in Groovy language |
| When to Use | Well-suited for Standard CI/CD Pipelines | Preferred for Complex Build and Deployment Scenarios
|
As we have seen above, groovy language makes a major difference and if you do not have expertise in it, you shouldn’t go with Scripted Pipeline as your preference.
Let’s see the syntax of the declarative and scripted pipelines for one problem statement to understand the syntax difference better:
Problem Statement: The project involves a Java application built using Maven, and the team wants a reliable and efficient pipeline to streamline these workflows.
Solution when Declarative Pipeline used:
pipeline {
agent any
stages {
stage('Build') {
steps {
checkout scm
script {
sh 'mvn clean package'
}
}
}
stage('Test') {
steps {
script {
sh 'mvn test'
}
}
}
stage('Deploy') {
steps {
script {
echo 'Deployment steps go here'
}
}
}
}
post {
success {
echo 'Build, tests, and deployment succeeded!'
}
failure {
echo 'Build or tests failed. Check logs for details.'
}
}
}
Solution when Scripted Pipeline used:
node {
stage('Build') {
checkout scm
sh 'mvn clean package'
}
stage('Test') {
sh 'mvn test'
}
stage('Deploy') {
echo 'Deployment steps go here'
}
post {
success {
echo 'Build, tests, and deployment succeeded!'
}
failure {
echo 'Build or tests failed. Check logs for details.'
}
}
}
Here you must have noticed that in a declarative pipeline, several stage phases have been defined within a ‘stages’ parameter but in a scripted pipeline, several stage phases are explicitly defined. There will be many more differences when you will work on different problem statements. Now, in both the examples above you must have noticed a few pre-defined terms like agent, stages, steps, and nodes, being used again and again. So, let’s understand several components used in the Jenkins Pipeline script in our next section.
Understanding Jenkins Pipeline Syntax Fundamentals
Let’s discuss some of the most frequently used terminologies in Jenkins Pipeline scripts:
Agent:
Agents refer to the execution environment for the Jenkins pipeline.
pipeline {
agent any
// or
agent {
label 'docker'
}
}
Nodes:
Machines in a Jenkins environment where tasks can be allocated. Usually, all the machines apart from Built-In Node are called Slaves or Worker Node.
node {
// Node-specific tasks go here
}
Stages:
Various phases in the pipeline where steps are defined to conduct different tasks of the software delivery process.
stages {
stage('Build') {
// Build tasks go here
}
stage('Test') {
// Test tasks go here
}
}
Steps:
Instructions or commands within a stage, define specific tasks.
steps {
script {
sh 'mvn clean package'
}
}
sh:
The step used to execute shell commands within the pipeline script.
steps {
script {
sh 'npm install'
}
}
Environment:
Term to define environment variables available to all steps within a specific block.
stages {
stage('Build') {
environment {
PATH = "/usr/local/bin:${env.PATH}"
}
steps {
// Build tasks go here
}
}
}
Labels:
Term to group nodes with common characteristics which helps to restrict the execution of different tasks to specific nodes.
agent {
label 'docker'
}
Plugins for Jenkins Pipeline:
Jenkins provides a wide array of plugins that can be seamlessly integrated into Jenkins Pipelines.
- Pipeline Plugin: The pipeline plugin itself enables the creation and execution of Jenkins Pipelines.
- Docker Plugin: It allows integration of Docker within Jenkins Pipelines allowing building and deployment of containerized applications.
- JUnit Plugin: Integrates JUnit test reports into Jenkins, providing visibility into test results within the pipeline.
- Maven Plugin: Facilitates the integration of Apache Maven, allowing Jenkins to build and manage Java projects.
- Node.js Plugin: Supports the building and testing of Node.js applications within Jenkins Pipelines.
- Artifactory Plugin: The JFrog plugin provides capabilities for artifact management and repository storage.
- Amazon EC2 Plugin: It allows Jenkins to allocate Amazon EC2 instances for scalable and flexible build environments.
- Email Extension Plugin: It allows Jenkins Pipelines to send detailed email notifications on build results.
- Blue Ocean Plugin: This plugin enhances the Jenkins user interface with a more intuitive dashboard for visualizing and managing Pipelines.
- Kubernetes Continuous Deploy Plugin: It facilitates continuous deployment to Kubernetes clusters.
- GitHub Integration Plugin: This enables automatic triggering of builds based on GitHub events.
- SonarQube Scanner for Jenkins: SonarQube can be integrated for static code analysis, helping maintain code quality in Jenkins Pipelines.
How to Setup Jenkins Pipeline?
To create a Jenkins Pipeline, you need to run at least two different jobs. To create a Jenkins job, you need to have Jenkins installed on your system. Please refer to the official documentation of Jenkins to install it according to your requirements. Once you have installed Jenkins, you need to add a slave if you do not wish to run deployments on the Built-In Node(Master).
Follow the next steps after accessing the Jenkins dashboard and adding nodes under ‘Manage Jenkins’.
Step 1: The first task to run a Jenkins pipeline is creating a Jenkins Job. A Jenkins Job is a project or unit of work where tasks are orchestrated and executed.
Step 2: Click on ‘+ New Item’ under Dashboard.
Step 3: Next, you have to select a Job type. If you are willing to run tasks using GUI, select ‘Freestyle Jobs’, else select ‘Pipeline Jobs’ for using declarative or scripted pipeline methods.
We have selected Pipeline in the above window. Now, you will be asked to provide configuration for the Job where you will find several checkboxes, options, and entry fields as shown below.
Each option provides you with more granular control over your job, including integration, triggers, source code, and task management. Scroll the page and you will get the workspace to write your declarative or scripted code.
For practice, Jenkins provides you sample scripts among which you can select any one of them and save the script by just clicking on Apply and Save. We have selected the “Hello World” in our case.
Once you click on Save, you will be redirected to a new screen where you need to click ‘Build Now’ to run the Job. Your Jenkins pipeline will run and the output will be displayed under ‘Stage View’.
Once your first job is ready, you can create a second job in the same way and define that the second job should run automatically, once the first job is successful. This action is called a POST-BUILD action and to perform this, you need to add the code mentioned below at the end of the first job.
post {
success {
// Trigger the second job only if the first job succeeds
build job: 'Name_of_Your_Second_Job', wait: false
}
}
In this way, we set up our Jenkins pipeline by linking different jobs.
To achieve a CI/CD Jenkins Pipelines, it becomes mandatory for you to integrate it with various tools and services. For this, you need to integrate the tools while defining a job with the correct syntax mentioned below for each tool:
Version Control Systems (VCS): Integration with Version Control System helps you to continuously integrate the source code. Git is one of the Version Control tools. Script to Integrate Git is given below:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
// Checkout code from Git
git 'https://github.com/username/repo.git'
}
}
// Other stages go here
}
}
Build Tools like Maven, Gradle, or NPM: These tools help you continuously build your project with the latest changes. Script to integrate Maven is given below:
stages {
stage('Build') {
steps {
// Build using Maven
sh 'mvn clean install'
}
}
// Other stages for testing, deploying, etc.
}
Artifact Repositories: Artifacts are generated during the build process which can be stored in artifact repositories like Nexus or Artifactory. Jenkins can publish and retrieve artifacts from these repositories which saves time during the next build. Script to integrate Nexus is given below:
stages {
stage('Publish Artifacts') {
steps {
// Publish artifacts to Nexus repository
nexusArtifactUploader artifacts: [['target/*.jar']], credentialsId: 'nexus-credentials', groupId: 'com.example', nexusUrl: 'http://nexus-server/repository/maven-releases', nexusVersion: 'nexus3'
}
}
// Other stages for testing, deploying, etc.
}
Testing Tools: Jenkins can execute test scripts and generate reports. For example, integrate with JUnit, TestNG, or other testing frameworks.
stages {
stage('Run Tests') {
steps {
// Run tests using a testing framework
sh 'npm test'
}
}
// Other stages for deploying, etc.
}
Deployment Tools: Development tools are available in Jenkins to support continuous integration and continuous delivery (CI/CD) processes. Jenkins can be integrated with Docker, Kubernetes, and Ansible.
stages {
stage('Deploy') {
steps {
// Use Ansible for deployment
sh 'ansible-playbook deploy.yml'
}
}
}
Project Ideas for Implementing Jenkins Pipeline
Here are several project ideas that you can try implementing using Jenkins Pipeline:
Web Application Deployment:
Create a Jenkins Pipeline to automate the build, test, and deployment of a web application.
Microservices CI/CD:
Implement a Jenkins Pipeline for a microservices architecture. Each microservice should have its pipeline, with stages for building, testing, and deploying independently.
Infrastructure as Code (IaC)
Explore Jenkins Pipeline for managing infrastructure deployments using tools like Terraform or Ansible. Automate the provisioning and configuration of cloud resources or on-premises infrastructure.
Containerized Applications:
Build a Jenkins Pipeline for containerized applications using Docker or Kubernetes including stages for building container images, running containerized tests, and deploying to a container orchestration platform.
Security Scanning Pipeline:
Build Jenkins Pipeline to integrate tools like SonarQube, or Snyk to analyze code for security vulnerabilities.
Multi-Branch Pipelines:
It enables automated CI/CD workflows by automatically triggering builds based on changes in different branches of version control repositories.
Create a Jenkins Pipeline for performance testing by integrating tools like Apache JMeter or Gatling that will automate load as part of your continuous integration process.
Serverless Application Deployment:
Include stages for packaging serverless functions, running tests, and deploying to serverless platforms like AWS Lambda or Azure Functions.
Conclusion
We have covered every minute detail required for understanding and practicing Jenkins Pipeline. Jenkins pipelines are a set of instructions written as a script that allows you to perform several tasks required for software delivery with continuous integration and continuous deployment, saving a lot of time and human effort.