Jenkins Master-Slave Architecture | automateNow (2024)

How to use Jenkins master-slave architecture for distributed builds

One of the key features that makes Jenkins a robust CI/CD solution is its Master-Slave architecture, allowing for the distribution of build and deployment tasks across multiple machines. This architecture is particularly beneficial in large-scale projects where resource optimization, parallelism, and scalability are crucial. This article will delve into the complexity of Jenkins Master-Slave architecture and explore how it facilitates distributed builds.

Table of contents

• Understanding Jenkins Master-Slave architecture
  • Master Node
  • Slave Node
• How Jenkins master and slave architecture works
• Advantages of Jenkins Master-Slave architecture
• How to set up Jenkins Master-Slave architecture
  • Install Jenkins master
  • Create Jenkins master node
  • Configure Jenkins master node
  • Jenkins master node launch and save
  • Launch Jenkins slave
  • Verify connection
• Best practices and considerations
• Conclusion

Understanding Jenkins Master-Slave architecture

Master-slave architecture is a hierarchical computing model featuring a central component (master) that allocates resources and assigns tasks to other interconnected components (slaves). This is a naturally efficient model because it promotes workload distribution and optimizes resource utilization.

Master Node

At the core of Jenkins Master-Slave architecture is the master node, often regarded as the brain of the CI/CD operation. The master node’s primary responsibility is to manage and coordinate tasks within the Jenkins environment, ensuring a smooth and efficient workflow.

Acting as the central hub, the master node oversees the entire CI/CD process, orchestrating the scheduling of builds and monitoring the health of the build environment.

The master node is crucial in the Jenkins ecosystem. It acts as a gateway to the user-friendly Jenkins web interface, through which users can configure jobs, get insights into ongoing builds in real-time, and interact with the system. It also enables collaboration and transparency throughout the software development lifecycle.

Slave Node

In contrast to the master node, the slave nodes, also called agents or worker nodes, embody the hands-on execution aspect of the Jenkins setup. These nodes are the unsung heroes responsible for the build and deployment tasks. By distributing these tasks across different physical machines or virtual environments, slave nodes enable parallel execution. This key feature significantly reduces build times and enhances overall system performance.

Slave nodes are adaptable to diverse environments and can be set up on different machines to optimize resources. They promote parallelism and enhance fault tolerance, and Jenkins can redirect tasks to other available nodes to ensure the resilience of the CI/CD system.

How does Jenkins master-slave architecture work?

Developer commits changes

The developer initiates the software development cycle by committing changes to the source code repository.

Jenkins Master checks repository

Jenkins Master is the central hub that continuously monitors the repository for new commits and automatically detects changes to initiate the Continuous Integration (CI) pipeline.

Slave compiles code

Jenkins Slave executes tasks such as fetching the latest code for compilation and testing. Once the code is fetched, it is compiled into an executable file, and the build process is initiated. If any issues arise during the build process, developers receive immediate feedback.

Deployment on test server

Applications that are successfully built are deployed on a test server, part of Jenkins Master. The deployment process is meticulously managed to ensure the applications function correctly in a controlled testing environment. Developers receive timely alerts if any complications arise during the deployment to the test server.

Deployment on production server

The Jenkins Master tool automates the deployment process to the production server, ensuring a smooth transition from development to production. Assuming no errors and all testing is successful, the application will be deployed to the production server, making it available for end-users.

Development server deployment

In addition to deployment to production, Jenkins can deploy versions of the application to a development server. This enables developers to perform additional testing and collaborate on features in a controlled environment.

Master-slave architecture for scalability

Jenkins Master-Slave architecture allows multiple builds to be executed simultaneously. The Master node distributes the workload to various Slave nodes, each operating independently and executing different builds concurrently, optimizing resource utilization and accelerating the build process.

Advantages of Jenkins Master-Slave architecture

Parallel execution

Jenkins’ Master-Slave architecture is designed to execute multiple jobs simultaneously across different slave nodes, making it an efficient software development system. This parallel execution enhances the speed of the development process and enables tasks that would typically be sequential to unfold concurrently, resulting in a quicker delivery of software updates.

The parallelism of Jenkins’ architecture is not just about speed; it reflects efficiency and responsiveness. Teams can achieve a level of agility essential in today’s fast-paced software development landscape by utilizing the power of multiple nodes.

Scalability

The Jenkins Master-Slave architecture is highly scalable, making it easy for development teams to adapt quickly to changing project workloads. This architecture allows for seamless adjustments, whether scaling up for a significant release or scaling down during maintenance periods. It ensures that the CI/CD pipeline remains responsive to the evolving demands of software development.

The scalability feature handles increased workloads and promotes efficient resource allocation. Teams can strategically distribute tasks across various slave nodes to prevent bottlenecks and optimize available resources. This ultimately contributes to a more agile and responsive development environment.

Resource optimization

Resource optimization in a Jenkins Master-Slave architecture is more than just distributing builds. It involves customizing the hardware specifications of each node to match the unique requirements of specific tasks. This level of control allows development teams to allocate resources more effectively by aligning computational power, memory, and other specifications with the demands of diverse jobs.

As a result, resources are utilized more efficiently, preventing over-provisioning and minimizing waste. Moreover, by configuring nodes with different specifications, the architecture can accommodate a broader range of workloads.

Resource optimization becomes a strategic advantage by promoting cost-effectiveness and ensuring that computational resources are precisely aligned with the needs of each stage in the development pipeline.

Isolation of environments

The concept of isolation in Jenkins Master-Slave architecture goes beyond preventing interference between jobs. It introduces a robust framework for creating and maintaining distinct environments for each task. Each slave node operates independently, encapsulating the job-specific dependencies, configurations, and runtime environments.

This isolation ensures that changes in one job do not inadvertently impact others, fostering reliability and reproducibility in builds. Developers can experiment with configurations and dependencies without worrying about unintended consequences, leading to more resilient and predictable CI/CD pipelines.

Fault tolerance

Fault tolerance is not just a safety net in Jenkins Master-Slave architecture; it’s a design principle. In the face of a failure on a slave node, Jenkins responds intelligently by redirecting tasks to other available nodes. This proactive approach to fault tolerance enhances the overall reliability of the CI/CD pipeline.

The architecture’s ability to gracefully handle node failures ensures that the development process remains robust and resilient despite unexpected challenges. This feature becomes increasingly crucial in mission-critical projects where uptime and reliability are non-negotiable.

How to set up Jenkins Master-Slave architecture

Setting up a Jenkins Master-Slave architecture involves configuring a Jenkins master server and one or more Jenkins slave nodes to distribute the workload.

Below is a step-by-step guide to help you set up Jenkins Master-Slave architecture on any machine:

Install Jenkins master

Please navigate to the official download page of Jenkins and refer to our installation guide to complete your Jenkins installation smoothly.

After installation, Jenkins will start automatically. Access it through your web browser athttp://localhost:8080.

Create Jenkins master node

In Jenkins Master, from the Dashboard, selectManage Jenkins.

Then, select Nodes under System Configuration and click onNew Node.

Enter a unique name for the node and choose thePermanent Agentoption. When you select thePermanent Agentoption in Jenkins, you are adding a long-lived worker node to Jenkins.

This type of agent is termedpermanentbecause Jenkins needs to provide a higher level of integration with these agents, such as dynamic provisioning.

Finally, clickCreateto proceed.

Configure Jenkins master node

After creating the node, you’ll be presented with a configuration page.

Set theNumber of executors, this determines how many parallel jobs the slave can handle. You can adjust this based on your machine’s capabilities.

Specify theRemote root directory– TheRemote Root Directoryis the location on the slave machine where Jenkins will create a workspace directory for performing its builds. It is recommended to provide an absolute path to the directory. For example, if your slave machine is running Windows, you might specify something like/Jenkins/workspace. And you might use/home/jenkins/workspaceif it’s a Unix-like system.

Set Labels:

Labels are essential; they allow you to tag Jenkins nodes to help categorize and group nodes based on certain characteristics or capabilities. When you configure a Jenkins job, you can specify the label(s) of the nodes on which the job can run.

Jenkins master node launch and save

Under theLaunch methodsection, chooseLaunch agent by connecting it to the controllerif you want to run the command on the slave system to connect to the master system. This method involves starting the agent manually using a Java Web Start command.

Then, click on theUse WebSocket.

Finally, scroll down and clickthe Savebutton to store the configuration.

Then, you will be redirected to the Nodes page, and our newly configured master node will be in the table. But it’s showing as offline or not activated with a cross sign.

Then, the next step is to download the agent.jar file through this URL –http://JENKINS_URL/jnlpJars/agent.jar. We are going to need this file for the slave node setup. Kindly use your respective Jenkins URL here instead of JENKINS_URL.

Finally, click on that not-activated master node. Then, you will be redirected to a new page. Inside that page, based on your OS (Windows or Unix ), copy the command code underRun from agent command line.

Launch Jenkins slave

Connect to slave machine

On the machine intended to be the Jenkins slave, ensure both Java and Jenkins are installed by following the installation steps specific to that machine.

Now, copy theagent.jarfile we downloaded from the Master system and paste it into the Jenkins folder of the slave.

Open a command prompt. Navigate to the directory where Jenkins is installed on the slave machine. This is typically the location where you have extracted Jenkins files.

cd path\to\jenkins

Replacepath\to\jenkinswith the path on your slave machine where Jenkins is installed.

Launch slave agent

Run the command provided by Jenkins Master, which we had copied before under Run from agent command line to launch the slave agent.

This command typically looks like this:

curl.exe -sO http://localhost:8080/jnlpJars/agent.jar & java -jar agent.jar -jnlpUrl http://localhost:8080/computer/My%20master%20node/jenkins-agent.jnlp -secret 6f173142afd4db6fcca5bf124d56e746f29589054f764f294a68b2a1baf0ddda -workDir "/Jenkins/workspace"Code language: JavaScript (javascript)

Once the command initiates successfully, it will show asConnected. That means the Jenkins agent on the slave machine is now connected to the Jenkins master.

Here, we have taken the Windows systems for both the Master system as well as the Slave system. The command editor may vary based on your respective OS.

Verify connection

Go back to the Jenkins master interface. Navigate toManage Jenkins>Nodes.

Check that the Master node is now appearing without any crossmark. That means it’s active now. And if you click on it, you can see the text asAgent is connected.

Your Jenkins Master-Slave setup is complete, and the master can distribute tasks to the slave node. You can now create and run jobs on Jenkins, with the workload distributed between the master and the configured slave.

Best practices and considerations

Node labeling

Use descriptive labels for nodes to ensure jobs are executed on nodes with the required capabilities. This practice enhances job assignment precision and allows for efficient resource utilization. For example, label nodes with attributes like Linux, Windows, or specific software versions to match job requirements accurately.

Resource management

Regularly monitor and adjust resource allocations for both Master and Slave nodes to prevent resource contention. Conduct periodic assessments of resource usage trends and adjust resource allocation based on project demands. This proactive approach ensures that each node operates optimally, preventing performance degradation and bottlenecks.

Security

Implement firewall rules, encryption, and proper authentication mechanisms to secure communication between the Master and Slave nodes. Utilize secure communication protocols, such as HTTPS for the Jenkins web interface and SSH for node communication. Configure firewall rules to permit only essential traffic, minimizing security risks and safeguarding sensitive information exchanged between nodes.

Plugin compatibility

Keep Jenkins and its plugins up to date to ensure compatibility and access to the latest features and security patches. Regularly check for updates to Jenkins core and installed plugins.

Adopt a systematic approach to plugin updates, testing them in a staging environment before applying them to the production environment. This proactive maintenance strategy reduces the risk of compatibility issues and ensures a stable Jenkins environment.

Monitoring and logging

Implement robust monitoring and logging practices to track system health, job statuses, and resource usage. Integrate Jenkins with monitoring tools to capture metrics related to build times, node performance, and job success rates.

Maintain detailed logs for each build job, aiding in post-mortem analysis and issue resolution. Monitoring and logging contribute to proactively identifying and addressing potential performance bottlenecks or failures.

Final thoughts on Jenkins master-slave architecture

The Jenkins Master-Slave architecture is a powerful solution for managing complex build and deployment tasks in a distributed environment. This architecture intelligently distributes workloads across multiple nodes, enabling organizations to achieve faster build times, increased scalability, and improved resource utilization.

However, setting up and managing this architecture can be challenging, so it is critical to have a good understanding of it to unlock the full potential of Jenkins for continuous integration and delivery. Happy testing!

Related articles

• Jenkins pipeline creation guide
• Declarative vs scripted pipelines in Jenkins
• Jenkins jobs detailed in detail
• Jenkins Monitoring
• How to install Jenkins

What is Jenkins master-slave architecture?