Background: A major telecommunications company in Germany, the Czech Republic, and Italy embarked on a project to maintain approximately 12,000 Red Hat Enterprise Linux (RHEL) systems. The goal was to ensure high availability, standardize configurations across different RHEL versions, and reduce maintenance costs through automation and efficient management.
Project Objectives:
- Maintain around 12,000 RHEL systems across three countries.
- Ensure high availability and redundancy using more than 10 Satellite Capsules.
- Standardize configurations across three different major RHEL versions.
- Implement inclusive and generic configuration management standards.
- Emphasize automation and fast virtual machine (VM) creation.
- Capture existing system configurations and move to a centralized configuration management system.
Tools and Technologies Used:
- Configuration Management: Puppet, Ansible
- System Management: Red Hat Satellite, Satellite Capsules
- Operating System: Red Hat Enterprise Linux (RHEL) (multiple versions)
- Virtualization: VMs from different vendors due to historical reasons, addressing multi-cloud in the form of several private VM clouds
Project Execution: The project started with one dedicated engineer from our company, who was later joined by one of our principal solution architects. The team meticulously used Red Hat Satellite and Puppet to manage the RHEL systems, ensuring high availability and redundancy through more than 10 Satellite Capsules.
One significant aspect of the project was capturing the existing system configurations into Puppet manifests, transitioning to a centralized configuration management system. This shift allowed the team to manage configurations centrally, moving away from semi-manual/semi-automated processes. The processes themselves had to be defined to match the new methodology, ensuring consistency and efficiency.
The project placed a strong emphasis on automation and fast VM creation, given the presence of different major RHEL versions and VMs from different vendors. The configuration management and related standards had to be highly inclusive and generic. The team implemented innovative solutions and automation techniques to standardize configurations and streamline maintenance processes.
During the project, the team expanded and collaborated closely with the client’s personnel. The combined efforts of the team and the client’s engineers ensured the project’s success and maintained leading roles in the production structures.
Challenges:
- Dealing with different major RHEL versions.
- Ensuring high availability and redundancy with multiple Satellite Capsules.
- Federating the load geographically across three countries.
- Managing VMs from different vendors due to historical reasons, addressing multi-cloud in the form of several private VM clouds.
- Capturing existing system configurations and transitioning to a centralized configuration management system.
Solutions and Innovations:
- Implementing inclusive and generic configuration management standards.
- Using custom scripts and automation techniques to standardize configurations.
- Ensuring redundancy and high availability through more than 10 Satellite Capsules.
- Emphasizing automation and fast VM creation.
- Defining and implementing processes to match the new centralized configuration management methodology.
Outcomes and Benefits:
- Improved system performance and reliability.
- Reduced maintenance costs through automation and efficient management.
- Streamlined maintenance processes and standardized configurations across different RHEL versions.
- Successful transition to a centralized configuration management system, enhancing overall efficiency.
- Enabled the operations team to provide a service catalogue with services that could be performed in a fraction of the time it took before the project.
Lessons Learned:
- The importance of inclusive and generic configuration management standards for handling multiple RHEL versions.
- The need for effective collaboration between teams to ensure seamless project execution.
Conclusion: The project to manage 12,000 RHEL systems across Germany, the Czech Republic, and Italy was a resounding success. Through meticulous planning, innovative solutions, and seamless collaboration, the team achieved high availability, streamlined configurations, and enhanced operational efficiency. The transition to a centralized configuration management system significantly reduced maintenance costs and enabled the operations team to deliver services in a fraction of the time. With plans for future improvements, the project sets a benchmark for managing complex IT infrastructures.