X-ISS Manages HPC Data Center Relocation and Expansion on Tight Deadline
ION GX Technology (GXT) is a leader in providing advanced seismic data processing and visualization services to oil and gas clients through a series of worldwide geophysical centers. The computing power that drives these processing services comes from a large internal cloud-based HPC data center hub in Houston with over 300 racks containing over 7,000 nodes, networking and over 10 petabytes of storage.
When GXT contracted X-ISS to oversee a relocation and expansion of its data center, the project had to be carried out with negligible downtime in a strict six-month time window. X-ISS already had a long standing relationship with GXT, supporting their network, xCAT cluster manager, and server virtualization environment under a ManagedHPC services agreement.
“GXT relies on X-ISS for much of its HPC design and operational support work worldwide,” said Bill Menger, Director of HPC at GXT.
Between the familiarity with GXT’s environment and technical depth, X-ISS was a logical choice for this project.
“The size of the data center combined with the tight window made this project a challenge,” said Deepak Khosla, President of X-ISS. “Our extensive planning, ability to work closely with GXT and adjust work schedules to see tasks through to completion was a critical to the success of this project.”
Keeping Clusters Operating
X-ISS designed and implemented the extension of GXT’s network between the old and new data centers utilizing Arista network switches. This allowed jobs to run without performance impact regardless of whether the CPUs and storage were physically in the same building, resulting in an impressive 99.97 percent uptime during the entire six-month move.
“X-ISS was instrumental in helping plan the move, especially in the area of keeping the infrastructure and network operational while transferring services,” said Menger. “Without the X-ISS staff supporting the GXT personnel daily for the six-month operation, we would have been forced to curtail operations.”
While GXT physically moved the clusters, X-ISS validated each cluster before it was returned to production use. Cluster validation scripts, which formerly ran overnight, were improved to run in under 10 minutes, allowing clusters to begin running jobs quicker. As part of the validation, each HPC node was upgraded to the latest approved operating system image.
X-ISS moved critical infrastructure servers between data centers, replacing obsolete network load balancers, as well as the critical servers providing DNS, LDAP, and NIS services. The software appliance Zen Load Balancer (ZLB) was utilized as a match between GXT’s budget and potential future capabilities.
X-ISS designed and implemented an upgrade of the xCAT cluster management system to the latest available version. This included customization of xCAT to use a BitTorrent and Mirrorbrain based file distribution, reducing cluster reboot times by as much as three to one. Upgrade of the cluster management nodes, as well as xCAT, support GXT’s future cluster operating system migration from their tried and true CentOS 5 images to CentOS 6, and is a model for their worldwide cluster image standardization.
Just as the relocation of existing clusters was being completed at the new facility, the installation of close to 1,500 new nodes began. Within two weeks of hardware installation, X-ISS validated the first half of these nodes, working closely with the hardware integrator and the computer and network equipment manufacturers to resolve issues. All new nodes were installed and validated in time to meet the hard deadline of six months.
“The professional support and project management from X-ISS supplemented GXT’s skillsets and personnel, allowing the combined team to accomplish the move,” said Menger.
X-ISS provided training in xCAT operations and created cluster provisioning services using xCAT and its own software to keep GXT’s HPC clusters up to date with the latest operating systems.
The new data center has been crucial for the company to meet growing demand for seismic processing services, according to Chris Usher, Executive Vice President and Chief Operating Officer of ION’s GeoScience division, commenting, “The opening of our new computing hub in Houston assures we can efficiently apply the most advanced imaging solutions in realistic timeframes and satisfy increasing customer demand.”
For more information, click on the link to read what GXT wrote about the data center move. http://www.iongeo.com/News_and_Events/Press_Releases/Press_Release/?id=1878613
Download this case study: DataCenterMove.CaseStudy4