X-ISS Removes Bottlenecks between Windows Desktops and GlusterFS Samba Servers

A provider of seismic data processing services to the oil & gas industry was experiencing slower-than-expected data transfer rates between its Windows desktop users and the GlusterFS Samba high-availability storage solution recently installed for its new Linux HPC cluster. X-ISS quickly diagnosed the cause of the bottleneck and improved data transfer speeds by more than 10X.

The seismic processing company has built its business on the quick turnaround of completed projects. In a typical job, the company receives enormous raw data files from its customer. A single project may include 100 unprocessed data files each in the 10 to 100 GB size range. Copying new data sets from the Windows desktops to the new cluster storage was sometimes taking days instead of hours.

“The slow transfer speed was unacceptable to the seismic processing firm because they often have to process the raw data, perform analysis and deliver end products within a few days,” said X-ISS president Deepak Khosla. “To meet client demands, data transfer rates had to be accelerated to minutes or hours.”

The firm had installed the GlusterFS to provide low-latency, high-throughput primary storage for the Linux cluster while allowing access to the cluster file system for Windows users. The Linux cluster routinely maintained file transfer rates of 900 Mbps, but when the Windows systems tried sending data, the transfer rate began at 25 Mbps and quickly slowed to just 2 to 5 Mbps.

X-ISS was called in to assess the problem and remove the bottleneck that was causing Windows desktop users to experience slow transfer to cluster storage. The X-ISS team first reviewed the specifications for the network hardware and configuration of the Gluster storage nodes, network switches, and Windows computers. It became readily apparent that several factors, all correctable, were contributing to the transfer slowdown. X-ISS recommended and then implemented a plan to fix the problem.

New Uplink, Reset Default Parameters

The first issue was the uplink switch between the networks. The Linux HPC cluster and the GlusterFS servers were connected to a 10 GbE Ethernet switch. The 10 GbE switch was connected via a 1 GbE link to a 1 GbE switch that drove the client’s Windows network. X-ISS upgraded the 1GbE link to 10GbE with a 10 GbE expansion module and checked the transfer speed.

The Iperf benchmarking tool measured the raw transfer speed at about 700 Mbps, but the Windows desktop transfer speed remained impaired. During the test, it copied at a peak of 25 Mbps and then fell to 5 Mbps. More tweaking had to be done.

After validating the network configurations against the desired network topology with particular attention paid to VLAN routing, the X-ISS team focused on changing default settings throughout the network which were set to handle small volumes of data, not the huge files sizes that were the norm for the client. The settings had to be changed to give priority to the network.

First, X-ISS examined how the file system was being used and modified performance settings in GlusterFS to optimize memory utilization. Default settings established for use of moderate file sizes were changed to minimize the IO wait time, a more optimal setting for Gluster usage of large sequential files.

Next, the team updated the Samba configurations for Windows 8. The default configurations were set to support the widest common denominator of Windows file transfer methods but with several performance features disabled that could be used with client’s Windows 8 operating system. Most notably, the team increased Samba buffers and enabled SMB2 protocol.

X-ISS then turned its attention to tuning the Windows TCP interface parameters. Again, the default settings there assumed the user’s network environment was likely similar to a home or small office situation where 100 Mb Ethernet networks are common. Unfortunately, that assumption didn’t hold true for the client’s 10 Gb network. The team changed the Netsh parameters for the TCP interface controlling heuristics to allow the Windows desktops to respond to the speed of the network.

With the fine tuning completed, X-ISS again ran performance tests on the system. The Iperf test showed an increase in the network to between 800 and 900 Mbps. More importantly, the Windows desktop file copy speed to the GlusterFS storage increased more than 10-fold to a rate of 150-200 Mbps. The client’s data files were soon moving between the desktops and the cluster at speeds needed to keep their customers happy with on time product delivery.

Download this case study: GlusterFS.CaseStudy10