Session: 10-01 Interactive Presentations

Paper Number: 99130

99130 - Comparative Study of Single-Phase Immersion Cooled Two Socket Server in Tank and Sled Configurations 

Submerging a cluster of servers inside a large tank is the customary way of employing single-phase immersion cooling. But this approach requires a complete renovation of existing air-cooled infrastructure. A practical approach to convert an air-cooled data center to immersion cooled data center can be retaining the rack and server arrangements and supplying each server with immersion liquid in sled configuration – retaining horizontal position. The present study aims at characterizing the thermal performance of a two-socket server in sled and tank configurations using CFD. An open compute server with a two-socket motherboard is chosen. The mother board has a form factor of 6.5”x 20” enclosed in a chassis of dimensions 176 X 90 X 883 mm. The chassis is designed to be compatible with Open Rack v2 rack architecture. The 2OU server is compatible with Intel Xeon processors that enhance the performance of the CPUs by increasing their frequency. These highly efficient servers can handle high demanding applications requiring high-power computing. This server is commercially used in the air-cooled data centers. Compatibility of it for immersion cooling is studied. To validate the CAD model, the server was initially run for air cooling and the results were validated against experimental results. A commercially available dielectric immersion cooling liquid EC 110 was used to study the behavior of this server when immersed in it.

A detailed baseline geometry of the server was first simplified, considering only the components that are significant source of heat and/or impact the server flow characteristics. Some of the components considered for analysis include CPU, storage drives and memory modules. The performance of the server in tank and sled configurations is compared to determine the efficiency of both the server configurations while ensuring the components do not exceed their respective thermal threshold. Component temperatures are obtained by varying the coolant flow rates and dielectric temperatures.​​​​​ Computational fluid dynamics-based software 6SigmaET was used to analyze the fluid flow pattern, peak temperatures and fluid velocity around various server components for both, air and immersion cooling. In the tank configuration model, the server is immersed vertically with the coolant supply from bottom to top as in the case of a typical single-phase immersion deployments. Traditionally a large tank filled with several servers placed upside-down, plays host to the immersion liquid loop where the whole of the bottom end of the server chassis area acts the entry area of the fluid. In the sled configuration, the server orientation is retained (horizontally) and the fluid supply is modeled as an inlet and outlet manifold connected to the same side of the server. When modelling the server in sled configuration, mounted on a rack, and supplied with fluid from one side of the server, 10 mm diameter pipe openings are assumed for supply and return openings. The location of the supply and return openings affects the flow direction, flow mixing and ultimately the peak temperatures of the CPUs. They are optimized by taking the peak temperature of CPUs in each case as the judging parameter.

The experimental results of air cooling show the peak temperature of the CPU is in the order of 80°C. With immersion cooling, even with elevated fluid inlet temperatures, the peak temperatures stay below 70°C. Higher thermal mass of the fluid compared to air, increases the Reynold’s number of the fluid. Higher Reynold’s number translates to higher convective heat transfer coefficient enabling the fluid to pick up more heat from the sever components. The study shows that the sled design in which the server design is horizontal has the same performance as that of tank based immersion cooling solution. The analysis paves the way to design future immersion cooled servers in a form factor compatible to that of an air-cooled server. The sled-based design would require minimal infrastructure changes thus making it relatively easy to reap the benefits of immersion cooling technology.

Presenting Author: Pratik Bansode The University of Texas at Arlington