Texas Advanced Computing Center The Texas Advanced Computing Center (TACC) at the University of Texas at Austin has been operating immersion cooling technology since 2009. They started with a single-rack installation and have since expanded their deployments including in the Lonestar6 supercomputer.
Six years of continuous mission-critical operation proves single-phase immersion cooling delivers the reliability enterprise data centers demand. This isn't experimental technology, it's proven infrastructure.
TACC Immersion Reliability Data 09/2019 to 03/2024*
Quantit y per Node
Quantity per 90 Nodes
Total Failures
Annual % Failure Rate
Component
Motherboard
1
90
1
0.25%
CPU (Intel)
2
180
0
0.00%
Memory
8
720
4
0.12%
Storage
1
90
0
0.00%
Network Card
1
90
0
0.00%
GPU (Nvidia)
4
360
7
0.43%
Power supply
1
90
1
0.25%
Built in network
1
90
1
0.00%
Heat Sink
2
180
0
0.00%
LED Lights
1
90
0
0.00%
Chassis
1
45
0
0.00%
The bulk of the system (70%) is housed in four immersion cooling tanks from Green Revolution Cooling (GRC), providing greater density than could be achieved otherwise. The remainder of the system is contained in 10 air- cooled racks. Each tank contains 21 2U chassis submerged in mineral oil with heat exchangers keeping the components and oil cool. TACC's experience has shown that immersion cooling can provide improvements in power efficiency with possible benefits to failure rates of components. At present, Lonestar6 provides a small subset of nodes that have two Nvidia A-100 GPUs in them. Depending on demand and usage patterns, other GPU configurations are possible, most likely more GPU nodes with fewer GPUs per node (32 x 1 GPU instead of 16 x 2 GPU). Some experimental subsystems are also being tested with the intention of providing more variations of possible node configurations for both GPU count as well as memory per node. TACC administrators monitor usage patterns on other TACC GPU systems to determine how they will ultimately set up Lonestar6 to best facilitate the community of researchers.
* Green Revolution Cooling, 9/2019 to 3/8/2024
7
Made with FlippingBook Ebook Creator