Designing transformer oil immersion cooling servers for machine learning and first principle calculations.

A transfomer oil immersion cooling server is designed and constructed for machine learning applications and first principle calculations that are carried out for materials-related research. CPU, motherboard, random access memory, hard disk drive, solid state drive, graphic card, and the power supply unit are submerged into the transformer oil in order to cool the entire system. Benchmark tests reveal that overall performance is improved while performance times for multicore calculations are dramatically improved. Furthermore, calculation times for machine learning with large data sets and density functional theory calculations are shortened during single core calculations. Thus, a transformer oil immersion cooling server is proposed to be an alternative cooling system used for improving the performance of first principle calculations and machine learning.

Introduction

Computer servers have become increasingly vital within materials science due to the rapid growth and increasing applications of machine learning and first principles calculations towards various research endeavors [1-5]. In particular, first principles calculations require large amounts of parallel computing where multiple central processing units (CPU) are engaged simultaneously while machine learning requires large amounts of input and output from both the CPU and random access memory (RAM) [6-11]. Consequently, these processes generate large amounts of heat within parts such as the CPU, motherboard, random access memory (RAM), hard disk drive (HDD), solid state drive (SDD), graphic card, and even the power supply units. Typically, cooling systems are designed for the CPU as they become the hottest part of the server. Air and liquid cooling are commonly used in cooling systems for removing heat from the CPU. Heat sinks are directly connected to the CPU in order to transfer the heat from the CPU to heat sinks where the heat sinks are then cooled via air flow generated by running fan in the air cooling system. For liquid, a water block is mounted against the CPU where water is circulated by a pump and the water is then cooled by a radiator, thereby transferring the heat generated by the CPU to the water. Although the heat of the CPU is efficiently cooled, other parts like the motherboard, random access memory, and power supply unit are commonly cooled by fans which are still limited to cooling those specific parts. Thus, it would be ideal to implement a cooling systems for the entire server.

Liquid immersion cooling is a progressive way to cool computer servers where the entire motherboard, random access memory, and CPU are commonly submerged into a dielectric liquid. Commonly, fluorocarbon and hydrocarbon based liquids are used. However, durability of the computer parts against the oil as well as the environmental effects of fluorocarbon are of concern, especially as fluorocarbon is considered to have a long atmospheric lifetime. Here, transformer oil is proposed as the base of a liquid immersion cooling system for computer servers as transformer oil has excellent electrical insulating and cooling properties.

Method

Benchmark methods

Benchmark tests are performed before and after submerging the computer server into the transformer oil immersion cooling system. FLIR ONE Pro, an infrared camera, is used to measure the temperature of the submerged oil cooling server. Geekbench 5 is used to test both single and multicore performances [12]. In particular, the following performances are tested: AES-XTS, text compression, image compression, navigation, HTML5, SQLite, PDF rendering, text rendering, clang, camera, N-body physics, rigid body physics, Gaussian blur, face detection, horizontal detection, image inpainting, HDS, ray tracing, structure from motion, speech recognition, and machine learning. Scores provided by Geekbench 5 are calculated based on a baseline score of 1000 which is the score of an Intel Core i3-8100. These performances are tested for both single and multicore cases before and after being submerged into the transformer oil.

Benchmark tests for first principles calculations are performed using grid based projector augmented wave (GPAW) method [9]. Relaxation of H2 molecules are calculated using linear combination of atomic orbitals within GPAW method. Exchange correlation of Perdew–Burke–Ernzerhof (PBE) with spin polarization is applied for all calculations [13].

Two types of supervised machine learning are implemented within sciki-learn [10]. Random forest regression (RF) and support vector regression (SVR) are used in order to test machine learning performance before and after submerging the computer server. The number of trees is set to 100 in RF while C and gamma are set to 100 and 0.001 in SVR, respectively. High throughput experimental oxidative coupling method catalysts data consisting of 27,622 data points is used as training data. 41 descriptor variables consist of 37 types of catalysts information created using one hot encoding and 4 types of experimental conditions (temperature, CH4 flow, O2 flow, and Ar flow) are used while C2 yield is set as an objective variable [14]. Cross validation is performed where data set is randomly divided into 80% train data and 20% test data. Calculation time is measured for cross validation of 10 random sets of train and test data.

Building the computer

A computer server is constructed based on the parts listed in Table 1. AMD Ryzen 5 3400G is chosen as the CPU and consists of 4 cores / 8 threads. The CPU, one terabyte-sized SSD, RAM totaling 32 gigabytes, and 4GB of graphic cards are connected to the motherboard. It is important to mention that none of the cooling devices such as fan and water block are attached to the CPU, leaving CPU in an exposed state. In addition, category 6a lan cable and HDMI cables are connected to the motherboard for wired network and outputing to the display, respectively. The constructed computer server is shown in Fig 1(a). Note that the server is not treated with a protective coating before being submerged in the oil. Before submersion, benchmark tests are run where benchmark tests using Geekbench 5, first principle calculations, and machine learning are performed without the presence of a cooling system.

Table 1

Specification details of parts used for submerging.

CPU: central processing unit, SSD: solid state drive, RAM: random access memory, GPU: graphics processing unit.

	Specification
CPU	AMD Ryzen 5 3400G
Motherboard	ASROCK B450
Power Unit	ANTEC 650W
SSD	Crucial SSD 1TB
RAM	CORSAIR DDR4-3200 16Gx2
GPU	ASUSGeForce GTX 1650 OC Edition 4GB

Fig 1

The constructed computer server (a) without the presence of the transformer oil and (b) fully submerged in the transformer oil. (c) Thermal imaging of the submerged computer server under operation.

The constructed computer server is then submerged into the transformer oil. Transformer oil “high pressure insulating oil A”, produced by ENEOS Corporation, is used [15]. Further specifications regarding the transformer oil are collected in Table 2. The chosen transformer oil has low viscosity while possessing high insulating and high antioxidative properties, which can benefit the server. In particular, its low viscosity allows for a high cooling rate while its insulating properties help protect operations by insulating electricity between the server and oil and its antioxidative properties give the oil a longer lifetime. Here, 80 liters of the transformer oil are poured into the computer server as shown in Fig 1(b). Note that all of the server parts– including the motherboard, CPU, RAM, SSD, power supply unit, grpahic card, lan cable, and HDMI cable– are submerged in order to cool the entire server. Both of power supply unit and graphic card have fans. Those fans create the circulation flow within the transformer oil, thus, heat is continuously removed.

Table 2

The details of the transformer oil [15].

	Specification
Density	0.87kg/l
kinematic viscosity(40°C)	8.09 mm2/s
kinematic viscosity(100°C)	2.21 mm2/s
Flash point	150°C
Acid Value	0.00mgKOH/g
Corrosive Sulfur	Non-corrosive
Electrical breakdown	70 kV
Dissipation factor	10%
Volume Resistivity(80°C)	45TΩ m
Benzotriazole	10/kG

Once submerged, the server is operated under the transformer oil. An infrared camera is used to measure the temperature of the transformer oil while under operation as shown in Fig 1(c). Fig 1(c) indicates that the oil temperature is measured to be 28.6°C. More importantly, not only is the CPU cooled by the oil but all other parts of the server are also simultaneously cooled by the transformer oil, thus, the oil acts as a heat sink. The rise in temperature may affect the viscosity of the oil in a manner that can be seen as beneficial for the server. In particular, the kinematic viscosity of the transformer oil is reported to be 8.09 mm2/s at 40C and decreases to 2.21 mm2/s when at 100C as shown in Table 2. This demonstrates that by increasing temperature, the kinematic viscosity decreases to 1/4 of its original viscosity with an increase of 60C. The server is potentially benefitting this as low viscosity allows for fluid to circulate much easier when compared to high viscosity. As the server runs and generates heat, the viscosity of the oil decreases and improves fluid circulation, which can thus help remove the heat from the server and help keep the server cool. These results thereby demonstrate that the constructed computer is operational while immersed within the transformer oil.

Benchmark

Benchmark tests of the constructed computer server are performed before and after being submerged into the transformer oil. Single core performance is first evaluated across 22 different categories as shown in Fig 2(a) which compared scores from before and after being submerged into transformer oil.

Fig 2

Benchmark tests by Geekbench 5 with and without the transformer oil (a) single core and (b) multi core test results by Geekbench 5, and (c) density functional theory (DFT), random forest (RF), and support vector machine (SVR) single core calculations where computational time is evaluated with and without the presence of transformer oil.

In general, one can see from Fig 2(a) that single core performance slightly increased for most test categories when the server is submerged in the transformer oil. When comparing the benchmark results of the multi-cores, however, performance differences become much more obvious. As seen in Fig 2(b), multi-core performance is dramatically increased once the server is submerged. Considering that multi-core calculations can generate more heat than single cores, one can believe that the transformer oil can contribute towards lowering the sever temperature, thereby resulting in better performance. In the same fashion, density functional theory (DFT) calculations, random forest (RF), and support vector regression (SVR) are evaluated using a single core and shown in Fig 2(c). Relaxation of a H2 molecule is performed where it takes 29.8 seconds to complete the relaxation calculation without the transformer oil while it takes 26.1 seconds to complete the relaxation calculation with the presence of transformer oil. Here, one can see that even with single core calculations, submerging the computer server into the transformer oil has shortened the time taken for density functional theory calculations by 3.7 seconds.

From here, performance of data science techniques are evaluated. 27,622 data points consisting of 41 descriptor variables and 1 objective variable are trained using RF and SVR. For the case of RF, conducting cross-validation with the exposed computer server takes 77.9 seconds to complete while the same cross-validation takes 67.0 second when the server is submerged. Submerging the server into the oil thus reduces completion time by 10.9 seconds. This effect becomes more pronounced in the case of SVR, which requires larger computational times than RF. For the case of SVR, the submitted job is completed within 691.3 seconds with the exposed computer server while the submerged server completes the submitted job within 336.3 seconds. By submerging the server into the transformer oil, it becomes very clear that the time required to complete said task is reduced by 336.0 seconds. This result suggests that within the same timeframe, the submerged server can potentially complete two jobs by the time the exposed server finishes its first job. The drastic decrease in calculation time can be attributed to the transformer oil, which is able to cool the large amount of heat generated by the single-core heavy calculation. With these results, one can conclude that the transformer oil can be an effective cooling system for single-core heavy calculations as well as multi-core calculations that are carried out during first princicples calculations and machine learning calculations.

Conclusion

A transformer oil immersion cooling server is proposed for first principles calculations and machine learning. A transformer oil with high insulative and high antioxidative properties is chosen where all server parts- including CPU, RAM, motherboard, power unit, graphic card, and all related cabled- are submerged into the transformer oil. The submerged server is successfully operational upon being submerged where the temperature of the oil during server operation remains at 28.6°C. It must be noted that server lifetime within the transformer oil must still be evaluated as there is possibility of the oil corroding the cables and plastics within the server. One can consider that one of the drawbacks of the transformer oil could be oxidation. As time passes, oxidation can potentially affect the durability of the oil. Further study is required to better understand the effects oxidation can have upon performance and durability of the oil over a longer period of time Benchmark tests demonstrate that single-core performance is moderately improved and multi-core performance is greatly improved when submerged within the transformer oil. Additionally, calculation times for first principles calculations and machine learning applications are shortened by the transformer oil, where this effect is magnified for the case of heavy calculations presented by SVR. Thus, a transformer oil immersion cooling server is designed and demonstrated to be an effective cooling option for servers that conduct first principles calculations and machine learning calculations.

20 Jan 2022

Submitted filename: Comment.pdf

Click here for additional data file.

9 Mar 2022

Dear Editors of PLOS ONE,

I have carried out revisions throughout the manuscript on the basis of the reviewers' comments. Comments to the reviewers are colored in red and mentioned after =>.

If there are any questions or concerns, please do not hesitate to contact me. I'm more than happy to revise the manuscript.

Best regards,

Keisuke Takahashi

Ph.D. Keisuke Takahashi

Department of Chemistry,

Hokkaido University,

North 10, West 8, Sapporo 060-8510, Japan

Here is the answers to reviewers comment.

Editorial Edit:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

=> Fixed

2- Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

=> Fixed

3-Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

=> Contribution is now witten

4- Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

=> Confirmed

Reviewer 1:

Computational servers are important in almost all fields of science and technology. These servers generate large amount of heat. Efficient cooling is required to keep the servers functional for a longer period of time. The performance of heated servers usually goes down. In this paper, authors have put their efforts in solving the issue of heated servers and have tried to use transformer oil as a heat sink. Authors have reported that speed of first principle calculations and machine learning applications are better in servers which are immersed in transformer oils. This is really a nice piece of work however authors are requested to clarify few doubts:

• Authors report that the server is functional in the oil. Did they use any type of coating on the server to protect the machine?

=> We haven't applied any coating. We added the following sentence.

“Note that the server is not treated with a protective coating before being submerged in the oil.”

• Authors see an improvement in server’s performance after keeping it inside the transformer oil. Authors should explain the reason behind this finding.

=> We believe that the transformer oil act as a heat sink.We added the following sentence.

“More importantly, not only is the CPU cooled by the oil but all other parts of the server are also simultaneously cooled by the transformer oil, thus, the oil acts as a heat sink.”

• Authors get better results in multi-core machines. The reason behind this observation is missing. The multi-core process generates more heat than the single core process. From this, we can consider the effect of cooling to be more crucial. We have added the following to the manuscript.

" Considering that multi-core calculations can generate more heat than single cores, one can believe that the transformer oil can contribute towards lowering the sever temperature, thereby resulting in better performance.."

• It is also interesting to know if the same oil can be recycled again and for how many times.

=>One of the drawbacks of the transformer oil is oxidation. As time passes, oxidation can potentially affect the durability of the oil. We have added the following sentence to the manuscript. "It must be noted that server lifetime within the transformer oil must still be evaluated as there is possibility of the oil corroding the cables and plastics within the server."

********************************************************************************

Reviewer 2:

Abstract: CPU, motherboard, ...... are submerged into the transformer oil.

Authors intent here is not clear. Need to reconstructed.

=> We meant that we wanted to cool the entire system where normally CPU is only cooled. We have added the following sentence to the abstruct.

“CPU, motherboard, random access memory, hard disk drive, solid state drive, graphic card, and the power supply unit are submerged into the transformer oil in order to cool the entire system.”

What is feasibility of transformer oil immersion cooling servers in real world considering implementation challenges?

=> The challenge would be lifetime and oxidation. One of the drawbacks of the transformer oil is oxidation. As time passes, oxidation can potentially affect the durability of the oil. We have added the following sentence to the manuscript. "It must be noted that server lifetime within the transformer oil must still be evaluated as there is possibility of the oil corroding the cables and plastics within the server. One of the drawbacks of the transformer oil could be oxidation. As time passes, oxidation can potentially affect the durability of the oil. Further study is required to better understand the effects oxidation can have upon performance and durability of the oil over a longer period of time"

Introduction:

unnecessary citations for will known facts should be avoided.

=> All citations at introduction are about materials informatics and computational materials science/Chemistry/physics now.

Methods:

It is not clear how heat from transformer oil would be removed? All other systems have circulation in place which continuously removes heat but herein no such details are mentioned.

=> Thank you for pointing this out. Both of power supply unit and graphic card have fans. Those fans create the circulation flow within the transformer oil, thus, heat is continuously removed.We have added the following to the manuscript.

“Both of power supply unit and graphic card have fans. Those fans create the circulation flow within the transformer oil, thus, heat is continuously removed.”

Experimental results could have been explained more especially effect of heating on density and viscosity. What is the effect of rise in temperature on the physical properties of the fluid.

=> It is quite challenging to identify exact physical properies of the fluid. However, as reviewers suggested, kinematic viscosity explains this well. In particular, kinematic viscosity of the transformer oil in this work is 8.09 mm2/s at 40C while its 2.21 mm2/s at 100C. This shows that kinematic viscosity become approximately ¼ with increase of 60C. In another word, low viscosity enable to circulate the fliud easily compared to high viscosity. Thus, low viscosity upon the rise of tempeature is one of the main feature of this work. The following sentences are now added.

“"The rise in temperature is found to affect the viscosity of the oil in a manner that can be seen as beneficial for the server. To start, the kinematic viscosity of the transformer oil is reported to be 8.09 mm2/s at 40C and decreases to 2.21 mm2/s when at 100C. This demonstrates that by increasing temperature, the kinematic viscosity decreases to 1/4 of its original viscosity with an increase of 60C. The server is potentially benefitting this as low viscosity allows for fluid to circulate much easier when compared to high viscosity. As the server runs and generates heat, the viscosity of the oil decreases and improves fluid circulation, which can thus help remove the heat from the server and help keep the server cool. "”

Submitted filename: comments to reviewer_final_revised.docx

Click here for additional data file.

30 Mar 2022

Designing Transformer Oil Immersion Cooling Servers for Machine Learning and First Principle Calculations

PONE-D-21-35184R1

Dear Dr. Takahashi,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mohammad Ali Haider, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

12 Apr 2022

PONE-D-21-35184R1

Designing Transformer Oil Immersion Cooling Servers for Machine Learning and First Principle Calculations

Dear Dr. Takahashi:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Mohammad Ali Haider

Academic Editor

PLOS ONE