CAE (Computer Aided Engineering) refers to the use of computers to simulate the structure, performance and operating status of engineering and industrial products, and CFD (Computational Fluid Dynamics) refers to the use of computers to numerically solve fluid dynamics problems.
At present, CAE/CFD simulation based on high-performance computers is widely used in aerospace, automobile and shipbuilding, petrochemical industry, civil engineering and other fields. Moreover, with the continuous development of computer technology, the role of CAE/CFD in the above fields has become increasingly important. Especially in the aerospace field, CAE/CFD has become one of the three research methods alongside theoretical analysis and wind tunnel experiments. NASA predicts that the design of aerospace vehicles will undergo revolutionary changes in the near future, that is, virtual flights in numerical wind tunnels based on CAE/CFD simulation data.
The primary goal of applying high-performance computing to CAE/CFD is to obtain results closest to reality at the lowest cost. At present, there are many softwares in the CAE/CFD industry, and customers' application requirements for these softwares are also very different, making it difficult to form a unified solution.
So how should high-performance computing apply optimization in CAE/CFD? First, let’s break down the characteristics of CAE/CFD.
CAE/CFD software basically uses the finite element method or the finite volume method to solve the control equations. The calculation involves a large number of iterations of front and back time steps and processing of the junction area. It is computationally intensive and requires CPU performance. Naturally, the higher the better.
Generally speaking, CAE/CFD software that uses explicit time formats has average memory capacity requirements, while CAE/CFD software that uses implicit time formats has relatively high requirements for memory capacity. Memory bandwidth requirements are directly related to the number of processes in a single node. The greater the number of processes, the higher the memory bandwidth requirements.
At the I/O level, when most CAE/CFD software performs a read operation, the master process reads data and then distributes the data to the slave processes; during a write operation, the master process collects data from the slave processes in a unified manner. data, and then the main process writes out the data. In other words, only the main process is directly responsible for I/O operations. Therefore, CAE/CFD puts moderate pressure on I/O storage.
As mentioned before, since the CAE/CFD software needs to iterate before and after time steps and process the junction area during calculation, it is also relatively demanding on network communication, so the InfiniBand network should be used.
Next, let’s take a look at how to optimize CAE/CFD operations for high-performance computing.
Since CAE/CFD is mostly commercial software, it can generally only perform hardware-level optimization and operation-level optimization. Considering the previous feature analysis, at the hardware level optimization level, we can improve the performance of CPU and memory respectively by configuring high-frequency CPU and four-channel memory. We can also improve the network connection topology, routing algorithm and RDMA. Optimize to improve the communication capabilities of InfiniBand networks.
At the run-level optimization level, MPI optimization strategies are generally used. For example, MPI process binding optimization, MPI message passing mechanism optimization, and MPI communication shared memory optimization, etc. With the development of the PGAS (Partitioned Global Address Space) programming model and accelerated co-processors such as GPU and MIC, the above traditional algorithms will be greatly improved.
Taking the Fluent calculation example as an example, Inspur used its independently developed "Teye" high-performance application characteristic monitoring and analysis system (English: Teye) to monitor and analyze the application characteristics of Fluent. On the Intel XeonE5-2680v3 CPU platform and 144 processes, a 10 million grid-scale Fluent calculation example was calculated.
In the figure below, it can be seen through Inspur Sky Eye that Fluent's CPU utilization is close to 100, and the physical cores involved in calculations are basically running at full load, indicating that Fluent is computationally intensive.
1
Inspur Sky Eye analyzes the CPU utilization of Fluent calculation examples
In terms of memory bandwidth, Sky Eye detects that Fluent’s memory bandwidth requirements mainly occur in the grid The total memory bandwidth after building, merging and partitioning is approximately 45GB/s. Among them, the read memory bandwidth is about 40GB/s, and the write memory bandwidth is about 5GB/s. Fluent reads and writes data from the main process, and the slave process does not directly participate in I/O operations. In addition, the amount of data read and written is directly related to the size of the Fluent calculation example.
2
Inspur Sky Eye analyzes the memory bandwidth of the Fluent calculation example
In addition, we can also see that except for the initial stage, the read operation rate of the master node reaches 40MB/ In addition, during the entire calculation process of Fluent, the I/O operation pressure is relatively small. As a typical CFD software, Fluent requires the use of InfiniBand network for data communication.
3
Inspur Sky Eye monitors the disk read and write operations of Fluent calculation examples
In the early stages of calculation, Fluent performs a large number of data sending and receiving operations through the network. In the subsequent calculation process, data transmission and reception show continuous flow characteristics, and the rate is maintained at 60MB/s ~ 100 MB/s. It can be seen that Fluent is a network-intensive software.
4
Inspur Sky Eye monitors the sending and receiving of InfiniBand grid data of the Fluent calculation example
The picture below shows the radar of the Fluent calculation example analyzed and monitored by Inspur Sky Eye The figure visually displays the various application features of Fluent.
5
Radar chart of Fluent software application characteristics analyzed by Inspur Sky Eye
Inspur has rich experience in CAE/CFD optimization in high-performance computing. Tianyan can analyze the characteristics of CAE/CFD software more comprehensively and finely, and provide a comprehensive optimization solution integrating software and hardware. At present, Inspur's CAE/CFD high-performance computing application solutions have been widely used in aerospace, automobile and shipbuilding, construction and other industries.