Prometheus – the only Polish supercomputer on the latest list TOP500

On 18th November 2019, the latest list TOP500 of the fastest supercomputers in the world was announced at the conference Supercomputing’19 held in Denver, USA. Prometheus of the AGH UST Academic Computer Centre CYFRONET has been listed in the ranking for the 10th time, taking 241st place, and (as in the previous editions of TOP500) it still remains the only supercomputer from Poland.  

Prometheus – the fastest Polish supercomputer 

Prometheus, working at the AGH UST Academic Computer Centre Cyfronet, has been the fastest supercomputer in Poland since its launch in 2015. This title has been ensured by the theoretical computing power of 2.4 PFlops (PetaFlops), which has been possible thanks to the use of efficient servers of the HP Apollo 8000 platform, connected with one another by a superfast InfiniBand network offering the transfer capacity of 56 Gb/s. The supercomputer has 53,604 processing cores (power-saving and efficient Intel Haswell processors), as well as 282 TB of internal memory made in the DDR4 technology. Prometheus has two file systems of total capacity equal to 10 PB, and the access speed of 180 GB/s. It is also equipped with NVIDIA Tesla cards with GPGPU graphics processing units. Prometheus was built by Hewlett-Packard according to the specifications developed by the experts of AGH UST Cyfronet.  

“To illustrate the operating speed of Prometheus, it can be said that in order to match its computing power, one would have to use the power of over 50,000 high-class personal computers in the best-possible configuration, additionally connected by a superfast network, and managed by specialist software,” says professor Kazimierz Wiatr, Director of the AGH UST Academic Computer Centre Cyfronet.  

A characteristic feature of Prometheus is also an innovative system of direct cooling by liquid. Thanks to the technology, it is one of the most energy-saving computers in its class in the world. In order to keep a suitable temperature of the cooling liquid in our climate, we only need economical dry coolers instead of technologically-advanced precision air-conditioning systems and chilled water generators, which use large amounts of energy. The liquid is used to cool both CPU processors and GPGPUs, as well as memory modules by means of a special airtight system of heat pipes and the contact exchange of heat between the servers and the system in which the liquid circulates. The servers themselves remain “dry” – they can be removed at any time without the risk of a leakage in the cooling system. Liquid-based cooling also allowed to reach an extremely high compactness of the installation – as many as 144 servers in a single rack.  

“The energy efficiency of Prometheus is comparable to the systems of the largest data centres in the world, such as Google, Amazon, or Facebook. Moreover, liquid-based cooling enables an extremely high density of installation, thanks to which the computing section, which weighs over 40 tons, takes up an area of only 18 square metres and fits in only 20 racks. To achieve a comparable computing power, Zeus, the predecessor of Prometheus, would have to take up even 160 racks. The predecessor of Zeus was Baribal, which fitted in 8 racks and had computing power equal to 1.5 TFlops (TeraFlops) – if Prometheus had been built in the same technology, it would have taken up 12,000 racks,” emphasizes the Director of Cyfronet.  

The architecture of the supercomputer Prometheus reflects the various needs of scientists by providing them with resources organised in partitions: 

  • classic computing servers with high-efficiency Intel Xeon Haswell and Intel Xeon Gold processors,  
  • a set of servers with NVIDIA K40 XL graphics processing units,  
  • an acceleration partition with GPGPU NVIDIA K80 cards, and Intel Xeon Phi and Nallatech FPGA accelerators,  
  • a set of servers designed for computing tasks in the cloud environment.  

Prometheus was installed in Cyfronet’s modern computer hall, specially adapted for its operation. The non-failure operation of the supercomputer is ensured by important elements of the technical infrastructure, such as a system of guaranteed power supply with an auxiliary power unit, and modern systems of technological air-conditioning and gaseous fire suppression.  

Zeus – support to Prometheus 

The supercomputer Zeus has been continually in operation at Cyfronet since 2008. Since its launch, in the years 2008-2015, it was listed in the ranking TOP500 twelve times in a row, of which four successive times among the first one hundred fastest supercomputers in the world (the highest rank was 81), and ten times as the fastest computer in Poland. The last time Zeus found a place on the list TOP500 – at position 387 – was in November 2015. At that time, on the list there were two supercomputers from Cyfronet. Prometheus was rated number 38. The heterogeneous architecture of the supercomputer consists of four dedicated hardware partitions and features 25,468 computing cores and 204 GPGPU processors of total computing power equal to 374 TFlops. The users have at their disposal 60 TB of random-access memory (RAM) and a disk storage capacity of 2.3 PB. Zeus servers are connected with one another by a fast InfiniBand network offering the transfer capacity of 40 Gb/s. 

It is worth emphasizing that even the highest positions on the list TOP500 or the latest technologies applied to building high performance computers do not fully reflect the importance of this kind of computing power for the Polish scientific environment. The usefulness of the supercomputers made available by the AGH UST Academic Computer Centre Cyfronet as a tool for conducting scientific research is best confirmed by the statistics concerning their use. In 2018, the two most powerful supercomputers of the AGH UST Academic Computer Centre Cyfronet – Zeus and Prometheus – performed 8,342,686 calculations of the total operation time equal to 42,436 years.  

Scientific calculations in different fields of science 

Prometheus and Zeus are a part of the European cloud and grid infrastructure within the framework of the European Grid Infrastructure (EGI). At the same time, Prometheus and Zeus are important supercomputers in the Polish computing infrastructure PLGrid – a platform that enables carrying out in silico scientific research and making calculations with the use of high performance computers, also in the cloud and grid infrastructure.  

Thanks to the PLGrid infrastructure, scientists and researchers can make an intensive use of the computing power of Prometheus and Zeus. Dedicated computing environments, the so-called “domain-specific grids”, and specialised information platforms make it possible to deal effectively with more and more complex computational problems.  

Scientists carrying out research with the use of the Cyfronet infrastructure represent many different branches of science. Advanced modelling and numerical calculations are mainly used in the fields of physics, chemistry, biology, medicine and materials engineering, as well as astronomy, geology, and environmental protection. Research projects cover, for example, modelling the electron infrastructure of composites containing graphene oxide, physicochemical properties of biodegradable polymers, modelling the activity of selected anticancer drugs by carbon nanoparticles, and the use of gravitational lenses as space telescopes. Scientists also conduct research into the activity of thunderclouds from the local and global perspective, the analysis of images with the use of quantum methods, as well as testing the strength of drill strings and the flow of drilling fluids. Thanks to the supercomputers located at Cyfronet, Polish scientists and researchers can participate in important international projects, including such experiments as CTA, LOFAR, EPOS, and CERN’s Large Hadron Collider, as well as recently-discovered gravitational waves in LIGO and VIRGO detectors.  

Use the power of supercomputers in scientific research projects 

Supercomputers make it possible to shorten substantially the time of calculations, which with the use of single computers would often take many years (in some real cases, over 150, 700, or even 1,000 years), and here they can usually be made in the course of just several days. Cyfronet users can receive professional support – from access to complete documentation, through various trainings, to individual guiding sessions with experts. Information related to the use of Prometheus and Zeus by scientists can be found on the websites: http://www.cyfronet.pl and http://www.plgrid.pl. It is also worthwhile registering on the PLGrid Portal (http://portal.plgrid.pl) and take advantage of many unique services available there.  

Poland compared to Europe and the world 

The fastest supercomputer in the world is again Summit from the USA, whose theoretical computing power exceeds 200 PFlops. The list TOP500 is clearly dominated by supercomputers from China (228 machines). Second place is taken by the USA (117 machines). 

European countries whose supercomputers were listed in the ranking (15 countries, including Russia), won 94 places in total. The fastest European supercomputer is still Piz Daint from Switzerland, which was rated number 6. In the top ten of the list, there was also the German supercomputer SuperMUC-NG, which reached the 9th place. The leaders in terms of the number of supercomputers on the list TOP500 in Europe are: France (18 machines), Germany (16 machines), and the Netherlands (15 machines).  

Complete list TOP500