31.05.2022

Three CYFRONET supercomputers on the TOP500 list


Photo by Adam Frączek, KSAF AGH

For the first time in history, three supercomputers from one Polish computer centre simultaneously made it to the TOP500 list of the fastest supercomputers in the world. The machines belong to the Academic Computer Centre CYFRONET and are called: Athena, Ares, and Prometheus. Athena is currently the fastest supercomputer in Poland and was ranked 105 in the classification.

Moreover, the three CYFRONET supercomputers also made it to the Green500 list – a ranking of the most eco-friendly supercomputers, with Athena in the 9th place.

Athena – the fastest supercomputer in Poland

Athena achieves a theoretical computing power of more than 7.7 PFlops (7,709 TFlops), which currently makes her the fastes supercomputer in Poland. Installed in 2021, the system aims to provide Polish science and economy with the most modern computing resources based on processors and the most recent generation of GPGPU accelerators with the necessary data storage subsystem based on very fast flash memory. Athena's configuration includes: 48 servers with AMD EPYC processors and 1 TB of RAM (6,144 CPU compute cores in total) and 384 NVIDIA A100 GPGPU cards.

To use such sizeable computing power efficiently, supercomputers must be connected to a high-speed internal network (InfiniBand HDR with 4 x 200 Gb/s throughput per server) and a super fast disk storage subsystem. Athena uses data storage based on the Lustre software, applied also to disk systems of Prometheus and Zeus, as well as dedicated disk servers equipped with NVMe flash memory. The system was installed in the existing CYFRONET data centre and integrated with the PLGrid infrastructure.

These networks meet the needs of users of CYFRONET’s supercomputers, who use the computing infrastructure to perform standard high-performance scientific simulations (HPC) and to apply artificial intelligence methods (AI) and machine learning (ML) to research in medicine, pharmacology, biology, chemistry, physics, and many other disciplines of science. Athena’s computing power for AI calculations amounts to almost 240 PFlops!

The anticipated effects of delivering Athena’s specialised computational resources include the expansion of research, the possibility of performing advanced simulations and analyses, and the increase in the processing efficiency of the continuous stream of data from laboratories around the world. A direct anticipated impact will be articles and papers, patents and, later on, novel discoveries, which will lay foundations for cutting-edge solutions for the economy.

Ares – a new addition to the Polish scientific stage

The total theoretical performance of Ares is more than 3.5 PFlops for the CPU parts (3,510 TFlops) and more than 500 TFlops for the GPU parts. Ares also supports a disk system with more than 11 PB capacity. Data transfer is possible due to the InfiniBand EDR network. The supercomputer has 37,824 computing cores and 147.7 TB of RAM. In the TOP500 (June 2022), Ares was given the 290th place.

Deployed in 2021, Ares is based on computing servers with Intel Xeon Platinum processors, divided into three groups:

  • 532 servers equipped with 192 GB of RAM
  • 256 servers equipped with 384 GB of RAM
  • 9 servers with 8 NVIDIA Tesla V100 cards each


Ares complements the CYFRONET computing resources by providing a newer generation of core processors and servers with more memory. All its components made it possible to shorten the computation time of scientific tasks and address issues that could not have been run on a large scale due to insufficient memory. In addition, placing Ares in a geographically different location from that of the Prometheus supercomputer guarantees the continuity of providing computing services.

Prometheus – fast and energy-efficient

Since its deployment in 2015, Prometheus has been constantly included in the TOP500 ranking of the fastes supercomputers in the world. In the most recent edition, it was given the 476th place. The computing power of Prometheus is more than 2.3 PFlops (2,349 TFlops), which could be achieved primarily due to efficient HP Apollo8000 servers combined with super fast InfiniBand network with 56 Gb/s throughput. The supercomputer has 53,748 computing cores (energy-efficient and potent Intel Haswell and Intel Skylake processors) and 283.5 TB of DDR4 RAM. The cluster’s computing servers are also equipped with a liquid cooling system, which adds to its energy efficiency. Prometheus was supplemented with two file systems with a maximum capacity of 10 PB and an access speed of 180 Gb/s. It is also equipped with NVIDIA Tesla cards with GPGPU processors.

The architecture of Prometheus answers the multifarious needs of scientists by providing resources in the following groups:

  • Cluster of computing servers with highly efficient CPU nodes equipped with Intel Xeon Haswell and Intel Xeon Gold processors
  • Cluster of servers equipped with graphic accelerators GPGPU NVIDIA Tesla K40 XL
  • Acceleration partition including GPGPU NVIDIA K80 cards as well as Intel Xeon Phi 7120P and Nallatech FPGA accelerators
  • Partition dedicated to calculations related to artificial intelligence, equipped with GPGPU NVIDIA Tesla V100 graphics accelerators


All supercomputers were installed in high-tech CYFRONET computer rooms adapted exclusively for their needs. The proper functioning of supercomputers is additionally supported by the accompanying infrastructure, including a guaranteed power supply with an additional power generator, modern air-conditioning, and gas extinguishing systems.

Supercomputer cooling devices

The top of the world’s energy efficiency

Simultaneously to publishing the TOP500 list of supercomputers with the highest computing power, the Green500 list of the most eco-friendly supercomputers was published. The latter classifies supercomputers with the highest energy efficiency calculated as the ratio of floating point operations per second (supercomputer computing power) and energy consumption: GFlops/W. We are proud to say that Athena ranked 9th in the Green500 list of the most energy-efficient supercomputers in the world. The place speaks volumes about an excellent ratio of available computing power to energy consumption. Ares and Prometheus also made it to the list: Ares was classified in the first hundred, in the 83rd place, and Prometheus was ranked 162nd. Both systems are equipped with liquid cooling systems, which allowed them to achieve such formidable energy efficiency.

Scientific computing in various disciplines

Scientists conducting research using the CYFRONET infrastructure represent a multitude of different scientific disciplines. Advanced modelling and numerical calculations are used predominantly in physics, chemistry, biology, medicine, and materials science, as well as in astronomy, geology, and environmental protection. The research themes include: dynamic analysis of ceramic composite materials, modelling and optimisation of an energy-efficient prototype vehicle powered with hydrogen, simulation of molecular dynamics of electrolytes for ion batteries, or using quantum methods to analyse paintings. Other topics pertain to research on gene expression in Baltic Sea cod, investigation of influenza fusion peptide interaction with lipid membrane, analysis of the influence of underground liquid injection on reductions in stresses of induced seismicity, and studies on microphysical processes in space plasma. CYFRONET supercomputers are also available within the PLGrid infrastructure and used for the needs of high-energy physics (projects: ATLAS, LHCb, ALICE, and CMS), astrophysicists (CTA, LOFAR), Earth sciences (EPOS), European Spallation Source (ESS), biologists (WeNMR), and scholars in the humanities and social sciences (CLARIN).

Poland, Europe, and the world

The title of the fastest supercomputer in the world went to Frontier, installed at Oak Ridge National Laboratory (ORNL) in the United States. Frontier is the first supercomputer to defeat the barrier of 1 EFlops (1018), and its theoretical computing power is 1685.65 Pflops (1.69 Eflops).

That being said, the fastest European supercomputer is LUMI, built due to a collaboration of 10 European countries, including Poland. In the most recent TOP500 list, LUMI was classified in the 3rd place, even though its GPU partition has not yet been fully deployed. Its current theoretical computing power is 537 Pflops. Polish scientists can access LUMI via the PLGrid infrastructure. The access is coordinated by CYFRONET (more details on LUMI). 

Other Polish supercomputers in TOP500

Next to Athena, Ares, and Prometheus, the TOP500 (June 2022) listed two more supercomputers from Poland. Full classification with computing powers looks like this:

  • 105 – Academic Computer Centre CYFRONET, Athena (7,709 TeraFlops)
  • 145 – Poznan Supercomputing and Networking Center, Altair (5,880 TeraFlops)
  • 290 – Academic Computer Centre CYFRONET, Ares (3,510 TeraFlops)
  • 462 – Centre of Informatics Tricity Academic Supercomputer and networK, Tryton Plus (2,824 TeraFlops)
  • 476 – Academic Computer Centre CYFRONET, Prometheus (2,349 TeraFlops)

    Machine Facility CYFRONET AGH

Use the power of supercomputers in your research

Supercomputers allow its users to significantly reduce the time of computation, which could take a few years to complete using individual computers (in some specific cases more than 150, 700, or even 1000 years). With supercomputers, such tasks can be done in a matter of a few hours, and for more complex challenges – several days. CYFRONET users have access to professional support – starting with training and ending with one-on-one consultations with experts. More details on CYFRONET resources and activities are available at cyfronet.pl.

***

The most recent TOP500 list was announced on May 30, 2022, during the ISC High Performance conference in Hamburg.

Full TOP500 list (June 2022)