Supercomputer procurement at CSC

Supercomputer procurement at CSC Janne Ignatius Project Manager, CSC HPC Purchase 2006 Development Manager, Scientific Data and Software 13 March 2007

Layers of the National Computing Capacity A. Supercomputer system at CSC: i. more tightly connected part (more expensive per CPU) capability part ii. massive cluster part (less expensive per CPU) capacity part B. Departmental (or cross-departmental) clusters at universities and research institutes

Two criteria 1. Summed throughput capacity National hardware compared to the rest of the world, ~ Top 500 list?

CSC's share of Top500-sum CSC / Top500 sum 0.008 0.006 0.004 0.002 0 1995 2000 2005 2010 Year [(c) CSC, J. Ignatius]

2. Large parallel jobs How well are the Finnish top scientists able to run massive parallel jobs, compared with international competition? => Frequent need for 100 1000 core runs

Preparatory actions 2003 fall: CSC becomes aware that the next Supercomputer will be needed 2003 end 2005 summer: reports on HPC need, visiting the vendors, discussions related to funding, 2005 summer: funding prospects look good (Ministry of Education)

From the budget of the State of Finland, as approved in the Parliament on 21 December 2005: Opetusministeriö saa vuonna 2006 tehdä uuden supertietokoneen hankkimiseksi sopimuksia siten, että niistä aiheutuu valtiolle menoja vuonna 2007 enintään 4 000 000 euroa, vuonna 2008 enintään 4 000 000 euroa ja vuonna 2009 enintään 2 000 000 euroa.

CSC HPC Purchase 2006 project Duration & manpower Duration of the actual purchase project roughly 13 months: September 2005 October 2006. Amount of work: 4 person-years at CSC Schedule was tight, but with very hard work it was possible to keep in the schedule Purchase and service contract were signed on 9 October 2006!

Project organization Project steering group: Janne Kanner chair, Kimmo Koski, Juha Haataja; Janne Ignatius as PM Significant decisions at CSC Steering Group, very significant add ly discussed with CSC Board of Directors & Ministry of Education Project Manager Janne Ignatius, Project Coordinator Dan Still BENCHMARK: Juha Fagerholm, Jura Tarus, Jan Åström, Kimmo Mattila, Olli-Pekka Lehto, Ari Lukkarinen, Paavo Ahonen, Jussi Heikonen; (Atte Sillanpää, Ville Savolainen, Mikael Johansson) LAW: Katri Luostarinen, Totti Mäkelä, Tiina Kupila-Rantala, (Urpo Kaila) PURCHASE: Totti Mäkelä, all team leaders (=underlined), Raimo Uusvuori, Yrjö Leino; (Kimmo Niittuaho) TECHNOLOGY: Joni Virtanen, Jussi Heikonen, TottiMäkelä, Ari Lukkarinen, Jura Tarus, (LAPA Group) COMMUNICATIONS: Paavo Ahonen Name = team leader, Name = additionally partial team leader role, Name = team member, Name = adopted to project membership, Name = adopted to team membership from within the project, (Name) = not formally in project but significantly contributed to it Notice: team borders were naturally constantly broken, on purpose

Starting point Two-fold approach in the purchase: more tightly connected system (more expensive per processor core): supercomputer massive cluster system (less expensive per processor core): supercluster Fully open to changes in this to optimize performance/price Investment: 10 M (excluding VAT) excluding: service contracts, maintenance work by CSC, space and electricity expenses a smallish fraction of the 10 M was allocated to data storage, as a separate project. However, the local work disks were purchased as part of the actual supercomputer systems

Procurement process 3 rounds of negotiations with the vendors 2 rounds of shortlistings The vendors needed to submit first benchmark results early in the process Official Invitation to Tender & Tender at a late stage in the process. However, much of this type of material was in practice covered in prior documents

Benchmarks Only in Lot 1: XLHIRLAM: weather forecasts DALTON: quantum chemistry VASP: density-functional theory POLAR: cosmic microwave background Both in Lot 1 and Lot 2: HPC Challenge Benchmarks (including LINPACK) IOzone (or similar) GROMACS: molecular dynamics SU3_AHIGGS: lattice gauge field theory Only in Lot 2: HMMER: protein sequence analysis in bioinformatics POROUS: fluid dynamics with lattice Boltzmann

Supercluster Murska : HP Peak performance 10.6 Tflops 2048 processor cores: 2.6 GHz AMD Opteron 2-core Interconnect: Infiniband (industry standard, middlerange scalability) Local fast work disk: 98 TB (SATA) Memory: in 512 core 4 GB/core, in 512 core 2 GB/core, in 1024 core 1 GB/core HP XC-cluster, operating system is Linux (based on Red Hat Enterprise Linux Advanced Server 4.0) Machine room and hosting provided by HP for at least the first 1.5 years

Cray XT4 ( Hood ) Supercomputer Louhi : Cray genuine homogeneous massively parallel processor (MPP) system Cray s proprietary interconnect, very good scalability At all phases memory 1 GB/core local fast work disk 70 TB (fiber channel) Operating system: in login nodes Linux (Unicos/lc, based on SuSE Linux) in computational cores (at least in phase 1) light-weight kernel Catamount: requires certain minor modifications to the source code of programs

Supercomputer Louhi : Cray Peak performance 10.5 Tflops -> 70 Tflops Phase 1 11 production cabinets, AMD Opteron 2.6 GHz 2-core, 10.5 Tflops peak performance Phase 2 Delivery in June 2008, acceptance tests in Q3/2008 replacing the computational blades (cores) in the 11 cabinets and adding 5 more cabinets: in all AMD Opteron 2.6 GHz 4- core, 62.1 Tflops peak power Phase 2b Adding 2 more cabinets in December 2008 Thus total of 18 cabinets: AMD Opteron 2.6 GHz 4-core, 70.1 Tflops peak power

CSC's share of Top500-sum CSC / Top500 sum 0.008 0.006 0.004 0.002 0 1995 2000 2005 2010 Year [(c) CSC, J. Ignatius]

CSC's share of Top500-sum CSC / Top500 sum 0.008 0.006 0.004 0.002 0 1995 2000 2005 2010 Year [(c) CSC, J. Ignatius]

Notes on the systems Consumption of electric power: HP 160 kw, Cray in the final configuration 400 kw. Both summed 0.56 MW, electricity bill approx. 350 k /year. Plus electricity needed for cooling, roughly + 30 %. In the contracts the vendors committed to: certain deadlines. If late from these, penalties for the vendor both minimum specified hardware configuration, and minimum specified performance in the benchmarks. If these not met, compensation. Altogether peak performance in supercomputers: 10.6 + 10.5 = 21 Tflops (Q2/2007) -> 10.6 + 70 = 80 Tflops (end of 2008).

CSC s history of supercomputers

Image: Juha Fagerholm, CSC

On profiling the systems HP supercluster Murska : For a large number of computational customers No porting of codes needed Data-intensive computing Jobs needing more memory Cray XT4 supercomputer Louhi : For a smaller number of research groups For runs/codes which benefit from Cray (in performance/price sense) Software: only as needed by those groups Customers will be helped in porting the codes to Cray In batch queue system more towards capability-type profiling: space for very large parallel jobs => a somewhat lower usage percentage

Let us make together a leap forward in computational science in Finland!

Vertailua eksponentiaalisessa kasvussa Edellinen superhankinta (IBM Power 4): huipputehon käyttöönotto n. 6 vuoden jälkeen sitä edeltävästä (Cray T3E) huippulaitteen hinta n. 1,5-kertaistui huippulaitteen teho n. 14-kertaistui (=2.25 TF/0.168 TF), käytännössä kuitenkin vastaten vain ~7-kertaistumista (koska verrataan 4 flop/cycle -laitetta 2 flop/cycle -laitteeseen) Nykyinen superhankinta (Cray Hood ): huipputehon käyttöönotto n. 6 vuoden jälkeen edeltävästä huippulaitteen hinta n. 1,5-kertaistui huippulaitteen teho n. 31-kertaistui (= 70 TF/2.25 TF), myös käytännössä vastaten 31-kertaistumista (verrataan 4 flop/cycle laitteita keskenään)