Realistic Simulations of Photoactive Systems on HPC Clusters with Many-Core Processors

• Applicants: Professor Dr. Tobias Kramer; Professor Dr. Alexander Reinefeld
• Subject Area: Statistical Physics, Nonlinear Dynamics, Complex Systems, Soft and Fluid Matter, Biological Physics ; Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: from 2014 to 2018
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 263051053

• Final Report Year: 2018

Final Report Abstract

Photosynthesis in plants fuels life on earth by converting solar radiation into chemical energy and by producing oxygen for respiration. Within the project the accurate simulation of energy transfer processes in photoactive pigment complexes has been studied. Understanding the directed energy transfer of an initial excitation of the antenna to the reaction center requires to model the dynamics on various time-scales very accurately. This is achieved by regarding the molecular pigments as coupled entities, which convert part of the absorbed energy into heat (vibrations), while also retaining some properties of a coherent quantum state. We compared the theoretical results with experimental data by computing the time-resolved optical response caused by a series of time-delayed pump-probe pulses for models of the reaction center of photosystem I and the photosynthetic apparatus of Green sulfur bacteria. The required simulations are computationally very demanding since we include electronic and vibrational degrees of freedom on an equal footing. Based on the Hierarchical Equation of Motion (HEOM) method for solving the quantum dynamics, we have developed a scalable, highly parallel computational package to simulate all optical spectra. With increasing number of pigments and decreasing temperature, the compute memory of the HEOM method increases exponentially. Our distributed memory DM-HEOM overcomes the memory barrier imposed by a single compute node by distributing the required memory across multiple nodes. This allowed us to consider larger molecular complexes (100 pigments) and a wider range of temperatures (30K-300K) than previous implementations. DM-HEOM facilitates the computation of time- and frequency resolved spectra upon pulsed laser excitations with specific polarization sequences. By comparing the HEOM results with more approximative methods, we studied the systematic limitations of perturbative approaches. DM-HEOM incorporates parallelism on various levels. The code is highly portable across multi- and many-core CPUs as well as GPUs. Using modern, vendor-independent standards like OpenCL and MPI 3.1 enable it to run efficiently on a wide range of (super-)computers. Methods developed throughout the project are generic enough to be adapted by other HPC codes, and have been disseminated through workshops and conferences. Some generic libraries developed throughout the project are already available as free open source: • KART: Runtime-compilation At RunTime: https://github.com/noma/kart • OpenCL-based numerical solvers: https://github.com/noma/num • OpenCL abstraction layer: https://github.com/noma/ocl By publishing and maintaining the DM-HEOM code as free open source software, we provide full reproducibility of results, provide an example for modern software engineering in HPC, and encourage other groups to use it and to contribute.

Publications

• Modeling of transient absorption spectra in exciton charge-transfer systems. Journal of Physical Chemistry B, 121:463, 2016
  Tobias Kramer, Mirta Rodríguez, and Yaroslav Zelinskyy
  
• KART – A Runtime Compilation Library for Improving HPC Application Performance, pages 389–403. Springer International Publishing, Cham, 2017 [Best Paper Award]
  Matthias Noack, Florian Wende, Georg Zitzlsberger, Michael Klemm, and Thomas Steinke
  
• OpenCL in Scientific High Performance Computing: The Good, the Bad, and the Ugly. In Proceedings of the 5th International Workshop on OpenCL, IWOCL 2017, pages 12:1–12:3, New York, NY, USA, 2017. ACM
  Matthias Noack
  
• Two-dimensional electronic spectra of the photosynthetic apparatus of green sulfur bacteria. Scientifc Reports, 7:45245, 2017
  Tobias Kramer and Mirta Rodríguez
  
• A Portable and Scalable Solver-Framework for the Hierarchical Equations of Motion. In 19th IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing (PDSEC 2018), 2018
  Matthias Noack, Alexander Reinefeld, Tobias Kramer, and Thomas Steinke
  
• Efficient calculation of open quantum system dynamics and time-resolved spectroscopy with distributed memory HEOM (DM-HEOM)
  Tobias Kramer, Matthias Noack, Alexander Reinefeld, Mirta Rodríguez, and Yaroslav Zelinskyy