A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition

Yuhe Wang, John E. Killough

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The reasons that cause load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so called dynamic imbalance can be further exacerbated in parallel compositional simulations. The flash calculations for equations of state in complex compositional simulations not only can consume over half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each grid block heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of grid blocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It over-decomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads which can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We employ this approach in a legacy reservoir simulator and demonstrate reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain over-decomposition together with a load balancer can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem.

Original languageEnglish
Title of host publicationSociety of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013
Pages121-137
Number of pages17
Volume1
Publication statusPublished - 2013
EventSPE Reservoir Simulation Symposium 2013 - The Woodlands, TX, United States
Duration: 18 Feb 201320 Feb 2013

Other

OtherSPE Reservoir Simulation Symposium 2013
CountryUnited States
CityThe Woodlands, TX
Period18/2/1320/2/13

Fingerprint

Load Balance
Dynamic loads
Simulators
decomposition
Decomposition
Decompose
Dynamic Load
simulation
Simulation
simulator
Hydrocarbons
Equations of state
Simulator
Scalability
Flash
Model
Execution Time
Bundle
Geometry
Communication

ASJC Scopus subject areas

  • Geochemistry and Petrology
  • Modelling and Simulation

Cite this

Wang, Y., & Killough, J. E. (2013). A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition. In Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013 (Vol. 1, pp. 121-137)

A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition. / Wang, Yuhe; Killough, John E.

Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013. Vol. 1 2013. p. 121-137.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, Y & Killough, JE 2013, A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition. in Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013. vol. 1, pp. 121-137, SPE Reservoir Simulation Symposium 2013, The Woodlands, TX, United States, 18/2/13.
Wang Y, Killough JE. A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition. In Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013. Vol. 1. 2013. p. 121-137
Wang, Yuhe ; Killough, John E. / A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition. Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013. Vol. 1 2013. pp. 121-137
@inproceedings{7c149d4f847c4bee8988e60809bdf23d,
title = "A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition",
abstract = "The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The reasons that cause load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so called dynamic imbalance can be further exacerbated in parallel compositional simulations. The flash calculations for equations of state in complex compositional simulations not only can consume over half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each grid block heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of grid blocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It over-decomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads which can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We employ this approach in a legacy reservoir simulator and demonstrate reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain over-decomposition together with a load balancer can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem.",
author = "Yuhe Wang and Killough, {John E.}",
year = "2013",
language = "English",
isbn = "9781627480246",
volume = "1",
pages = "121--137",
booktitle = "Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013",

}

TY - GEN

T1 - A new approach to load balance for parallel compositional simulation based on reservoir model over-decomposition

AU - Wang, Yuhe

AU - Killough, John E.

PY - 2013

Y1 - 2013

N2 - The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The reasons that cause load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so called dynamic imbalance can be further exacerbated in parallel compositional simulations. The flash calculations for equations of state in complex compositional simulations not only can consume over half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each grid block heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of grid blocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It over-decomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads which can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We employ this approach in a legacy reservoir simulator and demonstrate reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain over-decomposition together with a load balancer can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem.

AB - The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The reasons that cause load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so called dynamic imbalance can be further exacerbated in parallel compositional simulations. The flash calculations for equations of state in complex compositional simulations not only can consume over half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each grid block heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of grid blocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It over-decomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads which can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We employ this approach in a legacy reservoir simulator and demonstrate reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain over-decomposition together with a load balancer can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem.

UR - http://www.scopus.com/inward/record.url?scp=84881040283&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84881040283&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781627480246

VL - 1

SP - 121

EP - 137

BT - Society of Petroleum Engineers - SPE Reservoir Simulation Symposium 2013

ER -