On the improvement of a scalable sparse direct solver for unsymmetrical linear equations

M. Serdar Celebi*, Ahmet Duran, Figen Oztoprak, Mehmet Tuncel, Bora Akaydin

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


This paper focuses on the application level improvements in a sparse direct solver specifically used for large-scale unsymmetrical linear equations resulting from unstructured mesh discretization of coupled elliptic/hyperbolic PDEs. Existing sparse direct solvers are designed for distributed server systems taking advantage of both distributed memory and processing units. We conducted extensive numerical experiments with three state-of-the-art direct linear solvers that can work on distributed-memory parallel architectures; namely, MUMPS (MUMPS solver website, http://graal.ens-lyon.fr/MUMPS), WSMP (Technical Report TR RC-21886, IBM, Watson Research Center, Yorktown Heights, 2000), and SUPERLU_DIST (ACM Trans Math Softw 29(2):110–140, 2003). The performance of these solvers was analyzed in detail, using advanced analysis tools such as Tuning and Analysis Utilities (TAU) and Performance Application Programming Interface (PAPI). The performance is evaluated with respect to robustness, speed, scalability, and efficiency in CPU and memory usage. We have determined application level issues that we believe they can improve the performance of a distributed-shared memory hybrid variant of this solver, which is proposed as an alternative solver [SuperLU_MCDT (Many-Core Distributed)] in this paper. The new solver utilizing the MPI/OpenMP hybrid programming is specifically tuned to handle large unsymmetrical systems arising in reservoir simulations so that higher performance and better scalability can be achieved for a large distributed computing system with many nodes of multicore processors. Two main tasks are accomplished during this study: (i) comparisons of public domain solver algorithms; existing state-of-the-art direct sparse linear system solvers are investigated and their performance and weaknesses based on test cases are analyzed, (ii) improvement of direct sparse solver algorithm (SuperLU_MCDT) for many-core distributed systems is achieved. We provided results of numerical tests that were run on up to 16,384 cores, and used many sets of test matrices for reservoir simulations with unstructured meshes. The numerical results showed that SuperLU_MCDT can outperform SuperLU_DIST 3.3 in terms of both speed and robustness.

Original languageEnglish
Pages (from-to)1852-1904
Number of pages53
JournalJournal of Supercomputing
Issue number5
Publication statusPublished - 1 May 2017

Bibliographical note

Publisher Copyright:
© 2016, Springer Science+Business Media New York.


Authors acknowledge for the computing resources allocated by PRACE Research Infrastructures ‘Hydra’ at RZG (Rechenzentrum Garching) Germany, and ‘Karadeniz’ at UHeM (National High Performance Computing Center of Turkey) under Grant Agreement RI-28349. The authors thank to Aramco Overseas Company B.V. for their financial support under contract number 6600028651. The authors appreciate the helpful comments and suggestions by the Editor-in-Chief of The Journal of Supercomputing, Prof. Hamid R. Arabnia, and anonymous referees.

FundersFunder number
Aramco Overseas Company B.V.6600028651


    • Large scale simulations
    • Linear equations
    • Many-core distributed solver
    • Numerical factorization
    • Parallel linear direct solver
    • Reservoir simulations
    • Scalability
    • Sparse direct solver
    • SuperLU
    • Symbolic factorization


    Dive into the research topics of 'On the improvement of a scalable sparse direct solver for unsymmetrical linear equations'. Together they form a unique fingerprint.

    Cite this