A distributed approach for graph mining in massive networks

N. Talukder, M. J. Zaki

Research output: Contribution to journalArticle

22 Citations (Scopus)

Abstract

We propose a novel distributed algorithm for mining frequent subgraphs from a single, very large, labeled network. Our approach is the first distributed method to mine a massive input graph that is too large to fit in the memory of any individual compute node. The input graph thus has to be partitioned among the nodes, which can lead to potential false negatives. Furthermore, for scalable performance it is crucial to minimize the communication among the compute nodes. Our algorithm, DistGraph, ensures that there are no false negatives, and uses a set of optimizations and efficient collective communication operations to minimize information exchange. To our knowledge DistGraph is the first approach demonstrated to scale to graphs with over a billion vertices and edges. Scalability results on up to 2048 IBM Blue Gene/Q compute nodes, with 16 cores each, show very good speedup.

Original languageEnglish
Pages (from-to)1024-1052
Number of pages29
JournalData Mining and Knowledge Discovery
Volume30
Issue number5
DOIs
Publication statusPublished - 1 Sep 2016
Externally publishedYes

Keywords

  • Distributed graph mining
  • Frequent subgraph mining
  • High performance computing
  • Parallel graph mining
  • Single large graph

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'A distributed approach for graph mining in massive networks'. Together they form a unique fingerprint.

  • Cite this