Exploring convolutional auto-encoders for representation learning on networks

Pranav Ajeet Nerurkar, Madhav Chandane, Sunil Bhirud

Abstract


A multitude of important real-world or synthetic systems possess network structure. Extending learning techniques such as neural networks to process such non-euclidean data is therefore an important direction for machine learning research. However, till very recently this domain has received comparatively low levels of attention. There is no straight forward application of machine learning to network data as machine learning tools are designed for $i.i.d$ data,  simple euclidean data or grids. To address this challenge the technical focus of this dissertation is on use of graph neural networks for Network Representation Learning (NRL) i.e. learning vector representations of nodes in networks. Learning vector embeddings of graph-structured data is similar to embedding complex data into low-dimensional geometries. After the embedding process is completed, drawbacks associated with graph structured data are overcome. The current inquiry proposes two deep learning auto-encoder based approaches for generating node embeddings. The drawbacks in existing auto-encoder approaches such as shallow architectures and excessive parameters are tackled in the proposed architectures using fully convolutional layers. Extensive experiments are performed on publicly available benchmark network data-sets to highlight the validity of this approach.

Keywords


Data clustering, Deep learning, Graph convolutional neural networks

Full Text:

PDF

References


Narayanan A, Chandramohan M, Chen L, Liu Y, Saminathan S. subgraph2vec:

Learning distributed representations of rooted sub-graphs from large graphs.

arXiv preprint 2016; arXiv:1606.08928.

Nerurkar P, Shirke A, Chandane M, Bhirud, S. Empirical Analysis of Data Clus-

tering Algorithms. Procedia Comput Sci 2018; 125: 770-779.

Nerurkar P, Chandane M, Bhirud, S. A Comparative Analysis of Community De-

tection Algorithms on Social Networks. In Computational Intelligence: Theories,

Applications and Future Directions-Volume I; 2019; Springer. pp. 287-298.

Nerurkar P, Chandane M, Bhirud, S. Community Detection Using Node At-

tributes: A Non-negative Matrix Factorization Approach. In: Computational

Intelligence: Theories, Applications and Future Directions-Volume I; 2019;

Springer. pp. 275-285.

Nerurkar P, Shirke A, Chandane M, Bhirud, S. A Novel Heuristic for Evolutionary

Clustering. Procedia Comput Sci 2018; 125: 780-789.

Shi Y, Gui H, Zhu Q, Kaplan L, Han J. Aspem: Embedding learning by as-

pects in heterogeneous information networks. In: Proceedings of the 2018 SIAM

International Conference on Data Mining; 2018; SIAM. pp. 144152.

Perozzi B, Kulkarni V, Chen H, Skiena S. Dont walk, skip!: Online learning of

multi-scale network embeddings. In: Proceedings of the 2017 IEEE/ACM Inter-national Conference on Advances in Social Networks Analysis and Mining 2017;

; ACM. pp. 258265.

Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q. Line: Large-scale information

network embedding. In: Proceedings of the 24th International Conference on

World Wide Web; 2015; International World Wide Web Conferences Steering

Committee. pp. 10671077.

Huang X, Li J, Hu X. Label informed attributed network embedding. In: Pro-

ceedings of the Tenth ACM International Conference on Web Search and Data

Mining; 2017; ACM. pp. 731739.

Huang X, Li J, Hu X. Accelerated attributed network embedding. In: Proceedings

of the 2017 SIAM International Conference on Data Mining; 2017; ACM. pp.

Perozzi B, Al-Rfou R, Skiena S. Deepwalk: Online learning of social represen-

tations. In: Proceedings of the 20th ACM SIGKDD international conference on

Knowledge discovery and data mining; 2014; ACM. pp. 701710.

Grover A, Leskovec J. node2vec: Scalable feature learning for networks. In: Pro-

ceedings of the 22nd ACM SIGKDD international conference on Knowledge dis-

covery and data mining; 2016; ACM. pp. 855864.

Tang L, Liu H. Relational learning via latent social dimensions. In: Proceedings

of the 15th ACM SIGKDD international conference on Knowledge discovery and

data mining; 2009; ACM. pp. 817826.

Zitnik M, Agrawal M, Leskovec J. Modeling polypharmacy side effects with graph

convolutional networks. arXiv preprint arXiv:1802.00543, 2018.

Zitnik M, Leskovec J. Predicting multicellular function through multi-layer tissue

networks. Bioinformatics, 33:190198, 2017.

Liao L, He X, Zhang H, Chua TS. Attributed social network embedding. arXiv

preprint arXiv:1705.04969, 2017.

Bandyopadhyay S, Kara H, Kannan A, Murty MN. Fscnmf: Fusing structure

and content via non-negative matrix factorization for embedding information net-

works. arXiv preprint arXiv:1804.05313, 2018.

Tsitsulin A, Mottin D, Karras P, Muller E. Verse: Versatile graph embeddings

from similarity measures. In: Proceedings of the 2018 World Wide Web Con-

ference on World Wide Web; 2018; International World Wide Web Conferences

Steering Committee. pp. 539548.

Ou M, Cui P, Pei J, Zhang Z, Zhu W. Asymmetric transitivity preserving graph

embedding. In: Proceedings of the 22nd ACM SIGKDD international conference

on Knowledge discovery and data mining; 2016; ACM. pp. 11051114.

Rozemberczki B, Davies R, Sarkar R, Sutton C. Gemsec: Graph embedding with

self clustering. arXiv preprint 2018; arXiv:1802.03997.

Rozemberczki B, Sarkar R. Fast sequence based embedding with diffusion graphs.

In: International Conference on Complex Networks; 2018; Springer. pp. 99-107.

Yang Z, Cohen WW, Salakhutdinov R. Revisiting semi-supervised learning withgraph embeddings. arXiv preprint 2016; arXiv:1603.08861.

Sheikh N, Kefato Z, Montresor A. gat2vec: representation learning for attributed

graphs. Computing, 2018.

Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations

of words and phrases and their compositionality. Adv Neur In, 23:31113119, 2013.

Cao S, Lu W, Xu Q. Grarep: Learning graph representations with global struc-

tural information. In: Proceedings of the 24th ACM International on Conference

on Information and Knowledge Management; 2015; ACM. pp. 891900.

Liu Q, Li Z, Lui J, Cheng J. Powerwalk: Scalable personalized pagerank via

random walks with vertex centric decomposition. In: Proceedings of the 25th

ACM International on Conference on Information and Knowledge Management;

; ACM. pp. 195204.

Pandhre S, Mittal H, Gupta M, Balasubramanian VN. Stwalk: learning trajectory

representations in temporal graphs. In: Proceedings of the ACM India Joint

International Conference on Data Science and Management of Data; 2018; ACM.

pp. 210219.

Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representa-

tions in vector space. arXiv preprint 2013; arXiv:1301.3781.

Lin Y, Liu Z, Sun M, Liu Y, Zhu X. Learning entity and relation embeddings for

knowledge graph completion. AAAI, 15:21812187, 2015.

Wang Z, Ye X, Wang C, Wu Y, Wang C, Liang K. Rsdne: Exploring relaxed

similarity and dissimilarity from completely-imbalanced labels for network em-

bedding. Network 2018; 11: 14-26.

Zhang M, Cui Z, Neumann M, Chen Y. An end-to-end deep learning architec-

ture for graph classification. In: Proceedings of AAAI Conference on Artificial

Inteligence, 2018.

Kipf TN, Welling M. Semi-supervised classification with graph convolutional net-

works. arXiv preprint arXiv:1609.02907, 2016.

Chen J, Ma T, Xiao C. Fastgcn: fast learning with graph convolutional networks

via importance sampling. arXiv preprint 2018; arXiv:1801.10247.

Donnat C, Zitnik M, Hallac D, Leskovec J. Spectral graph wavelets for structural

role similarity in networks. arXiv preprint 2017; arXiv:1710.10321.

Wu L, Fisch A, Chopra S, Adams K, Bordes A, Weston J. Starspace: Embed all

the things! arXiv preprint 2017; arXiv:1709.03856.

Hamilton W, Ying Z, Leskovec J. Representation Learning on Graphs: Methods

and Applications. arXiv preprint, arXiv:1709.05584, 2017.

Balasubramanian M, Schwartz EL. The isomap algorithm and topological stabil-

ity. Science, 295:77, 2002.

Roweis ST, Saul LK. Nonlinear dimensionality reduction by locally linear embed-

ding. Science, 290:23232326, 2000.

Chen CM, Tsai MF, Lin YC, Yang YH. Query-based music recommendations via

preference embedding. In: Proceedings of the 10th ACM Conference on Recom-mender Systems; 2016; ACM. pp. 7982.

Dai H, Dai B, Song L. Discriminative embeddings of latent variable models for

structured data. In: International Conference on Machine Learning; 2018. pp.

Yang C, Liu Z, Zhao D, Sun M, Chang EY. Network representation learning with

rich text information. IJCAI, 56:21112117, 2015.

Hamilton W, Ying Z, Leskovec J. Inductive representation learning on

large

graphs. Adv Neur In, 31:1024-1034, 2017.

Scarselli F, Gori M, Tsoi AC, Hagenbuchner, M, Monfardini, G. The graph neural

network model. IEEE T Neural Networ, 20:61-80, 2005.

Monti F, Boscaini D, Masci J, Rodol E, Svoboda J, Bronstein MM. Geometric

Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In: IEEE

Conference on Computer Vision and Pattern Recognition (CVPR); 2017; IEEE.

pp. 5425-5434.

Li Q, Zhichao H, Xiao-Ming W. Deeper Insights into Graph Convolutional Net-

works for Semi-Supervised Learning. arXiv preprint arXiv:1801.07606, 2018.

Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY. Multimodal deep learning.

In: Proceedings of the 28th international conference on machine learning; 2011;

ICML; pp. 689-696.

Nikolentzos G, Polykarpos M, Tixier JPA, Konstantinos S, Vazirgiannis M. Kernel

graph convolutional neural networks. In: International Conference on Artificial

Neural Networks; 2018; Springer. pp. 22-32.

Pu Y, Gan Z, Henao R, Yuan X, Li C, Stevens A, Carin L. Variational autoen-

coder for deep learning of images, labels and captions. In: Advances in neural

information processing systems; 2016; NIPS; pp. 2352-2360.

Deng J, Zhang Z, Marchi E, Schuller B. Sparse autoencoder-based feature transfer

learning for speech emotion recognition. In: Humaine Association Conference on

Affective Computing and Intelligent Interaction; 2013; IEEE; pp. 511-516.

Lu X, Tsao Y, Matsuda S, Hori C. Binary coding of speech spectrograms using a

deep auto-encoder. In: Eleventh Annual Conference of the International Speech

Communication Association; 2010; Interspeech. pp. 436-440.




DOI: https://doi.org/10.7494/csci.2019.20.3.3167

Refbacks

  • There are currently no refbacks.