Using Redis supported by NVRAM in HPC applications

Artur Malinowski


Nowadays, the efficiency of a storage systems is a bottleneck in many moern HPC clusters. High performance in traditional approach – processing using files – is often difficult to obtain because of model complexity and its read/write patterns. Alternative approach is applying a key-value database, which usually has low latency and scales well. On the other hand, many key-value stores suffer from limitation of memory capacity and vulnerability to serious faiures, which is caused by processing in RAM. Moreover, some research suggests, that scientific data models are not applicable to storage structures of key-value databases. In this paper, the author proposes resolving mentioned issues by replacing RAM with NVRAM. Practical example is based on Redis NoSQL. The article contains also a three domain specific APIs, that show the idea bhind transformation from HPC data model to Redis structures, as well as two micro-benchmarks results.


high performance computing, storage systems, NoSQL, NVRAM

Full Text:



Butler D.M.: Scientific Computing Doesn’t Need noSQL. In: Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC ’12, pp. 1301–1302. IEEE Computer Society, Washington, DC, USA, 2012. ISBN 978-0-7695-4956-9. URL Companion.2012.158.

Foong A., Hady F.: Storage As Fast As Rest of the System. In: 2016 IEEE 8th International Memory Workshop (IMW), pp. 1–4. 2016. URL http://dx.doi. org/10.1109/IMW.2016.7495289.

Forum M.P.I.: MPI: A Message-Passing Interface Standard Version 3.1, 2015.

Gao S., Xu J., Hrder T., He B., Choi B., Hu H.: PCMLogging: Optimizing Transaction Logging and Recovery Performance with PCM. In: IEEE Transactions on Knowledge and Data Engineering, vol. 27(12), pp. 3332–3346, 2015. ISSN 1041-4347. URL

Han J., E H., Le G., Du J.: Survey on NoSQL database. In: Pervasive Computing and Applications (ICPCA), 2011 6th International Conference on, pp. 363–366. 2011. URL

Hanlon M.R., Dooley R., Mock S., Dahan M., Nuthulapati P., Hurley P.: A Case Study for NoSQL Applications and Performance Benefits: CouchDB vs. Postgres. In: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery. ACM, New York, NY, USA, 2011. ISBN 978-1-4503-0888-5.

Intel Corporation: Introducing Breakthrough Memory Technology, 2015. http: //

Kryder M.H., Kim C.S.: After Hard Drives. What Comes Next? In: IEEE Transactions on Magnetics, vol. 45(10), pp. 3406–3413, 2009. ISSN 0018-9464. URL

Li T., Verma R., Duan X., Jin H., Raicu I.: Exploring Distributed Hash Tables in HighEnd Computing. In: SIGMETRICS Perform. Eval. Rev., vol. 39(3), pp. 128–130, 2011. ISSN 0163-5999. URL 2160880.

Mateescu G., Gentzsch W., Ribbens C.J.: Hybrid ComputingWhere {HPC} meets grid and Cloud Computing. In: Future Generation Computer Systems, vol. 27(5), pp. 440 – 453, 2011. ISSN 0167-739X. URL

Mohan C.: History Repeats Itself: Sensible and NonsenSQL Aspects of the NoSQL Hoopla. In: Proceedings of the 16th International Conference on Extending Database Technology, EDBT ’13, pp. 11–16. ACM, New York, NY, USA, 2013. ISBN 978-1-4503-1597-5. URL 2452378.

Monnerat L., Amorim C.L.: An Effective Single-hop Distributed Hash Table with High Lookup Performance and Low Traffic Overhead. In: Concurr. Comput. : Pract. Exper., vol. 27(7), pp. 1767–1788, 2015. ISSN 1532-0626. URL http: //

Oral S., Dillow D.A., Fuller D., Hill J., Leverman D., Vazhkudai S.S., Wang F., Kim Y., Rogers J., Simmons J., Miller R.: OLCFs 1 TB/s, Next-Generation Lustre File System. In: Proceedings of Cray User Group Conference (CUG 2013). 2013.

Patterson D.: Past and Future of Hardware and Architecture. In: SOSP History Day 2015, SOSP ’15, pp. 9:1–9:63. ACM, New York, NY, USA, 2015. ISBN 978-1-4503-4017-5. URL

Qiu M., Ming Z., Li J., Gai K., Zong Z.: Phase-Change Memory Optimization for Green Cloud with Genetic Algorithm. In: IEEE Transactions on Computers, vol. 64(12), pp. 3528–3540, 2015. ISSN 0018-9340. URL

Strohmaier E., Dongarra J., Simon H., Meuer M.: TOP 10 Sites for June 2016. Accessed: 2016-07-10.

Zhao B.Y., Huang L., Stribling J., Rhea S.C., Joseph A.D., Kubiatowicz J.D.: Tapestry: A Resilient Global-scale Overlay for Service Deployment. In: IEEE Journal on Selected Areas in Communications, vol. 22(1), pp. 41–53, 2006. ISSN 0733-8716. URL

Zhao B.Y., Kubiatowicz J.D., Joseph A.D.: Tapestry: An Infrastructure for Fault-tolerant Wide-area Location and. Tech. rep., Berkeley, CA, USA, 2001.



  • There are currently no refbacks.