Analysis of Series of Measurements from Job-Centric Monitoring by Statistical Functions

Marcus Hilbrich; Markus Frank

doi:10.7494/csci.2017.18.1.2

Authors

Marcus Hilbrich s-lab -- Software Quality Lab, Universität Paderborn
Markus Frank Technische Universität Chemnitz

DOI:

https://doi.org/10.7494/csci.2017.18.1.2

Keywords:

Job-centric monitoring, Monitoring, Similarity, Series of measurements, Statistical functions, Grid, Cloud, Analysis

Abstract

The rising number of executed programs (jobs) enabled by the
growing amount of available resources from Clouds, Grids,
and HPC (for example) has resulted in an enormous number of
jobs. Nowadays, most of the executed jobs are mainly
unobserved, so unusual behavior, non-optimal resource usage,
and silent faults are not systematically searched and
analyzed. Job-centric monitoring enables permanent job
observation and, thus, enables the analysis of monitoring
data. In this paper, we show how statistic functions can be
used to analyze job-centric monitoring data and how the
methods compare to more-complex analysis methods.
Additionally, we present the usefulness of job-centric
monitoring based on practical experiences.

Downloads

Download data is not yet available.

References

Bellman R.: Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 2010.

Chan P., Stolfo S.J.: Toward Parallel and Distributed Learning by MetaLearning. In: AAAI Workshop in Knowledge Discovery in Databases, pp. 227– 240. 1993.

Charles J., Höcker A., Lacker H., Laplace S., Diberder F., Malcls J., Ocariz J., Pivk M., Roos L.: CP violation and the CKM matrix: assessing the impact of the asymmetric B factories. In: The European Physical Journal C - Particles and Fields, vol. 41(1), pp. 1–131, 2005. ISSN 1434-6044. URL http://dx.doi.org/ 10.1140/epjc/s2005-02169-1.

Denning D.E.: An intrusion-detection model. In: IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, vol. 13(2), pp. 222–232, 1987.

Dickerson J.E., Dickerson J.A.: Fuzzy network profiling for intrusion detection. In: Proc. of NAFIPS 19th International Conference of the North American Fuzzy Information Processing Society, Atlanta, pp. 301–306. 2000.

Dobai R., Balaz M.: Genetic method for compressed skewed-load delay test generation. In: Design and Diagnostics of Electronic Circuits Systems (DDECS), 2012 IEEE 15th International Symposium on, pp. 242–247. 2012. URL http: //dx.doi.org/10.1109/DDECS.2012.6219065.

Grefenstette J.: Optimization of Control Parameters for Genetic Algorithms. In: Systems, Man and Cybernetics, IEEE Transactions on, vol. 16(1), pp. 122–128, 1986. ISSN 0018-9472. URL http://dx.doi.org/10.1109/TSMC.1986.289288.

von Grünigen D.: Digitale Signalverarbeitung: Mit einer Einführung in die kontinuierlichen Signale und Systeme. Fachbuchverlag Leipzig, 2008. ISBN 9783446414631.

Gusfield D.: Algorithms on Stings, Trees, and Sequences. In: Computer Science and Computational Biology, 1997.

Hilbrich M.: Jobzentrisches Monitoring in Verteilten Heterogenen Umgebungen mit Hilfe Innovativer Skalierbarer Methoden. Dissertation, Fakultät Informatik der Technischen Universität Dresden, Germany, 2014.

Hilbrich M., Müller-Pfefferkorn R.: A Scalable Infrastructure for Job-Centric Monitoring Data from Distributed Systems. In: M. Bubak, M. Turala, K. Wiatr, eds., Proceedings Cracow Grid Workshop ’09, pp. 120–125. ACC CYFRONET AGH, ul. Nawojki 11, 30-950 Krakow 61, P.O. Box 386, Poland, 2010. ISBN 978-83-61433-01-9.

Hilbrich M., Müller-Pfefferkorn R.: Achieving scalability for job centric monitoring in a distributed infrastructure. In: G. Mühl, J. Richling, A. Herkersdorf, eds., ARCS Workshops, LNI, vol. 200, pp. 481–492. GI, 2012. ISBN 978-3-88579-294-9.

Hilbrich M., Müller-Pfefferkorn R.: Cross-Correlation as Tool to Determine the Similarity of Series of Measurements for Big-Data Analysis Tasks. In: accepted for 2015 International Conference on Cloud Computing and Big Data (CloudComAsia). 2015.

Hilbrich M., Weber M., Tschüter R.: Automatic Analysis of Large Data Sets: A Walk-Through on Methods from Different Perspectives. In: Cloud Computing and Big Data (CloudCom-Asia), 2013 International Conference on, pp. 373–380. 2013. URL http://dx.doi.org/10.1109/CLOUDCOM-ASIA.2013.47.

Höcker A., Lacker H., Laplace S., Le Diberder F.: A new approach to a global fit of the CKM matrix. In: The European Physical Journal C - Particles and Fields, vol. 21(2), pp. 225–259, 2001. ISSN 1434-6044. URL http://dx.doi.org/10. 1007/s100520100729.

Hoefler T., Schneider T., Lumsdaine A.: Characterizing the Influence of System Noise on Large-Scale Applications by Simulation. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’10, pp. 1–11. IEEE Computer Society, Washington, DC, USA, 2010. ISBN 978-1-4244-7559-9. URL http://dx.doi. org/10.1109/SC.2010.12.

Holland J.: Genetic Algorithms. In: Scientific American, vol. 267(1), 1992.

Lazarevic A., Ertoz L., Kumar V., Ozgur A., Srivastava J.: A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection. In: D. Barbará, C. Kamath, eds., Proceedings of SIAM Conference on Data Mining. 2003.

Lee W., Stolfo S., Mok K.: A data mining framework for building intrusion detection models. In: Security and Privacy, 1999. Proceedings of the 1999 IEEE Symposium on, pp. 120–132. 1999. ISSN 1081-6011. URL http://dx.doi.org/ 10.1109/SECPRI.1999.766909.

Lee W., Stolfo S.J.: Data Mining Approaches for Intrusion Detection. In: Proceedings of the 7th conference on USENIX Security Symposium - Volume 7, SSYM’98, pp. 6–6. USENIX Association, Berkeley, CA, USA, 1998. URL http://dl.acm.org/citation.cfm?id=1267549.1267555.

Lee W., Stolfo S.J.: A framework for constructing features and models for intrusion detection systems. In: ACM Transactions on Information and System Security, vol. 3(4), pp. 227–261, 2000. ISSN 1094-9224. URL http://dx.doi.org/10.1145/382912.382914.

Lorenz D., Borovac S., Buchholz P., Eichenhardt H., Harenberg T., Mättig P., Mechtel M., Müller-Pfefferkorn R., Neumann R., Reeves K., Uebing C., Walkowiak W., William T., Wismüller R.: Job monitoring and steering in DGrid’s High Energy Physics Community Grid. In: Future Gener. Comput. Syst., vol. 25, pp. 308–314, 2009. ISSN 0167-739X. URL http://dx.doi.org/http: //dx.doi.org/10.1016/j.future.2008.05.009.

Müller-Pfefferkorn R., Neumann R., William T.: AMon - a User-Friendly Job Monitoring for the Grid. In: T. Priol, M. Vanneschi, eds., CoreGRID, pp. 185– 192. Springer, 2007. ISBN 978-0-387-72497-3.

Myers E.W.: An O(ND) difference algorithm and its variations. In: Algorithmica, vol. 1, pp. 251–266, 1986.

Needleman S.B., Wunsch C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. In: Journal of Molecular Biology, vol. 48(3), pp. 443–453, 1970.

Paxson V.: Bro: a system for detecting network intruders in real-time. In: Computer Networks, vol. 31(2324), pp. 2435 – 2463, 1999. ISSN 1389-1286. URL http://dx.doi.org/10.1016/S1389-1286(99)00112-7.

Roesch M., Telecommunications S.: Snort - Lightweight Intrusion Detection for Networks. pp. 229–238. 1999.

Smaha S.E.: Haystack: An intrusion detection system. In: Proc. of the IEEE 4th Aerospace Computer Security Applications Conference. 1988.

Tang K., Man K., Kwong S., He Q.: Genetic algorithms and their applications. In: Signal Processing Magazine, IEEE, vol. 13(6), pp. 22–37, 1996. ISSN 10535888. URL http://dx.doi.org/10.1109/79.543973.