Competition-based rating system for medical website credibility

Grzegorz Kowalik

doi:10.7494/csci.2015.16.3.265

Authors

Grzegorz Kowalik Polish-Japanese Academy of Information Technology, Warsaw

DOI:

https://doi.org/10.7494/csci.2015.16.3.265

Keywords:

rating systems, Elo rating, credibility of online content, web credibility, rating aggregation

Abstract

In this paper, we propose a new approach to the aggregation of monadic ratings (5-step scale) done by crowdsourcing users for the evaluation of medical websites. We compare them pairwise with other evaluations done by the same users for other websites (whether they are higher or lower), and we will use an Elo rating algorithm to calculate website “credibility” values. Results show that this method of crowdsourcing evaluation is highly correlated with expert evaluations. As proposed, a competition-based model uses a 5-step scale as ordinal and only compares which website is rated higher or lower by the same user. This approach can solve many problems associated with a 5-point scale, such as different understanding by users, user bias, and distribution skewness that can be clearly observed in results.

Downloads

Download data is not yet available.

References

Borzymek P., Sydow M., Wierzbicki A.: Enriching trust prediction model in social network with user rating similarity. In: Computational Aspects of Social Networks, 2009. CASON’09. International Conference on, pp. 40–47, IEEE, 2009.

Elo A. E.: The rating of chessplayers, past and present, vol. 3, Batsford, London, 1978.

Glickman M. E.: The glicko system. Boston University, 1995.

Greco S., Matarazzo B., Słowiński R.: Decision rule approach. In: Multiple criteria decision analysis: state of the art surveys, pp. 507–555, Springer, 2005.

Greco S., Matarazzo B., Słowiński R.: Rough sets theory for multicriteria decision analysis. European Journal of Operational Research, vol. 129(1), pp. 1–47, 2001.

Greco S., Matarazzo B., Słowiński R.: Dominance-based rough set approach to case-based reasoning. In: Modeling Decisions for Artificial Intelligence, pp. 7–18, Springer, 2006.

Greco S., Matarazzo B., Słowiński R.: Dominance-based rough set approach on pairwise comparison tables to decision involving multiple decision makers. In: Rough Sets and Knowledge Technology, pp. 126–135, Springer, 2011.

Herbrich R., Minka T., Graepel T.: Trueskill TM: A Bayesian skill rating system. In: Advances in Neural Information Processing Systems, pp. 569–576, Cambridge, MA, USA, 2006.

Hinnant N. C.: Practicing Work, Perfecting Play: League of Legends and the Sentimental Education of E-Sports, 2013.

Hvattum L.M., Arntzen H.: Using ELO ratings for match result prediction in association football. International Journal of Forecasting, vol. 26(3), pp. 460–470, 2010.

Juźwin M., Adamska P., Rafalak M., Balcerzak B., Kąkol M., Wierzbicki A.: Threats of Using Gamification for Motivating Web Page Quality Evaluation. In: Proceedings of the 2014 Mulitmedia, Interaction, Design and Innovation International Conference on Multimedia, Interaction, Design and Innovation, pp. 1–8, ACM, 2014.

Kąkol M., Jankowski-Lorek M., Abramczuk K., Wierzbicki A., Catasta M.: On the subjectivity and bias of web content credibility evaluations. In: Proceedings of the 22nd international conference on World Wide Web companion, pp. 1131–1136, International World Wide Web Conferences Steering Committee, 2013.

Kaszuba T., Hupa A., Wierzbicki A.: Advanced feedback management for internet auction reputation systems. Internet Computing, IEEE, vol. 14(5), pp. 31–37, 2010.

Kowalik G., Adamska P., Nielek R., Wierzbicki A.: Simulations of Credibility Evaluation and Learning in a Web 2.0 Community. In: Artificial Intelligence and Soft Computing, pp. 373–384, Springer, 2014.

Liu J., Song Y. I., Lin C. Y.: Competition-based user expertise score estimation. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 425–434, ACM, 2011.

Louviere J. J., Islam T.: A comparison of importance weights and willingness-to-pay measures derived from choice-based conjoint, constant sum scales and best–worst scaling. Journal of Business Research, vol. 61(9), pp. 903–911, 2008.

Morzy M., Wierzbicki A.: The sound of silence: Mining implicit feedbacks to compute reputation. In: Internet and Network Economics, pp. 365–376. Springer, 2006.

Orme B.: Scaling multiple items: monadic ratings vs. paired comparisons. In: Sawtooth software conference proceedings, Sequim, pp. 43–59, Sequim, WA, USA, 2003.

Papaioannou T. G., Aberer K., Abramczuk K., Adamska P., Wierzbicki A.: Game-theoretic models of web credibility. In: Proceedings of the 2nd Joint WI-COW/AIRWeb Workshop on Web Quality, pp. 27–34, ACM, 2012.

Słowiński R., Greco S., Matarazzo B.: Rough sets in decision making. In: Encyclopedia of complexity and systems science, pp. 7753–7787, Springer, 2009.

Słowiński R., Greco S., Matarazzo B.: Rough-set-based decision support. In: Search Methodologies, pp. 557–609, Springer, 2014.

Wierzbicki A.: The case for fairness of trust management. Electronic Notes in Theoretical Computer Science, vol. 197(2), pp. 73–89, 2008.