Skip to Main Content
Skip Nav Destination

Bibliometrics - methods to quantitatively analyse the quality and impact of scientific or technical literature - are now a central part of the management of modern science. Through them, research managers seek to encourage quality and productivity and use scarce research funds effectively. Researchers are ranked on a range of quantitative assessments to measure the quality of their work, and the results influence employment prospects, grants, tenure and promotions. Unfortunately, researchers anxious to maximise their prospects may concentrate on good scores, not good science. This could change what they research, what they publish and where they publish. Natural history, baseline research and research of regional (but not international) significance could be marginalised despite the clear benefits of such research for monitoring, hypothesis generation and local management. These difficulties are compounded by inappropriate applications of common bibliometric statistics, such as the persistence of the discredited views that the quality of a paper may be judged by the journal in which it appears or that a simple citation count alone indicates the merit of a paper or a researcher. This paper takes a role-playing approach, centred on a fictitious interview as part of a promotion application, to explore some of the uses and misuses of bibliometrics and how researchers can present their case honestly, while defending against abuses and championing unfashionable but valuable areas of research.

Abbott, I., 1998. Conservation of the forest red-tailed black cockatoo, a hollow dependent species, in the eucalypt forests of Western Australia. Forest Ecology and Management 109: 175-185.
Abramo, G., Cicero, T. and D'Angelo, C. A., 2013. National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case. Scientometrics 95: 311-324.
Adler, R., Ewing, J. and Taylor, P., 2008. Citation statistics: A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Accessed 28th August, 2012.
Adler, N.J. and Harzing, A.-W., 2009. When knowledge wins: transcending the sense and nonsense of academic rankings. Academy of Management Learning and Education 8: 72-95.
Aksnes, D. W. and Rip, A., 2009. Researchers’ perceptions of citations. Research Policy 38: 895-905.
Amancio, D. R., Nunes, M. G. V., Oliveira Jr, O. N. and da Costa, L. F., 2012. Using complex networks concepts to assess approaches for citations in scientific papers. Scientometrics 91: 827-842.
Andersen, H. and 71 others. 2009. Journals under threat: A joint response from history of science, technology, and medicine editors. Science in Context 22: 1-4. (Also published in 54 other journals).
Ashkanasy, N. M., 2007. Playing the citations game. Journal of Organizational Behavior 28: 643-645.
Bensman, S. J., 2012. The impact factor: Its place in Garfield's thought, in science evaluation, and in library collection management. Scientometrics 92: 263-275.
Bloch, S. and Walter, G., 2001. The impact factor: time for change. Australian and New Zealand Journal of Psychiatry 35: 563-568.
Bollen, J., Van de Sompel, H., Hagberg, A. and Chute, R., 2009. A principal component analysis of 39 scientific impact measures. PLoS ONE 4: article number e6022.
Box, S., 2010. Performance-based funding for public research in tertiary education institutions: Country experiences. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, OECD Publishing.
Brown, H., 2007. How impact factors changed medical publishing - And science. British Medical Journal 334: 561-564.
Bryant, K. and Calver, M., 2012. Adaptive radiation in Australian journals in the Arbustocene ERA: an empty niche for JANCO? In Science under siege Eds. P.B. Banks, D. Lunney and C.R. Dickman, pp. 140-149. Sydney, Royal Zoological Society of New South Wales.
Buela-Casal, G. and Zych, I., 2012. What do the scientists think about the impact factor? Scientometrics 92: 281-292.
Buela-Casal, G., Zych, I., Sierra, J. C. and Bermûdez, M. P., 2007. The Internationality Index of the Spanish Psychology journals. International Journal of Clinical and Health Psychology 7: 899-910.
Butler, L., 2007. Assessing university research: a plea for a balanced approach. Science and Public Policy 34: 565-574.
Butler, L., 2010. Impacts of performance-based research funding systems: a review of the concerns and the evidence. In OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, OECD Publishing.
Butler, L., 2011. The devil is in the detail: Concerns about Vanclay's analysis of Australian journal rankings. Journal of Informetrics 5: 693-694.
Calver, M., Wardell-Johnson, G., Bradley, S. and Taplin, R., 2010. What makes a journal international? A case study using conservation biology journals. Scientometrics 85: 387-400.
Calver, M. C. and Bradley, J. S., 2009. Should we use the mean citations per paper to summarise a journal's impact or to rank journals in the same field? Scientometrics 81: 611-615.
Calver, M.C. and Bradley, J.S., 2010. Patterns of citations in open access and non-open access conservation biology journal papers and book chapters. Conservation Biology 24: 872-880.
Calver, M. C. and Bryant, K. A., 2008. Pacific Conservation Biology: An authorship and citation analysis. Pacific Conservation Biology 14: 285-303.
Calver, M. C., Beatty, S.J., Bryant, K.A., Dickman, C.R., Ebner, B.C. and Morgan, D.L., 2013a. Users beware: implications of database errors when assessing the individual research records of ecologists and conservation biologists. Pacific Conservation Biology 19: in press.
Calver, M. C., Fontaine, J. and Linke, T.E., 2013b. Bibliometric analysis of books and book chapters: a case study using publications by Surrey Beatty & Sons. Pacific Conservation Biology 19: in press.
Calver, M.C., Lilith, M. and Dickman, C.R., 2013c. A ‘perverse incentive’ from bibliometrics: could National Research Assessment Exercises (NRAEs) restrict literature availability for nature conservation? Scientometrics 93: 243-255.
Calver, M. C., O'Brien, P. A. and Lilith, M., 2012. Australasian plant Pathology: An analysis of authorship and citations in the 21st century. Australasian Plant Pathology 41: 179-187.
Chang, C. L. and McAleer, M., 2012. Citations and impact of ISI tourism and hospitality journals. Tourism Management Perspectives 1: 2-8.
Colledge, L., De Moya-Anegón, F., Guerrero-Bote, V., López-Illescas, C., El Aisati, M. and Moed, H.F., 2010. SJR and SNIP: two new journal metrics in Elsevier's Scopus. Serials 23: 215-221.
Corsi, M., D'Ippoliti, C. and Lucidi, F., 2010. Pluralism at risk? Heterodox economic approaches and the evaluation of economic research in Italy. American Journal of Economics and Sociology 69: 1495-1529.
D'Angelo, C. A., Giuffrida, C. and Abramo, G., 2011. A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology 62: 257-269.
De Sutter, B. and Van Den Oord, A., 2012. To be or not to be cited in computer science. Communications of the ACM 55: 69-75.
Elsevier, 2011. SciVerse Scopus: content coverage guide. Available from Accessed February 10, 2013.
Erskine, A. J., 1994. Editorial. Picoides 7: 2.
Jain, P. 2011. New trends and future applications/directions of institutional repositories in academic institutions. Library Review 60: 125-141.
Falagas, M. E. and Alexiou, V. G., 2008. The top-ten in journal impact factor manipulation. Archivum Immunologiae et Therapiae Experimentalis 56: 223-226.
Ford, H. A., 2011. Twinkling lights or turning down the dimmer switch? Are there two patterns of extinction debt in fragmented landscapes? Pacific Conservation Biology 17: 303-309.
Gagolewski, M., 2011. Bibliometric impact assessment with R and the CITAN package. Journal of Informetrics 5: 678-692.
García-Pérez, M. A., 2010. Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google scholar: A case study for the computation of h indices in psychology. Journal of the American Society for Information Science and Technology 61: 2070-2085.
Garfield, E., 2005. The agony and the ecstasy-the history and meaning of the journal impact factor. Paper presented at the International Congress on Peer Review and Biomedical Publication Chicago, September 16, 2005. Accessed 28th August 2012.
Gowrishankar, J. and Divakar, P., 1999. Sprucing up one's impact factor (multiple letters). Nature 401 (6751): 321-322.
Hargens, L. L. and Schuman, H., 1990. Citation counts and social comparisons: Scientists’ use and evaluation of citation index data. Social Science Research 19: 205-221.
Harrison, A. L., 2006. Who's who in conservation biology - An authorship analysis. Conservation Biology 20: 652-657.
Harzing, A. W. K. and van der Wal, R. 2008. Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics 8: 61-73.
Hirsch, J. E., 2005. An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102: 16569-16572.
Hodge, D. R. and Lacasse, J. R., 2011. Ranking disciplinary journals with the Google Scholar h-index: A new tool for constructing cases for tenure, promotion, and other professional decisions. Journal of Social Work Education 47: 579-596.
Holden, G., Rosenberg, G. and Barker, K., 2005. Bibliometrics: A potential decision making aid in hiring, reappointment, tenure and promotion decisions. Social Work in Health Care 41: 67-92.
Jacsó, P., 2005. As we may search-comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science 89: 1537-1547.
Jacsó, P., 2008a. Testing the calculation of a realistic h-index in Google Scholar, Scopus, and Web of Science for F. W. Lancaster. Library Trends 56: 784-815.
Jacsó, P., 2008b. Google Scholar's ghost authors. Library Journal 134: 26-27.
Jacsó, P., 2009. Calculating the h-index and other bibliometric and scientometric indicators from Google Scholar with the Publish or Perish software. Online Information Review 33: 1189-1200.
Jacsó, P., 2012. Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: reflections on Vanclay's criticism. Scientometrics 92: 325-354.
Kennedy, D., 2003. Multiple authors, multiple problems. Science 301(5634): 733.
Kohn, A., 1993. Punished by rewards: the trouble with gold stars, incentive plans, A's, praise and other bribes. Houghton Mifflin Company, Boston.
Krell, F. T., 2012. The Journal Impact Factor as a performance indicator. European Science Editing 38: 3-5.
Lane, J., 2010. Let's make science metrics more scientific. Nature 464: 488-489.
Lane, J. and Bertuzzi, S., 2011. Measuring the results of science investments. Science 331: 678-680.
Lawrence, P. A., 2002. Rank injustice: the misallocation of credit is endemic in science. Nature 415: 835-836
Lawrence, P. A., 2003. The politics of publication. Nature 422 (6929): 259-261.
Lawrence, P. A., 2007. The mismeasurement of science. Current Biology 17: (15).
Lawrence, P. A., 2009. Real lives and white lies in the funding of scientific research: The granting system turns young scientists into bureaucrats and then betrays them. PLoS Biology 7:
Li, J., Sanderson, M., Willett, P., Norris, M. and Oppenheim, C., 2010. Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments. Journal of Informetrics 4: 554-563.
MacRoberts, M. H. and MacRoberts, B. R., 1996. Problems of citation analysis. Scientometrics 36: 435-444.
MacRoberts, M. H. and MacRoberts, B. R., 2010. Problems of citation analysis: A study of uncited and seldom-cited influences. Journal of the American Society for Information Science and Technology 61: 1-12.
Marsh, H., Smith, B., King, M. and Evans, T., 2012. A new era for research education in Australia? Australian Universities’ Review 54, 83-93.
Meffe, G. K., 2006. The success - and challenges - of conservation biology. Conservation Biology 20: 931-933.
NHMRC, 2007. Australian code for the responsible conduct of research. National Health and Medical Research Council, Australian Research Council and Universities Australia. (Accessed September 2012).
Northcott, D. and Linacre, S., 2010. Producing spaces for academic discourse: The impact of research assessment exercises and journal quality rankings. Australian Accounting Review 20: 38-54.
OECD, 2010. Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, OECD Publishing.
Ouimet, M., Bédard, P. O. and Gélineau, F., 2011. Are the h-index and some of its alternatives discriminatory of epistemological beliefs and methodological preferences of faculty members? The case of social scientists in Quebec. Scientometrics 88: 91-106.
Oswald, A.J., 2010. A suggested method for the measurement of world-leading research (illustrated with data on economics). Scientometrics 84: 99-113.
Padial, A. A., Nabout, J. C., Siqueira, T., Bini, L. M. and Diniz-Filho, J. A. F., 2010. Weak evidence for determinants of citation frequency in ecological articles. Scientometrics 85: 1-12.
Parker, J. N., Allesina, S. and Lortie, C. J., 2013. Characterizing a scientific elite (B): Publication and citation patterns of the most highly cited scientists in environmental science and ecology. Scientometrics 94: 469-480.
Perakakis, P., Taylor, M., Buela-Casal, P. and Checa, P., 2006. A neuro-fuzzy system to calculate a journal internationality index. Proceedings CEDI 2005 Symposium. (Accessed September, 2012)
Primack, R. B., 2009. Why did we reject your paper? Biological Conservation 142: 1559.
Rafols, I., Leydesdorff, L., O'Hare, A., Nightingale, P. and Stirling, A., 2012. How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy 41: 1262-1282.
Roa, T., Beggs, J. R., Williams, J. and Moller, H., 2009. New Zealand's performance based research funding (PBRF) model undermines maori research. Journal of the Royal Society of New Zealand 39: 233-238.
Sampson, K. A. and Comer, K., 2010. When the governmental tail wags the disciplinary dog: Some consequences of national funding policy on doctoral research in New Zealand. Higher Education Research and Development 29: 275-289.
Sangwal, K., 2012. Application of progressive nucleation mechanism for the citation behavior of individual papers of different authors. Scientometrics 92: 643-655.
Schiemenz, H., 1969. Die Zikadenfauna mitteleuropäischer Trockenrasen (Homoptera, Auchenorrhyncha): Untersuchungen zu ihrer Phänologie, ökologie, Bionomie und Chorologie. Faunistische Abhandlungen Staatliches Museum für Tierkunde in Dresden 6: 201-280.
Schuch, S., Wesche, K. and Schaefer, M., 2012. Long-term decline in the abundance of leafhoppers and planthoppers (Auchenorrhyncha) in Central European protected dry grasslands. Biological Conservation 149: 75-83.
Schuch, S., 2012. Archives and conservation biology. Pacific Conservation Biology 18: 223-224.
Seglen, P.O., 1997. Why the impact factor of journals should not be used for evaluating research. British Medical Journal 314 (7079): 498-502.
Shearer, B. L., Shea, S. R. and Fairman, R. G., 1981. Infection of the stem and large roots of Eucalyptus marginata by Phytophthora cinnamomi. Australasian Plant Pathology 10: 2-3.
Shewan, L. G. and Coats, A. J. S., 2006. The research quality framework and its implications for health and medical research: time to take stock? Medical Journal of Australia 184: 463-466.
Short, J., 2004. Mammal decline in southern Western Australia - Perspectives from Shortridge's collections of mammals in 1904-07. Australian Zoologist 32: 605-624.
Shortridge, G. C., 1909. An account of the geographical distribution of the marsupials and monotremes of south-west Australia, having special reference to the specimens collected during the Balston Expedition of 1904-1907. Proceedings of the Zoological Society of London Series B, Biological Sciences: 803-848.
Steele, C., Butler, L. and Kingsley, D., 2006. The publishing imperative: The pervasive influence of publication metrics. Learned Publishing 19 (4): 277-290.
Thomson Reuters, 2011. Completing the research picture. Book Citation Index in Web of Science. Available from Accessed January 13th, 2013.
Torres-Salinas, D. and Moed, H. F., 2009. Library Catalog Analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in Economics. Journal of Informetrics 3: 9-26.
Underwood, A.J., 1997. Experiments in ecology: their logical design and interpretation using analysis of variance. Cambridge University Press, Cambridge.
Vanclay, J. K., 2012. Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics 92: 211-238.
Walters, W. H., 2011. Comparative recall and precision of simple and expert searches in google scholar and eight other databases. Portal 11: 972-1006.
Watson, R., Annells, M., Amella, E., and Wong, T. (2007), What makes a JCN paper international? Journal of Clinical Nursing, 16: 1-2.
White, H. D., Boell, S. K., Yu, H., Davis, M., Wilson, C. S. and Cole, F. T. H., 2009. Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences. Journal of the American Society for Information Science and Technology 60: 1083-1096.
Witten, K. and Hammond, K., 2010. What becomes of social science knowledge: New Zealand researchers’ experiences of knowledge transfer modes and audiences. Kotuitui 5: 3-12.
Zahavi, A., 1975. Mate selection: a selection for a handicap. Journal of Theoretical Biology 53: 205-214.
This content is PDF only. Please click on the PDF icon to access.
Close Modal

or Create an Account

Close Modal
Close Modal