@article{3106365,
    title = "Large expert-curated database for benchmarking document similarity detection in biomedical literature search",
    author = "Brown, P. and Zhou, Y. and Tan, A.-C. and El-Esawi, M.A. and Liehr, T. and Blanck, O. and Gladue, D.P. and Almeida, G.M.F. and Cernava, T. and Sorzano, C.O. and Yeung, A.W.K. and Engel, M.S. and Chandrasekaran, A.R. and Muth, T. and Staege, M.S. and Daulatabad, S.V. and Widera, D. and Zhang, J. and Meule, A. and Honjo, K. and Pourret, O. and Yin, C.-C. and Zhang, Z. and Cascella, M. and Flegel, W.A. and Goodyear, C.S. and van Raaij, M.J. and Bukowy-Bieryllo, Z. and Campana, L.G. and Kurniawan, N.A. and Lalaouna, D. and Hüttner, F.J. and Ammerman, B.A. and Ehret, F. and Cobine, P.A. and Tan, E.-C. and Han, H. and Xia, W. and McCrum, C. and Dings, R.P.M. and Marinello, F. and Nilsson, H. and Nixon, B. and Voskarides, K. and Yang, L. and Costa, V.D. and Bengtsson-Palme, J. and Bradshaw, W. and Grimm, D.G. and Kumar, N. and Martis, E. and Prieto, D. and Sabnis, S.C. and Amer, S.E.D.R. and Liew, A.W.C. and Perco, P. and Rahimi, F. and Riva, G. and Zhang, C. and Devkota, H.P. and Ogami, K. and Basharat, Z. and Fierz, W. and Siebers, R. and Tan, K.H. and Boehme, K.A. and Brenneisen, P. and Brown, J.A.L. and Dalrymple, B.P. and Harvey, D.J. and Ng, G. and Werten, S. and Bleackley, M. and Dai, Z. and Dhariwal, R. and Gelfer, Y. and Hartmann, M.D. and Miotla, P. and Tamaian, R. and Govender, P. and Gurney-Champion, O.J. and Kauppila, J.H. and Zhang, X. and Echeverría, N. and Subhash, S. and Sallmon, H. and Tofani, M. and Bae, T. and Bosch, O. and Cuív, P.O. and Danchin, A. and Diouf, B. and Eerola, T. and Evangelou, E. and Filipp, F. and Klump, H. and Kurgan, L. and Smith, S.S. and Terrier, O. and Tuttle, N. and Ascher, D.B. and Janga, S.C. and Schulte, L.N. and Becker, D. and Browngardt, C. and Bush, S.J. and Gaullier, G. and Ide, K. and Meseko, C. and Werner, G.D.A. and Zaucha, J. and Al-Farha, A.A. and Greenwald, N.F. and Popoola, S.I. and Rahman, S. and Xu, J. and Yang, S.Y. and Hiroi, N. and Alper, O.M. and Baker, C.I. and Bitzer, M. and Chacko, G. and Debrabant, B. and Dixon, R. and Forano, E. and Gilliham, M. and Kelly, S. and Klempnauer, K.-H. and Lidbury, B.A. and Lin, M.Z. and Lynch, I. and Ma, W. and Maibach, E.W. and Mather, D.E. and Nandakumar, K.S. and Ohgami, R.S. and Parchi, P. and Tressoldi, P. and Xue, Y. and Armitage, C. and Barraud, P. and Chatzitheochari, S. and Coelho, L.P. and Diao, J. and Doxey, A.C. and Gobet, A. and Hu, P. and Kaiser, S. and Mitchell, K.M. and Salama, M.F. and Shabalin, I.G. and Song, H. and Stevanovic, D. and Yadollahpour, A. and Zeng, E. and Zinke, K. and Alimba, C.G. and Beyene, T.J. and Cao, Z. and Chan, S.S. and Gatchell, M. and Kleppe, A. and Piotrowski, M. and Torga, G. and Woldesemayat, A.A. and Cosacak, M.I. and Haston, S. and Ross, S.A. and Williams, R. and Wong, A. and Abramowitz, M.K. and Effiong, A. and Lee, S. and Abid, M.B. and Agarabi, C. and Alaux, C. and Albrecht, D.R. and Atkins, G.J. and Beck, C.R. and Bonvin, A.M.J.J. and Bourke, E. and Brand, T. and Braun, R.J. and Bull, J.A. and Cardoso, P. and Carter, D. and Delahay, R.M. and Ducommun, B. and Duijf, P.H.G. and Epp, T. and Eskelinen, E.-L. and Fallah, M. and Farber, D.B. and Fernandez-Triana, J. and Feyerabend, F. and Florio, T. and Friebe, M. and Furuta, S. and Gabrielsen, M. and Gruber, J. and Grybos, M. and Han, Q. and Heinrich, M. and Helanterä, H. and Huber, M. and Jeltsch, A. and Jiang, F. and Josse, C. and Jurman, G. and Kamiya, H. and de Keersmaecker, K. and Kristiansson, E. and de Leeuw, F.-E. and Li, J. and Liang, S. and Lopez-Escamez, J.A. and Lopez-Ruiz, F.J. and Marchbank, K.J. and Marschalek, R. and Martín, C.S. and Miele, A.E. and Montagutelli, X. and Morcillo, E. and Nicoletti, R. and Niehof, M. and O'Toole, R. and Ohtomo, T. and Oster, H. and Palma, J.-A. and Paterson, R. and Peifer, M. and Portilla, M. and Portillo, M.C. and Pritchard, A.L. and Pusch, S. and Raghava, G.P.S. and Roberts, N.J. and Ross, K. and Schuele, B. and Sergeant, K. and Shen, J. and Stella, A. and Sukocheva, O. and Uversky, V.N. and Vanneste, S. and Villet, M.H. and Viveiros, M. and Vorholt, J.A. and Weinstock, C. and Yamato, M. and Zabetakis, I. and Zhao, X. and Ziegler, A. and Aizat, W.M. and Atlas, L. and Bridges, K.M. and Chakraborty, S. and Deschodt, M. and Domingues, H.S. and Esfahlani, S.S. and Falk, S. and Guisado, J.L. and Kane, N.C. and Kueberuwa, G. and Lau, C.L. and Liang, D. and Liu, E. and Luu, A.M. and Ma, C. and Ma, L. and Moyer, R. and Norris, A.D. and Panthee, S. and Parsons, J.R. and Peng, Y. and Pinto, I.M. and Reschke, C.R. and Sillanpää, E. and Stewart, C.J. and Uhle, F. and Yang, H. and Zhou, K. and Zhu, S. and Ashry, M. and Bergsland, N. and Berthold, M. and Chen, C.-E. and Colella, V. and Cuypers, M. and Eskew, E.A. and Fan, X. and Gajda, M. and Gonzálezlez-Prendes, R. and Goodin, A. and Graham, E.B. and Groen, E.J.N. and Gutiérrez-Sacristán, A. and Habes, M. and Heffler, E. and Higginbottom, D.B. and Janzen, T. and Jayaraman, J. and Jibb, L.A. and Jongen, S. and Kinyanjui, T. and Koleva-Kolarova, R.G. and Li, Z. and Liu, Y.-P. and Lund, B.A. and Lussier, A.A. and Ma, L. and Mier, P. and Moore, M.D. and Nagler, K. and Orme, M.W. and Pearson, J.A. and Prajapati, A.S. and Saito, Y. and Tröder, S.E. and Uchendu, F. and Verloh, N. and Voutchkova, D.D. and Abu-Zaid, A. and Bakkach, J. and Baumert, P. and Dono, M. and Hanson, J. and Herbelet, S. and Hobbs, E. and Kulkarni, A. and Kumar, N. and Liu, S. and Loft, N.D. and Reddan, T. and Senghore, T. and Vindin, H. and Xu, H. and Bannon, R. and Chen, B. and Cheung, J.T.K. and Cooper, J. and Esnakula, A.K. and Feghali, K.A. and Ghelardi, E. and Gnasso, A. and Horbar, J. and Lai, H.M. and Li, J. and Ma, L. and Ma, R. and Pan, Z. and Peres, M.A. and Pranata, R. and Seow, E. and Sydes, M. and Testoni, I. and Westermair, A.L. and Yang, Y. and Afnan, M. and Albiol, J. and Albuquerque, L.G. and Amir, S. and Amiya, E. and Amorim, R.M. and An, Q. and Andersen, S.U. and Aplin, J.D. and Argyropoulos, C. and Asmann, Y.W. and Assaeed, A.M. and Atanasov, A.G. and Atchison, D.A. and Avery, S.V. and Avillach, P. and Baade, P.D. and Backman, L. and Badie, C. and Baldi, A. and Ball, E. and Bardot, O. and Barnett, A.G. and Basner, M. and Batra, J. and Bazanova, O.M. and Beale, A. and Beddoe, T. and Bell, M.L. and Berezikov, E. and Berners-Price, S. and Bernhardt, P. and Berry, E. and Bessa, T.B. and Billington, C. and Birch, J. and Blakely, R.D. and Blaskovich, M.A.T. and Blum, R. and Boelaert, M. and Bogdanos, D. and Bosch, C. and Bourgoin, T. and Bouvard, D. and Boykin, L.M. and Bradley, G. and Braun, D. and Brownlie, J. and Brühl, A. and Burt, A. and Butler, L.M. and Byrareddy, S.N. and Byrne, H.J. and Cabantous, S. and Calatayud, S. and Candal, E. and Carlson, K. and Casillas, S. and Castelvetro, V. and Caswell, P.T. and Cavalli, G. and Cerovsky, V. and Chagoyen, M. and Chen, C.-S. and Chen, D.F. and Chen, H. and Chen, H. and Chen, J.-T. and Chen, Y. and Cheng, C. and Cheng, J. and Chinapaw, M. and Chinopoulos, C. and Cho, W.C.S. and Chong, L. and Chowdhury, D. and Chwalibog, A. and Ciresi, A. and Cockcroft, S. and Conesa, A. and Cook, P.A. and Cooper, D.N. and Coqueret, O. and Corea, E.M. and Costa, A. and Costa, E. and Coupland, C. and Crawford, S.Y. and Cruz, A.D. and Cui, H. and Cui, Q. and Culver, D.C. and D'Angiulli, A. and Dahms, T.E.S. and Daigle, F. and Dalgleish, R. and Danielsen, H.E. and Darras, S. and Davidson, S.M. and Day, D.A. and Degirmenci, V. and Demaison, L. and Devriendt, K. and Ding, J. and Dogan, Y. and Dong, X.C. and Donner, C.F. and Dressick, W. and Drevon, C.A. and Duan, H. and Ducho, C. and Dumaz, N. and Dwarakanath, B.S. and Ebell, M.H. and Eisenhardt, S. and Elkum, N. and Engel, N. and Erickson, T.B. and Fairhead, M. and Faville, M.J. and Fejzo, M.S. and Festa, F. and Feteira, A. and Flood-Page, P. and Forsayeth, J. and Fox, S.A. and Franks, S.J. and Frentiu, F.D. and Frilander, M.J. and Fu, X. and Fujita, S. and Galea, I. and Galluzzi, L. and Gani, F. and Ganpule, A.P. and García-Alix, A. and Gedye, K. and Giordano, M. and Giunta, C. and Gleeson, P.A. and Goarant, C. and Gong, H. and Gora, D. and Gough, M.J. and Goyal, R. and Graham, K.E. and Grande-Pérez, A. and Graves, P.M. and Greidanus, H. and Grice, D. and Grunau, C. and Gumulya, Y. and Guo, Y. and Gurevich, V.V. and Gusev, O. and Hacker, E. and Hage, S.R. and Hagen, G. and Hahn, S. and Haller, D.M. and Hammerschmidt, S. and Han, J. and Han, R. and Handfield, M. and Hapuarachchi, H.C. and Harder, T. and Hardingham, J.E. and Heck, M. and Heers, M. and Hew, K.F. and Higuchi, Y. and Hilaire, C.St. and Hilton, R. and Hodzic, E. and Hone, A. and Hongoh, Y. and Hu, G. and Huber, H.P. and Hueso, L.E. and Huirne, J. and Hurt, L. and Idborg, H. and Ikeo, K. and Ingley, E. and Jakeman, P.M. and Jensen, A. and Jia, H. and Jia, H. and Jia, S. and Jiang, J. and Jiang, X. and Jin, Y. and Jo, D. and Johnson, A.M. and Johnston, M. and Jonscher, K.R. and Jorens, P.G. and Jorgensen, J.O.L. and Joubert, J.W. and Jung, S.-H. and Junior, A.M. and Kahan, T. and Kamboj, S.K. and Kang, Y.-K. and Karamanos, Y. and Karp, N.A. and Kelly, R. and Kenna, R. and Kennedy, J. and Kersten, B. and Khalaf, R.A. and Khalid, J.M. and Khatlani, T. and Khider, T. and Kijanka, G.S. and King, S.R.B. and Kluz, T. and Knox, P. and Kobayashi, T. and Koch, K.-W. and Kohonen-Corish, M.R.J. and Kong, X. and Konkle-Parker, D. and Korpela, K.M. and Kostrikis, L.G. and Kraiczy, P. and Kratz, H. and Krause, G. and Krebsbach, P.H. and Kristensen, S.R. and Kumari, P. and Kunimatsu, A. and Kurdak, H. and Kwon, Y.D. and Lachat, C. and Lagisz, M. and Laky, B. and Lammerding, J. and Lange, M. and Larrosa, M. and Laslett, A.L. and Laverman, G.D. and Leclair, E.E. and Lee, K.-W. and Lee, M.-Y. and Lee, M.-S. and Li, G. and Li, J. and Lieb, K. and Lim, Y.Y. and Lindsey, M.L. and Line, P.-D. and Liu, D. and Liu, F. and Liu, H. and Liu, H. and Lloyd, V.K. and Lo, T.-W. and Locci, E. and Loidl, J. and Lorenzen, J. and Lorkowski, S. and Lovell, N.H. and Lu, H. and Lu, W. and Lu, Z. and Luengo, G.S. and Lundh, L.-G. and Lysy, P.A. and Mabb, A. and Mack, H.G. and Mackey, D.A. and Mahdavi, S.R. and Maher, P. and Maher, T. and Maity, S.N. and Malgrange, B. and Mamoulakis, C. and Mangoni, A.A. and Manke, T. and Manstead, A.S.R. and Mantalaris, A. and Marsal, J. and Marschall, H.-U. and Martin, F.L. and Martinez-Raga, J. and Martinez-Salas, E. and Mathieu, D. and Matsui, Y. and Maza, E. and McCutcheon, J.E. and McKay, G.J. and McMillan, B. and McMillan, N. and Meads, C. and Medina, L. and Merrick, B.A. and Metzger, D.W. and Meunier, F.A. and Michaelis, M. and Micheau, O. and Mihara, H. and Mintz, E.M. and Mizukami, T. and Moalic, Y. and Mohapatra, D.P. and Monteiro, A. and Montes, M. and Moran, J.V. and Morozov, S.Y. and Mort, M. and Murai, N. and Murphy, D.J. and Murphy, S.K. and Murray, S.A. and Naganawa, S. and Nammi, S. and Nasios, G. and Natoli, R.M. and Nguyen, F. and Nicol, C. and van Nieuwerburgh, F. and Nilsen, E.B. and Nobile, C.J. and O'Mahony, M. and Ohlsson, S. and Olatunbosun, O. and Olofsson, P. and Ortiz, A. and Ostrikov, K. and Otto, S. and Outeiro, T.F. and Ouyang, S. and Paganoni, S. and Page, A. and Palm, C. and Paradies, Y. and Parsons, M.H. and Parsons, N. and Pascal, P. and Paul, E. and Peckham, M. and Pedemonte, N. and Pellizzon, M.A. and Petrelli, M. and Pichugin, A. and Pinto, C.J.C. and Plevris, J.N. and Pollesello, P. and Polz, M. and Ponti, G. and Porcelli, P. and Prince, M. and Quinn, G.P. and Quinn, T.J. and Ramula, S. and Rappsilber, J. and Rehfeldt, F. and Reiling, J.H. and Remacle, C. and Rezaei, M. and Riddick, E.W. and Ritter, U. and Roach, N.W. and Roberts, D.D. and Robles, G. and Rodrigues, T. and Rodriguez, C. and Roislien, J. and Roobol, M.J. and Rowe, A. and Ruepp, A. and van Ruitenbeek, J. and Rust, P. and Saad, S. and Sack, G.H. and Santos, M. and Saudemont, A. and Sava, G. and Schrading, S. and Schramm, A. and Schreiber, M. and Schuler, S. and Schymkowitz, J. and Sczyrba, A. and Seib, K.L. and Shi, H.-P. and Shimada, T. and Shin, J.-S. and Shortt, C. and Silveyra, P. and Skinner, D. and Small, I. and Smeets, P.A.M. and So, P.-W. and Solano, F. and Sonenshine, D.E. and Song, J. and Southall, T. and Speakman, J.R. and Srinivasan, M.V. and Stabile, L.P. and Stasiak, A. and Steadman, K.J. and Stein, N. and Stephens, A.W. and Stewart, D.I. and Stine, K. and Storlazzi, C. and Stoynova, N.V. and Strzalka, W. and Suarez, O.M. and Sultana, T. and Sumant, A.V. and Summers, M.J. and Sun, G. and Tacon, P. and Tanaka, K. and Tang, H. and Tanino, Y. and Targett-Adams, P. and Tayebi, M. and Tayyem, R. and Tebbe, C.C. and Telfer, E.E. and Tempel, W. and Teodorczyk-Injeyan, J.A. and Thijs, G. and Thorne, S. and Thrift, A.G. and Tiffon, C. and Tinnefeld, P. and Tjahjono, D.H. and Tolle, F. and Toth, E. and Del Tredici, A.L. and Tsapas, A. and Tsirigotis, K. and Turak, A. and Tzotzos, G. and Udo, E.E. and Utsumi, T. and Vaidyanathan, S. and Vaillant, M. and Valsesia, A. and Vandenbroucke, R.E. and Veiga, F.H. and Vendrell, M. and Vesk, P.A. and Vickers, P. and Victor, V.M. and Villemur, R. and Vohl, M.-C. and Voolstra, C.R. and Vuillemin, A. and Wakelin, S. and Waldron, L. and Walsh, L.J. and Wang, A.Y. and Wang, F. and Wang, Y. and Watanabe, Y. and Weigert, A. and Wen, J.-C. and Wham, C. and White, E.P. and Wiener, J. and Wilharm, G. and Wilkinson, S. and Willmann, R. and Wilson, C. and Wirth, B. and Wojan, T.R. and Wolff, M. and Wong, B.M. and Wu, T.-W. and Wuerbel, H. and Xiao, X. and Xu, D. and Xu, J.W. and Xu, J. and Xue, B. and Yalcin, S. and Yan, H. and Yang, E.-C. and Yang, S. and Yang, W. and Ye, Y. and Ye, Z.-Q. and Yli-Kauhaluoma, J. and Yoneyama, H. and Yu, Y. and Yuan, G.-C. and Yuh, C.-H. and Zaccolo, M. and Zeng, C. and Zevnik, B. and Zhang, C. and Zhang, L. and Zhang, Y. and Zhang, Y. and Zhang, Z. and Zhang, Z.-Y. and Zhao, Y. and Zhou, M. and Zuberbier, T. and Aanei, C.M. and Ahmad, R. and Al-Lawama, M. and Alanio, A. and Allardyce, J. and Alonso-Caneiro, D. and Atack, J.M. and Baier, D. and Bansal, A. and Benezeth, Y. and Berbesque, C. and Berrevoet, F. and Biedermann, P.H.W. and Bijleveld, E. and Bittner, F. and Blombach, F. and van den Bos, W. and Boudreau, S.A. and Bramoweth, A.D. and Braubach, O. and Cai, Y. and Campbell, M. and Cao, Z. and Catry, T. and Chen, X. and Cheng, S. and Chung, H.-J. and Chávez-Fumagalli, M.A. and Conway, A. and Costa, B.M. and Cyr, N. and Dean, L.T. and Denzel, M.S. and Dlamini, S.V. and Dudley, K.J. and Dufies, M. and Ecke, T. and Eckweiler, D. and Eixarch, E. and El-Adawy, H. and Emmrich, J.V. and Eustace, A.J. and Falter-Wagner, C.M. and Farhoudi, R. and Fuss, J. and Gao, J. and Gill, M.R. and Gloyn, L. and Goggs, R. and Govinden, U. and Greene, G. and Greiff, V. and Grundle, D.S. and Grüneberg, P. and Gumede, N. and Haore, G. and Harrison, P. and Hoenner, X. and Hojsgaard, D. and Hori, H. and Ikonomopoulou, M.P. and Jeurissen, P. and Johnson, D.M. and Kabra, D. and Kamagata, K. and Karmakar, C. and Kasian, O. and Kaye, L.K. and Khan, M.M. and Kim, Y.-M. and Kish, J.K. and Kobold, S. and Kohanbash, G. and Kohls, G. and Kugler, J.-M. and Kumar, G. and Lacy-Colson, J. and Latif, A. and Lauschke, V.M. and Li, B. and Lim, C.J. and Liu, F. and Liu, X. and Lu, J.-J. and Lu, Q. and Mahavadi, P. and Marzocchi, U. and McGarrigle, C.A. and van Meerten, T. and Min, R. and Moal, I. and Molari, M. and Molleman, L. and Mondal, S.R. and van de Mortel, T. and Moss, W.N. and Moultos, O.A. and Mukherjee, M. and Nakayama, K. and Narayan, E. and Navaratnarajah and Neumann, P.-A. and Nie, J. and Nie, Y. and Niemeyer, F. and Nolan, F. and Nwaiwu, O. and Oldenmenger, W.H. and Olumayede, E. and Ou, J. and Pallebage-Gamarallage, M. and Pearce, S.P. and Pelkonen, T. and Pelleri, M.C. and Pereira, J.L. and Pheko, M. and Pinto, K.A. and Piovesan, A. and Pluess, M. and Podolsky, I.M. and Prescott, J. and Qi, D. and Qi, X. and Raikou, V.D. and Ranft, A. and Rhodes, J. and Rotge, J.-Y. and Rowe, A.D. and Saggar, M. and Schuon, R.A. and Shahid, S. and Shalchyan, V. and Shirvalkar, P. and Shiryayev, O. and Singh, J. and Smout, M.J. and Soares, A. and Song, C. and Srivastava, K. and Srivastava, R.K. and Sun, J. and Szabo, A. and Szymanski, W. and Tai, C.N.P. and Takeuchi, H. and Tanadini-Lang, S. and Tang, F. and Tao, W. and Theron, G. and Tian, C.F. and Tian, Y.-S. and Tuttle, L.M. and Valenti, A. and Verlot, P. and Walker, M. and Wang, J. and Welter, D. and Winslade, M. and Wu, D. and Wu, Y.-R. and Xiao, H. and Xu, B. and Xu, J. and Xu, Z. and Yang, D. and Yang, M. and Yankilevich, P. and You, Y. and Yu, C. and Zhan, J. and Zhang, G. and Zhang, K. and Zhang, T. and Zhang, Y. and Zhao, G. and Zhao, J. and Zhou, X. and Zhu, Z. and Ajani, P.A. and Anazodo, U.C. and Bagloee, S.A. and Bail, K. and Bar, I. and Bathelt, J. and Benkeser, D. and Bernier, M.L. and Blanchard, A.M. and Boakye, D.W. and Bonatsos, V. and Boon, M.H. and Bouboulis, G. and Bromfield, E. and Brown, J. and Bul, K.C.M. and Burton, K.J. and Butkowski, E.G. and Carroll, G. and Chao, F. and Charrier, E.E. and Chen, X. and Chen, Y.-C. and Chenguang and Choi, J.R. and Christoffersen, T. and Comel, J.C. and Cosse, C. and Cui, Y. and van Dessel, P. and Dhaval and Diodato, D. and Duffey, M. and Dutt, A. and Egea, L.G. and El-Said, M. and Faye, M. and Fernandez-Fernandez, B. and Foley, K.G. and Founou, L.L. and Fu, F. and Gadelkareem, R.A. and Galimov, E. and Garip, G. and Gemmill, A. and Gouil, Q. and Grey, J. and Gridneva, Z. and Grothe, M.J. and Grébert, T. and Guerrero, F. and Guignard, L. and Haenssgen, M.J. and Hasler, D. and Holgate, J.Y. and Huang, A. and Hulse-Kemp, A.M. and Jean-Quartier, C. and Jeon, S.-M. and Jia, Y. and Jutzeler, C. and Kalatzis, P. and Karim, M. and Karsay, K. and Keitel, A. and Kempe, A. and Keown, J.R. and Khoo, C.M. and Khwaja, N. and Kievit, R.A. and Kosanic, S. and Koutoukidis, D.A. and Kramer, P. and Kumar, D. and Kiraǧ, N. and Lanza, G. and Le, T.D. and Leem, J.W. and Leightley, D. and Leite, A. and Lercher, L. and Li, Y. and Lim, R. and Lima, L.R.A. and Lin, L. and Ling, T. and Liu, Y. and Liu, Z. and Lu, Y. and Lum, F.M. and Luo, H. and Machhi, J. and Macleod, A. and Macwan, I. and Madala, H.R. and Madani, N. and de Maio, N. and Makowiecki, K. and Mallinson, D.J. and Margelyte, R. and Maria, C. and Markonis, Y. and Marsili, L. and Mavoa, S. and McWilliams, L. and Megersa, M. and Souto-Maior, C. and Menichetti, J. and Mercieca-Bebber, R. and Miller, J.J. and Minde, D.-P.M. and Minges, A. and Mishra, E. and Mishra, V.R. and Moores, C. and Morrice, N. and Moskalensky, A.E. and Navarin, N. and Negera, E. and Nolet, P. and Nordberg, A. and Nordén, R. and Nowicki, J.P. and Olova, N. and Olszewski, P. and Onzima, R. and Pan, C.-L. and Park, C. and Park, D.I. and Park, S. and Patil, C.D. and Pedro, S.A. and Perry, S.R. and Peter, J. and Peterson, B.M. and Pezzuolo, A. and Pozdnyakov, I. and Qian, S. and Qin, L. and Rafe, A. and Raote, I. and Raza, A. and Rebl, H. and Refai, O. and Regan, T. and Richa, T. and Richardson, M.F. and Robinson, K.R. and Rossoni, L. and Rouet, R. and Safaei, S. and Schneeberger, P.H.H. and Schwotzer, D. and Sebastian, A. and Selinski, J. and Seltmann, S. and Sha, F. and Shalev, N. and Shang, J.-L. and Singer, J. and Singh, M. and Smith, T. and Solomon-Moore, E. and Song, L. and Soraggi, S. and Stanley, R. and Steckhan, N. and Strobl, F. and Subissi, L. and Supriyanto, I. and Surve, C.R. and Suzuki, T. and Syme, C. and Sörelius, K. and Tang, Y. and Tantawy, M. and Tennakoon, S. and Teseo, S. and Toelzer, C. and Tomov, N. and Tovar, M. and Tran, L. and Tripathi, S. and Tuladhar, A.M. and Ukubuiwe, A.C. and Ung, C.O.L. and Valgepea, K. and Vatanparast, H. and Vidal, A. and Wang, F. and Wang, Q. and Watari, R. and Webster, R. and Webster, R. and Wei, J. and Wibowo, D. and Wingenbach, T.S.H. and Xavier, R.M. and Xiao, S. and Xiong, P. and Xu, S. and Xu, S. and Yao, R. and Yao, W. and Yin, Q. and Yu, Y. and Zaitsu, M. and Zeineb, Z. and Zhan, X.-Y. and Zhang, J. and Zhang, R. and Zhang, W. and Zhang, X. and Zheng, S. and Zhou, B. and Zhou, X. and Ahmad, H. and Akinwumi, S.A. and Albery, G.F. and Alhowimel, A. and Ali, J. and Alshehri, M. and Alsuhaibani, M. and Anikin, A. and Azubuike, S.O. and Bach-Mortensen, A. and Baltiansky, L. and Bartas, M. and Belachew, K.Y. and Bhardwaj, V. and Binder, K. and Bland, N.S. and Boah, M. and Bullen, B. and Calabrò, G.E. and Callahan, T.J. and Cao, B. and Chalmers, K. and Chang, W. and Che, Z. and Chen, A.T.Y. and Chen, H. and Chen, H. and Chen, Y. and Chen, Z. and Choi, Y. and Chowdhury, M.A.K. and Christensen, M.R. and Cooke, R.S.C. and Cottini, M. and Covington, N.V. and Cunningham, C. and Delarocque, J. and Devos, L. and Dhar, A.R. and Ding, K.-F. and Dong, K. and Dong, Z. and Dreyer, N. and Ekstrand, C. and Fardet, T. and Feleke, B.E. and Feurer, T. and Freitas, A. and Gao, T. and Gebremedhin and Giganti, F. and Grabowski, P. and Guerra-Mora, J.R. and Guo, C. and Guo, X. and Gupta, H. and He, S. and Heijne, M. and Heinemann, S. and Hogrebe, A. and Huang, Z. and Iskander-Rizk, S. and Iyer, L.M. and Jahan, Y. and James, A.S. and Joel, E. and Joffroy, B. and Jégousse, C. and Kambondo, G. and Karnati, P. and Kaya, C. and Ke, A. and Kelly, D. and Kickert, R. and Kidibule, P.E. and Kieselmann, J.P. and Kim, H.J. and Kitazawa, T. and Lamberts, A. and Li, Y. and Liang, H. and Linn, S.N. and Litfin, T. and Liusuo, W. and Lygirou, V. and Mahato, A.K. and Mai, Z.-M. and Major, R.W. and Mali, S. and Mallis, P. and Mao, W. and Marvin-Dowle, K. and Mason, L.D. and Merideth, B. and Merino-Plaza, M.J. and Merlaen, B. and Messina, R. and Mishra, A.K. and Muhammad, J. and Musinguzi, C. and Nanou, A. and Naqash, A. and Nguyen, J.T. and Nguyen, T.T.H. and Ni, D. and Nida and Notcovich, S. and Ohst, B. and Ollivier, Q.R. and Osses, D.F. and Peng, X. and Plantinga, A. and Pulia, M. and Rafiq, M. and Raman, A. and Raucher-Chéné and Rawski, R. and Ray, A. and Razak, L.A. and Rudolf, K. and Rusch, P. and Sadoine, M.L. and Schmidt, A. and Schurr, R. and Searles, S. and Sharma, S. and Sheehan, B. and Shi, C. and Shohayeb, B. and Sommerlad, A. and Strehlow, J. and Sun, X. and Sundar, R. and Taherzadeh, G. and Tahir, N.D.M. and Tang, J. and Testa, J. and Tian, Z. and Tingting, Q. and Verheijen, G.P. and Vickstrom, C. and Wang, T. and Wang, X. and Wang, Z. and Wei, P. and Wilson, A. and Wyart and Yassine, A.-A. and Yousefzadeh, A. and Zare, A. and Zeng, Z. and Zhang, C. and Zhang, H. and Zhang, L. and Zhang, T. and Zhang, W. and Zhang, Z. and Zhou, J. and Zhu, D. and Adamo, V. and Adeyemo, A.A. and Aggelidou, M. and Al-Owaifeer, A.M. and Al-Riyami, A.Z. and Alzghari, S.K. and Andersen, V. and Angus, K. and Asaduzzaman, M. and Asady, H. and Ato, D. and Bai, X. and Baines, R.L. and Ballantyne, M. and Ban, B. and Beck, J. and Ben-Nafa, W. and Black, E. and Blancher, A. and Blankstein, R. and Bodagh, N. and Borges, P. and Brooks, A. and Brox-Ponce, J. and Brunetti, A. and Canham, C.D. and Carninci, P. and Carvajal, R. and Chang, S.C. and Chao, J. and Chatterjee, P. and Chen, H. and Chen, L. and Chen, Y.-C. and Chhatriwalla, A.K. and Chikowe, I. and Chuang, T.-J. and Collevatti, R.G. and Cornejo, D.A.V. and Cuenda, A. and Dao, M. and Dauga, D. and Deng, Z. and Devkota, K. and Doan, L.V. and Elewa, Y.H.A. and Fan, D. and Faruk, M. and Feifei, S. and Ferguson, T.S. and Fleres, F. and Foster, E.J. and Foster, S. and Furer, T. and Gao, Y. and Garcia-Rivera, E.J. and Gazdar, A. and George, R.B. and Ghosh, S. and Gianchecchi, E. and Gleason, J.M. and Hackshaw, A. and Hall, A. and Hall, R. and Harper, P. and Hogg, W.E. and Huang, G. and Hunter, K.E. and Ijzerman, A.P. and Jesus, C. and Jian, G. and Lewis, J.S., Jr. and Kanj, S.S. and Kaur, H. and Kelly, S. and Kheir, F. and Kichatova, V.S. and Kiyani, M. and Klein, R. and Kovesi, T. and Kraschnewski, J.L. and Kumar, A.P. and Labutin, D. and Lazo-Langner, A. and Leclercq, G. and Li, M. and Li, Q. and Li, T. and Li, Y. and Liao, W.-T. and Liao, Z.-Y. and Lin, J. and Lizer, J. and Lobreglio, G. and Lowies, C. and Lu, C. and Majeed, H. and Martin, A. and Martinez-Sobrido, L. and Meresh, E. and Middelveen, M. and Mohebbi, A. and Mota, J. and Mozaheb, Z. and Muyaya, L. and Nandhakumar, A. and Ng, S.H.X. and Obeidat, M. and Oh, D.-H. and Owais, M. and Pace-Asciak, P. and Panwar, A. and Park, C. and Patterson, C. and Penagos-Tabaree, F. and Pianosi, P.T. and Pinzi, V. and Pridans, C. and Psaroulaki, A. and Pujala, R.K. and Pulido-Arjona, L. and Qi, P.-F. and Rahman, P. and Rai, N.K. and Rassaf, T. and Refardt, J. and Ricciardi, W. and Riess, O. and Rovas, A. and Sacks, F.M. and Saleh, S. and Sampson, C. and Schmutz, A. and Sepanski, R. and Sharma, N. and Singh, M. and Spearman, P. and Subramaniapillai, M. and Swali, R. and Tan, C.M. and Tellechea, J.I. and Thomas, L.-M. and Tong, X. and Vavvas, D.G. and Veys, R. and Vitriol, V. and Wang, H.-D. and Wang, J. and Wang, J. and Waugh, J. and Webb, S.A. and Williams, B.A. and Workman, A.D. and Xiang, T. and Xie, L.-X. and Xu, J. and Xu, T. and Yang, C. and Yoon, J.G. and Yuan, C.M. and Zaritsky, A. and Zhang, Y. and Zhao, H. and Zuckerman, H. and Lyu, R. and Pullan, W. and RELISH Consortium",
    journal = "ACM SIGMIS Database",
    year = "2019",
    volume = "2019",
    pages = "1-67",
    publisher = "Oxford University Press",
    issn = "0095-0033",
    doi = "10.1093/database/baz085",
    keywords = "article;  benchmarking;  human;  medical research;  Medical Subject Headings;  Medline;  nonhuman;  okapi;  plant seed;  search engine;  statistical bias;  systematic review",
    abstract = "Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical science. © The Author(s) 2019. Published by Oxford University Press."
}