LOGIN TO YOUR ACCOUNT

Username
Password
Remember Me
Or use your Academic/Social account:

CREATE AN ACCOUNT

Or use your Academic/Social account:

Congratulations!

You have just completed your registration at OpenAire.

Before you can login to the site, you will need to activate your account. An e-mail will be sent to you with the proper instructions.

Important!

Please note that this site is currently undergoing Beta testing.
Any new content you create is not guaranteed to be present to the final version of the site upon release.

Thank you for your patience,
OpenAire Dev Team.

Close This Message

CREATE AN ACCOUNT

Name:
Username:
Password:
Verify Password:
E-mail:
Verify E-mail:
*All Fields Are Required.
Please Verify You Are Human:
fbtwitterlinkedinvimeoflicker grey 14rssslideshare1
Roller, R.A.; Stevenson, M. (2015)
Publisher: Association for Computational Linguistics
Languages: English
Types: Other
Subjects:

Classified by OpenAIRE into

ACM Ref: ComputingMethodologies_PATTERNRECOGNITION
Distant supervision is a useful technique for creating relation classifiers in the absence of labelled data. The approaches are often evaluated using a held-out portion of the distantly labelled data, thereby avoiding the need for lablelled data entirely. However, held-out evaluation means that systems are tested against noisy data, making it difficult to determine their true accuracy. This paper examines the effectiveness of using held-out data to evaluate relation extraction systems by comparing the results that are produced with those generated using manually labelled versions of the same data. We train classifiers to detect two UMLS Metathesaurus relations (may-treat and may-prevent) in Medline abstracts. A new evaluation data set for these relations is made available. We show that evaluation against a distantly labelled gold standard tends to overestimate performance and that no direct connection can be found between improved performance against distantly and manually labelled gold standards.
  • The results below are discovered through our pilot algorithms. Let us know how we are doing!

    • A. Aronson and F. Lang. 2010. An overview of MetaMap: historical perspective and recent advances. Journal of the American Medical Association, 17(3):229-236.
    • Isabelle Augenstein, Diana Maynard, and Fabio Ciravegna. 2014. Relation extraction from the web using distant supervision. In Proceedings of the 19th International Conference on Knowledge Engineering and Knowledge Management (EKAW 2014), Linko¨ping, Sweden, November.
    • Kedar Bellare and Andrew Mccallum. 2007. Learning Extractors from Unlabeled Text using Relevant Databases. In Sixth International Workshop on Information Integration on the Web (IIWeb).
    • Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46.
    • Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In In Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology (ISMB), pages 77-86. AAAI Press.
    • Raphael Hoffmann, Congle Zhang, and Daniel S. Weld. 2010. Learning 5000 relational extractors. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10, pages 286-295, Stroudsburg, PA, USA. Association for Computational Linguistics.
    • Ander Intxaurrondo, Mihai Surdeanu, Oier Lopez de Lacalle, and Eneko Agirre. 2013. Removing noisy mentions for distant supervision. Procesamiento del Lenguaje Natural, 51:41-48.
    • Sebastian Krause, Hong Li, Hans Uszkoreit, and Feiyu Xu. 2012. Large-scale learning of relationextraction rules with distant supervision from the web. In Proceedings of the 11th International Conference on The Semantic Web - Volume Part I, ISWC'12, pages 263-278, Berlin, Heidelberg. Springer-Verlag.
    • Bonan Min, Ralph Grishman, Li Wan, Chang Wang, and David Gondek. 2013. Distant supervision for relation extraction with an incomplete knowledge base. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 777-782, Atlanta, Georgia, June. Association for Computational Linguistics.
    • Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL '09, pages 1003-1011, Stroudsburg, PA, USA. Association for Computational Linguistics.
    • Claire Ne´dellec, Robert Bossy, Jin-Dong Kim, JungJae Kim, Tomoko Ohta, Sampo Pyysalo, and Pierre Zweigenbaum. 2013. Overview of bionlp shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 1-7, Sofia, Bulgaria, August. Association for Computational Linguistics.
    • Truc-Vien T. Nguyen and Alessandro Moschitti. 2011. Joint distant and direct supervision for relation extraction. In Proceedings of The 5th International Joint Conference on Natural Language Processing (IJCNLP'11), pages 732-740. Association for Computational Linguistics.
    • Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD '10).
    • Alan Ritter, Luke Zettlemoyer, Mausam, and Oren Etzioni. 2013. Modeling missing data in distant supervision for information extraction. In Association for Computational Linguistics Vol. 1 (TACL).
    • Roland Roller and Mark Stevenson. 2014. Applying umls for distantly supervised relation detection. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi), pages 80-84, Gothenburg, Sweden, April. Association for Computational Linguistics.
    • Roland Roller, Eneko Agirre, Aitor Soroa, and Mark Stevenson. 2015. Improving distant supervision using inference learning. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 273-278, Beijing, China, July. Association for Computational Linguistics.
    • Benjamin Roth and Dietrich Klakow. 2013. Combining generative and discriminative model scores for distant supervision. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 24-29, Seattle, Washington, USA, October. Association for Computational Linguistics.
    • Isabel Segura-Bedmar, Paloma Mart´ınez, and Mar´ıa Herrero Zazo. 2013. Semeval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 341-350, Atlanta, Georgia, USA, June. Association for Computational Linguistics.
    • Mihai Surdeanu, David McClosky, Mason Smith, Andrey Gusev, and Christopher Manning. 2011. Customizing an information extraction system to a new domain. In Proceedings of the ACL 2011 Workshop on Relational Models of Semantics, pages 2- 10, Portland, Oregon, USA, June. Association for Computational Linguistics.
    • Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLPCoNLL '12, pages 455-465, Stroudsburg, PA, USA. Association for Computational Linguistics.
    • Shingo Takamatsu, Issei Sato, and Hiroshi Nakagawa. 2012. Reducing wrong labels in distant supervision for relation extraction. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL '12, pages 721-729, Stroudsburg, PA, USA. Association for Computational Linguistics.
    • Philippe Thomas, Ille´s Solt, Roman Klinger, and Ulf Leser. 2011. Learning protein protein interaction extraction using distant supervision. In Proceedings of Robust Unsupervised and Semi-Supervised Methods in Natural Language Processing, pages 34-41.
    • Andreas Vlachos and Stephen Clark. 2014. Application-driven relation extraction with limited distant supervision. In Proceedings of the First AHA!-Workshop on Information Discovery in Text, pages 1-6, Dublin, Ireland, August. Association for Computational Linguistics and Dublin City University.
    • Fei Wu and Daniel S. Weld. 2007. Autonomously semantifying wikipedia. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management, CIKM '07, pages 41-50, New York, NY, USA. ACM.
    • Wei Xu, Raphael Hoffmann, Le Zhao, and Ralph Grishman. 2013. Filling knowledge base gaps for distant supervision of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 665-670, Sofia, Bulgaria, August. Association for Computational Linguistics.
    • Xingxing Zhang, Jianwen Zhang, Junyu Zeng, Jun Yan, Zheng Chen, and Zhifang Sui. 2013. Towards accurate distant supervision for relational facts extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 810-815, Sofia, Bulgaria, August. Association for Computational Linguistics.
  • No related research data.
  • Discovered through pilot similarity algorithms. Send us your feedback.

Share - Bookmark

Cite this article