GERBIL: General Entity Annotation Benchmark Framework

GERBIL is a general entity annotation system.

Source Code Demo Issues Wiki

Overview of GERBIL
GERBIL is a general Linked Data benchmarking system (formerly used for entity annotation systems based on the BAT-Framework). GERBIL offers an easy-to-use web-based platform for the agile comparison of annotators using multiple datasets and uniform measuring approaches. To add a tool to GERBIL, all the end user has to do is to provide a URL to a REST interface to its tool which abides by a given specification. The integration and benchmarking of the tool against user-specified datasets is then carried out automatically by the GERBIL platform.

BAT-Framework GERBIL 1.0.0 GERBIL1.2.5 Experiment
Wikipedia Miner (✔) A2KB
Illinois Wikifier (✔) A2KB
Spotlight A2KB
AIDA A2KB
TagMe 2 A2KB
NERD-ML A2KB
KEA A2KB
WAT A2KB
Dexter A2KB
AGDISTIS (✔) D2KB
Babelfy A2KB
FOX OKE Task 1
FRED OKE Task 1
FREME OKE Task 1
entityclassifier.eu A2KB
CETUS OKE Task 2
xLisa A2KB
DoSer D2KB
PBOH D2KB
NERFGUN D2KB
NIF-based Annotator any

The following table lists the annotators that are currently available and the experiment types they support. Note that some of the A2KB annotators support the D2KB experiment by offering an own API method. Other A2KB annotators can be chosen for a D2KB experiment as well as described in the wiki. However, since the comparison might not be fair, we marked these annotators with (✔) in the table. The same is done for Entity Typing.
A2KB, C2KB,
Entity Recognition
D2KB Entity
Typing
OKE Task 1 OKE Task 2
AIDA (✔)
AGDISTIS
Babelfy
CETUS
CETUS (FOX)
Dexter (✔)
entityclassifier.eu (✔)
FRED (✔) (✔)
FREME e-Entity
FOX (✔) (✔)
KEA
NERD-ML (✔)
Spotlight
TagMe 2 (✔)
WAT
xLisa (✔)
PBoH
NERFGUN
DoSER

The following table lists the datasets that are currently available and the experiment types they support.

A2KB, C2KB, D2KB,
Entity Recognition
Entity Typing OKE Task 1 OKE Task 2
AIDA/CoNLL-Complete
AIDA/CoNLL-Test A
AIDA/CoNLL-Test B
AIDA/CoNLL-Training
AQUAINT
DBpediaSpotlight
Dercyznski
IITB
KORE50
MSNBC
Microposts 2013-Test
Microposts 2013-Train
Microposts 2014-Test
Microposts 2014-Train
Microposts 2015-Test
Microposts 2015-Train
Microposts 2015-Dev
Microposts 2016-Test
Microposts 2016-Train
Microposts 2016-Dev
N3-RSS-500
N3-Reuters-128
OKE 2015 Task 1 evaluation dataset
OKE 2015 Task 1 example set
OKE 2015 Task 1 gold standard sample
OKE 2015 Task 2 evaluation dataset
OKE 2015 Task 2 example set
OKE 2015 Task 2 gold standard sample
Senseval 2
Senseval 3
UMBC
WSDM 2012

Long term stability

The idea of GERBIL emerged in September 2014 when a couple of articles released at the same time claimed to be state-of-the-art. Especially, those approaches were not easily comparable due to their heterogeneous set-up, dataset use and evaluation metrics. Thus, we decided to build GERBIL and extend the BAT-Framework to break the barriers for people not able to write source code.

GERBIL is now more than 3 years old and has hosted more than 50.000 experiments. It is currently hosted at the research and development unit of the University Leipzig Computation Center and the Paderborn University which keep daily backups to ensure long-term quotability.

The survey data from our paper can be found at GERBIL's GitHub repository.

Contributors

  • Ciro Baron (University Leipzig, Germany)
  • Andreas Both (R&D, Unister GmbH, Germany)
  • Martin Brümmer (University Leipzig, Germany)
  • Diego Ceccarelli (Unversity Pisa, Italy)
  • Marco Cornolti (University of Pisa, Italy)
  • Didier Cherix (R&D, Unister GmbH, Germany)
  • Bernd Eickmann (R&D, Unister GmbH, Germany)
  • Paolo Ferragina (University of Pisa, Italy)
  • Christiane Lemke (R&D, Unister GmbH, Germany)
  • Andrea Moro (Sapienza University of Rome, Italy)
  • Roberto Navigli (Sapienza University of Rome, Italy)
  • Francesco Piccinno (University of Pisa, Italy)
  • Giuseppe Rizzo (EURECOM, France)
  • Harald Sack (HPI Potsdam, Germany)
  • René Speck (Institute for Applied Informatics, Germany)
  • Raphaël Troncy (EURECOM, France)
  • Jörg Waitelonis (HPI Potsdam, Germany)
  • Lars Wesemann (R&D, Unister GmbH, Germany)

Publications

by (Editors: ) [BibTex of ]

News

SANSA 0.5 (Semantic Analytics Stack) Released ( 2018-12-13T09:25:34+01:00 by Prof. Dr. Jens Lehmann)

2018-12-13T09:25:34+01:00 by Prof. Dr. Jens Lehmann

We are happy to announce SANSA 0.5 – the fifth release of the Scalable Semantic Analytics Stack. Read more about "SANSA 0.5 (Semantic Analytics Stack) Released"

AKSW at web.br in São Paulo ( 2018-10-22T09:37:49+02:00 by Natanael Arndt)

2018-10-22T09:37:49+02:00 by Natanael Arndt

From October 1st until 6th a delegation from AKSW Group, Leipzig University of Applied Sciences (HTWK), eccenca GmbH, and Max Planck Institute for Human Cognitive and Brain Sciences went to São Paulo, Brazil to meet people from the Web Technologies … Continue reading → Read more about "AKSW at web.br in São Paulo"

AskNow 0.1 Released ( 2018-09-13T15:35:04+02:00 by Prof. Dr. Jens Lehmann)

2018-09-13T15:35:04+02:00 by Prof. Dr. Jens Lehmann

Dear all, we are very happy to announce AskNow 0.1 – the initial release of Question Answering Components and Tools over RDF Knowledge Graphs. Website: http://asknow.sda.tech/ Demo: http://asknowdemo.sda.tech GitHub: https://github. Read more about "AskNow 0.1 Released"

Jekyll RDF Tutorial Screencast ( 2018-08-07T11:11:12+02:00 by Natanael Arndt)

2018-08-07T11:11:12+02:00 by Natanael Arndt

Since 2016 we are developing Jekyll-RDF a plugin for the famous Jekyll–static website generator. Read more about "Jekyll RDF Tutorial Screencast"

DBpedia Day @ SEMANTiCS 2018 ( 2018-07-20T14:37:25+02:00 by Johannes Frey)

2018-07-20T14:37:25+02:00 by Johannes Frey

Don’t miss the 12th edition of the DBpedia Community Meeting in Vienna, the city with the highest quality of life in the world. Read more about "DBpedia Day @ SEMANTiCS 2018"