GERBIL: General Entity Annotation Benchmark Framework

GERBIL is a general entity annotation system based on the BAT-Framework.

Source Code Demo Issues Wiki

Overview of GERBIL
The need to bridge between the unstructured data on the document Web and the structured data on the Data Web has led to the development of a considerable number of annotation tools. Those tools are hard to compare since published results are calculated on diverse datasets and measured in different units. A first approach to structure the parameter- and tool-space of semantic entity annotation systems was published by Cornolti et al.. However, the BAT-framework is hard to setup and does not allow for an easy comparison of tools and datasets.

We present GERBIL, a general entity annotation system based on the BAT-Framework. GERBIL offers an easy-to-use web-based platform for the agile comparison of annotators using multiple datasets and uniform measuring approaches. To add a tool to GERBIL, all the end user has to do is to provide a URL to a REST interface to its tool which abides by a given specification. The integration and benchmarking of the tool against user-specified datasets is then carried out automatically by the GERBIL platform. Currently, our platform provides results for 9 annotators and 11 datasets with more coming. Internally, GERBIL is based on the Natural Language Programming Interchange Format (NIF) and provide Java classes for implementing APIs for datasets and annotators to NIF.

The following table lists the annotators supported in different version of the BAT-Framework and GERBIL.

Annotators BAT-Framework Gerbil 1.0.0 Gerbil 1.2.2
Wikipedia Miner
Illionois Wikifier
TagMe 2
NIF-based Annotator
AIDA (✔)
FREME e-Entity

The following table lists the annotators that are currently available and the experiment types they support. Note that some of the A2KB annotators support the D2KB experiment by offering an own API method. Other A2KB annotators can be chosen for a D2KB experiment as well as described in the wiki. However, since the comparison might not be fair, we marked these annotators with (✔) in the table. The same is done for Entity Typing.

Entity Recognition
D2KB Entity
OKE Task 1 OKE Task 2
AIDA (✔)
Dexter (✔) (✔)
FRED (✔) (✔)
FREME e-Entity
FOX (✔) (✔)
TagMe 2 (✔)
xLisa (✔)

The following table lists the datasets that are currently available and the experiment types they support.

Entity Recognition
Entity Typing OKE Task 1 OKE Task 2
Microposts 2014-Test
Microposts 2014-Train
OKE 2015 Task 1 evaluation dataset
OKE 2015 Task 1 example set
OKE 2015 Task 1 gold standard sample
OKE 2015 Task 2 evaluation dataset
OKE 2015 Task 2 example set
OKE 2015 Task 2 gold standard sample

Long term stability

The idea of GERBIL emerged in September 2014 when a couple of articles released at the same time claimed to be state-of-the-art. Especially, those approaches were not easily comparable due to their heterogeneous set-up, dataset use and evaluation metrics. Thus, we decided to build GERBIL and extend the BAT-Framework to break the barriers for people not able to write source code. GERBIL is still a young project and thus we are trying to explore the borders of our endeavour. As GERBIL has been launched within two PhD projects funded by European Social Fund we are confident that it will be a long lasting web service. The fallback is our working group AKSW which currently already hosts more than 30 open source projects. Finally, GERBIL is open source software which can be maintained and hosted by anybody.

Furthermore, the research and developement unit of the University Leipzig Computation Center keeps daily backups to ensure long-term quotability.

With this project we aim at establishing a highly available, easy quotable and liable focal point for NER and NED evaluations. Additionally, we build our framework to be rapidly extensible and adaptable for future uses.

The survey data from our paper can be found at GERBIL's GitHub repository.


  • Ciro Baron (University Leipzig, Germany)
  • Andreas Both (R&D, Unister GmbH, Germany)
  • Martin Brümmer (University Leipzig, Germany)
  • Diego Ceccarelli (Unversity Pisa, Italy)
  • Marco Cornolti (University of Pisa, Italy)
  • Didier Cherix (R&D, Unister GmbH, Germany)
  • Bernd Eickmann (R&D, Unister GmbH, Germany)
  • Paolo Ferragina (University of Pisa, Italy)
  • Christiane Lemke (R&D, Unister GmbH, Germany)
  • Andrea Moro (Sapienza University of Rome, Italy)
  • Roberto Navigli (Sapienza University of Rome, Italy)
  • Francesco Piccinno (University of Pisa, Italy)
  • Giuseppe Rizzo (EURECOM, France)
  • Harald Sack (HPI Potsdam, Germany)
  • René Speck (Institute for Applied Informatics, Germany)
  • Raphaël Troncy (EURECOM, France)
  • Jörg Waitelonis (HPI Potsdam, Germany)
  • Lars Wesemann (R&D, Unister GmbH, Germany)

Project Team


by (Editors: ) [BibTex of ]


AKSW Colloquium, 17.10.2016, Version Control for RDF Triple Stores + NEED4Tweet ( 2016-10-17T09:55:50+02:00 by Marvin Frommhold)

2016-10-17T09:55:50+02:00 by Marvin Frommhold

In the upcoming Colloquium, October the 17th at 3 PM, two papers will be presented: Version Control for RDF Triple Stores Marvin Frommhold will discuss the paper “Version Control for RDF Triple Stores” by Steve Cassidy and James Ballantine which forms the foundation … Continue reading → Read more about "AKSW Colloquium, 17.10.2016, Version Control for RDF Triple Stores + NEED4Tweet"

LIMES 1.0.0 Released ( 2016-10-14T11:38:31+02:00 by Kleanthi Georgala)

2016-10-14T11:38:31+02:00 by Kleanthi Georgala

Dear all, the LIMES Dev team is happy to announce LIMES 1.0.0. LIMES, the Link Discovery Framework for Metric Spaces, is a link discovery framework for the Web of Data. Read more about "LIMES 1.0.0 Released"

DL-Learner 1.3 (Supervised Structured Machine Learning Framework) Released ( 2016-10-11T21:41:00+02:00 by Dr. Jens Lehmann)

2016-10-11T21:41:00+02:00 by Dr. Jens Lehmann

Dear all, the Smart Data Analytics group at AKSW is happy to announce DL-Learner 1.3. DL-Learner is a framework containing algorithms for supervised machine learning in RDF and OWL. Read more about "DL-Learner 1.3 (Supervised Structured Machine Learning Framework) Released"

OntoWiki 1.0.0 released ( 2016-10-05T16:50:05+02:00 by Natanael Arndt)

2016-10-05T16:50:05+02:00 by Natanael Arndt

Dear Semantic Web and Linked Data Community, we are proud to finally announce the releases of OntoWiki 1.0.0 and the underlying Erfurt Framework in version 1.8.0. Read more about "OntoWiki 1.0.0 released"

AKSW Colloquium, 05.09.2016. LOD Cloud Statistics, OpenAccess at Leipzig University. ( 2016-08-31T11:23:10+02:00 by Ivan Ermilov)

2016-08-31T11:23:10+02:00 by Ivan Ermilov

On the upcoming Monday (05.09.2016), AKSW group will discuss topics related to Semantic Web and LOD Cloud Statistics. Also, we will have invited speaker from University of Leipzig Library (UBL) Dr. Astrid Vieler talking about OpenAccess at Leipzig University. Read more about "AKSW Colloquium, 05.09.2016. LOD Cloud Statistics, OpenAccess at Leipzig University."