tutorial status: under revision

builds on an earlier version by Michael Hanl

• Stanford NER from Python

This tutorial builds on the basic Stanford NER tutorial. If you would like to learn more about the basic running and options of the Stanford NER tagger, please refer to that tutorial first. In this tutorial, we are going to explore ways of displaying the NER tagged output in a visually pleasing and informative fashion by means of a tool called WordFreak as well as by means of XML / XHTML technologies.

Basic output from the NER tagging process looks like this:

The/O fate/O of/O Lehman/ORGANIZATION Brothers/ORGANIZATION ,/O the/O beleaguered/O investment/O bank/O ,/O hung/O in/O the/O balance/O on/O Sunday/O as/O Federal/ORGANIZATION Reserve/ORGANIZATION officials/O and/O the/O leaders/O of/O major/O financial/O institutions/O continued/O to/O gather/O in/O emergency/O meetings/O trying/O to/O complete/O a/O plan/O to/O rescue/O the/O stricken/O bank/O ./O Several/O possible/O plans/O emerged/O from/O the/O talks/O ,/O held/O at/O the/O Federal/ORGANIZATION Reserve/ORGANIZATION Bank/ORGANIZATION of/ORGANIZATION New/ORGANIZATION York/ORGANIZATION and/O led/O by/O Timothy/PERSON R./PERSON Geithner/PERSON ,/O the/O president/O of/O the/O New/ORGANIZATION York/ORGANIZATION Fed/ORGANIZATION ,/O and/O Treasury/ORGANIZATION Secretary/O Henry/PERSON M./PERSON Paulson/PERSON Jr./PERSON ./O

You can also extract the data in CoNNL format which is a tab separated (tsv) table-like format according to the guidelines of a computational shared task:

The entities detected by the classifier may be stored in xml for further inspection by means of regular expression searches, concordancing etc. or highlighted in xhtml in different colors by means of css in order to highlight them in a text.

<s>Then a man from <LOCATION>Belfast<LOCATION> bought the field […].</s>

A possibility is a deeper analysis of the data by a concordance program such as AntConc or WConcord. Thereby you can search, for instance, for the different entities. This can be useful especially when using larger amount of texts to preserve clarity. The following figure shows the results searching for the entity location in the NER output by applying WConcord.

Besides its compatibility with other tools provided by Stanford University an advantage of the Stanford NER is the fast generation of results. Furthermore, it is principally language independent which means numerous possibilities for application to new languages that depend on training the tool on new data. For English texts, accuracy may range between 90-95% with or without distributional similarity features (Spanish texts 80%, Dutch 70-80%). But results for other text types such as literary texts or historical texts may differ.

Furthermore, you have the possibility to train the Stanford NER on your own data. This complex task requires a lot of memory and further knowledge. Useful pieces of information can be found in the FAQ.

For a deeper analysis combined with other tools for text analysis, there is the Stanford NER UIMA annotator which uses a modified version of Stanford NER. Developed by Florian Law you can download it here.

All in all, the Stanford NER offers a fast solution for automatic detection and visualization of named entities, which you can achieve easily.

[1] Compatible means that the output of the Stanford NER can be used as input for further analysis as tagging and parsing which is useful for building a processing pipeline.