Stemmer
|
|
A stemmer is a program or algorithm which determines the morphological root of a given inflected (or, sometimes, derived) word form -- generally a written word form.
A stemmer for English, for example, should identify the string "cats" (and possibly "catlike", "catty" etc.) as based on the root "cat", and "stemmer", "stemming", "stemmed" as based on "stem".
English stemmers are fairly trivial (with only occasional problems, such as "dries" being the third-person singular present form of the verb "dry", "axes" being the plural of "ax" as well as "axis"); but stemmers become harder to design as the morphology, orthography, and character encoding of the target language becomes more complex. For example, an Italian stemmer is more complex than an English one (because of more possible verb inflections), a Russian one is more complex (more possible noun declensions), a Hebrew one is even more complex (a hairy writing system), and so on.
Stemmers are common elements in query systems, since a user who runs a query on "daffodils" probably cares about documents that contain the word "daffodil" (without the s).
A more complex approach to the problem of determining a morphological root of a word is lemmatisation. This process involves first determining the part of speech of a word, and applying different normalisation rules for each part of speech.
The first ever published stemmer was written by Julie Beth Lovins: Lovins JB (1968) Development of a stemming algorithm, Mechanical Translation and Computational Linguistics, 11: 22-31. This paper was remarkable for its early date, and had great influence on later work in this area.
A later stemmer was written by Martin Porter, and published in Program, Vol 14 no. 3 pp 130-137, July 1980. This stemmer became very widely used, and became the de-facto standard algorithm used for English stemming. Dr Porter received the Tony Kent Strix award in 2000 for his work on stemming and Information retrieval.
Many implementations of this algorithm were written and freely distributed. Unfortunately, many of these implementations contained subtle flaws, and as a result systems using these stemmers performed less well than they ought. To eliminate this source of error, around the year 2000 Martin Porter released an official free-software implementation of the algorithm. Over the next few years, he extended this work by building Snowball, a framework for writing stemming algorithms, and implemented an improved English stemmer together with stemmers for several other languages.
- This article was originally based on material from the Free On-line Dictionary of Computing, which is licensed under the GFDL.
External links
- Snowball - free stemming algorithms for many languages (http://snowball.tartarus.org)
- Stemming library in C++ released under LGPL (http://sourceforge.net/projects/porterstemmers/)
- Unofficial home page of the Lovins stemming algorithm (http://www.cs.waikato.ac.nz/~eibe/stemmers/index.html)
- Official home page of the Porter stemming algorithm (http://www.tartarus.org/~martin/PorterStemmer/index.html)