Lucene

http://en.wikipedia.org/wiki/Lucene

http://news.ycombinator.com/item?id=137802

Lucene is a fast text search engine. Nutch is a derivative project that also has web crawling etc. There is pyLucene too - Python interface for Java based Lucene.