Word Filter Plugin Test

This is a collection of web pages used to test the Word Filter Plugin that
comes with Yioop. To see how Word Filter Plugin works you can do two web crawls
of this folder. Place this folder unzipped under the document root of a machine
you have a web server on. Then have as your seed sites for a Yioop crawl only
the URL of this folder. For example, if this test was being carried out on
localhost, and you placed this folder under document root, then the URL
to use in seed sites is:

http://localhost/word-filter-test-crawl/

You can control whether the Word Filter Plugin is being used  by checking/
unchecking its checkbox under Admin => Page Options => (Crawl Time Tab)
Indexing Plugins. Do a crawl without checking this plugin and you should
see index.html and all of the files p1.html - p8.html being indexed.
If you do a crawl with the plugin checked then p1.html is not in the index
at all because it has term1. p4.html is not indexed as only p1.html links
to it. In search results, p2.html, since it had the word 
term2 on it, has no snippets, and as the only page that links to p5.html
is p2.html, because of the nofollow, p5.html is not indexed. p3.html
and p6.html should be indexed normally. All of the pages have term0 on them,
except p7.html. p7.html is linked to from p1.html as it does not contain
term0, minus rule applies and so Yioop does JUSTFOLLOW for this page.
This means p7.html's summary is not stored, but links are followed.
JUSTFOLLOW has a slightly different semantics than NOINDEX. When NOINDEX
is used the document is actually stored in the index (unlike JUSTFOLLOW).
If another document links to this document, it can be detected. If at
search time a NOINDEX document or a link to a NOINDEX document is about to
be returned, the NOINDEX is detected and the result won't be returned.
With JUSTFOLLOW since the data is not stored in the index we can't tell if
a link pointing to a JUSTFOLLOW page just hasn't been crawled yet or if it
is a link to a JUSTFOLLOW page, so links to JUSTFOLLOW pages might appear
in the index. One can see this effect by doing a search on site:any. The
link that found the p7.html page shows up.
