% (two rows with three columns) beneath the article title.

% More than six makes the first-page appear very cluttered indeed.

%

% Use the \alignauthor commands to handle the names

% and affiliations for an 'aesthetic maximum' of six authors.

% Add names, affiliations, addresses for

% the seventh etc. author(s) as the argument for the

% \additionalauthors command.

% These 'additional authors' will be output/set for you

% without further effort on your part as the last section in

% the body of your article BEFORE References or any Appendices.

% of EIGHT authors. SIX appear on the 'first-page' (for formatting

% reasons) and the remaining two appear in the \additionalauthors section.

%

% \author{

% % You can go ahead and credit any number of authors here,

% % e.g. one 'row of three' or two rows (consisting of one row of three

% % and a second row of one, two or three).

% %

% % The command \alignauthor (no curly braces needed) should

% % precede each author name, affiliation/snail-mail address and

% % e-mail address. Additionally, tag each line of

% % affiliation/address with \affaddr, and tag the

%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\terms{Theory}

\keywords{Information Filtering; Cumulative Citation Recommendation; knowledge maintenance; Stream Filtering;  emerging entities} % NOT required for Proceedings

\section{Introduction}

In 2012, the Text REtrieval Conferences (TREC) introduced the Knowledge Base Acceleration (KBA) track  to help Knowledge Bases(KBs) curators. The track is crucial to address a critical need of KB curators: given KB (Wikipedia or Twitter) entities, filter  a stream  for relevant documents, rank the retrieved documents and recommend them to the KB curators. The track is crucial and timely because  the number of entities in a KB on one hand, and the huge amount of new information content on the Web on the other hand make the task of manual KB maintenance challenging.   TREC KBA's main task, Cumulative Citation Recommendation (CCR), aims at filtering a stream to identify   citation-worthy  documents, rank them,  and recommend them to KB curators.

 Filtering is a crucial step in CCR for selecting a potentially

 relevant set of working documents for subsequent steps of the

represented by persistent KB entities (Wikipedia pages or Twitter

users, in the TREC scenario).

 TREC-KBA 2013's participants applied Filtering as a first step  to

 produce a smaller working set for subsequent experiments. As the

 subsequent steps of the pipeline use the output of the filter, the

 final performance of the system is dependent on this step.  The

 filtering step particularly determines the recall of the overall

 system. However, all 141 runs submitted by 13 teams did suffer from

 poor recall, as pointed out in the track's overview paper 

 \cite{frank2013stream}. 

The most important components of the filtering step are cleansing

(referring to pre-processing noisy web text into a canonical ``clean''

text format), and

entity profiling (creating a representation of the entity that can be

used to match the stream documents to). For each component, different

choices can be made. In the specific case of TREC KBA, organisers have

provided two different versions of the corpus: one that is already cleansed,

and one that is the raw data as originally collected by the organisers. 

Also, different

approaches use different entity profiles for filtering, varying from

using just the KB entities' canonical names to looking up DBpedia name

variants, and from using the bold words in the first paragraph of the Wikipedia

using the exact name as given to WordNet derived synonyms. The type of entities

(Wikipedia or Twitter) and the category of documents in which they

occur (news, blogs, or tweets) cause further variations.

% A variety of approaches are employed  to solve the CCR

% challenge. Each participant reports the steps of the pipeline and the

% final results in comparison to other systems.  A typical TREC KBA

% poster presentation or talk explains the system pipeline and reports

% the final results. The systems may employ similar (even the same)

% steps  but the choices they make at every step are usually

% different. 

In such a situation, it becomes hard to identify the factors that

result in improved performance. There is  a lack of insight across

In this paper, we therefore fix the subsequent steps of the pipeline,

and zoom in on \emph{only} the filtering step; and conduct an in-depth analysis of its

main components.  In particular, we study the effect of cleansing,

entity profiling, type of entity filtered for (Wikipedia or Twitter), and

document category (social, news, etc) on the filtering components'

performance. The main contribution of the

paper are an in-depth analysis of the factors that affect entity-based

stream filtering, identifying optimal entity profiles without

compromising precision, describing and classifying relevant documents

that are not amenable to filtering , and estimating the upper-bound

of recall on entity-based filtering.

We base this analysis on the TREC-KBA 2013 dataset%

\footnote{\url{http://trec-kba.org/trec-kba-2013.shtml}}

that consists of three main parts: a time-stamped stream corpus, a set of

KB entities to be curated, and a set of relevance judgments. A CCR

system now has to identify for each KB entity which documents in the

stream corpus are to be considered by the human curator.

\subsection{Stream corpus} The stream corpus comes in two versions:

raw and cleaned. The raw and cleansed versions are 6.45TB and 4.5TB

respectively,  after xz-compression and GPG encryption. The raw data

is a  dump of  raw HTML pages. The cleansed version is the raw data

after its HTML tags are stripped off and only English documents

are very similar to each other. Mainstream-news and news are

similar. The reason they exist separately, in the first place,  is

because they were collected from two different sources, by different

groups and at different times. we call them news from now on.  The

same is true with weblog and social, and we call them social from now

on.   2) some groups have so small number of annotations that treating

them independently does not make much sense. Majority of vital or

relevant annotations are social (social and weblog) (63.13\%). News

(mainstream +news) make up 30\%. Thus, news and social make up about

93\% of all annotations.  The rest make up about 7\% and are all

grouped as others.

 The TREC Filtering track defines filtering as a ``system that sifts

 through stream of incoming information to find documents that are

 relevant to a set of user needs represented by profiles''

 \cite{robertson2002trec}. Its information needs are long-term and are

 represented by persistent profiles, unlike the traditional search system

 whose adhoc information need is represented by a search

 query. Adaptive Filtering, one task of the filtering track,  starts

 with  a persistent user profile and a very small number of positive

 examples. A filtering system can improve its user profiles with a

 feedback obtained from interaction with users, and thereby improve

 its performance. The  filtering stage of entity-based stream

relavant documents and irrrelevant documensts. Bcause of this,

filtering in this case can only be studied by binding it to the later

stages of the entity-centric pipeline. This bond influnces how we do

evaluation.

To achieve this, we use recall percentages in the filtering stage for

the different choices of entity profiles. However, we use the overall

performance to select the best entity profiles.To generate the overall

pipeline performance we use the official TREC KBA evaluation metric

and scripts \cite{frank2013stream} to report max-F, the maximum

F-score obtained over all relevance cut-offs.

There has been a great deal of interest  as of late on entity-based filtering and ranking. One manifestation of that is the introduction of TREC KBA in 2012. Following that, there have been a number of research works done on the topic \cite{frank2012building, ceccarelli2013learning, taneva2013gem, wang2013bit, balog2013multi}.  These works are based on KBA 2012 task and dataset  and they address the whole problem of entity filtering and ranking.  TREC KBA continued in 2013, but the task underwent some changes. The main change between  the 2012 and 2013 are in the number of entities, the type of entities, the corpus and the relevance rankings.

The number of entities increased from 29 to 141, and it included 20 Twitter entities. The TREC KBA 2012 corpus is 1.9TB after xz-compression and has  400M documents. By contrast, the KBA 2013 corpus is 6.45 after XZ-compression and GPG encryption. A version with all-non English documented removed  is 4.5 TB and consists of 1 Billion documents. The 2013 corpus subsumed the 2012 corpus and added others from spinn3r, namely main-stream news, forum, arxiv, classified, reviews and meme-tracker.  A more important difference is, however, a change in the definitions of relevance ratings vital and relevant. While in KBA 2012, a document was judged vital if it has citation-worthy content for a given entity, in 2013 it must have the freshliness, that is the content must trigger an editing of the given entity's KB entry. 

While the tasks of 2012 and 2013 are fundamentally the same, the approaches  varied due  to the size of the corpus. In 2013, all participants used filtering to reduce the size of the big corpus.   They used different ways of filtering: many of them used two or more of different name variants from DBpedia such as labels, names, redirects, birth names, alias, nicknames, same-as and alternative names \cite{wang2013bit,dietzumass,liu2013related, zhangpris}.  Although most of the participants used DBpedia name variants none of them used all the name variants.  A few other participants used bold words in the first paragraph of the Wikipedia entity's profiles and anchor texts from other Wikipedia pages  \cite{bouvierfiltering, niauniversity}. One participant used Boolean \emph{and} built from the tokens of the canonical names \cite{illiotrec2013}.  

All of the studies used filtering as their first step to generate a smaller set of documents. And many systems suffered from poor recall and their system performances were highly affected \cite{frank2012building}. Although  systems  used different entity profiles to filter the stream, and achieved different performance levels, there is no study on and the factors and choices that affect the filtering step itself. Of course filtering has been extensively examined in TREC Filtering \cite{robertson2002trec}. However, those studies were isolated in the sense that they were intended to optimize recall. What we have here is a different scenario. Documents have relevance rating. Thus we want to study filtering in connection to  relevance to the entities and thus can be done by coupling filtering to the later stages of the pipeline. This is new to the best of our knowledge and the TREC KBA problem setting and data-sets offer a good opportunity to examine this aspect of filtering. 

Moreover, there has not been a chance to study at this scale and/or a study into what type of documents defy filtering and why? In this paper, we conduct a manual examination of the documents that are missing and classify them into different categories. We also estimate the general upper bound of recall using the different entities profiles and choose the best profile that results in an increased over all performance as measured by F-measure. 

All analyses in this paper are carried out on the documents that have

relevance assessments associated to them. For this purpose, we

extracted those documents from the big corpus. We experiment with all

KB entities. For each KB entity, we extract different name variants

from DBpedia and Twitter.

\

\subsection{Entity Profiling}

We build entity profiles for the KB entities of interest. We have two

types: Twitter and Wikipedia. Both entities have been selected, on

purpose by the track organisers, to occur only sparsely and be less-documented.

For the Wikipedia entities, we fetch different name variants

%Most (more than 80\%) of the annotation documents are in the test set.

The 2013 training and test data contain 68405

annotations, of which 50688 are unique document-entity pairs.   Out of

these, 24162 unique document-entity pairs are vital (9521) or relevant

(17424).

 We conducted experiments to study  the effect of cleansing, different entity profiles, types of entities, category of documents, relevance ranks (vital or relevant), and the impact on classification.  In the following subsections, we present the results in different categories, and describe them.

 \subsection{Cleansing: raw or cleansed}

\begin{table}

\caption{Percentage of vital or relevant documents retrieved under different name variants (upper part from cleansed, lower part from raw)}

\begin{center}

\begin{tabular}{l@{\quad}rrrrrrr}

\hline

&cano&cano-part  &all &all-part  \\

\hline

aggregate and document category levels, we observe that recall

increases as we move from canonicals to canonical partial, to

name-variant, and to name-variant partial. The only case where this

does not hold is in the transition from Wikipedia's canonical partial

to name-variant. At the aggregate level (as can be inferred from Table

\ref{tab:name}), the difference in performance between  canonical  and

name-variant partial is 31.9\% on all entities, 20.7\% on Wikipedia

entities, and 79.5\% on Twitter entities. 

Section \ref{sec:analysis} discusses the most plausible explanations for these findings.

%% TODO: PERHAPS SUMMARY OF DISCUSSION HERE

In the overall experimental setup, classification, ranking, and

evaluation are kept constant. Following \cite{balog2013multi}

settings, we use

WEKA's\footnote{\url{http://www.cs.waikato.ac.nz/~ml/weka/}} Classification

Random Forest. However, we use fewer numbers of features which we

found to be more effective. We determined the effectiveness of the

features by running the classification algorithm using the fewer

features we implemented and their features. Our feature

implementations achieved better results.  The total numbers of

features we used are 13 and are listed below.

\paragraph*{Google's Cross Lingual Dictionary (GCLD)}

%%%%%%%%%%%%

In vital ranking, across all entity profiles and types of corpus,

Wikipedia's canonical partial  achieves better performance than any

other Wikipedia entity profiles. In vital-relevant documents too,

Wikipedia's canonical partial achieves the best result. In the raw

corpus, it achieves a little less than name-variant partial. For

Twitter entities, the name-variant partial profile achieves the

highest F-score across all entity profiles and types of corpus.

entities and relevant documents.  This  is validated by the

observation that recall  gains in Twitter entities and the relevant

categories in the raw corpus also translate into overall performance

gains. This observation implies that cleansing removes relevant and

social documents than it does vital and news. That it removes relevant

documents more than vital can be explained by the fact that cleansing

removes the related links and adverts which may contain a mention of

the entities. One example we saw was the the cleansing removed an

image with a text of an entity name which was actually relevant. And

that it removes social documents can be explained by the fact that

most of the missing of the missing  docuemnts from cleansed are

social. And all the docuemnts that are missing from raw corpus

necessary if we do not want to accept a low recall of the filtering

step.

%With this understanding, there  is actually no

%need to go and fetch different names variants from DBpedia, a saving

%of time and computational resources.

%%%%%%%%%%%%

\begin{table}

\caption{The number of documents missing  from raw and cleansed extractions. }

\begin{center}

\begin{tabular}{l@{\quad}llllll}

\hline

\multicolumn{1}{l}{\rule{0pt}{12pt}category}&\multicolumn{1}{l}{\rule{0pt}{12pt}Vital }&\multicolumn{1}{l}{\rule{0pt}{12pt}Relevant }&\multicolumn{1}{l}{\rule{0pt}{12pt}Total }\\[5pt]

\hline

Cleansed &1284 & 1079 & 2363 \\

Raw & 276 & 4951 & 5227 \\

\hline

 missing only from cleansed &1065&2016&3081\\

  missing only from raw  &57 &160 &217 \\

  Missing from both &219 &1927&2146\\

\hline

\end{tabular}

\end{center}

\label{tab:miss}

\end{table}

%%%%%%%%%%%%%%%%%%%%%%

\paragraph*{Outgoing link mentions} A post (tweet) with an outgoing link which mentions the entity.

\paragraph*{Entity -related entity} A document about an important figure such as artist, athlete  can be vital to another. This is specially true if the two are contending for the same title, one has snatched a title, or award from the other. 

\paragraph*{Organization - main activity} A document that talks about about an area on which the company is active is vital for the organization. For example, Atacocha is a mining company  and a news item on mining waste was annotated vital. 

\paragraph*{Entity - group} If an entity belongs to a certain group (class),  a news item about the group can be vital for the individual members. FrankandOak is  named innovative company and a news item that talks about the group  of innovative companies is relevant for a  it. Other examples are: a  big event  of which an entity is related such an Film awards for actors. 

\paragraph*{Artist - work} Documents that discuss the work of artists can be relevant to the artists. Such cases include  books or films being vital for the book author or the director (actor) of the film. Robocop is film whose screenplay is by Joshua Zetumer. A blog that talks about the film was judged vital for Joshua Zetumer. 

\paragraph*{Politician - constituency} A major political event in a certain constituency is vital for the politician from that constituency. 

 A good example is a weblog that talks about two north Dakota counties being drought disasters. The news is vital for Joshua Boschee, a politician, a member of North Dakota democratic party.  

\paragraph*{head - organization} A document that talks about an organization of which the entity is the head can be vital for the entity.  Jasper\_Schneider is USDA Rural Development state director for North Dakota and an article about problems of primary health centers in North Dakota is judged vital for him. 

\paragraph*{World Knowledge} Some things are impossible to know without your world knowledge. For example ''refreshments, treats, gift shop specials, "bountiful, fresh and fabulous holiday decor," a demonstration of simple ways to create unique holiday arrangements for any home; free and open to the public`` is judged relevant to Hjemkomst\_Center. This is a social media post, and unless one knows the person posting it, there is no way that this text shows that. Similarly ''learn about the gray wolf's hunting and feeding behaviors and watch the wolves have their evening meal of a full deer carcass; $15 for members, $20 for nonmembers`` is judged vital to Red\_River\_Zoo.  

\paragraph*{No document content} A small number of documents were found to have no content.

\paragraph*{Disagreement} For a few remaining documents, the authors disagree with the assessors as to why these are vital to the entity.

In this paper, we examined the filtering stage of the entity-centric stream filtering and ranking  by holding the later stages of fixed. In particular, we studied the cleansing step, different entity profiles, type of entities(Wikipedia or Twitter), categories of documents(news, social, or others) and the relevance ratings. We attempted to address the following research questions: 1) does cleansing affect filtering and subsequent performance? 2) what is the most effective way of entity profiling? 3) is filtering different for Wikipedia and Twitter entities? 4) are some type of documents easily filterable and others not? 5) does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline? and 6) what are the circumstances under which vital documents can not be retrieved? 

Cleansing does remove parts or entire contents of documents making them irretrievable. However, because of the introduction of false positives, recall gains by  raw corpus and some  richer entity profiles do not necessarily translate to overall performance gain. The results conclusion on this is mixed in the sense that cleansing helps improve the recall on vital documents and Wikipedia entities, but reduces the recall on Twitter entities and the relative category of relevance ranking. Vital and relevant documents show a difference in retrieval nonperformance documents are easier to filter than relevant.  

Despite an aggressive attempt to filter as many vital-relevant documents as possible,  we observe that there are still documents that we miss. While some are possible to retrieve with some modifications, some others are not. There are some document that indicate that an information filtering system does not seem to get them no matter how rich representation of entities they use. These circumstances under which this happens are many. We found that some documents have no content at all, subjectivity(it is not clear why some are judged vital). However, the main circumstances under which vital  documents can defy filtering is: outgoing link mentions, 

venue-event, entity - related entity, organization - main area of operation, entity - group, artist - artist's work,  party-politician, and world knowledge.  

%ACKNOWLEDGMENTS are optional

%\section{Acknowledgments}