126,952         &11,851         &arxiv \\

394,381,405      &   688,974        & social \\

134,933,117       &  280,658       &  news \\

5,448,875         &12,946         &linking \\

57,391,714         &164,160      &   MAINSTREAM\_NEWS (spinn3r)\\

36,559,578         &85,769      &   FORUM (spinn3r)\\

14,755,278         &36,272     &    CLASSIFIED (spinn3r)\\

52,412         &9,499         &REVIEW (spinn3r)\\

7,637         &5,168         &MEMETRACKER (spinn3r)\\

1,040,520,595   &      2,222,554 &        Total\\

\end{tabular}

\end{center}

\label{tab:streams}

\end{table}

\subsection{KB entities}

The KB entities in the dataset consist of 20 Twitter entities and 121

Wikipedia entities. These selected entities are, on purpose, ``sparse

entities'' in the sense that they only occur in relatively few

documents. The collection of entities consists of 71 people entities,

1 organization entity, and 24 facilities entities.  

\subsection{Relevance judgments}

TREC-KBA provided relevance judgments for training and

testing. Relevance judgments are given as a document-entity

pairs. Documents with citation-worthy content to a given entity are

annotated  as \emph{vital},  while documents with tangentially

relevant content, or documents that lack freshliness o  with content

that can be useful for initial KB-dossier are annotated as

\emph{relevant}. Documents with no relevant content are labeled

\emph{neutral} and spam is labeled as \emph{garbage}. 

%The inter-annotator agreement on vital in 2012 was 70\% while in 2013 it

%is 76\%. This is due to the more refined definition of vital and the

%distinction made between vital and relevant.

\subsection{Breakdown of results by document source category}

%The results of the different entity profiles on the raw corpus are

%broken down by source categories and relevance rank% (vital, or

%relevant).  

In total, the dataset contains 24162 unique entity-document

pairs, vital or relevant; 9521 of these have been labelled as vital,

and the remaining 17424 as relevant.

All documents are categorized into 8 source categories: 0.98\%

arxiv(a), 0.034\% classified(c), 0.34\% forum(f), 5.65\% linking(l),

50.2\% weblog(w). We have regrouped these source categories into three

groups ``news'', ``social'', and ``other'', for two reasons: 1) some groups

are very similar to each other. Mainstream-news and news are

similar. The reason they exist separately, in the first place,  is

because they were collected from two different sources, by different

groups and at different times. we call them news from now on.  The

same is true with weblog and social, and we call them social from now

on.   2) some groups have so small number of annotations that treating

them independently does not make much sense. Majority of vital or

relevant annotations are social (social and weblog) (63.13\%). News

(mainstream +news) make up 30\%. Thus, news and social make up about

93\% of all annotations.  The rest make up about 7\% and are all

grouped as others.

 \section{Stream Filtering}\label{sec:fil}

 The TREC Filtering track defines filtering as a ``system that sifts

 through stream of incoming information to find documents that are

 relevant to a set of user needs represented by profiles''

 \cite{robertson2002trec}. Its information needs are long-term and are

 represented by persistent profiles, unlike the traditional search system

 whose adhoc information need is represented by a search

 query. Adaptive Filtering, one task of the filtering track,  starts

 with  a persistent user profile and a very small number of positive

 examples. A filtering system can improve its user profiles with a

 feedback obtained from interaction with users, and thereby improve

 its performance. The  filtering stage of entity-based stream

 filtering and ranking can be likened to the adaptive filtering task

 of the filtering track. The persistent information needs are the KB

 entities, and the relevance judgments are the small number of postive

 examples.

Stream filtering is then the task to, given a stream of documents of news items, blogs

 and social media on one hand and a set of KB entities on the other,

 to filter the stream for  potentially relevant documents  such that

 the relevance classifier(ranker) achieves as maximum performance as

 possible.  Specifically, we conduct in-depth analysis on the choices

 and factors affecting the cleansing step, the entity-profile

 construction, the document category of the stream items, and the type

 of entities (Wikipedia or Twitter) , and finally their impact overall

 performance of the pipeline. Finally, we conduct manual examination

 of the vital documents that defy filtering. We strive to answer the

 following research questions:

\begin{enumerate}

  \item Does cleansing affect filtering and subsequent performance

  \item What is the most effective way of entity profile representation

  \item Is filtering different for Wikipedia and Twitter entities?

  \item Are some type of documents easily filterable and others not?