<<<<<<< HEAD

<<<<<<< HEAD

the filtering stage by examining their contents. This way, we give

estimate of a practical upper-bound of recall for entity-centric stream

=======

>>>>>>> 68fbea2f0372ab9b4199b88f980dbf5e97f49063

=======

the filtering stage by examing their contents. This way, we

estimate a practical upper-bound of recall for entity-centric stream

>>>>>>> 3eb20e9cca3d074a4001a593e626a9269cb5608c

 Filtering is a crucial step in CCR for selecting a potentially

 relevant set of working documents for subsequent steps of the

 pipeline out of a big collection of stream documents. The TREC

 Filtering track defines filtering as a ``system that sifts through

 stream of incoming information to find documents that are relevant to

 a set of user needs represented by profiles''

 \cite{robertson2002trec}. 

In the specific setting of CCR, these profiles are

represented by persistent KB entities (Wikipedia pages or Twitter

 TREC-KBA 2013's participants applied Filtering as a first step  to

 produce a smaller working set for subsequent experiments. As the

 subsequent steps of the pipeline use the output of the filter, the

 final performance of the system is dependent on this step.  The

 filtering step particularly determines the recall of the overall

 system. However, all 141 runs submitted by 13 teams did suffer from

 poor recall, as pointed out in the track's overview paper 

 \cite{frank2013stream}. 

The most important components of the filtering step are cleansing

(referring to pre-processing noisy web text into a canonical ``clean''

text format), and

entity profiling (creating a representation of the entity that can be

used to match the stream documents to). For each component, different

choices can be made. In the specific case of TREC KBA, organisers have

provided two different versions of the corpus: one that is already cleansed,

and one that is the raw data as originally collected by the organisers. 

Also, different

approaches use different entity profiles for filtering, varying from

using just the KB entities' canonical names to looking up DBpedia name

variants, and from using the bold words in the first paragraph of the Wikipedia

entities’ page to using anchor texts from other Wikipedia pages, and from

using the exact name as given to WordNet derived synonyms. The type of entities

(Wikipedia or Twitter) and the category of documents in which they

occur (news, blogs, or tweets) cause further variations.

% A variety of approaches are employed  to solve the CCR

% challenge. Each participant reports the steps of the pipeline and the

% final results in comparison to other systems.  A typical TREC KBA

% poster presentation or talk explains the system pipeline and reports

% the final results. The systems may employ similar (even the same)

% steps  but the choices they make at every step are usually

% different. 

In such a situation, it becomes hard to identify the factors that

result in improved performance. There is  a lack of insight across

different approaches. This makes  it hard to know whether the

improvement in performance of a particular approach is due to

preprocessing, filtering, classification, scoring  or any of the

sub-components of the pipeline.

In this paper, we therefore fix the subsequent steps of the pipeline,

and zoom in on \emph{only} the filtering step; and conduct an in-depth analysis of its

main components.  In particular, we study the effect of cleansing,

entity profiling, type of entity filtered for (Wikipedia or Twitter), and

document category (social, news, etc) on the filtering components'

performance. The main contribution of the

paper are an in-depth analysis of the factors that affect entity-based

stream filtering, identifying optimal entity profiles without

compromising precision, describing and classifying relevant documents

that are not amenable to filtering , and estimating the upper-bound

of recall on entity-based filtering.

\textbf{TODO!!}

<<<<<<< HEAD

We base this analysis on the TREC-KBA 2013 dataset%

\footnote{http://http://trec-kba.org/trec-kba-2013.shtml}

that consists of three main parts: a time-stamped stream corpus, a set of

KB entities to be curated, and a set of relevance judgments. A CCR

system now has to identify for each KB entity which documents in the

stream corpus are to be considered by the human curator.

\subsection{Stream corpus} The stream corpus comes in two versions:

raw and cleaned. The raw and cleansed versions are 6.45TB and 4.5TB

respectively,  after xz-compression and GPG encryption. The raw data

is a  dump of  raw HTML pages. The cleansed version is the raw data

after its HTML tags are stripped off and only English documents

identified with Chromium Compact Language Detector

\footnote{https://code.google.com/p/chromium-compact-language-detector/}

are included.  The stream corpus is organized in hourly folders each

of which contains many  chunk files. Each chunk file contains between

hundreds and hundreds of thousands of serialized  thrift objects. One

thrift object is one document. A document could be a blog article, a

news article, or a social media post (including tweet).  The stream

corpus comes from three sources: TREC KBA 2012 (social, news and

linking) \footnote{http://trec-kba.org/kba-stream-corpus-2012.shtml},

arxiv\footnote{http://arxiv.org/}, and

spinn3r\footnote{http://spinn3r.com/}.

Table \ref{tab:streams} shows the sources, the number of hourly

directories, and the number of chunk files.

=======

>>>>>>> 3eb20e9cca3d074a4001a593e626a9269cb5608c

TREC-KBA provided relevance judgments for training and

testing. Relevance judgments are given as a document-entity

pairs. Documents with citation-worthy content to a given entity are

annotated  as \emph{vital},  while documents with tangentially

relevant content, or documents that lack freshliness o  with content

that can be useful for initial KB-dossier are annotated as

\emph{relevant}. Documents with no relevant content are labeled

\emph{neutral} and spam is labeled as \emph{garbage}. 

%The inter-annotator agreement on vital in 2012 was 70\% while in 2013 it

%is 76\%. This is due to the more refined definition of vital and the

%distinction made between vital and relevant.

In total, the dataset contains 24162 unique entity-document

pairs, vital or relevant; 9521 of these have been labelled as vital,

and the remaining 17424 as relevant.

All documents are categorized into 8 source categories: 0.98\%

arxiv(a), 0.034\% classified(c), 0.34\% forum(f), 5.65\% linking(l),

11.53\% mainstream-news(m-n), 18.40\% news(n), 12.93\% social(s) and

50.2\% weblog(w). We have regrouped these source categories into three

groups ``news'', ``social'', and ``other'', for two reasons: 1) some groups

are very similar to each other. Mainstream-news and news are

similar. The reason they exist separately, in the first place,  is

because they were collected from two different sources, by different

groups and at different times. we call them news from now on.  The

same is true with weblog and social, and we call them social from now

on.   2) some groups have so small number of annotations that treating

them independently does not make much sense. Majority of vital or

relevant annotations are social (social and weblog) (63.13\%). News

(mainstream +news) make up 30\%. Thus, news and social make up about

93\% of all annotations.  The rest make up about 7\% and are all

grouped as others.

 \section{Stream Filtering}

 The TREC Filtering track defines filtering as a ``system that sifts

 through stream of incoming information to find documents that are

 relevant to a set of user needs represented by profiles''

 \cite{robertson2002trec}. Its information needs are long-term and are

 represented by persistent profiles, unlike the traditional search system

 whose adhoc information need is represented by a search

 query. Adaptive Filtering, one task of the filtering track,  starts

 with  a persistent user profile and a very small number of positive

 examples. A filtering system can improve its user profiles with a

 feedback obtained from interaction with users, and thereby improve

 its performance. The  filtering stage of entity-based stream

 filtering and ranking can be likened to the adaptive filtering task

 of the filtering track. The persistent information needs are the KB

 entities, and the relevance judgments are the small number of postive

 examples.

Stream filtering is then the task to, given a stream of documents of news items, blogs

 and social media on one hand and a set of KB entities on the other,

 to filter the stream for  potentially relevant documents  such that

 the relevance classifier(ranker) achieves as maximum performance as

 possible.  Specifically, we conduct in-depth analysis on the choices

 and factors affecting the cleansing step, the entity-profile

 construction, the document category of the stream items, and the type

 of entities (Wikipedia or Twitter) , and finally their impact overall

 performance of the pipeline. Finally, we conduct manual examination

 of the vital documents that defy filtering. We strive to answer the

 following research questions:

\begin{enumerate}

  \item Does cleansing affect filtering and subsequent performance

  \item What is the most effective way of entity profile representation

  \item Is filtering different for Wikipedia and Twitter entities?

  \item Are some type of documents easily filterable and others not?

  \item Does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline?

  \item What characterizes the vital (and relevant) documents that are

    missed in the filtering step?

\end{enumerate}

The TREC filtering and the filtering as part of the entity-centric

stream filtering and ranking pipepline have different purposes. The

TREC filtering track's goal is the binary classification of documents:

for each incoming docuemnt, it decides whether the incoming document

is relevant or not for a given profile. The docuemnts are either

relevant or not. In our case, the documents have relevance ranking and

the goal of the filtering stage is to filter as many potentially

relevant documents as possible, but less  irrelevant documents as

possible not to obfuscate the later stages of the piepline.  Filtering

as part of the pipeline needs that delicate balance between retrieving

relavant documents and irrrelevant documensts. Bcause of this,

filtering in this case can only be studied by binding it to the later

stages of the entity-centric pipeline. This bond influnces how we do

evaluation.

To achieve this, we use recall percentages in the filtering stage for

the different choices of entity profiles. However, we use the overall

performance to select the best entity profiles.To generate the overall

pipeline performance we use the official TREC KBA evaluation metric

and scripts \cite{frank2013stream} to report max-F, the maximum

F-score obtained over all relevance cut-offs.

\section{Literature Review}

All analyses in this paper are carried out on the documents that have

relevance assessments associated to them. For this purpose, we

extracted those documents from the big corpus. We experiment with all

KB entities. For each KB entity, we extract different name variants

from DBpedia and Twitter.

<<<<<<< HEAD

We have a total of 121 Wikipedia entities.  Every entity has a DBpedia label.  Only 82 entities have a name string and only 49 entities have redirect strings. Most of the entities have only one string, but some have several redirect sterings. One entity, Buddy\_MacKay, has the highest (12) number of redirect strings. 6 entities have  birth names, 1 entity has a nick name, 1 entity has alias and  4 entities have alternative names.

We combined the different name variants  we extracted to form a set of strings for each KB entity.  For Twitter entities, we used the display names that we collected . We consider the names of the entities that are part of the URL as canonical. For example in http://en.wikipedia.org/wiki/Benjamin\_Bronfman, Benjamin Bronfman is a canonical name of the entity.  From the combined name variants and the canonical names, we  created four sets of profiles for each entity: canonical(cano) canonical partial (cano-part), all name variants combined (all) and partial names of all name variants(all-part). We refer to the last two profiles as name-variant and name-variant partial. The names in paranthesis are used in table captions.

\begin{table*}

\caption{Example entity profiles (upper part Wikipedia, lower part Twitter)}

\begin{center}

\begin{tabular}{l*{3}{c}}

 &Wikipedia&Twitter \\

\hline

 &Benjamin\_Bronfman& roryscovel\\

  cano&[Benjamin Bronfman] &[roryscovel]\\

  cano-part &[Benjamin, Bronfman]&[roryscovel]\\

  all&[Ben Brewer, Benjamin Zachary Bronfman] &[Rory Scovel] \\

  all-part& [Ben, Brewer, Benjamin, Zachary, Bronfman]&[Rory, Scovel]\\

   \hline                      

\end{tabular}

\end{center}

\label{tab:breakdown}

\end{table*}

=======

>>>>>>> 3eb20e9cca3d074a4001a593e626a9269cb5608c