and one that is the raw data as originally collected by the organisers. 

Also, different

approaches use different entity profiles for filtering, varying from

using just the KB entities' canonical names to looking up DBpedia name

variants, and from using the bold words in the first paragraph of the Wikipedia

entities’ page to using anchor texts from other Wikipedia pages, and from

using the exact name as given to WordNet derived synonyms. The type of entities

(Wikipedia or Twitter) and the category of documents in which they

occur (news, blogs, or tweets) cause further variations.

% A variety of approaches are employed  to solve the CCR

% challenge. Each participant reports the steps of the pipeline and the

% final results in comparison to other systems.  A typical TREC KBA

% poster presentation or talk explains the system pipeline and reports

% the final results. The systems may employ similar (even the same)

% steps  but the choices they make at every step are usually

% different. 

In such a situation, it becomes hard to identify the factors that

result in improved performance. There is  a lack of insight across

different approaches. This makes  it hard to know whether the

improvement in performance of a particular approach is due to

preprocessing, filtering, classification, scoring  or any of the

sub-components of the pipeline.

In this paper, we therefore fix the subsequent steps of the pipeline,

and zoom in on \emph{only} the filtering step; and conduct an in-depth analysis of its

main components.  In particular, we study the effect of cleansing,

entity profiling, type of entity filtered for (Wikipedia or Twitter), and

document category (social, news, etc) on the filtering components'

performance. The main contribution of the

paper are an in-depth analysis of the factors that affect entity-based

stream filtering, identifying optimal entity profiles without

compromising precision, describing and classifying relevant documents

that are not amenable to filtering , and estimating the upper-bound

of recall on entity-based filtering.

The rest of the paper is is organized as follows: 

\textbf{TODO!!}

 \section{Data Description}

We base this analysis on the TREC-KBA 2013 dataset%

\footnote{\url{http://trec-kba.org/trec-kba-2013.shtml}}

that consists of three main parts: a time-stamped stream corpus, a set of

KB entities to be curated, and a set of relevance judgments. A CCR

system now has to identify for each KB entity which documents in the

stream corpus are to be considered by the human curator.

\subsection{Stream corpus} The stream corpus comes in two versions:

raw and cleaned. The raw and cleansed versions are 6.45TB and 4.5TB

respectively,  after xz-compression and GPG encryption. The raw data

is a  dump of  raw HTML pages. The cleansed version is the raw data

after its HTML tags are stripped off and only English documents

identified with Chromium Compact Language Detector

\footnote{\url{https://code.google.com/p/chromium-compact-language-detector/}}

are included.  The stream corpus is organized in hourly folders each

of which contains many  chunk files. Each chunk file contains between

hundreds and hundreds of thousands of serialized  thrift objects. One

thrift object is one document. A document could be a blog article, a

news article, or a social media post (including tweet).  The stream

corpus comes from three sources: TREC KBA 2012 (social, news and

linking) \footnote{\url{http://trec-kba.org/kba-stream-corpus-2012.shtml}},

arxiv\footnote{\url{http://arxiv.org/}}, and

spinn3r\footnote{\url{http://spinn3r.com/}}.

Table \ref{tab:streams} shows the sources, the number of hourly

directories, and the number of chunk files.

\begin{table}

\caption{Retrieved documents to different sources }

\begin{center}

 \begin{tabular}{l*{4}{l}l}

 documents     &   chunk files    &    Sub-stream \\

\hline

126,952         &11,851         &arxiv \\

394,381,405      &   688,974        & social \\

134,933,117       &  280,658       &  news \\

5,448,875         &12,946         &linking \\

57,391,714         &164,160      &   MAINSTREAM\_NEWS (spinn3r)\\

36,559,578         &85,769      &   FORUM (spinn3r)\\

14,755,278         &36,272     &    CLASSIFIED (spinn3r)\\

52,412         &9,499         &REVIEW (spinn3r)\\

7,637         &5,168         &MEMETRACKER (spinn3r)\\

1,040,520,595   &      2,222,554 &        Total\\

\end{tabular}

\end{center}

\label{tab:streams}

\end{table}

\subsection{KB entities}

 The KB entities consist of 20 Twitter entities and 121 Wikipedia entities. The selected entities are, on purpose, sparse. The entities consist of 71 people, 1 organization, and 24 facilities.  

\subsection{Relevance judgments}

TREC-KBA provided relevance judgments for training and

testing. Relevance judgments are given as a document-entity

pairs. Documents with citation-worthy content to a given entity are

annotated  as \emph{vital},  while documents with tangentially

relevant content, or documents that lack freshliness o  with content

that can be useful for initial KB-dossier are annotated as

\emph{relevant}. Documents with no relevant content are labeled

\emph{neutral} and spam is labeled as \emph{garbage}. 

%The inter-annotator agreement on vital in 2012 was 70\% while in 2013 it

%is 76\%. This is due to the more refined definition of vital and the

%distinction made between vital and relevant.

\subsection{Breakdown of results by document source category}

%The results of the different entity profiles on the raw corpus are

%broken down by source categories and relevance rank% (vital, or

%relevant).  

In total, the dataset contains 24162 unique entity-document

pairs, vital or relevant; 9521 of these have been labelled as vital,

and the remaining 17424 as relevant.

All documents are categorized into 8 source categories: 0.98\%

arxiv(a), 0.034\% classified(c), 0.34\% forum(f), 5.65\% linking(l),

11.53\% mainstream-news(m-n), 18.40\% news(n), 12.93\% social(s) and

50.2\% weblog(w). We have regrouped these source categories into three

groups ``news'', ``social'', and ``other'', for two reasons: 1) some groups

are very similar to each other. Mainstream-news and news are

similar. The reason they exist separately, in the first place,  is

because they were collected from two different sources, by different

groups and at different times. we call them news from now on.  The

same is true with weblog and social, and we call them social from now

on.   2) some groups have so small number of annotations that treating

them independently does not make much sense. Majority of vital or

relevant annotations are social (social and weblog) (63.13\%). News

(mainstream +news) make up 30\%. Thus, news and social make up about

93\% of all annotations.  The rest make up about 7\% and are all

grouped as others.

 \section{Stream Filtering}

 The TREC Filtering track defines filtering as a ``system that sifts

 through stream of incoming information to find documents that are

 relevant to a set of user needs represented by profiles''

 \cite{robertson2002trec}. Its information needs are long-term and are

 represented by persistent profiles, unlike the traditional search system

 whose adhoc information need is represented by a search

 query. Adaptive Filtering, one task of the filtering track,  starts

 with  a persistent user profile and a very small number of positive

 examples. A filtering system can improve its user profiles with a

 feedback obtained from interaction with users, and thereby improve

 its performance. The  filtering stage of entity-based stream

 filtering and ranking can be likened to the adaptive filtering task

 of the filtering track. The persistent information needs are the KB

 entities, and the relevance judgments are the small number of postive

 examples.

Stream filtering is then the task to, given a stream of documents of news items, blogs

 and social media on one hand and a set of KB entities on the other,

 to filter the stream for  potentially relevant documents  such that

 the relevance classifier(ranker) achieves as maximum performance as

 possible.  Specifically, we conduct in-depth analysis on the choices

 and factors affecting the cleansing step, the entity-profile

 construction, the document category of the stream items, and the type

 of entities (Wikipedia or Twitter) , and finally their impact overall

 performance of the pipeline. Finally, we conduct manual examination

 of the vital documents that defy filtering. We strive to answer the

 following research questions:

\begin{enumerate}

  \item Does cleansing affect filtering and subsequent performance

  \item What is the most effective way of entity profile representation

  \item Is filtering different for Wikipedia and Twitter entities?

  \item Are some type of documents easily filterable and others not?

  \item Does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline?

  \item What characterizes the vital (and relevant) documents that are

    missed in the filtering step?

\end{enumerate}

The TREC filtering and the filtering as part of the entity-centric

stream filtering and ranking pipepline have different purposes. The

TREC filtering track's goal is the binary classification of documents:

for each incoming docuemnt, it decides whether the incoming document

is relevant or not for a given profile. The docuemnts are either

relevant or not. In our case, the documents have relevance ranking and

the goal of the filtering stage is to filter as many potentially

relevant documents as possible, but less  irrelevant documents as

possible not to obfuscate the later stages of the piepline.  Filtering

as part of the pipeline needs that delicate balance between retrieving

relavant documents and irrrelevant documensts. Bcause of this,

filtering in this case can only be studied by binding it to the later

stages of the entity-centric pipeline. This bond influnces how we do

evaluation.

To achieve this, we use recall percentages in the filtering stage for

the different choices of entity profiles. However, we use the overall

performance to select the best entity profiles.To generate the overall

pipeline performance we use the official TREC KBA evaluation metric

and scripts \cite{frank2013stream} to report max-F, the maximum

F-score obtained over all relevance cut-offs.

\section{Literature Review}

There has been a great deal of interest  as of late on entity-based filtering and ranking. One manifestation of that is the introduction of TREC KBA in 2012. Following that, there have been a number of research works done on the topic \cite{frank2012building, ceccarelli2013learning, taneva2013gem, wang2013bit, balog2013multi}.  These works are based on KBA 2012 task and dataset  and they address the whole problem of entity filtering and ranking.  TREC KBA continued in 2013, but the task underwent some changes. The main change between  the 2012 and 2013 are in the number of entities, the type of entities, the corpus and the relevance rankings.

The number of entities increased from 29 to 141, and it included 20 Twitter entities. The TREC KBA 2012 corpus is 1.9TB after xz-compression and has  400M documents. By contrast, the KBA 2013 corpus is 6.45 after XZ-compression and GPG encryption. A version with all-non English documented removed  is 4.5 TB and consists of 1 Billion documents. The 2013 corpus subsumed the 2012 corpus and added others from spinn3r, namely main-stream news, forum, arxiv, classified, reviews and meme-tracker.  A more important difference is, however, a change in the definitions of relevance ratings vital and relevant. While in KBA 2012, a document was judged vital if it has citation-worthy content for a given entity, in 2013 it must have the freshliness, that is the content must trigger an editing of the given entity's KB entry. 

While the tasks of 2012 and 2013 are fundamentally the same, the approaches  varied due  to the size of the corpus. In 2013, all participants used filtering to reduce the size of the big corpus.   They used different ways of filtering: many of them used two or more of different name variants from DBpedia such as labels, names, redirects, birth names, alias, nicknames, same-as and alternative names \cite{wang2013bit,dietzumass,liu2013related, zhangpris}.  Although most of the participants used DBpedia name variants none of them used all the name variants.  A few other participants used bold words in the first paragraph of the Wikipedia entity's profiles and anchor texts from other Wikipedia pages  \cite{bouvierfiltering, niauniversity}. One participant used Boolean \emph{and} built from the tokens of the canonical names \cite{illiotrec2013}.  

All of the studies used filtering as their first step to generate a smaller set of documents. And many systems suffered from poor recall and their system performances were highly affected \cite{frank2012building}. Although  systems  used different entity profiles to filter the stream, and achieved different performance levels, there is no study on and the factors and choices that affect the filtering step itself. Of course filtering has been extensively examined in TREC Filtering \cite{robertson2002trec}. However, those studies were isolated in the sense that they were intended to optimize recall. What we have here is a different scenario. Documents have relevance rating. Thus we want to study filtering in connection to  relevance to the entities and thus can be done by coupling filtering to the later stages of the pipeline. This is new to the best of our knowledge and the TREC KBA problem setting and data-sets offer a good opportunity to examine this aspect of filtering. 

Moreover, there has not been a chance to study at this scale and/or a study into what type of documents defy filtering and why? In this paper, we conduct a manual examination of the documents that are missing and classify them into different categories. We also estimate the general upper bound of recall using the different entities profiles and choose the best profile that results in an increased over all performance as measured by F-measure. 

\section{Method}

All analyses in this paper are carried out on the documents that have

relevance assessments associated to them. For this purpose, we

extracted those documents from the big corpus. We experiment with all

KB entities. For each KB entity, we extract different name variants

from DBpedia and Twitter.

\

\subsection{Entity Profiling}

We build entity profiles for the KB entities of interest. We have two

types: Twitter and Wikipedia. Both entities have been selected, on

purpose by the track organisers, to occur only sparsely and be less-documented.

For the Wikipedia entities, we fetch different name variants

from DBpedia: name, label, birth name, alternative names,

redirects, nickname, or alias. 

These extraction results are summarized in Table

\ref{tab:sources}.

For the Twitter entities, we visit

their respective Twitter pages and fetch their display names. 

\begin{table}

\caption{Number of different DBpedia name variants}

\begin{center}

 \begin{tabular}{l*{4}{c}l}

 Name variant& No. of strings  \\

\hline

 Name  &82\\

 Label   &121\\

Redirect  &49 \\

 Birth Name &6\\

 Nickname & 1&\\

 Alias &1 \\

 Alternative Names &4\\

\hline

\end{tabular}

\end{center}

\label{tab:sources}

\end{table}

The collection contains a total number of 121 Wikipedia entities.

Every entity has a corresponding DBpedia label.  Only 82 entities have

a name string and only 49 entities have redirect strings. (Most of the

entities have only one string, except for a few cases with multiple

redirect strings; Buddy\_MacKay, has the highest (12) number of

redirect strings.) 

We combine the different name variants we extracted to form a set of

strings for each KB entity. For Twitter entities, we used the display

names that we collected. We consider the names of the entities that

are part of the URL as canonical. For example in entity\\

\url{http://en.wikipedia.org/wiki/Benjamin_Bronfman}\\

Benjamin Bronfman is a canonical name of the entity. 

An example is given in Table \ref{tab:profile}.

From the combined name variants and

the canonical names, we  created four sets of profiles for each

entity: canonical(cano) canonical partial (cano-part), all name

variants combined (all) and partial names of all name

variants(all-part). We refer to the last two profiles as name-variant

and name-variant partial. The names in parentheses are used in table

captions.

\begin{table*}

\caption{Example entity profiles (upper part Wikipedia, lower part Twitter)}

\begin{center}

\begin{tabular}{l*{3}{c}}

 &Wikipedia&Twitter \\

\hline

 &Benjamin\_Bronfman& roryscovel\\

  cano&[Benjamin Bronfman] &[roryscovel]\\

  cano-part &[Benjamin, Bronfman]&[roryscovel]\\

  all&[Ben Brewer, Benjamin Zachary Bronfman] &[Rory Scovel] \\

  all-part& [Ben, Brewer, Benjamin, Zachary, Bronfman]&[Rory, Scovel]\\

   \hline                      

\end{tabular}

\end{center}

\label{tab:profile}

\end{table*}

\subsection{Annotation Corpus}

The annotation set is a combination of the annotations from before the Training Time Range(TTR) and Evaluation Time Range (ETR) and consists of 68405 annotations.  Its breakdown into training and test sets is  shown in Table \ref{tab:breakdown}.

\begin{table}

\caption{Number of annotation documents with respect to different categories(relevance rating, training and testing)}

\begin{center}

\begin{tabular}{l*{3}{c}r}

 &&Vital&Relevant  &Total \\

\hline

\multirow{2}{*}{Training}  &Wikipedia & 1932  &2051& 3672\\

			  &Twitter&189   &314&488 \\

			   &All Entities&2121&2365&4160\\

\hline 

\multirow{2}{*}{Testing}&Wikipedia &6139   &12375 &16160 \\

                         &Twitter&1261   &2684&3842  \\

                         &All Entities&7400   &12059&20002 \\

             \hline 

\multirow{2}{*}{Total} & Wikipedia       &8071   &14426&19832  \\

                       &Twitter  &1450  &2998&4330  \\

                       &All Entities&9521   &17424&24162 \\

\hline

\end{tabular}

\end{center}

\label{tab:breakdown}

\end{table}

%Most (more than 80\%) of the annotation documents are in the test set.

The 2013 training and test data contain 68405

annotations, of which 50688 are unique document-entity pairs.   Out of

these, 24162 unique document-entity pairs are vital (9521) or relevant

(17424).

\section{Experiments and Results}

 We conducted experiments to study  the effect of cleansing, different entity profiles, types of entities, category of documents, relevance ranks (vital or relevant), and the impact on classification.  In the following subsections, we present the results in different categories, and describe them.

 \subsection{Cleansing: raw or cleansed}

\begin{table}

\caption{Percentage of vital or relevant documents retrieved under different name variants (upper part from cleansed, lower part from raw)}

\begin{center}

\begin{tabular}{l@{\quad}rrrrrrr}

\hline

&cano&cano-part  &all &all-part  \\

\hline

   Wikipedia      &61.8  &74.8  &71.5  &77.9\\

   Twitter        &1.9   &1.9   &41.7  &80.4\\

   All Entities   &51.0  &61.7  &66.2  &78.4 \\	

\hline

\hline

   Wikipedia      &70.0  &86.1  &82.4  &90.7\\

   Twitter        & 8.7  &8.7   &67.9  &88.2\\

  All Entities    &59.0  &72.2  &79.8  &90.2\\

\hline

\end{tabular}

\end{center}

\label{tab:name}

\end{table}

The upper part of Table \ref{tab:name} shows the recall performances on the cleansed version and the lower part on the raw version. The recall performances for all entity types  are increased substantially in the raw version. Recall increases on Wikipedia entities  vary from 8.2 to 12.8, and in Twitter entities from 6.8 to 26.2. In all entities, it ranges from 8.0 to 13.6.  The recall increases are substantial. To put it into perspective, an 11.8 increase in recall on all entities is a retrieval of 2864 more unique document-entity pairs. %This suggests that cleansing has removed some documents that we could otherwise retrieve. 

\subsection{Entity Profiles}

If we look at the recall performances for the raw corpus,   filtering documents by canonical names achieves a recall of  59\%.  Adding other name variants  improves the recall to 79.8\%, an increase of 20.8\%. This means  20.8\% of documents mentioned the entities by other names  rather than by their canonical names. Canonical partial  achieves a recall of 72\%  and name-variant partial achives 90.2\%. This says that 18.2\% of documents mentioned the entities by  partial names of other non-canonical name variants. 

%\begin{table*}

%\caption{Breakdown of recall percentage increases by document categories }

%\begin{center}\begin{tabular}{l*{9}{c}r}

% && \multicolumn{3}{ c| }{All entities}  & \multicolumn{3}{ c| }{Wikipedia} &\multicolumn{3}{ c| }{Twitter} \\ 

% & &others&news&social & others&news&social &  others&news&social \\

%\hline

% 

%\multirow{4}{*}{Vital}	 &cano-part $-$ cano  	&8.2  &14.9    &12.3           &9.1  &18.6   &14.1             &0      %&0       &0  \\

%                         &all$-$ cano         	&12.6  &19.7    &12.3          &5.5  &15.8   &8.4             &73   &35%.9    &38.3  \\

%	                 &all-part $-$ cano\_part&9.7    &18.7  &12.7       &0    &0.5  &5.1        &93.2 & 93 &64.4 \\%

%	                 &all-part $-$ all     	&5.4  &13.9     &12.7           &3.6  &3.3    &10.8              &20.3 %  &57.1    &26.1 \\

%	                 \hline

%	                 

%\multirow{4}{*}{Relevant}  &cano-part $-$ cano  	&10.5  &15.1    &12.2          &11.1  &21.7   &14.1            % &0   &0    &0  \\

%                         &all $-$ cano         	&11.7  &36.6    &17.3          &9.2  &19.5   &9.9             &%54.5   &76.3   &66  \\

%	                 &all-part $-$ cano-part &4.2  &26.9   &15.8          &0.2    &0.7    &6.7           &72.2   &8%7.6 &75 \\

%	                 &all-part $-$ all     	&3    &5.4     &10.7           &2.1  &2.9    &11              &18.2   &%11.3    &9 \\

%	                 

%	                 \hline

%\multirow{4}{*}{total} 	&cano-part $-$ cano   	&10.9   &15.5   &12.4         &11.9  &21.3   &14.4          &0 %    &0       &0\\

%			&all $-$ cano         	&13.8   &30.6   &16.9         &9.1  &18.9   &10.2          &63.6  &61.8%    &57.5 \\

%                        &all-part $-$ cano-part	&7.2   &24.8   &15.9          &0.1    &0.7    &6.8           &8%2.2  &89.1    &71.3\\

%                        &all-part $-$ all     	&4.3   &9.7    &11.4           &3.0  &3.1   &11.0          &18.9  &27.3%    &13.8\\	                 

%	                 

%                                  	                 

%\hline

%\end{tabular}

%\end{center}

%\label{tab:source-delta2}

%\end{table*}

 \begin{table*}

\caption{Breakdown of recall performances by document source category}

\begin{center}\begin{tabular}{l*{9}{c}r}

 && \multicolumn{3}{ c| }{All entities}  & \multicolumn{3}{ c| }{Wikipedia} &\multicolumn{3}{ c| }{Twitter} \\ 

 & &others&news&social & others&news&social &  others&news&social \\

\hline

\multirow{4}{*}{Vital} &cano                 &82.2& 65.6& 70.9& 90.9&  80.1& 76.8&   8.1&  6.3&  30.5\\

&cano part & 90.4& 80.6& 83.1& 100.0& 98.7& 90.9&   8.1&  6.3&  30.5\\

&all  & 94.8& 85.4& 83.1& 96.4&  95.9& 85.2&   81.1& 42.2& 68.8\\

&all part &100& 99.2& 95.9& 100.0&  99.2& 96.0&   100&  99.3& 94.9\\

\hline

\multirow{4}{*}{Relevant} &cano & 84.2& 53.4& 55.6& 88.4& 75.6& 63.2& 10.6& 2.2& 6.0\\

&cano part &94.7& 68.5& 67.8& 99.6& 97.3& 77.3& 10.6& 2.2& 6.0\\

&all & 95.8& 90.1& 72.9& 97.6& 95.1& 73.1& 65.2& 78.4& 72.0\\

&all part &98.8& 95.5& 83.7& 99.7& 98.0& 84.1& 83.3& 89.7& 81.0\\

	                 \hline

\multirow{4}{*}{total} 	&cano    &   81.1& 56.5& 58.2& 87.7& 76.4& 65.7& 9.8& 3.6& 13.5\\

&cano part &92.0& 72.0& 70.6& 99.6& 97.7& 80.1& 9.8& 3.6& 13.5\\

&all & 94.8& 87.1& 75.2& 96.8& 95.3& 75.8& 73.5& 65.4& 71.1\\

&all part & 99.2& 96.8& 86.6& 99.8& 98.4& 86.8& 92.4& 92.7& 84.9\\

\hline

\end{tabular}

\end{center}

\label{tab:source-delta}

\end{table*}

%The break down of the raw corpus by document source category is presented in Table

%\ref{tab:source-delta}.  

 \subsection{ Relevance Rating: vital and relevant}

When comparing recall for vital and relevant, we observe that

canonical names are more effective for vital than for relevant

entities, in particular for the Wikipedia entities. 

%For example, the recall for news is 80.1 and for social is 76, while the corresponding recall in relevant is 75.6 and 63.2 respectively.

We conclude that the most relevant documents mention the

entities by their common name variants.

%  \subsection{Difference by document categories}

%  

%  Generally, there is greater variation in relevant rank than in vital. This is specially true in most of the Delta's for Wikipedia. This  maybe be explained by news items referring to  vital documents by a some standard name than documents that are relevant. Twitter entities show greater deltas than Wikipedia entities in both vital and relevant. The greater variation can be explained by the fact that the canonical name of Twitter entities retrieves very few documents. The deltas that involve canonical names of Twitter entities, thus, show greater deltas.  

%  

% If we look in recall performances, In Wikipedia entities, the order seems to be others, news and social. This means that others achieve a higher recall than news than social.  However, in Twitter entities, it does not show such a strict pattern. In all, entities also, we also see almost the same pattern of other, news and social. 

\subsection{Recall across document categories: others, news and social}

The recall for Wikipedia entities in Table \ref{tab:name} ranged from

61.8\% (canonicals) to 77.9\% (name-variants).  Table

\ref{tab:source-delta} shows how recall is distributed across document

categories. For Wikipedia entities, across all entity profiles, others

have a higher recall followed by news, and then by social.  While the

recall for news ranges from 76.4\% to 98.4\%, the recall for social

documents ranges from 65.7\% to 86.8\%. In Twitter entities, however,

the pattern is different. In canonicals (and their partials), social

documents achieve higher recall than news.

%This indicates that social documents refer to Twitter entities by their canonical names (user names) more than news do. In name- variant partial, news achieve better results than social. The difference in recall between canonicals and name-variants show that news do not refer to Twitter entities by their user names, they refer to them by their display names.

Overall, across all entities types and all entity profiles, documents

in the others category achieve a higher recall than news, and news documents, in turn, achieve higher recall than social documents. 

% This suggests that social documents are the hardest  to retrieve.  This  makes sense since social posts such as tweets and blogs are short and are more likely to point to other resources, or use short informal names.

%%NOTE TABLE REMOVED:\\\\

%

%We computed four percentage increases in recall (deltas)  between the

%different entity profiles (Table \ref{tab:source-delta2}). The first

%delta is the recall percentage between canonical partial  and

%canonical. The second  is  between name= variant and canonical. The

%third is the difference between name-variant partial  and canonical

%partial and the fourth between name-variant partial and

%name-variant. we believe these four deltas offer a clear meaning. The

%delta between name-variant and canonical means the percentage of

%documents that the new name variants retrieve, but the canonical name

%does not. Similarly, the delta between  name-variant partial and

%partial canonical-partial means the percentage of document-entity

%pairs that can be gained by the partial names of the name variants. 

% The  biggest delta  observed is in Twitter entities between partials

% of all name variants and partials of canonicals (93\%). delta. Both

% of them are for news category.  For Wikipedia entities, the highest

% delta observed is 19.5\% in cano\_part - cano followed by 17.5\% in

% all\_part in relevant. 

  \subsection{Entity Types: Wikipedia and Twitter}

Table \ref{tab:name} summarizes the differences between Wikipedia and

Twitter entities.  Wikipedia entities' canonical representation

achieves a recall of 70\%, while canonical partial achieves a recall of 86.1\%. This is an

increase in recall of 16.1\%. By contrast, the increase in recall of

name-variant partial over name-variant is 8.3\%.

%This high increase in recall when moving from canonical names to their

%partial names, in comparison to the lower increase when moving from

%all name variants to their partial names can be explained by

%saturation: documents have already been extracted by the different

%name variants and thus using their partial names do not bring in many

%new relevant documents.

For Wikipedia entities, canonical

partial achieves better recall than name-variant in both the cleansed and

the raw corpus.  %In the raw extraction, the difference is about 3.7.

In Twitter entities, recall of canonical matching is very low.%

\footnote{Canonical

and canonical partial are the same for Twitter entities because they

are one word strings. For example in https://twitter.com/roryscovel,

``roryscovel`` is the canonical name and its partial is identical.}

%The low recall is because the canonical names of Twitter entities are

%not really names; they are usually arbitrarily created user names. It

%shows that  documents  refer to them by their display names, rarely

%by their user name, which is reflected in the name-variant recall

%(67.9\%). The use of name-variant partial increases the recall to

%88.2\%.

The tables in \ref{tab:name} and \ref{tab:source-delta} show a higher recall

for Wikipedia than for Twitter entities. Generally, at both

aggregate and document category levels, we observe that recall

increases as we move from canonicals to canonical partial, to

name-variant, and to name-variant partial. The only case where this

does not hold is in the transition from Wikipedia's canonical partial

to name-variant. At the aggregate level (as can be inferred from Table

\ref{tab:name}), the difference in performance between  canonical  and

name-variant partial is 31.9\% on all entities, 20.7\% on Wikipedia

entities, and 79.5\% on Twitter entities. 

Section \ref{sec:analysis} discusses the most plausible explanations for these findings.

%% TODO: PERHAPS SUMMARY OF DISCUSSION HERE

\section{Impact on classification}

In the overall experimental setup, classification, ranking, and

evaluation are kept constant. Following \cite{balog2013multi}

settings, we use

WEKA's\footnote{\url{http://www.cs.waikato.ac.nz/~ml/weka/}} Classification

Random Forest. However, we use fewer numbers of features which we

found to be more effective. We determined the effectiveness of the

features by running the classification algorithm using the fewer

features we implemented and their features. Our feature

implementations achieved better results.  The total numbers of

features we used are 13 and are listed below.

\paragraph*{Google's Cross Lingual Dictionary (GCLD)}

This is a mapping of strings to Wikipedia concepts and vice versa

\cite{spitkovsky2012cross}. 

(1) the probability with which a string is used as anchor text to

a Wikipedia entity 

\paragraph*{jac} 

  Jaccard similarity between the document and the entity's Wikipedia page

\paragraph*{cos} 

  Cosine similarity between the document and the entity's Wikipedia page

\paragraph*{kl} 

  KL-divergence between the document and the entity's Wikipedia page

  \paragraph*{PPR}

For each entity, we computed a PPR score from

a Wikipedia snapshot  and we kept the top 100  entities along

with the corresponding scores.

\paragraph*{Surface Form (sForm)}

For each Wikipedia entity, we gathered DBpedia name variants. These

are redirects, labels and names.

\paragraph*{Context (contxL, contxR)}

From the WikiLink corpus \cite{singh12:wiki-links}, we collected

all left and right contexts (2 sentences left and 2 sentences

right) and generated n-grams between uni-grams and quadro-grams

for each left and right context. 

Finally,  we select  the 5 most frequent n-grams for each context.

\paragraph*{FirstPos}

  Term position of the first occurrence of the target entity in the document 

  body 

\paragraph*{LastPos }

  Term position of the last occurrence of the target entity in the document body

\paragraph*{LengthBody} Term count of document body

\paragraph*{LengthAnchor} Term count  of document anchor

\paragraph*{FirstPosNorm} 

  Term position of the first occurrence of the target entity in the document 

  body normalised by the document length 

\paragraph*{MentionsBody }

  No. of occurrences of the target entity in the  document body

  Features we use incude similarity features such as cosine and jaccard, document-entity features such as docuemnt mentions entity in title, in body, frequency  of mention, etc., and related entity features such as page rank scores. In total we sue  The features consist of similarity measures between the KB entiities profile text, document-entity features such as  

  In here, we present results showing how  the choices in corpus, entity types, and entity profiles impact these latest stages of the pipeline.  In tables \ref{tab:class-vital} and \ref{tab:class-vital-relevant}, we show the performances in max-F. 

\begin{table*}

\caption{vital performance under different name variants(upper part from cleansed, lower part from raw)}

\begin{center}

\begin{tabular}{ll@{\quad}lllllll}

\hline

%&\multicolumn{1}{l}{\rule{0pt}{12pt}}&\multicolumn{1}{l}{\rule{0pt}{12pt}cano}&\multicolumn{1}{l}{\rule{0pt}{12pt}canonical partial }&\multicolumn{1}{l}{\rule{0pt}{12pt}name-variant }&\multicolumn{1}{l}{\rule{0pt}{50pt}name-variant partial}\\[5pt]

  &&cano&cano-part&all  &all-part \\

   all-entities &max-F& 0.241&0.261&0.259&0.265\\

%	      &SU&0.259  &0.258 &0.263 &0.262 \\	

   Wikipedia &max-F&0.252&0.274& 0.265&0.271\\

%	      &SU& 0.261& 0.259&  0.265&0.264 \\

   twitter &max-F&0.105&0.105&0.218&0.228\\

%     &SU &0.105&0.250& 0.254&0.253\\

\hline

\hline

  all-entities &max-F & 0.240 &0.272 &0.250&0.251\\

%	  &SU& 0.258   &0.151  &0.264  &0.258\\

   Wikipedia&max-F &0.257&0.257&0.257&0.255\\

%   &SU	     & 0.265&0.265 &0.266 & 0.259\\

   twitter&max-F &0.188&0.188&0.208&0.231\\

%	&SU&    0.269 &0.250 &0.250&0.253\\

\hline

\end{tabular}

\end{center}

\label{tab:class-vital}

\end{table*}

  \begin{table*}

\caption{vital-relevant performances under different name variants(upper part from cleansed, lower part from raw)}

\begin{center}

\begin{tabular}{ll@{\quad}lllllll}

\hline

%&\multicolumn{1}{l}{\rule{0pt}{12pt}}&\multicolumn{1}{l}{\rule{0pt}{12pt}canonical}&\multicolumn{1}{l}{\rule{0pt}{12pt}canonical partial }&\multicolumn{1}{l}{\rule{0pt}{12pt}name-variant }&\multicolumn{1}{l}{\rule{0pt}{50pt}name-variant partial}\\[5pt]

 &&cano&cano-part&all  &all-part \\

   all-entities &max-F& 0.497&0.560&0.579&0.607\\

%	      &SU&0.468  &0.484 &0.483 &0.492 \\	

   Wikipedia &max-F&0.546&0.618&0.599&0.617\\

%   &SU&0.494  &0.513 &0.498 &0.508 \\

   twitter &max-F&0.142&0.142& 0.458&0.542\\

%    &SU &0.317&0.328&0.392&0.392\\

\hline

\hline

  all-entities &max-F& 0.509 &0.594 &0.590&0.612\\

%    &SU       &0.459   &0.502  &0.478  &0.488\\

   Wikipedia &max-F&0.550&0.617&0.605&0.618\\

%   &SU	     & 0.483&0.498 &0.487 & 0.495\\

   twitter &max-F&0.210&0.210&0.499&0.580\\

%	&SU&    0.319  &0.317 &0.421&0.446\\

\hline

\end{tabular}

\end{center}

\label{tab:class-vital-relevant}

\end{table*}

Table \ref{tab:class-vital} shows the recall performance for vitally judged documents.  On Wikipedia entities, except in the canonical profile, the cleansed version achieves  better results than the raw version.  However, on Twitter entities, the raw corpus achieves  better  in all entity profiles (except  in name-variant partial).  At an aggregate (both Wikipedia and Twitter) level, we see that in three profiles, cleansed achieves better.  Only in canonical partial, does raw perform better. Overall cleansed achieves better results than raw.  This result is interesting because we saw in previous sections that the raw corpus achieves  higher recall than cleansed. In the case name-variant partial, for example, 10\% more relevant documents are retrieved in the raw corpus. The gain in recall in raw corpus does not translate into a gain in F\_measure. In fact, in most cases F\_measure decreased. % One explanation for this is that it brings in many false positives from, among related links, adverts, etc.  

For Wikipedia entities,  canonical partial  achieves the highest performance. For Twitter, name-variant partial achieves  better results.

In vital-relevant category (Table \ref{tab:class-vital-relevant}), the performances are different.  Except in canonical partial,  raw achieves better results in all cases. For Twitter entities, the raw corpus achieves better results in all cases.  In terms of  entity profiles, Wikipedia's canonical partial  achieves  the best F-score. For Twitter, as before, canonical partial. The raw corpus has more effect on relevant documents and Twitter entities.  

%The fact that canonical partial names achieve better results is interesting.  We know that partial names were used as a baseline in TREC KBA 2012, but no one of the KBA participants actually used partial names for filtering.

%    

%    

%    \begin{table*}

% \caption{Breakdown of missing documents by sources for cleansed, raw and cleansed-and-raw}

% \begin{center}\begin{tabular}{l*{9}r}

%   &others&news&social \\

% \hline

% 

% 			&missing from raw only &   0 &0   &217 \\

% 			&missing from cleansed only   &430   &1321     &1341 \\

% 

%                          &missing from both    &19 &317     &2196 \\

%                         

%                          

% 

% \hline

% \end{tabular}

% \end{center}

% \label{tab:miss-category}

% \end{table*}

%    To gain more insight, I sampled for each 35 entities, one document-entity pair and looked into the contents. The results are in \ref{tab:miss from both}

%    

%    \begin{table*}

% \caption{Missing documents and their mentions }

% \begin{center}

% 

%  \begin{tabular}{l*{4}{l}l}

%  &entity&mentioned by &remark \\

% \hline

%  Jeremy McKinnon  & Jeremy McKinnon& social, mentioned in read more link\\

% Blair Thoreson   & & social, There is no mention by name, the article talks about a subject that is political (credit rating), not apparent to me\\

%   Lewis and Clark Landing&&Normally, maha music festival does not mention ,but it was held there \\

% Cementos Lima &&It appears a mistake to label it vital. the article talks about insurance and centos lima is a cement company.entity-deleted from wiki\\

% Corn Belt Power Cooperative & &No content at all\\

% Marion Technical Institute&&the text could be of any place. talks about a place whose name is not mentioned. 

%  roryscovel & &Talks about a video hinting that he might have seen in the venue\\

% Jim Poolman && talks of party convention, of which he is member  politician\\

% Atacocha && No mention by name The article talks about waste from mining and Anacocha is a mining company.\\

% Joey Mantia & & a mention of a another speeedskater\\

% Derrick Alston&&Text swedish, no mention.\\

% Paul Johnsgard&& not immediately clear why \\

% GandBcoffee&& not immediately visible why\\

% Bob Bert && talks about a related media and entertainment\\

% FrankandOak&& an article that talks about a the realease of the most innovative companies of which FrankandOak is one. \\

% KentGuinn4Mayor && a theft in a constituency where KentGuinn4Mayor is vying.\\

% Hjemkomst Center && event announcement without mentioning where. it takes a a knowledge of \\

% BlossomCoffee && No content\\

% Scotiabank Per\%25C3\%25BA && no content\\

% Drew Wrigley && politics and talk of oilof his state\\

% Joshua Zetumer && mentioned by his film\\

% Théo Mercier && No content\\

% Fargo Air Museum && No idea why\\

% Stevens Cooperative School && no content\\

% Joshua Boschee && No content\\

% Paul Marquart &&  No idea why\\

% Haven Denney && article on skating competition\\

% Red River Zoo && animal show in the zoo, not indicated by name\\

% RonFunches && talsk about commedy, but not clear whyit is central\\

% DeAnne Smith && No mention, talks related and there are links\\

% Richard Edlund && talks an ward ceemony in his field \\

% Jennifer Baumgardner && no idea why\\

% Jeff Tamarkin && not clear why\\

% Jasper Schneider &&no mention, talks about rural development of which he is a director \\

% urbren00 && No content\\

% \hline

% \end{tabular}

% \end{center}

% \label{tab:miss from both}

% \end{table*}

\section{Analysis and Discussion}\label{sec:analysis}

We conducted experiments to study the impacts on recall of 

different components of the filtering stage of entity-based filtering and ranking pipeline. Specifically 

we conducted experiments to study the impacts of cleansing,