%SUGGESTION:

%\title{The Impact of Entity-Centric Stream Filtering on Recall and

%  Missed Documents}

We use the TREC-KBA 2013 dataset \footnote{http://http://trec-kba.org/trec-kba-2013.shtml}. The dataset consists of a time-stamped  stream corpus, a set of KB entities, and a set of relevance judgments. 

\subsection{Stream corpus} The stream corpus comes in two versions: raw and cleaned. The raw  and cleansed versions are 6.45TB and 4.5TB respectively,  after xz-compression and GPG encryption. The raw data is a  dump of  raw HTML pages. The cleansed version is the raw data after its HTML tags are stripped off and only English documents identified with Chromium Compact Language Detector \footnote{https://code.google.com/p/chromium-compact-language-detector/} are included.  The stream corpus is organized in hourly folders each of which contains many  chunk files. Each chunk file contains between hundreds and hundreds of thousands of serialized  thrift objects. One thrift object is one document. A document could be a blog article, a news article, or a social media post (including tweet).  The stream corpus comes from three sources: TREC KBA 2012 (social, news and linking) \footnote{http://trec-kba.org/kba-stream-corpus-2012.shtml}, arxiv\footnote{http://arxiv.org/}, and spinn3r\footnote{http://spinn3r.com/}. Table \ref{tab:streams}   shows the sources, the number of hourly directories, and the number of chunk files. 

 \begin{tabular}{r*{4}{r}l}

\subsection{Breakdown of results by document source category}

%The results of the different entity profiles on the raw corpus are

%broken down by source categories and relevance rank% (vital, or

%relevant).  

In total, there are 24162 vital or relevant unique entity-document

pairs. 9521 of them are vital  and  17424 are relevant. These

documents  are categorized into 8 source categories: 0.98\% arxiv(a),

0.034\% classified(c), 0.34\% forum(f), 5.65\% linking(l), 11.53\%

mainstream-news(m-n), 18.40\% news(n), 12.93\% social(s) and 50.2\%

weblog(w). We have regrouped these source categories into three groups

``news'', ``social'', and ``other'', for two reasons: 1) some groups

are very similar to each other. Mainstream-news and news are

similar. The reason they exist separately, in the first place,  is

because they were collected from two different sources, by different

groups and at different times. we call them news from now on.  The

same is true with weblog and social, and we call them social from now

on.   2) some groups have so small number of annotations that treating

them independently does not make much sense. Majority of vital or

relevant annotations are social (social and weblog) (63.13\%). News

(mainstream +news) make up 30\%. Thus, news and social make up about

93\% of all annotations.  The rest make up about 7\% and are all

grouped as others. 

To achieve this,  we use recall percentages in the filtering stage for the different choices of entity profiles. However, we use the overall performance to select the best entity profiles.To generate the overall pipeline performance we use the official TREC KBA evaluation metric and scripts \cite{frank2013stream} to report max-F, the maximum F-score obtained over all relevance cut-offs. 

%\begin{table*}

%\caption{Breakdown of recall percentage increases by document categories }

%\begin{center}\begin{tabular}{l*{9}{c}r}

% && \multicolumn{3}{ c| }{All entities}  & \multicolumn{3}{ c| }{Wikipedia} &\multicolumn{3}{ c| }{Twitter} \\ 

% & &others&news&social & others&news&social &  others&news&social \\

%\hline

% 

%\multirow{4}{*}{Vital}	 &cano-part $-$ cano  	&8.2  &14.9    &12.3           &9.1  &18.6   &14.1             &0      %&0       &0  \\

%                         &all$-$ cano         	&12.6  &19.7    &12.3          &5.5  &15.8   &8.4             &73   &35%.9    &38.3  \\

%	                 &all-part $-$ cano\_part&9.7    &18.7  &12.7       &0    &0.5  &5.1        &93.2 & 93 &64.4 \\%

%	                 &all-part $-$ all     	&5.4  &13.9     &12.7           &3.6  &3.3    &10.8              &20.3 %  &57.1    &26.1 \\

%	                 \hline

%	                 

%\multirow{4}{*}{Relevant}  &cano-part $-$ cano  	&10.5  &15.1    &12.2          &11.1  &21.7   &14.1            % &0   &0    &0  \\

%                         &all $-$ cano         	&11.7  &36.6    &17.3          &9.2  &19.5   &9.9             &%54.5   &76.3   &66  \\

%	                 &all-part $-$ cano-part &4.2  &26.9   &15.8          &0.2    &0.7    &6.7           &72.2   &8%7.6 &75 \\

%	                 &all-part $-$ all     	&3    &5.4     &10.7           &2.1  &2.9    &11              &18.2   &%11.3    &9 \\

%	                 

%	                 \hline

%\multirow{4}{*}{total} 	&cano-part $-$ cano   	&10.9   &15.5   &12.4         &11.9  &21.3   &14.4          &0 %    &0       &0\\

%			&all $-$ cano         	&13.8   &30.6   &16.9         &9.1  &18.9   &10.2          &63.6  &61.8%    &57.5 \\

%                        &all-part $-$ cano-part	&7.2   &24.8   &15.9          &0.1    &0.7    &6.8           &8%2.2  &89.1    &71.3\\

%                        &all-part $-$ all     	&4.3   &9.7    &11.4           &3.0  &3.1   &11.0          &18.9  &27.3%    &13.8\\	                 

%	                 

%                                  	                 

%\hline

%\end{tabular}

%\end{center}

%\label{tab:source-delta2}

%\end{table*}

The break down of the raw corpus by document source category is presented in Table

\ref{tab:source-delta}.  

%%NOTE TABLE REMOVED:\\\\

%

%We computed four percentage increases in recall (deltas)  between the

%different entity profiles (Table \ref{tab:source-delta2}). The first

%delta is the recall percentage between canonical partial  and

%canonical. The second  is  between name= variant and canonical. The

%third is the difference between name-variant partial  and canonical

%partial and the fourth between name-variant partial and

%name-variant. we believe these four deltas offer a clear meaning. The

%delta between name-variant and canonical measn the percentage of

%documents that the new name variants retrieve, but the canonical name

%does not. Similarly, the delta between  name-variant partial and

%partial canonical-partial means the percentage of document-entity

%pairs that can be gained by the partial names of the name variants. 

% The  biggest delta  observed is in Twitter entities between partials

% of all name variants and partials of canonicals (93\%). delta. Both

% of them are for news category.  For Wikipedia entities, the highest

% delta observed is 19.5\% in cano\_part - cano followed by 17.5\% in

% all\_part in relevant. 

%% TODO: PERHAPS SUMMARY OF DISCUSSION HERE

\section{Impact on classification}

  In here, we present results showing how  the choices in corpus, entity types, and entity profiles impact these latest stages of the pipeline.  In tables \ref{tab:class-vital} and \ref{tab:class-vital-relevant}, we show the performances in max-F. 

   all-entities &max-F& 0.241&0.261&0.259&0.265\\

%	      &SU&0.259  &0.258 &0.263 &0.262 \\	

   Wikipedia &max-F&0.252&0.274& 0.265&0.271\\

%	      &SU& 0.261& 0.259&  0.265&0.264 \\

   twitter &max-F&0.105&0.105&0.218&0.228\\

%     &SU &0.105&0.250& 0.254&0.253\\

  all-entities &max-F & 0.240 &0.272 &0.250&0.251\\

%	  &SU& 0.258   &0.151  &0.264  &0.258\\

   Wikipedia&max-F &0.257&0.257&0.257&0.255\\

%   &SU	     & 0.265&0.265 &0.266 & 0.259\\

   twitter&max-F &0.188&0.188&0.208&0.231\\

%	&SU&    0.269 &0.250 &0.250&0.253\\

   all-entities &max-F& 0.497&0.560&0.579&0.607\\

%	      &SU&0.468  &0.484 &0.483 &0.492 \\	

   Wikipedia &max-F&0.546&0.618&0.599&0.617\\

%   &SU&0.494  &0.513 &0.498 &0.508 \\

   twitter &max-F&0.142&0.142& 0.458&0.542\\

%    &SU &0.317&0.328&0.392&0.392\\

  all-entities &max-F& 0.509 &0.594 &0.590&0.612\\

%    &SU       &0.459   &0.502  &0.478  &0.488\\

   Wikipedia &max-F&0.550&0.617&0.605&0.618\\

%   &SU	     & 0.483&0.498 &0.487 & 0.495\\

   twitter &max-F&0.210&0.210&0.499&0.580\\

%	&SU&    0.319  &0.317 &0.421&0.446\\

%%%%% MOVED FROM LATER ON - CHECK FLOW

There is a trade-off between using a richer entity-profile and retrieval of irrelevant documents. The richer the profile, the more relevant documents it retrieves, but also the more irrelevant documents. To put it into perspective, lets compare the number of documents that are retrieved with  canonical partial and with name-variant partial. Using the raw corpus, the former retrieves a total of 2547487 documents and achieves a recall of 72.2\%. By contrast, the later retrieves a total of 4735318 documents and achieves a recall of 90.2\%. The total number of documents extracted increases by 85.9\% for a recall gain of 18\%. The rest of the documents, that is 67.9\%, are newly introduced irrelevant documents. 

%%%%%%%%%%%%

There are 3 interesting observations: 

1) cleansing impacts Twitter

entities and relevant documents.  This  is validated by the

observation that recall  gains in Twitter entities and the relevant

categories in the raw corpus also translate into overall performance

gains. This observation implies that cleansing removes relevant and

social documents than it does vital and news. That it removes relevant

documents more than vital can be explained by the fact that cleansing

removes the related links and adverts which may contain a mention of

the entities. One example we saw was the the cleansing removed an

image with a text of an entity name which was actually relevant. And

that it removes social documents can be explained by the fact that

most of the missing of the missing  docuemnts from cleansed are

social. And all the docuemnts that are missing from raw corpus

social. So in both cases socuial seem to suffer from text

transformation and cleasing processes. 

%%%% NEEDS WORK:

2) Taking both performance (recall at filtering and overall F-score

during evaluation) into account, there is a clear trade-off between using a richer entity-profile and retrieval of irrelevant documents. The richer the profile, the more relevant documents it retrieves, but also the more irrelevant documents. To put it into perspective, lets compare the number of documents that are retrieved with  canonical partial and with name-variant partial. Using the raw corpus, the former retrieves a total of 2547487 documents and achieves a recall of 72.2\%. By contrast, the later retrieves a total of 4735318 documents and achieves a recall of 90.2\%. The total number of documents extracted increases by 85.9\% for a recall gain of 18\%. The rest of the documents, that is 67.9\%, are newly introduced irrelevant documents. 

Wikipedia's canonical partial is the best entity profile for Wikipedia entities. This is interesting  to see that the retrieval of of  thousands vital-relevant document-entity pairs by name-variant partial does not translate to an increase in over all performance. It is even more interesting since canonical partial was not considered as contending profile for stream filtering by any of participant to the best of our knowledge. With this understanding, there  is actually no need to go and fetch different names variants from DBpedia, a saving of time and computational resources.

%%%%%%%%%%%%

\subsection{Missing vital-relevant documents \label{miss}}

 The use of name-variant partial for filtering is an aggressive attempt to retrieve as many relevant documents as possible at the cost of retrieving irrelevant documents. However, we still miss about  2363(10\%) of the vital-relevant documents.  Why are these documents missed? If they are not mentioned by partial names of name variants, what are they mentioned by? Table \ref{tab:miss} shows the documents that we miss with respect to cleansed and raw corpus.  The upper part shows the number of documents missing from cleansed and raw versions of the corpus. The lower part of the table shows the intersections and exclusions in each corpus.  

\begin{table}

\caption{The number of documents missing  from raw and cleansed extractions. }

\begin{center}

\begin{tabular}{l@{\quad}llllll}

\hline

\multicolumn{1}{l}{\rule{0pt}{12pt}category}&\multicolumn{1}{l}{\rule{0pt}{12pt}Vital }&\multicolumn{1}{l}{\rule{0pt}{12pt}Relevant }&\multicolumn{1}{l}{\rule{0pt}{12pt}Total }\\[5pt]

\hline

Cleansed &1284 & 1079 & 2363 \\

Raw & 276 & 4951 & 5227 \\

\hline

 missing only from cleansed &1065&2016&3081\\

  missing only from raw  &57 &160 &217 \\

  Missing from both &219 &1927&2146\\

\hline

\end{tabular}

\end{center}

\label{tab:miss}

\end{table}

One would  assume that  the set of document-entity pairs extracted from cleansed are a sub-set of those   that are extracted from the raw corpus. We find that that is not the case. There are 217  unique entity-document pairs that are retrieved from the cleansed corpus, but not from the raw. 57 of them are vital.    Similarly,  there are  3081 document-entity pairs that are missing  from cleansed, but are present in  raw. 1065 of them are vital.  Examining the content of the documents reveals that it is due to a missing part of text from a corresponding document.  All the documents that we miss from the raw corpus are social. These are documents such as tweets and blogs, posts from other social media. To meet the format of the raw data (binary byte array), some of them must have been converted later, after collection and on the way lost a part or the entire content. It is similar for the documents that we miss from cleansed: a part or the entire content  is lost in during the cleansing process (the removal of HTML tags and non-English documents).  In both cases the mention of the entity happened to be on the part of the text that is cut out during transformation. 

 The interesting set  of relevance judgments are those that  we miss from both raw and cleansed extractions. These are 2146 unique document-entity pairs, 219 of them are with vital relevance judgments.   The total number of entities in the missed vital annotations is  28 Wikipedia and 7  Twitter, making a total of 35. The  great majority (86.7\%) of the documents are social. This suggests that social (tweets and blogs) can talk about the entities without mentioning  them by name more than news and others do. This is, of course, inline with intuition. 

%%%%%%%%%%%%%%%%%%%%%%

We observed that there are vital-relevant documents that we miss from raw only, and similarly from cleansed only. The reason for this is transformation from one format to another. The most interesting documents are those that we miss from both raw and cleansed corpus. We first identified the number of KB entities who have a vital relevance judgment and  whose documents can not be retrieved (they were 35 in total) and conducted a manual examination into their content to find out why they are missing.