documents (news, blog, tweets) can influence filtering.

 A variety of approaches are employed  to solve the CCR challenge. Each participant reports the steps of the pipeline and the final results in comparison to other systems.  A typical TREC KBA poster presentation or talk explains the system pipeline and reports the final results. The systems may employ similar (even the same) steps  but the choices they make at every step are usually different. In such a situation, it becomes hard to identify the factors that result in improved performance. There is  a lack of insight across different approaches. This makes  it hard to know  whether the improvement in performance of a particular approach is due to preprocessing, filtering, classification, scoring  or any of the sub-components of the pipeline. 

 Identifying optimal entity profiles vis-avis not compromising precision

 Describing relevant documents that are not amenable to filtering and thereby estimating the upper-bound on entity-based filtering

  \item Are there vital (relevant) documents that are not filterable by a reasonable system?

The main change between the the 2012 and 2013 are in the number of entities, the type of entities, the corpus and the relevance rankings.The number of entities increased from 29 to 141, and it included 20 Twitter entities. The TREC KBA 2012 corpus was 1.9TB after xz-compression and had  400M documents. By contrast, the KBA 2013 corpus was 6.45 after XZ-compression and GPG encryption. A version with all-non English documented removed  is 4.5 TB and consists of 1 Billion documents. The 2013 corpus subsumed the 2012 corpus and added others from spinn3r, namely main-stream news, forum, arxiv,classified, reviews and meme-tracker.  A more important difference is, however, that the definition of the relevance ranking changed. 

The change in between KBA 2012 and 2013 is that in the definitions of vital and relevant. While in KBA 2012, a document was judged vital if it has citation-worthy content, In 2013 it must have the freshliness, that is the content must trigger an editing of the KB entry. 

While the task of 2012 and 2013 are fundamentally the same, the approaches for the tasks varied due  to the size of the corpus. In the 2013, all participants used filtering to reduce the size of the big corpus.   They used different ways of filtering: Many of them used two or more of different name variants from DBpedia such as labels, names, redirects, birth names, alias, nicknames, same-as and alternative names \cite{wang2013bit, dietzumass ,liu2013related, zhangpris}.  Although all of the participants used DBpedia name variants such as name, label, birth name, alternative names, redirects, nickname, or alias, none of them used all them.  A few other participants used bold words in the first paragraph of the Wikipedia entity's profiles and anchor texts from other Wikipedia pages  \cite{bouvierfiltering, niauniversity}.  Very few participants used Boolean And built from the tokens of the canonical names \cite{illiotrec2013}.  

All of the studies mentions used filtering as their first step to generate a smaller set of documents. However, none of them conducted a study on it.   Of all the systems submitted for TREC conference, the highest recall achieved is 83\% \cite{frank2012building}. However, many systems suffered from poor recall and their system performances were highly affected \cite{frank2012building}.

Although as mentioned before, systems have used different entity profiles to filter the stream, and achieved different performance levels, there is no study on and the factors and choices that affect the filtering step itself. Of course filtering has been extensively examined in TREC Filtering \cite{robertson2002trec}. However, those studies were isolated in the sense that they were intended to optimize recall. What we have here is a different scenario. Documents have relevance rating. Thus we want to study filtering in connection to the relevance to the entities and thus can be done by coupling filtering to the later stages of the pipeline. This is new to the best of our knowledge and the TREC KBA problem setting and data-sets offer a good opportunity to examine this aspect of filtering. 

Moreover, there has not been a chance to study at this scale and/or a study into what type of documents defy filtering and why? In this paper, we conduct a manual examination of the documents that are missing and classify them into different categories. We also estimate the general upper bound of recall for different entities profiles and choose the best profile for maximum precision.

The results of the breakdown by document categopries is presented in a multi-dimensional table shown in \ref{tab:source-delta}. There are three outer columns for  all entities, Wikipedia and Twitter. Each of the outer columns consist of the document categories of other,news and social. The rows consist of Vital, relevant and total each of which have the four percentage deltas.   The first delta is the recall percentage between partial names of canonical names and canonical names. The second  is the delta between name variants and canonical names. The third is the difference between partial names of name variants  and partial names of canonical names and the fourth between partial names of name variants and name variants.  

 When comparing the recall performances in vital and relevant, we observe that canonical names achieve better in vital than in relaxant. This is specially true with Wikipedia entities. For example, the recall for news is 80.1 and for social is 76, while the corresponding recall in relevant is 75.6 and 63.2 respectively. We can generally see that the recall in vital are better than the recall in relevant suggesting that relevant documents are more probable to mention the entities and if they do, using some of their common name variants. 

 \subsection{Difference by document categories}

 There are is some consistent difference between others, news and social. In the Wikipedia entities category, one can see that the recall of others is higher than news. The recall of news is higher than social. This holds across all name variants suggesting that social documents are more difficult to retrieve that other and news. Notice that the others category stands for arxiv (scientific documents), classifieds, forums and linking. However, there is no clear pattern when it comes to Twitter entities. On all entities, we see the effect of the Wikipedia entities overriding the Twitter entities and hence the order of others, news and social. 

If we look in recall performances, In Wikipedia entities, the order seems to be others, news and social. This means that others achieve a higher recall than news than social.  However, in Twitter entities, it does not show such a strict pattern. In all, entities also, we also see almost the same pattern of other, news and social. It seems that news documents are the hardest to retrieve. This is of course makes sense since social posts are short and are more likely to point to other resources, or use short informal names.

\subsection{Document category: others. news and social}

There is a trade-off between using a richer entity-profile and retrieval of irrelevant documents. The richer the profile, the more relevant documents it retrieves, but also the more irrelevant documents. To put into perspective, lets compare the number of documents that are retrieved with partial names of name variants and partial names of canonical names. Using the raw corpus, the partial names of canonical names extracts a total of 2547487 and achieves a recall of 72.2\%. By contrast, the partial names of name variants extracts a total 4735318 documents and achieves a recall of 90.2\%. The total number of documents extracted increases by 85.9\% for a recall gain of 18\%. The rest of the documents, that is 67.9\%, are newly introduced irrelevant documents. There is an advantage in excluding irrelevant documents from filtering because they confuse the later stages of the pipeline. 

 The use of the partial names of name variants for filtering is, therefore, an aggressive attempt to retrieve as many relevant documents as possible at the cost retrieving irrelevant documents. However, we still miss about  2363(10\%) documents.  Why are these documents missed? If they are not mentioned by partial names of name variants, what are they mentioned by? Table \ref{tab:miss} shows the documents that miss with respect to cleansed and raw corpus.  The upper part shows the number of documents missing from cleansed and raw versions of the corpus. The lower part of the table shows the intersections and exclusions in each corpus.  

\caption{The number of documents that are missing from raw and cleansed extractions. }

 The interesting set of of relevance judgments are those that  we miss from both raw and cleansed extractions. These are 2146 unique document-entity pairs, 219 of them are with vital relevance judgments.   The total number of entities in the missed vital annotations is  28 Wikipedia and 7  Twitter, making a total of 35. Looking into document categories shows that the  great majority (86.7\%) of the documents are social. This suggests that social (tweets and blogs) can talk about the entities without mentioning  them by name. This is, of course, inline with intuition. 

However, it is interesting to look into the actual content of the documents to gain an insight into the ways a document can talk about an entity without mentioning the entity . We collected 35 documents, one for each entity, for manual examination. Here below we present the reasons.

\paragraph{Event place - Event} a document that talks about an event is vital to the location entity where it takes place.  For example Maha Music Festival takes place in Lewis and Clark\_Landing, and a document talking about the festival is vital for the park. There are also cases where an event's address places the event in a park and due to the the document becomes vital to the park. This basically being mentioned by address

\paragraph{Entity -related entity} A document about an important figure such as artist, athlete  can be vital to another. This is specially true if the two are contending for the same title, one has snatched a title, and award from the other. 

\paragraph{Organization - main activity} A document that talks about about an area on which the company is active is vital for the organization. For example, Atacocha is a mining company and and an news item on mining waste was annotated vital. 

\paragraph{Artist - work} documents that discuss the work of artists can be relevant to the artists. Such cases include  books or films being vital for the book author or the director (actor) of the film. robocop is film whose screenplay is by Joshua Zetumer. An blog that talks about the film was judged vital for Joshua Zetumer. 

\paragraph{No document content} Some documents were found to have no content

Although they have different document ids, many of the documents have the same content. In the vital annotation, there are only three (88 mainstream, social, 401 weblog). In the 35 document vital document-entity pairs we examined, 22 are social, and 13 are news.