the methods to create entity profiles, the

types of entities of interest, document type, and the grade of

relevance of the document-entity pair under consideration.

We analyze how these factors (and the design choices made in their

corresponding system components) affect filtering performance.

We identify and characterize the relevant documents that do not pass

estimate a practical upper-bound of recall for entity-centric stream

filtering.

\end{abstract}

% A category with the (minimum) three required fields

\category{H.4}{Information Filtering}{Miscellaneous}

that are not amenable to filtering , and estimating the upper-bound

of recall on entity-based filtering.

The rest of the paper  is organized as follows. Section \ref{sec:desc}

describes the dataset and section \ref{sec:fil} defines the task. In

discussion of our method in \ref{sec:mthd}. Following that,  we

present the experimental results in \ref{sec:expr}, and discuss and

analyze them in \ref{sec:analysis}. Towards the end, we discuss the

impact of filtering choices on classification in section

\ref{sec:impact}, examine and categorize unfilterable documents in

section \ref{sec:unfil}. Finally, we present our conclusions in

\subsection{Breakdown of results by document source category}

%The results of the different entity profiles on the raw corpus are

%broken down by source categories and relevance rank% (vital, or

%relevant).  

In total, the dataset contains 24162 unique entity-document

and the remaining 17424 as relevant.

All documents are categorized into 8 source categories: 0.98\%

arxiv(a), 0.034\% classified(c), 0.34\% forum(f), 5.65\% linking(l),

11.53\% mainstream-news(m-n), 18.40\% news(n), 12.93\% social(s) and

50.2\% weblog(w). We have regrouped these source categories into three

groups ``news'', ``social'', and ``other'', for two reasons: 1) some groups

 with  a persistent user profile and a very small number of positive

 examples. A filtering system can improve its user profiles with a

 feedback obtained from interaction with users, and thereby improve

 its performance. The  filtering stage of entity-based stream

 filtering and ranking can be likened to the adaptive filtering task

 of the filtering track. The persistent information needs are the KB

 examples.

Stream filtering is then the task to, given a stream of documents of news items, blogs

 and social media on one hand and a set of KB entities on the other,

 to filter the stream for  potentially relevant documents  such that

 the relevance classifier(ranker) achieves as maximum performance as

  \item Does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline?

  \item What characterizes the vital (and relevant) documents that are

    missed in the filtering step?

\end{enumerate}

The TREC filtering and the filtering as part of the entity-centric

TREC filtering track's goal is the binary classification of documents:

relevant or not. In our case, the documents have relevance ranking and

the goal of the filtering stage is to filter as many potentially

relevant documents as possible, but less  irrelevant documents as

as part of the pipeline needs that delicate balance between retrieving

filtering in this case can only be studied by binding it to the later

evaluation.

To achieve this, we use recall percentages in the filtering stage for

the different choices of entity profiles. However, we use the overall

performance to select the best entity profiles.To generate the overall

pipeline performance we use the official TREC KBA evaluation metric

and scripts \cite{frank2013stream} to report max-F, the maximum

F-score obtained over all relevance cut-offs.

\section{Literature Review} \label{sec:lit}

 A more recent manifestation of that is the introduction of TREC KBA in 2012.  Following that, there have been a number of research works done on the topic \cite{frank2012building, ceccarelli2013learning, taneva2013gem, wang2013bit, balog2013multi}.  These works are based on KBA 2012 task and dataset  and they address the whole problem of entity filtering and ranking.  TREC KBA continued in 2013, but the task underwent some changes. The main change between  the 2012 and 2013 are in the number of entities, the type of entities, the corpus and the relevance rankings.

The number of entities increased from 29 to 141, and it included 20 Twitter entities. The TREC KBA 2012 corpus is 1.9TB after xz-compression and has  400M documents. By contrast, the KBA 2013 corpus is 6.45 after XZ-compression and GPG encryption. A version with all-non English documented removed  is 4.5 TB and consists of 1 Billion documents. The 2013 corpus subsumed the 2012 corpus and added others from spinn3r, namely main-stream news, forum, arxiv, classified, reviews and meme-tracker.  A more important difference is, however, a change in the definitions of relevance ratings vital and relevant. While in KBA 2012, a document was judged vital if it has citation-worthy content for a given entity, in 2013 it must have the freshliness, that is the content must trigger an editing of the given entity's KB entry. 

While the tasks of 2012 and 2013 are fundamentally the same, the approaches  varied due  to the size of the corpus. In 2013, all participants used filtering to reduce the size of the big corpus.   They used different ways of filtering: many of them used two or more of different name variants from DBpedia such as labels, names, redirects, birth names, alias, nicknames, same-as and alternative names \cite{wang2013bit,dietzumass,liu2013related, zhangpris}.  Although most of the participants used DBpedia name variants none of them used all the name variants.  A few other participants used bold words in the first paragraph of the Wikipedia entity's profiles and anchor texts from other Wikipedia pages  \cite{bouvierfiltering, niauniversity}. One participant used Boolean \emph{and} built from the tokens of the canonical names \cite{illiotrec2013}.  

\end{table}

The upper part of Table \ref{tab:name} shows the recall performances on the cleansed version and the lower part on the raw version. The recall performances for all entity types  are increased substantially in the raw version. Recall increases on Wikipedia entities  vary from 8.2 to 12.8, and in Twitter entities from 6.8 to 26.2. In all entities, it ranges from 8.0 to 13.6.  The recall increases are substantial. To put it into perspective, an 11.8 increase in recall on all entities is a retrieval of 2864 more unique document-entity pairs. %This suggests that cleansing has removed some documents that we could otherwise retrieve. 

\subsection{Entity Profiles}

%\begin{table*}

%\caption{Breakdown of recall percentage increases by document categories }

%\begin{center}\begin{tabular}{l*{9}{c}r}

% && \multicolumn{3}{ c| }{All entities}  & \multicolumn{3}{ c| }{Wikipedia} &\multicolumn{3}{ c| }{Twitter} \\ 

\paragraph*{MentionsBody }

  No. of occurrences of the target entity in the  document body

  In here, we present results showing how  the choices in corpus, entity types, and entity profiles impact these latest stages of the pipeline.  In tables \ref{tab:class-vital} and \ref{tab:class-vital-relevant}, we show the performances in max-F. 

\begin{table*}

\caption{vital performance under different name variants(upper part from cleansed, lower part from raw)}

\begin{center}

\begin{tabular}{ll@{\quad}lllllll}

\hline

social documents than it does vital and news. That it removes relevant

documents more than vital can be explained by the fact that cleansing

removes the related links and adverts which may contain a mention of

the entities. One example we saw was the the cleansing removed an

image with a text of an entity name which was actually relevant. And

that it removes social documents can be explained by the fact that

social. So in both cases social seem to suffer from text

%%%% NEEDS WORK:

Taking both performance (recall at filtering and overall F-score

during evaluation) into account, there is a clear trade-off between

using a richer entity-profile and retrieval of irrelevant

 We  observed  that among the missing documents, different document ids can have the same content, and be judged multiple times for a given entity.  %In the vital annotation, there are 88 news, and 409 weblog. 

 Avoiding duplicates, we randomly selected 35 documents, one for each entity.   The documents are 13 news and  22  social. Here below we have classified the situation under which a document can be vital for an entity without mentioning the entities with the different entity  profiles we used for filtering. 

\paragraph*{Outgoing link mentions} A post (tweet) with an outgoing link which mentions the entity.

\paragraph*{Entity -related entity} A document about an important figure such as artist, athlete  can be vital to another. This is specially true if the two are contending for the same title, one has snatched a title, or award from the other. 

\paragraph*{Organization - main activity} A document that talks about about an area on which the company is active is vital for the organization. For example, Atacocha is a mining company  and a news item on mining waste was annotated vital. 

\paragraph*{Entity - group} If an entity belongs to a certain group (class),  a news item about the group can be vital for the individual members. FrankandOak is  named innovative company and a news item that talks about the group  of innovative companies is relevant for a  it. Other examples are: a  big event  of which an entity is related such an Film awards for actors. 

\paragraph*{Artist - work} Documents that discuss the work of artists can be relevant to the artists. Such cases include  books or films being vital for the book author or the director (actor) of the film. Robocop is film whose screenplay is by Joshua Zetumer. A blog that talks about the film was judged vital for Joshua Zetumer. 

\paragraph*{Politician - constituency} A major political event in a certain constituency is vital for the politician from that constituency. 

 A good example is a weblog that talks about two north Dakota counties being drought disasters. The news is vital for Joshua Boschee, a politician, a member of North Dakota democratic party.