% THIS IS SIGPROC-SP.TEX - VERSION 3.1

% WORKS WITH V3.2SP OF ACM_PROC_ARTICLE-SP.CLS

% APRIL 2009

%

% It is an example file showing how to use the 'acm_proc_article-sp.cls' V3.2SP

% LaTeX2e document class file for Conference Proceedings submissions.

% ----------------------------------------------------------------------------------------------------------------

% This .tex file (and associated .cls V3.2SP) *DOES NOT* produce:

%       1) The Permission Statement

%       2) The Conference (location) Info information

%       3) The Copyright Line with ACM data

%       4) Page numbering

% ---------------------------------------------------------------------------------------------------------------

% It is an example which *does* use the .bib file (from which the .bbl file

% is produced).

% REMEMBER HOWEVER: After having produced the .bbl file,

% and prior to final submission,

% you need to 'insert'  your .bbl file into your source .tex file so as to provide

% ONE 'self-contained' source file.

%

% Questions regarding SIGS should be sent to

% Adrienne Griscti ---> griscti@acm.org

%

% Questions/suggestions regarding the guidelines, .tex and .cls files, etc. to

% Gerald Murray ---> murray@hq.acm.org

%

% For tracking purposes - this is V3.1SP - APRIL 2009

\documentclass{acm_proc_article-sp}

\usepackage{booktabs}

\usepackage{multirow}

\usepackage{todonotes}

\begin{document}

\title{Entity-Centric Stream Filtering and ranking: Filtering and Unfilterable Documents 

}

%

% You need the command \numberofauthors to handle the 'placement

% and alignment' of the authors beneath the title.

%

% For aesthetic reasons, we recommend 'three authors at a time'

% i.e. three 'name/affiliation blocks' be placed beneath the title.

%

% NOTE: You are NOT restricted in how many 'rows' of

% "name/affiliations" may appear. We just ask that you restrict

% the number of 'columns' to three.

%

% Because of the available 'opening page real-estate'

% we ask you to refrain from putting more than six authors

% (two rows with three columns) beneath the article title.

% More than six makes the first-page appear very cluttered indeed.

%

% Use the \alignauthor commands to handle the names

% and affiliations for an 'aesthetic maximum' of six authors.

% Add names, affiliations, addresses for

% the seventh etc. author(s) as the argument for the

% \additionalauthors command.

% These 'additional authors' will be output/set for you

% without further effort on your part as the last section in

% the body of your article BEFORE References or any Appendices.

% of EIGHT authors. SIX appear on the 'first-page' (for formatting

% reasons) and the remaining two appear in the \additionalauthors section.

%

% \author{

% % You can go ahead and credit any number of authors here,

% % e.g. one 'row of three' or two rows (consisting of one row of three

% % and a second row of one, two or three).

% %

% % The command \alignauthor (no curly braces needed) should

% % precede each author name, affiliation/snail-mail address and

% % e-mail address. Additionally, tag each line of

% % affiliation/address with \affaddr, and tag the

% % e-mail address with \email.

% %

% % 1st. author

% \alignauthor

% Ben Trovato\titlenote{Dr.~Trovato insisted his name be first.}\\

%        \affaddr{Institute for Clarity in Documentation}\\

%        \affaddr{1932 Wallamaloo Lane}\\

%        \affaddr{Wallamaloo, New Zealand}\\

%        \email{trovato@corporation.com}

% % 2nd. author

% \alignauthor

% G.K.M. Tobin\titlenote{The secretary disavows

% any knowledge of this author's actions.}\\

%        \affaddr{Institute for Clarity in Documentation}\\

%        \affaddr{P.O. Box 1212}\\

%        \affaddr{Dublin, Ohio 43017-6221}\\

%        \email{webmaster@marysville-ohio.com}

% }

% There's nothing stopping you putting the seventh, eighth, etc.

% author on the opening page (as the 'third row') but we ask,

% for aesthetic reasons that you place these 'additional authors'

% in the \additional authors block, viz.

% Just remember to make sure that the TOTAL number of authors

% is the number that will appear on the first page PLUS the

% number that will appear in the \additionalauthors section.

\maketitle

\begin{abstract}

Entity-centric information processing requires complex pipelines

involving both natural language processing and information retrieval

components. In entity-centric stream filtering and ranking, the

pipeline involves four stages: filtering, classification,

ranking (scoring) and evaluation. Filtering is an initial step, that

extracts a working-set of documents from the web-scale corpus, aiming

for a smaller size collection that would be more manageable in the

subsequent stages of the pipeline. This filtering step therefore

determines the maximally attainable performance of the overall system.

This paper investigates the filtering stage in isoltation, in context

of a cumulative citation recommendation problem. We conduct an

in-depth analysis of the main factors that determine filtering

effectiveness: cleansing noisy web data, methods to create entity

profiles, the types of entity of interest, document category, and the

relevance level of the entity-document pair under consideration.

We analyze how these factors (and the design choices made in their

corresponding system components) affect filtering performance.

We identify and characterize the relevant documents that do not pass the

filtering stage, and conduct a manual examination into their

contents. The paper classifies the ways unfilterable documents  

are mentioned in text and estimates the practical upper-bound of

recall in entity-based filtering.  

\end{abstract}

% A category with the (minimum) three required fields

\category{H.4}{Information Filtering}{Miscellaneous}

%A category including the fourth, optional field follows...

%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\terms{Theory}

\keywords{Information Filtering; Cumulative Citation Recommendation; knowledge maintenance; Stream Filtering;  emerging entities} % NOT required for Proceedings

\section{Introduction}

  In 2012, the Text REtrieval Conferences (TREC) introduced the Knowledge Base Acceleration (KBA) track  to help Knowledge Bases(KBs) curators. The track is crucial to address a critical need of KB curators: given KB (Wikipedia or Twitter) entities, filter  a stream  for relevant documents, rank the retrieved documents and recommend them to the KB curators. The track is crucial and timely because  the number of entities in a KB on one hand, and the huge amount of new information content on the Web on the other hand make the task of manual KB maintenance challenging.   TREC KBA's main task, Cumulative Citation Recommendation (CCR), aims at filtering a stream to identify   citation-worthy  documents, rank them,  and recommend them to KB curators.

 Filtering is a crucial step in CCR for selecting a potentially relevant set of working documents for subsequent steps of the pipeline out of a big collection of stream documents. The TREC Filtering track defines filtering as a ``system that sifts through stream of incoming information to find documents that are relevant to a set of user needs represented by profiles'' \cite{robertson2002trec}. Adaptive Filtering, one task of the filtering track,  starts with   a persistent user profile and a very small number of positive examples. The  filtering step used in CCR systems fits under adaptive filtering: the profiles are represented by persistent KB (Wikipedia or Twitter) entities and there is a small set of relevance judgments representing positive examples. 

 TREC-KBA 2013's participants applied Filtering as a first step  to produce a smaller working set for subsequent experiments. As the subsequent steps of the pipeline use the output of the filter, the final performance of the system is dependent on this important step.  The filtering step particularly determines the recall of the overall system. However, all submitted systems suffered from poor recall \cite{frank2013stream}.  The most important components of the filtering step are cleansing, and entity profiling. Each component has choices to make. For example, there are two versions of corpus: cleansed and raw. Different approaches used different entity profiles for filtering. These entity profiles varied from  KB entities' canonical names, to  DBpedia name variants, to using bold words in the first paragraph of the Wikipedia entities’ profiles and anchor texts from other Wikipedia pages, to using exact name and wordNet synonyms. Moreover, the Type of entities (Wikipedia or Twitter), the category of 

documents (news, blog, tweets) can influence filtering.

 A variety of approaches are employed  to solve the CCR challenge. Each participant reports the steps of the pipeline and the final results in comparison to other systems.  A typical TREC KBA poster presentation or talk explains the system pipeline and reports the final results. The systems may employ similar (even the same) steps  but the choices they make at every step are usually different. In such a situation, it becomes hard to identify the factors that result in improved performance. There is  a lack of insight across different approaches. This makes  it hard to know  whether the improvement in performance of a particular approach is due to preprocessing, filtering, classification, scoring  or any of the sub-components of the pipeline. 

 In this paper,  we hold the subsequent steps of the pipeline fixed, zoom in on the filtering step and  conduct an in-depth analysis of the main components in it.  In particular, we study  cleansing, different entity profiling,  type of entities (Wikipedia or Twitter), and document categories (social, news, etc).  The main contribution of the paper are an in-depth analysis of the factors that affect entity-based stream filtering, identifying optimal entity profiles vis-avis not compromising precision, describing and classifying relevant documents that are not amenable to filtering , and  estimating the upper-bound of recall on entity-based filtering.

 The rest of the paper is is organized as follows: 

% \label{tab:miss-category}

% \end{table*}

%    To gain more insight, I sampled for each 35 entities, one document-entity pair and looked into the contents. The results are in \ref{tab:miss from both}

%    

%    \begin{table*}

% \caption{Missing documents and their mentions }

% \begin{center}

% 

%  \begin{tabular}{l*{4}{l}l}

%  &entity&mentioned by &remark \\

% \hline

%  Jeremy McKinnon  & Jeremy McKinnon& social, mentioned in read more link\\

% Blair Thoreson   & & social, There is no mention by name, the article talks about a subject that is political (credit rating), not apparent to me\\

%   Lewis and Clark Landing&&Normally, maha music festival does not mention ,but it was held there \\

% Cementos Lima &&It appears a mistake to label it vital. the article talks about insurance and centos lima is a cement company.entity-deleted from wiki\\

% Corn Belt Power Cooperative & &No content at all\\

% Marion Technical Institute&&the text could be of any place. talks about a place whose name is not mentioned. 

%  roryscovel & &Talks about a video hinting that he might have seen in the venue\\

% Jim Poolman && talks of party convention, of which he is member  politician\\

% Atacocha && No mention by name The article talks about waste from mining and Anacocha is a mining company.\\

% Joey Mantia & & a mention of a another speeedskater\\

% Derrick Alston&&Text swedish, no mention.\\

% Paul Johnsgard&& not immediately clear why \\

% GandBcoffee&& not immediately visible why\\

% Bob Bert && talks about a related media and entertainment\\

% FrankandOak&& an article that talks about a the realease of the most innovative companies of which FrankandOak is one. \\

% KentGuinn4Mayor && a theft in a constituency where KentGuinn4Mayor is vying.\\

% Hjemkomst Center && event announcement without mentioning where. it takes a a knowledge of \\

% BlossomCoffee && No content\\

% Scotiabank Per\%25C3\%25BA && no content\\

% Drew Wrigley && politics and talk of oilof his state\\

% Joshua Zetumer && mentioned by his film\\

% Théo Mercier && No content\\

% Fargo Air Museum && No idea why\\

% Stevens Cooperative School && no content\\

% Joshua Boschee && No content\\

% Paul Marquart &&  No idea why\\

% Haven Denney && article on skating competition\\

% Red River Zoo && animal show in the zoo, not indicated by name\\

% RonFunches && talsk about commedy, but not clear whyit is central\\

% DeAnne Smith && No mention, talks related and there are links\\

% Richard Edlund && talks an ward ceemony in his field \\

% Jennifer Baumgardner && no idea why\\

% Jeff Tamarkin && not clear why\\

% Jasper Schneider &&no mention, talks about rural development of which he is a director \\

% urbren00 && No content\\

% \hline

% \end{tabular}

% \end{center}

% \label{tab:miss from both}

% \end{table*}

\section{Analysis and Discussion}

We conducted experiments to study  the impacts on recall of 

different components of the filtering stage of entity-based filtering and ranking pipeline. Specifically 

we conducted experiments to study the impacts of cleansing, 

entity profiles, relevance ratings, categories of documents, entity profiles. We also measured  impact of the different factors and choices  on later stages of the pipeline. 

Experimental results show that cleansing can remove entire or parts of the content of documents making them difficult to retrieve. These documents can, otherwise, be retrieved from the raw version. The use of the raw corpus brings in documents that can not be retrieved from the cleansed corpus. This is true for all entity profiles and for all entity types. The  recall difference between the cleansed and raw ranges from  6.8\% t 26.2\%. These increases, in actual document-entity pairs,  is in thousands. We believe this is a substantial increase. However, the recall increases do not always translate to improved F-score in overall performance.  In the vital relevance ranking for both Wikipedia and aggregate entities, the cleansed version performs better than the raw version.  In Twitter entities, the raw corpus achieves better except in the case of all name-variant, though the difference is negligible.  However, for vital-relevant, the raw corpus performs  better across all entity profiles and entity types except in partial canonical names of Wikipedia entities. 

The use of different profiles also shows a big difference in recall. Except in the case of Wikipedia where the use of canonical partial achieves better than name-variant, there is a steady increase in recall from canonical to  canonical partial, to name-variant, and to name-variant partial. This pattern is also observed across the document categories.  However, here too, the relationship between   the gain in recall as we move from less richer profile to a more richer profile and overall performance as measured by F-score  is not linear. 

In vital ranking, across all entity profiles and types of corpus, Wikipedia's canonical partial  achieves better performance than any other Wikipedia entity profiles. In vital-relevant documents too, Wikipedia's canonical partial achieves the best result. In the raw corpus, it achieves a little less than name-variant partial. For Twitter entities, the name-variant partial profile achieves the highest F-score across all entity profiles and types of corpus.  

There are 3 interesting observations: 1) cleansing impacts Twitter entities and relevant documents.  This  is validated by the observation that recall  gains in Twitter entities and the relevant categories in the raw corpus also translate into overall performance gains. This observation implies that cleansing removes relevant and social documents than it does vital and news. That it removes relevant documents more than vital can be explained by the fact that cleansing removes the related links and adverts which may contain a mention of the entities. One example we saw was the the cleansing removed an image with a text of an entity name which was actually relevant. And that it removes social documents can be explained by the fact that most of the missing of the missing  docuemnts from cleansed are social. And all the docuemnts that are missing from raw corpus social. So in both cases socuial seem to suffer from text transformation and cleasing processes. 2) Taking both performance (recall at filtering and overall F-score during evaluation) into account, Wikipedia's canonical partial is the best entity profile for Wikipedia entities. This is interesting  to see that the retrieval of of  thousands vital-relevant document-entity pairs by name-variant partial does not translate to an increase in over all performance. It is even more interesting since canonical partial was not considered as contending profile for stream filtering by any of participant to the best of our knowledge. With this understanding, there  is actually no need to go and fetch different names variants from DBpedia, a saving of time and computational resources.

The deltas between entity profiles, relevance ratings, and document categories reveal four differences between Wikipedia and Twitter entities. 1) For Wikipedia entities, the difference between canonical partial and canonical is higher(16.1\%) than between name-variant partial and  name-variant(18.3\%).  This can be explained by saturation. This is to mean that documents have already been extracted by  name-variants and thus using their partials does not bring in many new relevant documents.  2) Twitter entities are mentioned by name-variant or name-variant partial and that is seen in the high recall achieved  compared to the low recall achieved by canonical(or their partial). This indicates that documents (specially news and others) almost never use user names to refer to Twitter entities. Name-variant partials are the best entity profiles for Twitter entities. 3) However, comparatively speaking, social documents refer to Twitter entities by their user names than news and others suggesting a difference in adherence to standard in names and naming. 4) Wikipedia entities achieve higher recall and higher overall performance. 

The high recall and subsequent higher overall performance of Wikipedia entities can  be due to two reasons. 1) Wikipedia entities are relatively well described than Twitter entities. The fact that we can retrieve different name variants from DBpedia is a measure of relatively rich description. Rich description plays a role in both filtering and computation of features such as similarity measures in later stages of the pipeline.   By contrast, we have only two names for Twitter entities: their user names and their display names which we collect from their Twitter pages. 2) There is not DBpedia-like resource for Twitter entities from which alternative names cane be collected.   

In the experimental results, we also observed that recall scores in the vital category are higher than in the relevant category. This observation  confirms one commonly held assumption:(frequency) mention is related to relevance.  this is the assumption why term frequency is used an indicator of document relevance in many information retrieval systems. The more  a document mentions an entity explicitly by name, the more likely the document is vital to the entity.

Across document categories, we observe a pattern in recall of others, followed by news, and then by social. Social documents are the hardest to retrieve. This can be explained by the fact that social documents (tweets and  blogs) are more likely to point to a resource where the entity is mentioned, mention the entities with some short abbreviation, or talk without mentioning the entities, but with some context in mind. By contrast news documents mention the entities they talk about using the common name variants more than social documents do. However, the greater difference in percentage recall between the different entity profiles in the news category indicates news refer to a given entity with different names, rather than by one standard name. By contrast others show least variation in referring to news. Social documents falls in between the two.  The deltas, for Wikipedia entities, between canonical partials and canonicals,  and name-variants and canonicals are high, an indication that canonical partials and name-variants bring in new relevant documents that can not be retrieved by canonicals. The rest of the two deltas are very small,  suggesting that partial names of name variants do not bring in new relevant documents. 

\section{Unfilterable documents}

There is a trade-off between using a richer entity-profile and retrieval of irrelevant documents. The richer the profile, the more relevant documents it retrieves, but also the more irrelevant documents. To put it into perspective, lets compare the number of documents that are retrieved with  canonical partial and with name-variant partial. Using the raw corpus, the former retrieves a total of 2547487 documents and achieves a recall of 72.2\%. By contrast, the later retrieves a total of 4735318 documents and achieves a recall of 90.2\%. The total number of documents extracted increases by 85.9\% for a recall gain of 18\%. The rest of the documents, that is 67.9\%, are newly introduced irrelevant documents. 

% 

 We observed that there are vital-relevant documents that we miss from raw only, and similarly from cleansed only. The reason for this is transformation from one format to another. The most interesting documents are those that we miss from both raw and cleansed corpus. We first identified the number of KB entities who have a vital relevance judgment and  whose documents can not be retrieved (they were 35 in total) and conducted a manual examination into their content to find out why they are missing. 

 We  observed  that among the missing documents, different document ids can have the same content, and be judged multiple times for a given entity.  %In the vital annotation, there are 88 news, and 409 weblog. 

 Avoiding duplicates, we randomly selected 35 documents, one for each entity.   The documents are 13 news and  22  social. Here below we have classified the situation under which a document can be vital for an entity without mentioning the entities with the different entity  profiles we used for filtering. 

 A good example is a weblog that talks about two north Dakota counties being drought disasters. The news is vital for Joshua Boschee, a politician, a member of North Dakota democratic party.  

\section{Conclusions}

In this paper, we examined the filtering stage of the entity-centric stream filtering and ranking  by holding the later stages of fixed. In particular, we studied the cleansing step, different entity profiles, type of entities(Wikipedia or Twitter), categories of documents(news, social, or others) and the relevance ratings. We attempted to address the following research questions: 1) does cleansing affect filtering and subsequent performance? 2) what is the most effective way of entity profiling? 3) is filtering different for Wikipedia and Twitter entities? 4) are some type of documents easily filterable and others not? 5) does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline? and 6) what are the circumstances under which vital documents can not be retrieved? 

Cleansing does remove parts or entire contents of documents making them irretrievable. However, because of the introduction of false positives, recall gains by  raw corpus and some  richer entity profiles do not necessarily translate to overall performance gain. The results conclusion on this is mixed in the sense that cleansing helps improve the recall on vital documents and Wikipedia entities, but reduces the recall on Twitter entities and the relative category of relevance ranking. Vital and relevant documents show a difference in retrieval nonperformance documents are easier to filter than relevant.  

Despite an aggressive attempt to filter as many vital-relevant documents as possible,  we observe that there are still documents that we miss. While some are possible to retrieve with some modifications, some others are not. There are some document that indicate that an information filtering system does not seem to get them no matter how rich representation of entities they use. These circumstances under which this happens are many. We found that some documents have no content at all, subjectivity(it is not clear why some are judged vital). However, the main circumstances under which vital  documents can defy filtering is: outgoing link mentions, 

venue-event, entity - related entity, organization - main area of operation, entity - group, artist - artist's work,  party-politician, and world knowledge.  

%ACKNOWLEDGMENTS are optional

%\section{Acknowledgments}

%

% The following two commands are all you need in the

% initial runs of your .tex file to

% produce the bibliography for the citations in your paper.

\bibliographystyle{abbrv}

\bibliography{sigproc}  % sigproc.bib is the name of the Bibliography in this case

% You must have a proper ".bib" file

%  and remember to run:

% latex bibtex latex latex

% to resolve all references

%

% ACM needs 'a single self-contained file'!

%

%APPENDICES are optional

%\balancecolumns

\end{document}