HCDA/cikm-paper Changeset - 2049a41bcedf · Centrum Wiskunde & Informatica (CWI)

Changeset - 2049a41bcedf

Parent rev.

Child rev.

[Not reviewed]

0 1 0

Arjen de Vries (arjen) - 11 years ago 2014-06-12 05:43:29
arjen.de.vries@cwi.nl

working on conclusions

1 file changed with 40 insertions and 8 deletions:

mypaper-final.tex

0 comments (0 inline, 0 general)

mypaper-final.tex

➞

Show inline comments

 In the experimental results, we also observed that recall scores in the vital category are higher than in the relevant category. This observation  confirms one commonly held assumption:(frequency) mention is related to relevance.  this is the assumption why term frequency is used an indicator of document relevance in many information retrieval systems. The more  a document mentions an entity explicitly by name, the more likely the document is vital to the entity.
 Across document categories, we observe a pattern in recall of others, followed by news, and then by social. Social documents are the hardest to retrieve. This can be explained by the fact that social documents (tweets and  blogs) are more likely to point to a resource where the entity is mentioned, mention the entities with some short abbreviation, or talk without mentioning the entities, but with some context in mind. By contrast news documents mention the entities they talk about using the common name variants more than social documents do. However, the greater difference in percentage recall between the different entity profiles in the news category indicates news refer to a given entity with different names, rather than by one standard name. By contrast others show least variation in referring to news. Social documents falls in between the two.  The deltas, for Wikipedia entities, between canonical partials and canonicals,  and name-variants and canonicals are high, an indication that canonical partials
 and name-variants bring in new relevant documents that can not be retrieved by canonicals. The rest of the two deltas are very small,  suggesting that partial names of name variants do not bring in new relevant documents.
 %\section{Unfilterable documents}\label{sec:unfil}
 \section{Missing vital-relevant documents}\label{sec:unfil}
+%
- The use of name-variant partial for filtering is an aggressive attempt to retrieve as many relevant documents as possible at the cost of retrieving irrelevant documents. However, we still miss about  2363(10\%) of the vital-relevant documents.  Why are these documents missed? If they are not mentioned by partial names of name variants, what are they mentioned by? Table \ref{tab:miss} shows the documents that we miss with respect to cleansed and raw corpus.  The upper part shows the number of documents missing from cleansed and raw versions of the corpus. The lower part of the table shows the intersections and exclusions in each corpus.
+ The use of name-variant partial for filtering is an exhaustive attempt to retrieve as many relevant documents as possible at the cost of retrieving irrelevant documents. However, we still miss about  2363(10\%) of the vital-relevant documents.  Why are these documents missed? If they are not mentioned by partial names of name variants, what are they mentioned by? Table \ref{tab:miss} shows the documents that we miss with respect to cleansed and raw corpus.  The upper part shows the number of documents missing from cleansed and raw versions of the corpus. The lower part of the table shows the intersections and exclusions in each corpus.
 \begin{table}
 \caption{The number of documents missing  from raw and cleansed extractions. }
 \begin{center}
 \begin{tabular}{l@{\quad}llllll}
 \hline
 \multicolumn{1}{l}{\rule{0pt}{12pt}category}&\multicolumn{1}{l}{\rule{0pt}{12pt}Vital }&\multicolumn{1}{l}{\rule{0pt}{12pt}Relevant }&\multicolumn{1}{l}{\rule{0pt}{12pt}Total }\\[5pt]
 \hline
 Cleansed &1284 & 1079 & 2363 \\
 Raw & 276 & 4951 & 5227 \\
 \hline
 \paragraph*{Entity - group} If an entity belongs to a certain group (class),  a news item about the group can be vital for the individual members. FrankandOak is  named innovative company and a news item that talks about the group  of innovative companies is relevant for a  it. Other examples are: a  big event  of which an entity is related such an Film awards for actors.
 \paragraph*{Artist - work} Documents that discuss the work of artists can be relevant to the artists. Such cases include  books or films being vital for the book author or the director (actor) of the film. Robocop is film whose screenplay is by Joshua Zetumer. A blog that talks about the film was judged vital for Joshua Zetumer.
 \paragraph*{Politician - constituency} A major political event in a certain constituency is vital for the politician from that constituency.
  A good example is a weblog that talks about two north Dakota counties being drought disasters. The news is vital for Joshua Boschee, a politician, a member of North Dakota democratic party.
 \paragraph*{head - organization} A document that talks about an organization of which the entity is the head can be vital for the entity.  Jasper\_Schneider is USDA Rural Development state director for North Dakota and an article about problems of primary health centers in North Dakota is judged vital for him.
 \paragraph*{World Knowledge} Some things are impossible to know without your world knowledge. For example ''refreshments, treats, gift shop specials, "bountiful, fresh and fabulous holiday decor," a demonstration of simple ways to create unique holiday arrangements for any home; free and open to the public`` is judged relevant to Hjemkomst\_Center. This is a social media post, and unless one knows the person posting it, there is no way that this text shows that. Similarly ''learn about the gray wolf's hunting and feeding behaviors and watch the wolves have their evening meal of a full deer carcass; $15 for members, $20 for nonmembers`` is judged vital to Red\_River\_Zoo.
 \paragraph*{No document content} A small number of documents were found to have no content.
 \paragraph*{Disagreement} For a few remaining documents, the authors disagree with the assessors as to why these are vital to the entity.
 \section{Conclusions} \label{sec:conc}
 In this paper, we examined the filtering stage of the entity-centric stream filtering and ranking  by holding the later stages of fixed. In particular, we studied the cleansing step, different entity profiles, type of entities(Wikipedia or Twitter), categories of documents(news, social, or others) and the relevance ratings. We attempted to address the following research questions: 1) does cleansing affect filtering and subsequent performance? 2) what is the most effective way of entity profiling? 3) is filtering different for Wikipedia and Twitter entities? 4) are some type of documents easily filterable and others not? 5) does a gain in recall at filtering step translate to a gain in F-measure at the end of the pipeline? and 6) what are the circumstances under which vital documents can not be retrieved?
 Cleansing does remove parts or entire contents of documents making them irretrievable. However, because of the introduction of false positives, recall gains by  raw corpus and some  richer entity profiles do not necessarily translate to overall performance gain. The results conclusion on this is mixed in the sense that cleansing helps improve the recall on vital documents and Wikipedia entities, but reduces the recall on Twitter entities and the relative category of relevance ranking. Vital and relevant documents show a difference in retrieval nonperformance documents are easier to filter than relevant.
 Despite an aggressive attempt to filter as many vital-relevant documents as possible,  we observe that there are still documents that we miss. While some are possible to retrieve with some modifications, some others are not. There are some document that indicate that an information filtering system does not seem to get them no matter how rich representation of entities they use. These circumstances under which this happens are many. We found that some documents have no content at all, subjectivity(it is not clear why some are judged vital). However, the main circumstances under which vital  documents can defy filtering is: outgoing link mentions,
 venue-event, entity - related entity, organization - main area of operation, entity - group, artist - artist's work,  party-politician, and world knowledge.
 In this paper, we examined the filtering stage of the entity-centric
 stream filtering and ranking  by holding the later stages of fixed. In
 particular, we studied the cleansing step, different techniques to
 construct entity profiles, and the effects of entity type (Wikipedia
 or Twitter) and document category (news, social, or other). We attempted to address
 the following research questions: 1) does cleansing affect filtering
 and subsequent performance? 2) what is the most effective way of
 entity profiling? 3) is filtering different for Wikipedia and Twitter
 entities? 4) are some type of documents easily filterable and others
 not? 5) does a gain in recall at filtering step translate to a gain in
 max-F at the end of the pipeline? and 6) what are the
 circumstances under which vital documents can not be retrieved?
 Cleansing may remove (parts of) the contents of documents, making
 them irretrievable. However, because of the introduction of false
 positives, gaining recall by filtering the raw corpus instead of the
 cleansed one and developing richer entity profiles, does not necessarily translate to overall
 performance gains. The overall conclusion on this is mixed in the
 sense that cleansing has helped to improve the recall on vital
 documents and Wikipedia entities, but at the same time reduces the
 recall on Twitter entities and the relative category of
 relevance ranking. Vital and relevant documents show a difference in
 retrieval performance, where vital documents appear to be easier to filter than
 relevant ones. Notice that in the context of the CCR task, the vital documents are
 most important.
 Despite an exhaustive attempt to identify as many vital-relevant
 documents as possible,  we observe that there are still documents that
 we miss. While some can clearly be retrieved by modifying the
 filtering procedure, some relevant and even vital documents can be
 considered irretrievable. The circumstances under
 which this happens are many. A few documents have no content, or it is
 unclear why they have been judged vital. However, the main
 circumstances under which vital documents
 can defy filtering include: outgoing link mentions,
 venue-event, entity - related entity, organization - main area of
 operation, entity - group, artist - artist's work,  party-politician,
 and world knowledge.
 %ACKNOWLEDGMENTS are optional
 %\section{Acknowledgments}
+%
 % The following two commands are all you need in the
 % initial runs of your .tex file to
 % produce the bibliography for the citations in your paper.
 \bibliographystyle{abbrv}
 \bibliography{sigproc}  % sigproc.bib is the name of the Bibliography in this case
 % You must have a proper ".bib" file

0 comments (0 inline, 0 general)