HCDA/cikm-paper Changeset - a07620ff7ab1 · Centrum Wiskunde & Informatica (CWI)

@@ -906,147 +906,145 @@ In vital-relevant category (Table \ref{tab:class-vital-relevant}), the performan

% Bob Bert && talks about a related media and entertainment\\

% FrankandOak&& an article that talks about a the realease of the most innovative companies of which FrankandOak is one. \\

% KentGuinn4Mayor && a theft in a constituency where KentGuinn4Mayor is vying.\\

% Hjemkomst Center && event announcement without mentioning where. it takes a a knowledge of \\

% BlossomCoffee && No content\\

% Scotiabank Per\%25C3\%25BA && no content\\

% Drew Wrigley && politics and talk of oilof his state\\

% Joshua Zetumer && mentioned by his film\\

% Théo Mercier && No content\\

% Fargo Air Museum && No idea why\\

% Stevens Cooperative School && no content\\

% Joshua Boschee && No content\\

% Paul Marquart &&  No idea why\\

% Haven Denney && article on skating competition\\

% Red River Zoo && animal show in the zoo, not indicated by name\\

% RonFunches && talsk about commedy, but not clear whyit is central\\

% DeAnne Smith && No mention, talks related and there are links\\

% Richard Edlund && talks an ward ceemony in his field \\

% Jennifer Baumgardner && no idea why\\

% Jeff Tamarkin && not clear why\\

% Jasper Schneider &&no mention, talks about rural development of which he is a director \\

% urbren00 && No content\\

% \hline

% \end{tabular}

% \end{center}

% \label{tab:miss from both}

% \end{table*}

\section{Analysis and Discussion}\label{sec:analysis}

We conducted experiments to study the impacts on recall of

different components of the filtering stage of entity-based filtering and ranking pipeline. Specifically

we conducted experiments to study the impacts of cleansing,

entity profiles, relevance ratings, categories of documents, entity

profiles. We also measured  impact of the different factors and

choices  on later stages of the pipeline of our own system.

Experimental results show that cleansing can remove entire or parts of

the content of documents making them difficult to retrieve. These

documents can, otherwise, be retrieved from the raw version. The use

of the raw corpus brings in documents that can not be retrieved from

the cleansed corpus. This is true for all entity profiles and for all

entity types. The  recall difference between the cleansed and raw

ranges from  6.8\% t 26.2\%. These increases, in actual

ranges from  6.8\% to 26.2\%. These increases, in actual

document-entity pairs,  is in thousands. We believe this is a

substantial increase. However, the recall increases do not always

translate to improved F-score in overall performance.  In the vital

translate to improved max-F on the overall system performance.  In the vital

relevance ranking for both Wikipedia and aggregate entities, the

cleansed version performs better than the raw version.  In Twitter

entities, the raw corpus achieves better except in the case of all

name-variant, though the difference is negligible.  However, for

name-variant, though the difference is negligible. However, for

vital-relevant, the raw corpus performs  better across all entity

profiles and entity types except in partial canonical names of

profiles and entity types except for the case of partial canonical names of

Wikipedia entities.

The use of different profiles also shows a big difference in

recall. While in Wikipedia the use of canonical

partial achieves better than name-variant, there is a steady increase

in recall from canonical to canonical partial, to name-variant, and

to name-variant partial. This pattern is also observed across the

document categories.  However, here too, the relationship between

the gain in recall as we move from less richer profile to a more

richer profile and overall performance as measured by F-score  is not

linear.

The use of different entity profiles can have a large effect on

recall. While in the case of Wikipedia entities the use of canonical

partial achieves better recall than using name-variants, there seems a

steady increase in recall from canonical to canonical partial, to

name-variant, and to name-variant partial, a pattern that is observed

across the document categories.  However, here too, the relationship between

the gain in recall as we move from less richer profile to a

richer profile and the overall CCR performance as measured by max-F is

not simply positive.

%%%%%%%%%%%%

In vital ranking, across all entity profiles and types of corpus,

Wikipedia's canonical partial  achieves better performance than any

Wikipedia's canonical partial representation achieves better performance than any

other Wikipedia entity profiles. In vital-relevant documents too,

Wikipedia's canonical partial achieves the best result. In the raw

corpus, it achieves a little less than name-variant partial. For

Twitter entities, the name-variant partial profile achieves the

highest F-score across all entity profiles and types of corpus.

There are 3 interesting observations:

1) cleansing impacts Twitter

entities and relevant documents.  This  is validated by the

observation that recall  gains in Twitter entities and the relevant

Cleansing impacts Twitter

entities and relevant documents.  This is validated by the

observation that recall gains in Twitter entities and the relevant

categories in the raw corpus also translate into overall performance

gains. This observation implies that cleansing removes relevant and

social documents than it does vital and news. That it removes relevant

documents more than vital can be explained by the fact that cleansing

gains. This observation implies that cleansing removes more relevant and

social documents than it does vital and news, which may be

explained by the fact that cleansing

removes the related links and adverts which may contain a mention of

the entities. One example we saw was the the cleansing removed an

image with a text of an entity name which was actually relevant. And

that it removes social documents can be explained by the fact that

most of the missing of the missing  docuemnts from cleansed are

social. And all the docuemnts that are missing from raw corpus

social. So in both cases social seem to suffer from text

image with a text of an entity name which was actually relevant. The

removal of predominantly social documents can be explained by the fact that

all of the missing documents from the raw corpus and the majority of

the missing documents from the cleansed corpus belong to

the social category. In both cases, especially the social channel seems to suffer from text

transformation and cleasing processes.

%%%% NEEDS WORK:

Taking both performance (recall at filtering and overall F-score

during evaluation) into account, there is a clear trade-off between

using a richer entity-profile and retrieval of irrelevant

documents. The richer the profile, the more relevant documents it

retrieves, but also the more irrelevant documents. To put it into

perspective, lets compare the number of documents that are retrieved

with  canonical partial and with name-variant partial. Using the raw

corpus, the former retrieves a total of 2547487 documents and achieves

a recall of 72.2\%. By contrast, the later retrieves a total of

4735318 documents and achieves a recall of 90.2\%. The total number of

documents extracted increases by 85.9\% for a recall gain of 18\%. The

rest of the documents, that is 67.9\%, are newly introduced irrelevant

documents.

Perhaps surprising, Wikipedia's canonical partial is the best entity profile for Wikipedia

entities. Here, the retrieval of

thousands vital-relevant document-entity pairs by name-variant partial

does not materialize into an increase in over all performance. Notice

that none of the participants in TREC KBA considered canonical partial

as a viable strategy though. We conclude that, at least for our

system, the remainder of the pipeline needs a different approach to

handle the correct scoring of the additional documents -- that are

necessary if we do not want to accept a low recall of the filtering

step.

%With this understanding, there  is actually no

%need to go and fetch different names variants from DBpedia, a saving

%of time and computational resources.

%%%%%%%%%%%%

The deltas between entity profiles, relevance ratings, and document categories reveal four differences between Wikipedia and Twitter entities. 1) For Wikipedia entities, the difference between canonical partial and canonical is higher(16.1\%) than between name-variant partial and  name-variant(8.3\%).  This can be explained by saturation. This is to mean that documents have already been extracted by  name-variants and thus using their partials does not bring in many new relevant documents.  2) Twitter entities are mentioned by name-variant or name-variant partial and that is seen in the high recall achieved  compared to the low recall achieved by canonical(or their partial). This indicates that documents (specially news and others) almost never use user names to refer to Twitter entities. Name-variant partials are the best entity profiles for Twitter entities. 3) However, comparatively speaking, social documents refer to Twitter entities by their user names than news and others suggesting a difference in

adherence to standard in names and naming. 4) Wikipedia entities achieve higher recall and higher overall performance.

The high recall and subsequent higher overall performance of Wikipedia entities can  be due to two reasons. 1) Wikipedia entities are relatively well described than Twitter entities. The fact that we can retrieve different name variants from DBpedia is a measure of relatively rich description. Rich description plays a role in both filtering and computation of features such as similarity measures in later stages of the pipeline.   By contrast, we have only two names for Twitter entities: their user names and their display names which we collect from their Twitter pages. 2) There is not DBpedia-like resource for Twitter entities from which alternative names cane be collected.

In the experimental results, we also observed that recall scores in the vital category are higher than in the relevant category. This observation  confirms one commonly held assumption:(frequency) mention is related to relevance.  this is the assumption why term frequency is used an indicator of document relevance in many information retrieval systems. The more  a document mentions an entity explicitly by name, the more likely the document is vital to the entity.

Across document categories, we observe a pattern in recall of others, followed by news, and then by social. Social documents are the hardest to retrieve. This can be explained by the fact that social documents (tweets and  blogs) are more likely to point to a resource where the entity is mentioned, mention the entities with some short abbreviation, or talk without mentioning the entities, but with some context in mind. By contrast news documents mention the entities they talk about using the common name variants more than social documents do. However, the greater difference in percentage recall between the different entity profiles in the news category indicates news refer to a given entity with different names, rather than by one standard name. By contrast others show least variation in referring to news. Social documents falls in between the two.  The deltas, for Wikipedia entities, between canonical partials and canonicals,  and name-variants and canonicals are high, an indication that canonical partials

and name-variants bring in new relevant documents that can not be retrieved by canonicals. The rest of the two deltas are very small,  suggesting that partial names of name variants do not bring in new relevant documents.