HCDA/sigir2016repo Changeset - dec17e349fb3 · Centrum Wiskunde & Informatica (CWI)

@@ -135,224 +135,224 @@

% (The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).}

% \date{30 July 1999}

% Just remember to make sure that the TOTAL number of authors

% is the number that will appear on the first page PLUS the

% number that will appear in the \additionalauthors section.

\maketitle

%opening

\title{Items that trigger clicks on recommendation}

\author{}

\maketitle

\begin{abstract}

In a setting where  recommendations are provided to users when they are viewing particular items, what are the  factors that contribute to clicks on recommendations?  We examine factors that  trigger clicks on  recommended items in relation to the items the user  viewing and to which the recommendations are provided. More specifically, we examine the items from which clicks happen and what type of items get clicked. Are some items more likely to cause the user to click  on recommendations, and are some other recommendations more likely to be clicked? In short, are clicks on recommendations a function of the base item, or are they a function of the recommended items?  We attempt to explain the factors that trigger clicks on recommendations from different angles.

\end{abstract}

\section{Introduction}

 In a  study that investigated the relationship between the number of times  items are viewed and the the number of times clicks happened from those items  in several online publishers  \cite{said2013month}, it was reported that traditional news portals providing news and opinions on politics and current events are more likely to generate clicks on recommendation than special interest portals such as sports,  gardening, and auto mechanic forums.  Another study \cite{esiyok2014users}, using a similar dataset, investigated the impressions and clicks at level of the category  of items of one of the traditional news portals - Tagesspiegel (a popular German national news portal).  The finding was that there is a relationship between what the user is currently reading and what they  read next. They  reported that the category local and sports enjoyed the most loyal readers, that is that a user reading on local items will more likely keep reading items of the same category.  % recommendations that were made to the different websites raising a queation as to whether the clicks on recommendations were because of nature of the online publishers or the recommendation items.this study, we focus on one traditional news portal, tagespiegel and examine it to find out factors that trigger recommendations on clicks or lack thereof. %wether some categories are more likely to recieve clicks on recommendations. We also even go further and look at what type of items are more likely to trigger more clicks than others.

 While both these studies are very related and relevant, they  did not investigate the relationship between the base items, the recommended items and the resulting clicks or lack thereof. In a recommendation setting where recommendation items are provided to  users on the items that the user is currently  viewing (henceforth referred to as base items),   what are the factors that trigger user to click on recommendations?  Are the clicks a function of the base items or of the recommended items? Do some base items and some recommended items cause users to click on recommendations more than others, and if they do what explains this difference?

In this study we examine the factors that might trigger clicks on recommendations  from several angles. One angle  is from  the categories of the base items the user is currently reading. More specifically, are some categories of items the user is currently on  more likely to  cause the user to click on recommendations? Similarly, we examine the categories of the recommended items and investigate whether some are more likely to trigger clicks on themselves upon recommendation. We also investigate how  the categories of the base items and the categories of the recommendation items are related in the way they trigger clicks. %Are some categories  more likely to trigger clicks on some categories? For example, is political category more likely to trigger clicks on political categories, or another category such as local category?

We also go down to the item level and look at the relationships of the base items and the recommendation items with respect to how likely they are to trigger clicks.  More specifically, we examine whether those base items that are more likely to trigger clicks on recommendations are the same with those recommendation items that are more likely to receive  clicks. The study contributes to the understanding of factors that influence recommendation systems. The insights from investigating from different angles help 1) to understand what aspects of the base  item the user is viewing makes user click on a recommendations 2) to understand what aspects of the recommended items make the user click on those recommendations 3) to target those items that generate clicks and to ignore those that do not trigger recommendations.

% are more likely to trigger clicks, and what recommendation items are more likely to be clicked. To accomplish this task, we focus on items of categories that trigger more clicks. We identify items that triggered more clicks and items that caused less clicks. We also examine the relationship of the items that triggered clicks and the recommendations that are clicked and not clicked.

% We also examine  the relationship of the contenet of the base items and the items that are clicked to glean any relationship. For this, we employ contenet similarity measures between the base item and those items that are clicked from the base item. %A third angle is to look at the relationship between the  content of the base item and the items that are clicked from it. %This is interesting because sometimes it is not clear whether there i a direct relationship between the content similarity and behavioral factors.

% we can also look at  whether users actually clicked on those items which have some geographical relevance in the sense that the items are about the geographical region that they come from too.

\section{Dataset}

We used Plista\footnote{http://orp.plista.com/documentation}  dataset collected from user-item interaction with the tagesspiegel.com news portal, German online news and opinions portal,  over more than two months, from 15-04-2015 to 04-07-2015. Items in tagesspiegel are manually placed under $\mathit{10}$ categories, $\mathit{9}$ of which we investigate in this study.  The dataset is aggregated from the logs of the recommender systems that we used during our participation in the CLEF NewsREEL 2015 challenge \cite{kille2015overview}. The challenge offered   participants the opportunity to plug their recommendation algorithms to PLISTA\footnote{http://orp.plista.com/documentation} and provide recommendations to  real users visiting online publishers. PLISTA is a framework that connects recommendation providers such as ourselves and recommendation service requester such as online news portals. Participation in the challenge enabled us to collect information of user -item interaction such as impressions (a user viewing an item), update (appearance of news item, or change of  content of existing item) and clicks[user clicking on recommendation.

We used Plista\footnote{http://orp.plista.com/documentation}  dataset collected from user-item interaction with the tagesspiegel.com news portal, German online news and opinions portal,  over more than two months, from 15-04-2015 to 04-07-2015. Items in tagesspiegel are manually placed under $\mathit{10}$ categories, $\mathit{9}$ of which we investigate in this study.  The dataset is aggregated from the logs of the  recommender systems that we used during our participation in the CLEF NewsREEL 2015 challenge \cite{kille2015overview}. The challenge offered   participants the opportunity to plug their recommendation algorithms to Plista\footnote{http://orp.plista.com/documentation} and provide recommendations to  real users visiting online publishers. Plista is a framework that connects recommendation providers such as ourselves and recommendation service requester such as online news portals. Participation in the challenge enabled us to collect information of user-item interaction such as impression (a user viewing an item), update (appearance of new item, or change of  content of existing item) and click (a user clicking on recommendation item).

The three  recommendation that we used are two instances of Recency, and RecencyRandom. The recency algorithm keeps the most recently viewed or updated items and recommends the top most $mathit{k}$  recent items every time a recommendation request is made. The RecencyRandom recommender keeps the most recent $\mathit{100}$ items at any time and recommends, randomly, the requested number of recommendations every time  recommendation request is made

The three  recommendation algorithms that we used are two instances of \textbf{Recency}, and \textbf{RecencyRandom}. The Recency algorithm keeps the most recently viewed or updated items and recommends the top most $mathit{k}$  recent items every time a recommendation request is made. The RecencyRandom recommender keeps the most recent $\mathit{100}$ items at any time and recommends, randomly, the requested number of items every time a  recommendation request is made.

 Unfortunately, the click information did not include whether the click on recommendation was on our recommendations or on someone else's recommendations. Since we know the user and the base item for whom we recommended and the recommended items, we considered a click notification on one of our recommended items as a click on our recommendation if that click happened with in $\mathit{5}$ minutes  from the time of recommendation.  From the combined collected dataset,  we extracted the base item, the category of the base item, the recommended item and the category of the recommended item, the number of times a recommendation item has been recommended to a base item and the number of times that the recommended item has been clicked from the base item. A sample dataset is presented in Table \ref{tab:sample}.

 Unfortunately, the click information did not include whether the click on recommendation was on our recommendations or on someone else's recommendations. Since we know the user and the base item for which we recommended and the recommended items, we considered a click notification on one of our recommended items as a click on our recommendation if that click happened with in $\mathit{5}$ minutes  from the time of our recommendation.  From the combined collected dataset,  we extracted the base item, the category of the base item, the recommended item and the category of the recommended item, the number of times a recommendation item has been recommended to a base item (view) and the number of times that the recommended item has been clicked from the base item. A sample dataset is presented in Table \ref{tab:sample}.

\begin{table}

\caption{A sample dataset. B is the base item id, R  is the recommendation item id, and B-Cat and R-cat are the categories of the base item and the recommendation item respectively. }

\caption{A sample dataset. B is the base item id, R  is the recommendation item id, and B-Cat and R-cat are the categories of the base item and the recommendation item, respectively. \label{tab:sample}}

\centering

  \begin{tabular}{|l|l|l|l|l|l|l|}

\hline

           B  &  B-Cat & R& R-Cat &View&Click & CTR \\

           \hline

229397219 &229495114   &   Berlin &     Berlin &  17   &  1  & 5.88\\

230306628 &230291175&     politics &     wissen &   14 &    1 & 7.14\\

40485126 & 225589114  &      Berlin &    politics  & 2    & 0  & 0.00\\

   \hline

  \end{tabular}

  \label{tab:sample}

\end{table}

%Plista is a company that provides a recommendation platform where recommendation providers are linked with online publishers in need of recommendation sertvice.

% It is not easy to get the exact number of times a recommendation item is recommended to a   certain base item since the logs did not include wWe assume that the number of times a base item has been viewed as the number of times recommendations were shown. We assume this to be a fair assumption as recommendation were sought each time a an item was viewed by a user. % Although each time an item is viewed, more than one item (usually 5 items) are shown to the user as recommendations, we just count the number of clicks that have happened from those items

% regardless of which items are clicked.

\section{Results and Analysis}

Our dataset consists of a  a total of $\mathit{288979}$ base- item recommendation-item pairs. To see the relationship between views and recommendations, we first  sorted dataset according to views and then  normalized the view and click counts by the total number of views and total number of click, respectively. We then sleeted the top $\mathit{1000}$ pairs and plotted the views  and the clicks. the reason for normalization is to be able to plot them together and compare them. The selection of only 100 items is because the more items we use, the more difficult it is to compare them.

Our dataset consists of a total of $\mathit{288979}$ base-item recommendation-item pairs. To see the relationship between views and clicks, we first  sorted the dataset according to views and then  normalized the \textbf{view} and \textbf{click} counts by the total number of views and total number of click, respectively. We then slected the top $\mathit{1000}$ pairs and plotted the views  and the clicks. The reason for normalization is to be able to plot them together for easy comparision. %The selection of only $\mathit{1000}$ pairs is because the more items we use, the more difficult is to see .

Figure \ref{fig:view_click} shows the plot of views and clicks for the $\mathit{1000}$ pairs. The blue plot  is for views and is smooth since the data was sorted by views. The red plot is for the corresponding clicks on recommendations. We observe that the clicks do not follow the views, an indication  that t clicks do not correspond with the number of of times that a recommendation items is recommended to a base item. This is the reason we set out to investigate, to begin with.  The ragged click plot shows that some items are more likely to trigger clicks on recommendations than others.  What can possibly explain this observation?  What causes these difference between the number of views and the number of clicks?

Figure \ref{fig:view_click} shows the plot of views and clicks for the $\mathit{1000}$ pairs. The blue plot  is for views and is smooth since the data was sorted by views. The red plot is for the corresponding clicks on recommendations. We observe that the clicks do not follow the views, an indication  that t clicks do not correspond with the number  of times that a recommendation items is recommended to a base item. This is the reason we set out to investigate, to begin with.  The ragged click plot shows that some items are more likely to trigger clicks on recommendations than others.  What can possibly explain this observation?  What causes these difference between the number of views and the number of clicks?

\begin{figure} [t]

\centering

\includegraphics[scale=0.5]{img/tage_view1_click000.pdf}

\caption{Plots of views and clicks on Tagesspiegel. The Plots are normalzized by the total views and total clicks. The Blue plot (bottom) ais the sorted view plot and the red plot is the corresponding click plot.\label{fig:view_click}}

\end{figure}

%  \begin{figure} [t]

% \centering

% \includegraphics[scale=0.5]{img/tage_view100.pdf}

% \label{fig:view100}

% \caption{Plot of the most viewed 100 items}

% \end{figure}

%  \begin{figure} [t]

% \centering

% \includegraphics[scale=0.5]{img/tage_click100.pdf}

% \label{fig:click100}

% \caption{Plot of the clicks triggered from the 100 most viewed items}

% \end{figure}

\subsection{Categories of Base and Recommendation Items}

To start to explain the difference between the view plot and the click plot,  we aggregated  views and clicks by the $\mathit{9}$ categories of   items  that the items are placed under in the Tagesspiegel website. The aggregation gives us two results: view and click counts aggregated by base categories of the base items and  by the categories of recommended items.   In other words, we are attempting to answer two questions: 1)  is there a relationship between the category of the base item and the likelihood that  it triggers  a click on recommendation, regardless of the type of recommendation and 2) is there a relationship between the category of the recommended item and the likelihood that it triggers a click upon its recommendation?    Tables \ref{tab:base} and \ref{tab:reco} present  the views, clicks and CTR scores. The results are sorted by CTR scores.

To start to explain the difference between the view plot and the click plot observed in \ref{fig:view_click},  we aggregated  views and clicks by the $\mathit{9}$ categories of   items  that the items are placed under in the Tagesspiegel website. The aggregation gives us two results: view and click counts the  categories in base and in recommendation.   With the categories, we attemopt to answer two questions: 1)  is there a relationship between the category of the base item and the likelihood of triggering  a click on recommendation, and  2) is there a relationship between the category of the recommended item and the likelihood of triggering a click upon its recommendation?    Tables \ref{tab:base} and \ref{tab:reco} present  the views, clicks and CTR scores. The results are sorted by CTR scores.

We observe that there is a difference between the base categories and the recommendation categories with respect to generating clicks.  In the base categories, the category of politics triggers clicks more than any other category. After the category of politics, the categories of opinion and the world  trigger more clicks on recommendations. Special categories such as culture and and knowledge trigger the least clicks on recommendations. This is consistent with previous findings that reported special interest portals enjoyed less clicks than traditional and mainstream news and opinion portals.

We observe that there is a difference between the base categories and the recommendation categories with respect to the likelihood of triggering clicks.  In the base categories, the  \textbf{politics} is more likely to triggers clicks  than any other category \textbf{opinion} and \textbf{world}. Special categories such as \textbf{culture} and and \textbf{knowledge} are the least likely to trigger clicks on recommendations. This is consistent with previous findings that reported special interest portals generated  less clicks on recommendations than traditional and mainstream news and opinion portals.

\begin{table*}

\caption{A table showing the views, clicks, and ctr of the 12 categories of Tagesspiegel on the basis of the base items. This table shows the views, clicks and CTRs of the base item. A click for base item happens when an item recommended to it is clicked. }

\parbox{.45\linewidth}{

\centering

\begin{tabular}{|l|l|l|l|l|}

\hline

           category  &  Views & Clicks & CTR (\%)\\

           \hline

politics&73197&178&0.24\\

media&22426&50&0.22\\

weltspiegel&37413&77&0.21\\

wirtschaft&30045&61&0.2\\

sport&29812&58&0.19\\

berlin&123595&129&0.1\\

meinung&4611&3&0.07\\

kultur&21840&11&0.05\\

wissen&13500&4&0.03\\

   \hline

  \end{tabular}

\caption{Base Category \label{tab:base}}

\hfill

\parbox{.45\linewidth}{

\centering

 \begin{tabular}{|l|l|l|l|}

\hline

           category  &  Views & Clicks & CTR (\%)\\

           \hline

medien&22147&68&0.31\\

politik&68230&170&0.25\\\

berlin&123559&188&0.15\\

weltspiegel&37535&58&0.15\\

sport&28160&36&0.13\\

meinung&4925&5&0.1\\

kultur&23278&21&0.09\\

wissen&15650&10&0.06\\

wirtschaft&32955&15&0.05\\

   \hline

  \end{tabular}

\caption{Recommendation Category \label{tab:reco}}

\end{table*}

On the recommendation categories, however, it is the media category that triggers more clicks upon recommendation, followed by  politics and the local categories. The two least performing recommendation categories are economy and knowledge,  similar to the least performing base categories. So, overall, it seems that the likelihood of triggering clicks by the categories shows a difference when they are in base and recommendation.  Overall, it seems the categories have higher CTRs in recommendation that in base. To gain further insight, we looked at the CTRs of transitions from base category to recommendation category. The aim of this is to find out whether some base categories are more likely to trigger clicks on some recommendation categories. The results are presented in Table \ref{heatmap}.

On the recommendation side, however, it is  \textbf{media} that is the more likely to triggers  clicks upon recommendation, followed by  \textbf{politics} and the local category (\textbf{Berlin}. The two least performing categories are \textbf{business} and \textbf{knowledge},  similar to the least performing  categories in base. So, overall, it seems that the likelihood of triggering clicks by the categories shows a difference when they are in base and recommendation.  In general, the  categories have higher CTRs in recommendation that in base. To gain further insight, we looked at the CTRs of transitions from base category to recommendation category. The aim of this is to find out whether some base categories are more likely to trigger clicks on some recommendation categories. The results are presented in Table \ref{heatmap}.

There are some interesting observations in the category-to-category transitions. While the highest transition CTRs for the base categories of  Berlin and Politics are to Media, for economy, it is to opinion, for sport it is to sport. The highest transition CTR for Culture is to the local category Berlin, and for world it is to politics followed by to Berlin.  The media category is the one that receive clicks from more categories than any others. The local category Berlin is the one that clicks on recommendations from more categories.

\begin{table*}

\caption{Transition CTR scores from base categories to recommendation categories.  The row categories represent the categories of base items and the column categories represent the recommendation categories. \label{heatmap}}

  \begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|}

\hline

&Berlin&politics&wirtschaft&sport&kultur&weltspiegel&meinung&medien&wissen\\

\hline

berlin&0.14&0.08&0.06&0.05&0.06&0.12&0.12&0.16&0.06\\

politik&0.2&0.39&0.06&0.12&0.04&0.3&0&0.73&0.1\\

wirtschaft&0.15&0.4&0.07&0.13&0.36&0.13&0.46&0.21&0\\

sport&0.14&0.27&0&0.68&0.05&0.18&0&0.27&0.07\\

kultur&0.11&0&0&0.06&0.07&0&0&0.07&0\\

weltspiegel&0.24&0.27&0.06&0.13&0.17&0.13&0&0.4&0.18\\

meinung&0.06&0&0&0&0&0&1.85&0.32&0\\

medien&0.1&0.85&0&0.06&0&0.08&0&0.16&0\\

wissen&0.02&0&0&0&0.11&0.15&0&0&0\\

 \hline

  \end{tabular}

\end{table*}

%  For example  if we look at the category of politics , we see that the CTR from politics to politics is the highest than from politics to any other category.  We also observe that the CTR from local category Berlin to politics is higher than from the local category Berlin to any other category including to itself. A little surprising result is the high CTR from media to politics.

% The way we extracted our recommendations and clciks is a little uncertan. In the Plista setting, when click results are reported to users, they are not known whose recommendations are being clicked. So while we know our recommendation, we do not know for sure how much of the click notifications that we recieve belong to our recommendations. To extract our clciks, we introduced a time frame of 5 minutes. That is if the click notification happens in with in a range of time, in our case 5 minutes, we consider the clcik is on our recommendations. We consider the click information is a bit inflated for users might not stay for more than 5 minutes. While the actual CTR might be a bit inflated as a result of the inflated number of clicks, we consider the relative scores as indicative of the true difference.

% To find out therelationship between base item recommendation pairs that resulted in high CTR scoores, we selected some item-recommendations pairs. To avoid selecting item-recommendation pairs that have very low views and clicks which is usually the type of combination that results in high CTR scores, we first sort our data according to views, and according to clicks. Using  cutt off values, we repeat the intersection until we find the items that have both the highest view and the hight clicks. Using this approach we selected 12 item-recommendation pairs and out of them we selected the 5 pairs that have the highest score. These pairs are presented in Table \ref{}

\subsection{Item-level Base and Recommendation CTRs}

% We look at the two types of item-level CTR's:the base item CTRs and the recommendation CTRs.  The base item CTR measures how likely the base item is to trigger clicks on recommendation. We assume that part if clicking on recommendations is a function of the item the user is reading. this is corroborated by the category-level CTr's that we looked at above in thesense that some categories do not generate clicks. even if the item are from clickable categories. The recommendation CTR's ameasures how likely the item is to recieve a click when recomened to a user regardless of the category of the base item.  But, should we not be concerned about the base item?