%\section{ and Experiments} \label{exp}
\section{Datasets, Results and Analysis} \label{result}
We used two datasets of recommendations and clicks collected from Plista.
% The second dataset is Yahoo! stream dataset. In this dataset, the dimension of the vectors are entities. Entities are relatively perennial, as compared to items.
%Or maybe we should consider some cities and Tagesspiegel? I think that seems to make more sense. With the second case, we considered a total of 14 cities, seven of them around the Bay Area and 7 of them around New York city. The results both are multidimensional scaled as shown in Figure \ref{}.
Plista is a recommendation service provider that offers the Open Recommendation Platform (ORP)\footnote{http://orp.plista.com/documentation}, a framework that brings together online content publishers in need of recommendation service and news recommendation service providers (referred by participants from now on) that provide recommendation by plugging their recommendation algorithms to the platform. When a user starts reading a news item, a recommendation request is sent to one of the recommendation providers while the other participants receive the impression information - information about the user visiting the item. When a recommended item is clicked by the user, all participants receive the click information. Every participant has access to all user-news item interaction information. It is however, not possible to find out who recommended the clicked item.
For we can not know whose recommendation was clicked, we collected all the clicks regardless of who recommended them. Another way to look at it is to think of all the recommenders as one ensemble recommender. We also assumed that the recommenders from the participants did not employ personalization, for most of the recommenders were variations of the recency algorithm. This in effect means that we used the same view vector for different states. For our analysis we choose two German news and opinion portals: Tagesspiegel\footnote{www.tagesspiegel.de} and Kstatager\footnote{http://www.ksta.de}. For users, we chose the 16 states of Germany.
%\subsection{Yahoo! Dataset}
We applied the PullPush equation Introduced in \ref{pullpushaggr} to the view and click vectors of two publisher: Tagesspiegel and Ksta. This means we had $mathit{16}$ view and click vectors, one pair for each German state. The components of the vectors are items. One can also use other components such as entities or some other meta data. The PullPush scores are presented in Table \ref{tage-ksta}, and for visual ease in figures \ref{fig:tage} and \ref{fig:ksta}.
There are two levels of results we can obtain with the proposed method. The first level is the aggregate personalization score for the overall personalization in a recommender system as obtained using Equation \ref{pullpushaggr}. The scores of the recommender systems for the two publishers of Tagesspiegel and Ksta are $\mathit{-0.224}$ and $mathit{-0.213}$ respectively. The smaller score for Ksta can be explained by the fact that ksta is a more geographically local publisher as compared to Tagesspiegel which is a nationally read publisher. A more local readership means less diversity in users and in turn less diversity in interests and preferences. Tagesspiegel has larger and more diversified readership, which means more need to cater to different needs. The smaller PullPush score for Ksta means there is less need for personalization in the publisher than there is in the case of Tagesspiegel.
The second level score is the individual results for the pairs of users, which shows how the system's personalization performs in each individual pairs. The results for select $\mathit{11}$ German states are presented in Table \ref{tage-ksta}. The upper diagonal shows the PullPush scores for Tagesspiegel and the lower diagonal for Ksta. The first thing that we observe is that all the scores are negative, indicating that there was overall a potential for personalization in both publishers. Another observation is that when we compare the results for corresponding pairs of states in Tagesspiegel and in Ksta, we find that in most cases the scores in Tagesspiegel have greater absolute values than those in Ksta. This is again an indication of the national character of Tagesspiegel and thus a more diverse audience and a more need for personalization to meet that diverse need.
Looking at specific scores though, we find that Westphalia has the largest absolute PullPush score with all of the other states in the case of Ksta. In Tagesspiegel, Berlin followed by Brandenburg have the largest absolute scores. These scores can be seen in the multidimensional visualizations in Figure \ref{fig:tage} and Figure \ref{fig:ksta}. This can be explained by the fact that Ksta is based in Cologne and most of its readers come from the state in which Cologne is found - the state of Westphalia. There is more need for personalization for audiences coming from Westphalia than there is for audiences coming from other states. Similarly,
%Just looking at the multidimensionally scaled results, we can see differences between the two publishers. In Tagesspiegel (Figure \ref{fig:tage}), we observe that Berlin followed by Brandenburg are further apart from the rest of the states.
personalized recommendations are much more needed in Berlin and then in Brandenburg than in other states for Tagesspiegel. This has to do with the fact that Tagesspiegel is seen as primarily the local portal for Berlin and Brandenburg as it is also based in Berlin. So clearly, as most people from these states consume news from Tagesspiegel, making a distinction between users by means of personalization would make much more sense here, than on other states with less consumers. %Applying the proposed method on 16 German states, we found that they all wanted to drift away from each other suggesting that the recommendation algorithms were overloading the states with information that is not engaged with.
% The next observation we see is that the distances among the other states is a bit far apart
%
% In the case of Ksta (Figure \ref{fig:ksta}), we observe that Westphalia stands out as the furthest states from the others. The other states are much more closer to each other than how they are in the case of Tagesspiegel. This is an indication that
% there is more need for personalization in the Westphalia than in other states. The same is true for Berlin and Brandenburg.
%A third level score is to choose a pair that for inspection and look at the items that are doing worse/better. In \ref{} are the results for aggregate performance levels for the recommendations on the two Plista datasets (Tagesspiegel and Ksta). %and on the yahoo dataset.
%Figures \ref{fig:tage} and \ref{fig:ksta} show the results of the distances between each pair in a multidimensionally scaled to two-dimension for viewing.
\begin{table*}
\caption{Adjacency matrix of PushPull scores for select states of Germany. The upper part of the matrix is for Tagesspiegel and the lower part for Ksta. Comparing the corresponding scores for Tagesspiegel and Ksta, we observe that, for most pairs, the absolute value of the scores in Tagesspiegel are larger than those in Ksta, an indication that overall there is more potential for personalization in the former. Individually, Westphalia in Ksta has the largest absolute PullPush scores, an indication that there is a bigger potential for personalization in this state than in any other states. }
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
&Baden&Bavaria&Berlin&Bremen&Hamburg&Hessen&MeckPom&Saarland&Saxony&Thuringia&Westphal.\\
Baden&0&-0.127&-0.221&-0.302&-0.198&-0.144&-0.302&-0.344&-0.192&-0.26&-0.133\\
Bavaria&-0.178&0&-0.221&-0.307&-0.201&-0.146&-0.306&-0.349&-0.196&-0.263&-0.129\\
Berlin&-0.198&-0.187&0&-0.366&-0.264&-0.233&-0.359&-0.404&-0.256&-0.321&-0.204\\
Bremen&-0.291&-0.271&-0.204&0&-0.206&-0.269&-0.161&-0.156&-0.231&-0.174&-0.334\\
Hamburg&-0.227&-0.215&-0.158&-0.149&0&-0.176&-0.212&-0.245&-0.166&-0.185&-0.221\\
Hessen&-0.168&-0.167&-0.164&-0.247&-0.184&0&-0.271&-0.31&-0.177&-0.227&-0.156\\
MeckPom&-0.279&-0.267&-0.192&-0.087&-0.134&-0.235&0&-0.165&-0.228&-0.173&-0.334\\
Saarland&-0.281&-0.267&-0.194&-0.087&-0.137&-0.235&-0.086&0&-0.265&-0.194&-0.377\\
Saxony&-0.233&-0.218&-0.158&-0.138&-0.126&-0.188&-0.125&-0.125&0&-0.195&-0.21\\
Thuringia&-0.268&-0.257&-0.185&-0.098&-0.127&-0.227&-0.083&-0.091&-0.121&0&-0.288\\
Westphal.&-0.382&-0.389&-0.452&-0.569&-0.502&-0.401&-0.561&-0.56&-0.509&-0.554&0\\
\hline
\end{tabular}
\label{tage-ksta}
\end{table*}
\begin{figure} [t]
\centering
\includegraphics[scale=0.5]{img/mds_tage.pdf}
\label{fig:tage}
\caption{Multidimensional scaling of the PullPush scores for Tagesspiegel. We observe that the highest potential for personalization is to be found in the state of Berlin followed by Brandenburg.}
\end{figure}
\begin{figure} [t]
\centering
\includegraphics[scale=0.5]{img/mds_ksta.pdf}
\caption{Multidimensional scaling of the PullPush scores for Ksta. We observe that the highest potential for personalization is to be found in the state of Westphalia.}
\label{fig:ksta}
\end{figure}
%The next natural question was which entities should be served more, which entities should be served less, or which entities should be left unchanged and for which cities.
% % So for each entity shared between two users, we have 9 combinations of increases, decreases and do-not-changes(do nothing). Note that each increase or decrease is to be based on the view and click scores.
% The goal of recommending increase, decrease or do-not-changes for each entity is in order to achieve an equilibrium, that is $\mathit{PullPush=0}$ between the two geographical units under consideration.
% %This part of the work is not done, but we feel, if solved, can be a good addition to and a completion of the above method we proposed.