% \section{Discussion and Conclusion} \label{conc}
% 
% In this work, we discussed two different perspectives on personalization in recommender system: one that views personalization as morally good and beneficial, and another that views personalization as bad and disadvantageous. We discussed that these two perspectives call for two different measures of personalization: system-centric and user-centric.  In this study, we approached it from the perspective of those who seek a perfect personalization. For users, we abstracted away from the actual user and considered geographical units, German states to be specific. These provided us with two advantages: less data sparsity and an opportunity to examine personalization at a higher level.
% We specifically proposed a comparative and reactive metric which we called PullPush as a measure of the users tendency to drift away or come closer towards each other in terms of interests (e.g. concepts, entities, topics) included within their personalized recommended content.
% Using German states as users, items as elements of vectors of   views(recommendations) and clicks,  we examined the state of personalization in two online publishers using  the proposed metric. We obtained PullPush scores for personalization at aggregate and pairs-of-states level. We discussed that the PushPull scores can be viewed as the potential for personalization/depersonalization at the particular user (cohort). 
% 
% 
% Whether we want to avoid under- or over-personalization in recommendation - with either too much non-relevant content being shown, or where users potentially end up in their own separate content bubbles - each of these perspectives call for measuring the extent to which a system actually personalizes recommendation. In this work, we attempted to quantify the degree of personalization in a recommendation system by comparing the 'distance' between different users. 
% The method presented here uses user engagement history, such as click or dwell history, as an approximate measure of user preference and compares the recommended content against this preference.  
% 
% The proposed measure allows a quantification of whether the preferences between (cohorts of) users are more or less different, \textit{'further apart'}, than the difference of the content that is shown by a recommendation system. 
% As a demonstration, we  applied the method to cohorts of users in different geographical areas for a news recommender. In this demonstration, the click history compared to recommended content suggested the potential for more personalization - if the goal is to increase engagement. And, if the goal is for a system to keep people slightly closer in terms of the content that they see, as compared to the difference that would 'organically' occur when fully optimizing for engagement - we can choose to have the system push the cohorts slightly back towards each other.


\section{Discussion and Conclusion} \label{conc}
Personalized recommendation can lead to increased engagement, but in some cases can also mean that users may end up reading drastically different content. There is a concern that the later may lead to filter bubbles. Whether we want to avoid, by a manner of speaking, under- or over-personalization - with either too much non-relevant content being shown, or where users potentially end up in their own separate content bubbles - each of these perspectives call for measuring the extent to which a system actually personalizes recommendation.

In this work, we attempted to quantify the degree of personalization in a recommendation system by comparing the 'distance' between different users.  The method presented here uses user engagement history, such as click or dwell history, as an approximate measure of user preference, and presents a comparison with serving history. We specifically proposed a comparative and reactive metric which we called PullPush as a measure of the users tendency to drift away or come closer towards each other in terms of interests (e.g. concepts, entities, topics) included within their personalized recommended content. The proposed measure allows a quantification of whether the preferences between (cohorts of) users are more or less different, \textit{'further apart'}, than the difference of the content that is shown to them by a recommendation system. 

As a demonstration, we  applied the method to cohorts of users in different geographical areas for news recommendation. For users, we abstracted away from individual users to (users in) geographical units, German states to be specific. This provided us with two advantages: less data sparsity and an opportunity to examine personalization at the cohort level.
Using users in German states as user cohorts, items as elements of vectors of views (served recommendations) and clicks, we examined the state of personalization in two online publishers using  the proposed metric. 
We obtained PullPush scores for personalization at aggregate and pairs-of-states level. We discussed that the PushPull scores can be viewed as the potential for personalization/depersonalization at the particular user or user cohort. 

In the demonstration, the click history compared to recommended content suggested the potential for more personalization when aiming for increasing engagement. 
In this case, it turns out that user cohorts were served with information outside of their regular interest spheres (or bubble). Interestingly, this could suggest either a conclusion that there is a potential for more personalization or localization to increase engagement further, or it could suggest that using this particular news recommendation website actually keeps users from 'organically' drifting as far apart as they would if they only were served content befitting their interests. In any case, using this methodology allows to gain insight in these processes, by quantifying the level of personalization towards only users' interests.