Судя по описанию, shared task’и выходят на новый уровень: организаторы не только проводят сореванования рекомендательных систем, но и позволяют отлаживать их в реальном времнеи с реальными юзерами.
CLEF NewsREEL 2016
CLEF registration opens: 30 October 2015
Complete CLEF registration at the latest: 2 April 2016
Paper Submission Deadline: 25 May 2016
Feedback on Submission Status: 17 June 2016
Camera-ready Version Due: 1 July 2016
CLEF Conference and Labs (Evora, Portugal): 5-8 September 2016
We invite you to develop stream recommendation solutions to address a news recommendation task in two exciting settings. First, in Task 1, your solution has the chance to serve real users in a living lab. Second, in Task 2, you can “re-play” logged interactions to compare different algorithms on identical data. Task 2 can support you in developing your solution before deploying it online in Task 1. Continue reading for more details.
—- Task 1: Online Evaluation in a Living Lab —
Task 1 is a living lab that provides task participants with access to A/B testing facilities of a commercial service. In this way, the task allows participants to evaluate the response of a real-world user population to different recommender system algorithms.
As soon as you sign up to participate, you can implement a system, and your system performance will appear on the public leaderboard. This will allow you to compare your performance as soon as possible to a baseline recommender.
The winner for this task will be determined on the basis of performance during the four week Challenge Window in April 2016. The official evaluation metric is the click-through-rate (CTR). CTR measures clicks/requests for all those requests to which a participating system has responded successfully within the time limit.
We encourage you to start developing and optimizing your algorithms as soon as possible. To support you in this respect, we suggest that you participate in Task 2 and also take advantage of the three Development Windows, during which we will record performance in order to give you a picture on where you stand.
The time frames are:
6-12 February 2016: Development Window 1
20-26 February 2016: Development Window 2
5-11 March 2016: Development Window 3
2-29 April 2016: Challenge Window
The Challenge Window is timed so that you can submit your paper to the CLEF Lab by the 25 May 2016 deadline. Note that you sign up for CLEF after 2 April, you will be allowed to submit a paper on your solution to the CLEF Lab, but you cannot win the competition. For this reason, we advise you to sign up as soon as possible, and make sure that your system is running stably by 2 April. If you need a virtual machine to run you solution, we will provide you with one.
— Task 2: Offline Evaluation using Simulation Framework—
Task 2 allows you to run a comprehensive evaluation offline, comparable to the online evaluation of Task 1. The official evaluation metric is the click-through-rate (CTR), but we encourage participants to consider other metrics as well. We provide you with a data set and a framework that allows you to compare a variety of algorithms on identical data and reliably conclude which one performs best. Additionally, in Task 2, you gain insights on the technical challenges of running recommendation algorithms including complexity, scalability, and responsiveness. Besides prediction accuracy, you can run extensive experiments to determine which algorithm handles the most simultaneous load, which algorithm requires the least memory, or which algorithm can be most easily distributed. These factors reflect important constraints in industrial recommender systems.
You can find more information and also the CLEF registration link at:
With questions or for more information, please contact: