The Evolution of WTF: Follower Recommendation Services at Twitter

Speaker: Jimmy Lin, University of Maryland

Abstract:

WTF (Who to Follow) is Twitter's user recommendation service, which is responsible for creating millions of connections daily between users based on shared interests, common connections, and other related factors. In this talk I will discuss the evolution of the WTF service: the first generation architecture depended on a system called Cassovary, an open-source in-memory graph processing engine built from scratch by Twitter specifically for WTF. This approach gave way to a Hadoop-based machine learning framework, which has recently been supplemented by a custom architecture for generating real-time recommendations. I will discuss the tradeoffs between different architectures, provide a general overview of algorithms, and share lessons learned in running a large-scale production service.

Bio:

Jimmy Lin is an Associate Professor in the College of Information Studies (The iSchool) at the University of Maryland, with a joint appointment in the Institute for Advanced Computer Studies (UMIACS) and an affiliate appointment in the Department of Computer Science. He graduated with a Ph.D. in Electrical Engineering and Computer Science from MIT in 2004. Lin's research lies at the intersection of information retrieval and natural language processing; his current work focuses on large-scale distributed algorithms and infrastructure for data analytics. From 2010-2012, Lin spent an extended sabbatical at Twitter, where he worked on services designed to surface relevant content to users and analytics infrastructure to support data science. He continues to engage with Twitter on various aspects of big data and data science.