Datacast Episode 46: From Building Recommendation Systems To Teaching Online Courses with Frank Kane
The 46th episode of Datacast is my conversation with Frank Kane - the owner of Sundog Education. Give it a listen to hear about his experience working in game development, building recommender systems at Amazon, venturing into self-employment, teaching data science online courses, and more.
Listen to the episode on: (1) Spotify, (2) Apple Podcasts, (3) Google Podcasts, (4) Stitcher, (5) Breaker, (6) iHeart Radio, and (7) TuneIn
Key Takeaways
Here are the highlights from my chat with Frank Kane:
On Studying Electrical Engineering
I went to school for electrical engineering, but I have never done anything with it throughout my career. But that doesn't mean it was wasted.
The main thing that I got out of my undergraduate education is developing a strong work ethic and time management. To pay me throughout college, I also worked full-time at a health insurance company. I went to school in the morning for electrical engineering, which was not easy, and then went to work night shifts for another 8 hours. Doing that without going crazy taught me about grit, diligence, and perseverance.
On Working In The Game Industry
I moved from Boston to the middle of nowhere in California. My team was an eclectic group of engineers, game designers, artists, and musicians. I worked alongside such a diverse group of folks, from Disney animators who worked in old Disney cell-animation films to music composers with a collection of Emmy awards.
The work environment was demanding. I remembered sleeping at the office a few times.
Back then, we wrote programs in very low-level code such as Assembly Language. Today, game developers built programs on top of game engines such as Unreal or Unity.
On Pioneering The Amazon's Recommendation System
Amazon didn't really care much about the specific technical skills. They want to find smart people who have the determination and fundamental engineering skills to solve problems that weren't solved before. Thankfully, they saw that in my resume and brought me in as a senior software engineer.
Initially, I worked on the systems that kept track of the top sellers on Amazon.com. Then I quickly moved to the system that recommends items to customers. This was a totally new problem back then.
The first challenge was to figure out the algorithms. There was some academic research that we built upon coming out of the University of Michigan. Still, we needed to figure out how those applied to the real-world with dynamic customer data.
The second challenge was to scale the algorithms to the massive amount of inputs, like 10,000 requests per second. Vending the recommendations at that rate turned out to be extremely hard. Back then, we didn't have tools like Hadoop, Docker, or Kubernetes. We ended up inventing those technologies from scratch.
We made slow and steady progress over time with new algorithms, new UI, new technologies, etc.
On Proudest Accomplishments at Amazon
From a technical standpoint, I am proud of the incremental progress that my team made towards Amazon's underlying recommendation system. These recommendations generate millions to billions of revenue for the company. Being responsible for that sort of an impact, not just on Amazon's bottom line but also on the customer experience, meant a lot to me.
From a managerial standpoint, I am proud of the team environment that I created there. People felt like they were listened to. Furthermore, any engineering challenges that came up were met with a positive attitude and a willingness to solve them.
On Amazon's Leadership Principles
Amazon takes its leadership principles very seriously. Every year, employees are evaluated on how well they embody those principles. The principles are not just lip-service. They denote a set of very carefully thought-out principles that guide how the Amazon employees work.
The number one principle on that list is to be customer-centric by focusing on the customer experience. That makes Amazon's engineering culture very unique. They don't develop technology for the sake of technology, ever. They always start with what impact the technology might have on the customer experience.
On Venturing Into Self-Employment
Don't just quit your day job if you have bills to pay. Make sure you have at least 3 months of savings. Make sure you have at least something that generates revenue before transitioning into self-employment.
Whenever possible, invest in creating products that can sell itself when you sleep. Software libraries and online courses are great examples.
Most businesses are self-funded lifestyle businesses. If you can create a business using your own money, you don't own anybody anything. That's a comfortable living year after year, as opposed to the high-growth businesses.
On Becoming An Online Instructor
The pros: My online courses have made an impact on hundreds of thousands of people. Furthermore, they scale up well, giving me the freedom to work whatever and whenever I want.
The con: When you have an audience of 500,000 people, you start to run into the same problem that celebrities have. Managing my privacy turns out to be a necessity at that scale.
My advice for aspiring online instructors is to be very careful about what topics they want to teach. Don't teach a popular topic that has been soaked up by top instructors. There are always new technologies coming out that no one has taught before. Invest time in these green fields instead of competing with established players.
On Recommendation Systems
The big trend recently is the shift to applying neural networks to recommendation systems. I don't think qualitatively, that has made much of a huge difference.
The real thing that recommendation systems need to deal with in the future is how they fit into our society and our privacy. Recommendation systems are only as good as the data that we give them, so drawing that line would be an interesting direction.
Show Notes
(2:05) Frank reflected on his undergraduate experience studying Electrical Engineering at the University of Massachusetts - Dartmouth.
(3:33) Frank commented on his experience working in the game industry after school.
(6:28) Frank went over the opportunity to work as a software engineer at Amazon, where he contributed to the personalization system that recommends products to customers at a scale of tens of thousands of requests per second.
(8:44) Frank brought up the challenges of building Amazon’s recommendation systems back in the early days.
(10:14) Frank discussed how Amazon’s recommendations and content optimization technology evolved incrementally during his time as a Senior Manager.
(12:05) Frank touched on the core engineering challenges during his time as a Senior Manager of Technology at IMDB.
(14:19) Frank spoke about his proudest accomplishments at Amazon, both from the technical and the management perspectives.
(18:19) Frank shared the story behind his professional transition into self-employment (check out his book “Self-Employment: Building an Internet Business of One”).
(24:07) Frank shared a brief overview of his business (Sundog Software)'s virtual reality products.
(25:15) Frank shared how he came to be an online instructor, discussed the pros/cons, and provided advice for aspiring ones.
(29:38) Frank has created various courses that focus on Apache Spark, ranging from Python and Scala support to Spark Streaming capability.
(31:34) Frank discussed how the Hadoop ecosystem has fallen out of favor (check out his popular Udemy courses titled “The Ultimate Hands-On Hadoop”).
(33:20) Frank touched on ElasticSearch - an industry-standard open-source search engine (check out his Manning live videos on ElasticSearch 6 and ElasticSearch 7).
(37:08) Frank provided his perspectives on the current landscape of recommendation systems research and applications.
(42:17) Frank advised scientists and engineers on how to communicate with non-technical colleagues effectively.
(43:25) Closing segment.
His Contact Info
His Recommended Resources
"Lean Startup" by Eric Ries
"Architecting Modern Data Platforms" by Jan Kunigk, Ian Buss, Paul Wilkinson, Lars George