Datacast Episode 26: From Cognitive Neuroscience to Reinforcement Learning with Arthur Juliani
Datacast’s 26th Episode is my conversation with Arthur Juliani, a Senior Machine Learning Engineer at Unity Technologies — a well-known video game software development company. Listen to dig into his background in Cognitive Neuroscience, his popular blog series on Reinforcement Learning, the open-source ML-Agents Toolkit that his team built at Unity, his advice for grad students who want to make a dent in the research community, and more.
Arthur Juliani is a Senior Machine Learning Engineer at Unity Technologies, where he has worked as a founding member of the ml-agents GitHub project as well as the leader of the Obstacle Tower project. He is also currently a Ph.D. candidate in Cognitive Neuroscience at the University of Oregon, where he studies computation models of spatial representation learning in humans.
Unity ML-Agents Toolkit (https://awesomeopensource.com/project/Unity-Technologies/ml-agents)
Show Notes
(2:00) Arthur talked about his undergraduate studying Psychology at North Carolina State University.
(3:28) Arthur mentioned his time working as a research assistant at the LACElab in NCSU that does human factor and cognition research.
(5:08) Arthur discussed his decision to pursue a graduate degree in Cognitive Neuroscience at the University of Oregon right after college.
(6:35) Arthur went over his Master’s thesis (Navigation performance in virtual environments varies with a fractal dimension of landscape) in more detail.
(10:30) Arthur unpacked his popular blog series called “Simple Reinforcement Learning in TensorFlow” on Medium.
(12:56) Arthur recalled his decision to join Unity to work on its reinforcement learning problems.
(14:31) Arthur recalled his choice to do a Ph.D. part-time while working full-time.
(16:24) Arthur discussed problems with existing reinforcement learning simulation platforms and how the Unity Machine Learning Agents Toolkit addresses those.
(18:30) Arthur went over the challenges of maintaining and continuously iterating the Unity ML-Agents toolkit.
(20:36) Arthur emphasized the benefit of training the agents with an additional curiosity-based intrinsic reward, which is inspired by a paper from UC Berkeley researchers (check out the Unity blog post).
(22:33) Arthur talked about the challenges of implementing such curiosity-based techniques.
(25:15) Arthur unpacked the introduction of the Obstacle Tower — a high fidelity, 3D, third-person, procedurally generated environment — released in the latest version of the toolkit (read his blog post “On “solving” Montezuma’s Revenge”).
(29:15) Arthur discussed the Obstacle Tower Challenge, a contest that offers researchers and developers the chance to compete to train the best-performing agents on the Obstacle Tower Environment.
(32:49) Referring to his fun tutorial called “GANs explained with a classic sponge bob square pants episode,” Arthur walked through the theory behind the Generative Adversarial Network algorithm via an explanation using an episode of Spongebob Squarepants.
(34:30) Arthur extrapolated on his post “RL or Evolutionary Strategies? Nature has a solution: Both.”
(38:36) Arthur shared a couple of approaches to balance the bias and variance tradeoff in reinforcement learning models, referring to his article “Making sense of the bias/variance tradeoff in Deep RL.”
(41:19) Arthur talked about successor representations and their applications in deep learning, psychology, and neuroscience (read his post “The present in terms of the future: Successor representations in RL”).
(42:38) Arthur reflected on the benefits of his Psychology and Neuroscience background for his research career.
(44:21) Arthur shared his advice for graduate students who want to make a dent in the AI / ML research community.
(45:30) Closing segment.
His Contact Info
His Recommended Resources
Being and Time (by Martin Heidegger)