The 67th episode of Datacast is my conversation with Aparna Dhinakaran — the Chief Product Officer at Arize AI, a startup focused on ML Observability.
Our wide-ranging conversation touches on her educational background in Electrical Engineering and Computer Science at UC Berkeley; her work on the model store for Uber’s Marketplace; the difference between academic ML and production ML; the case for researching AI Bias; her current journey with Arize to tackle real-time ML observability; her predictions for the ML infrastructure ecosystem, and much more.
Please enjoy my conversation with Aparna!
Listen to the show on (1) Spotify, (2) Apple Podcasts, (3) Google Podcasts, (4) TuneIn, (5) RadioPublic, (6) Stitcher, and (7) Breaker.
Key Takeaways
Here are highlights from my conversation with Aparna:
On Her Undergraduate at UC Berkeley
I originally went through the Electrical Engineering and Computer Science programs because I really liked Math. I was looking for applications of Math that would be interesting majors. UC Berkeley’s combined EECS is the ideal one.
I did an even split between Electrical Engineering and Computer Science, but I got really interested in the Machine Learning courses at Berkeley (such as CS189, EE126, EE127, etc.). I was also working on research projects focused on ML. UC Berkeley exposed me to the growing interest in AI and all the applications people can build using AI.
UC Berkeley has research professors who work on different projects in which undergraduates can get involved with. Sometimes you can walk up to the professors and demonstrate your interest to contribute. Sometimes graduate students can take you on as an undergraduate researcher.
One professor I worked with was Alice Agogino, who heads up the BEST Lab at Berkeley. We worked on a lighting system on a smart grid, figuring out how to optimally place the lightning sensors in order to replicate smart grid optimizations. We used ML algorithms (even in 2013) for this task. I enjoyed this whole process of working with a research team and publishing a few papers. This project contains the implementation of both hardware and software components (Alice is a Mechanical Engineering professor).
On Academic ML and Real-World ML
I enjoyed a good mix of doing research and completing real-world internships.
A lot of the time, researchers work on not necessarily how to take something to production, but rather a proof-of-concept (how it works, how to explain it, how to prove theoretically that it is faster or more optimal, etc.).
In the real world, the data is a lot messier. You have to think about the cost of implementation and the simplicity of your code. These are considerations that you do not think about in the research realm.
During my summer at TubeMogul, I worked on the click-bot detection algorithms for fraud campaigns. It was my first time working with production systems. We spent a lot of time iterating on different algorithms and deploying the final version by the end of the summer. Building the algorithms took about a month while putting them into production took the remaining months of my internship. I learned that most of my time isn’t spent building the algorithms but rather on bringing them into the real world in a scalable manner.
While working in a research environment, I did not think much about these aspects. For example, another research project I worked on at Berkeley was through the Hybrid Systems Lab headed by professor Claire Tomlin. We attempted to build a hybrid framework for multi-vehicle collision avoidance. We spent a lot of time looking at various papers and statistical approaches. We had to prove that vehicles won’t collide.
In research, the papers and proofs are the outputs. In the real world, you have to bring the applications you’ve built into the hands of customers.
On Being a Software Engineer at Uber
After graduating from Berkeley, I debated between going towards a Ph.D. program or joining the industry. I opted for the latter option.
Uber was an incredible place to work right after graduation. I’d recommend this to anybody who just starts their careers. At large organizations like Google or Facebook, most of the systems have already been built, so you would probably work on optimizing them or keeping them running. At mid-stage startups, you would build new systems and launch them from scratch, which is a precious experience.
I joined Uber’s Marketplace org. Initially, I worked on a couple of core ML product teams — applying ML to the products that Uber was deploying. Later on, I joined a new team that was building a platform called MLaaS (later known as Michelangelo). The platform holds all the models that are fetched from a model store and then served/deployed in production. We had finance, EATs, marketplace, and more models on that platform. The system's design is complex because of the different model types: neural networks, gradient boosted trees, regressors, etc.
When storing a model, how can we build a general framework that can handle such a variety and complexity (in terms of hyper-parameters, training data, and architectures)?
Another challenge is to scale the platform to Uber’s level. Every year, Uber onboard more rides, meaning more models are deployed. So we had to think about designing the platform to withhold such a growing scale.
Those are the two main challenges: the upfront system design and the ongoing maintenance work.
On Model Monitoring at Uber
In ML development, the data scientists develop the models while the engineers put them into production. Once the models were put into production, we always noticed issues. Given Uber’s dynamic marketplace of riders and drivers, we got real-time feedback on product metrics. Given such metrics, we attempted to troubleshoot why things worked/did not work manually, like constructing dashboards and reviewing one-off scripts. Data scientists would pull things from production systems to understand what were the issues, when was the last time the models were retrained, did we launch a new model or pipeline that could impact the whole system, etc.
What initially sparked my interest in monitoring is the intuition that there gotta be a better way to deploy models. If we look at software systems, tools like Datadog and PagerDuty enable you to write tests and alerts to monitor the software performance. Nothing like them existed for ML.
ML was still a research discipline, not a well-oiled machine the way software engineering is.
We are still trying to figure out how to build a good ML engineering system to gather insights into the models and know when to improve them.
On Researching Bias in Computer Vision
I applied to Cornell University’s Ph.D. program a while back and got an offer. I wanted to give the Ph.D. experience a shot. After working for a couple of years, I thought that it’d be a good time to transition back into academia.
During my time at Berkeley, the last research group I was a part of focuses extensively on robotics and autonomous vehicles. I thought that the core issue of why there won’t be more intelligent automated devices was technical challenges in computer vision — not only being able to see but also to perceive the world. As humans, we have a lot of contextual clues to understand the scenario. These were still not there for computer vision applications. I thought that was the big frontier we still had to learn about and improve upon. That’s another reason I decided to join the Ph.D. program and worked on the exciting space of computer vision.
I joined Cornell and worked under Serge Belongie, also a Berkeley alumnus. I was at Cornell very briefly. In computer vision, there was a lot more interest in making sure that the models are not discriminatory/biased. Cornell has an excellent AI ethics program with incredible people like Rediet Abebe. They did a good job of asking questions about bias as part of the research process.
We already live in a society that is not equal. Even the type of people who go into a Ph.D. program typically does not look like me.
We have these inherent systemic inequalities. We also have the historical biased data used to build the models. Are we going to end up building models that continue to propagate the system of inequality in the world? That felt like a huge problem to me. We have no idea what the models are doing. And they can make critical decisions about hiring or financial opportunities.
On Creating MonitorML
Before thinking about model bias, we had no way of getting insights or visibility into the model. Forget about fairness. We could not even ask a simple question: Is this model working, or how is the model performing.
The day before the YC application closed, I saw a LinkedIn ad saying: “Hey, apply to this batch of YC.” I thought it would be interesting to apply and see if this idea of better monitoring ML could be a potential product. So I put together an application. At the time, my brother had been listening to what I was doing research on, so he was also on-boarded.
We ended up getting into YC.
I would recommend that experience of going through YC for any first-time founder ever. As someone with a technical background, I would call it a 3-month MBA program with hands-on doing versus reading about what others did. We would learn about well-known tech startups and what they looked like when they were doing YC (what they focused on, what metrics they optimized for, how they built and launched products quickly to get usage and traction, etc.). Having that group of ex-founders (who were constantly advising) and current founders (who were going through the same experience) was incredible.
On Co-founding Arize AI
I met my now co-founder Jason, who also faced a lot of the same pain points: not knowing models are working as expected and not knowing if models in production are going to work the same way as seen in the research environment. Jason also founded TubeMogul (where I interned) and took it to IPO.
As the two of us caught up and discussed the space, we noticed a lot of similarities. So we figured that it would be best to join both of our experiences together. Ultimately, MonitorML was acquired by Arize AI.
It’s been a crazy experience, actually: going through YC, teaming up with Jason, and now having a full team of folks working on ML observability at Arize. But, for me, the main rationale was finding a really incredible co-founder.
On Model Observability
The statement that we made, model observability is the foundational platform of the core ML stack, is my fundamental belief. Maybe two to three years from now, no one will deploy models in production and fly blind, not knowing why issues happen.
The big divide today is that data scientists build models in an offline environment. Then, when we have to operationalize these models, we have to think about all of the things we do offline (feature encoding, feature transformations, feature imputation, etc.) and how to make them work in real-time and at scale. Then, once the model is deployed (either on its own service or via an MLOps platform), how do we ensure it is still working reliably?
We won’t get to the point where we continuously deliver ML as part of the software stack if we do not treat ML as an engineering discipline.
It is not done after you build the model. It is done when the model is in the real world. So how can we build a feedback loop to validate things quickly and retrain the model promptly?
The key components of an ML observability platform are the tools that can analyze model performance in production the same way in an offline way. This entails monitoring and alerting to capture data quality issues, performance issues, distribution changes in the data, etc. You also need tools that troubleshoot and explain when things are not going well (the equivalent of Datadog, Splunk, and New Relic in software).
On The ML Infrastructure Ecosystem
The ML infrastructure space is very exciting right now!
It took us almost 20 years to define the current software stack. Now we are doing it all over again with machine learning. This includes everything ranging from what are the best tools for data preparation all the way down to what we should use for observability. The tools in the last 5 years or so have really focused on preparing data and building models. I think in the next 2 to 3 years, we will see a lot more focus on what’s known as “MLOps” tools, those that help you operationalize models and actually put models in the real world.
Furthermore, in the last few years, many ML platforms have been end-to-end (H2O, Dataiku, DataRobot). There are definitely use cases for them, especially those that gear towards less technical folks. However, they are not the most flexible tools. In the early innings, they help you go from A to Z, so people have gravitated towards them.
But as people become more specialized and want more configurability, the more vertical-focused platforms that focus on specific functions or personas will arise. Examples include Tecton for feature store, Weights and Biases for experiment tracking, and Algorithmia for model serving. These verticalized solutions will become the go-to options for ML teams instead of the big end-to-end platforms in the past.
On Finding Initial Customers
The first thing is to figure out who has the pain points that you are building towards. Of course, as a founder, you might have faced these pain points yourself. But who is going to be the core community that really identifies with that pain points as well?
You want to find the design partners or lighthouse customers. These people feel the pain points so much that they are willing to try the early product, which obviously won’t have everything. You want to find people who feel the pain points so deeply and solve the pain very well, such that they forget about all the things that your product does not have.
So as a founder, how do you find these design partners you can use to ground yourself and build a product that solves their pain points really well? Then you can use those insights to build the moat around your product and nail down the use cases.
A lot of that is hustle: You have to go out and talk to as many prospects as you possibly can. Make sure that you have materials, mocks, or MVPs that they can play with.
On Hiring
A valuable hiring lesson that I learned is that there are amazing people who are really excited about the space. There are also people interested in learning about the space.
I have been reaching out to my network, speaking about the pain points that I experienced, and getting them to understand what the pain points that I tried to solve are. As a founder, your constant job is to convince people about the company’s focus, mission, and the big picture.
Arize is a Cal Berkeley-heavy team. Recruiting from UC Berkeley has been a big success for us.
On Participating in The Amazing Race
While I was at Uber, I ended up being in the Amazing Race Season 32. It was a once-in-a-lifetime experience. My brother and I went to an open casting call and auditioned for it. We ended up getting cast.
During the filming of the show, we couldn’t really prepare too much for it. We were on the ground, went to any clue boxes, got the clues, and performed the challenges. Some preparation was useful — being fit, knowing geography, speaking a couple of languages, etc. But a lot of it was quick decision-making on the road and being able to trust teammates.
There are a lot of commonalities between participating in the race and running a startup. You have to think about your company’s strengths and how you can tackle this problem in a way that no one else can. Again this drills down to being able to hustle and execute, not being afraid of failing.
That mentality of anticipating the crazy adventure ahead is what’s common between a reality TV show and a startup journey.
Show Notes
(01:39) Aparna talked about her Bachelor’s degree in Electrical Engineering and Computer Science at UC Berkeley.
(02:50) Aparna shared her undergraduate research experience at the Energy and Sustainable Technologies lab.
(04:34) Aparna discussed valuable lessons learned from her industry internships at TubeMogul and compared the objective with that of a research environment.
(08:26) Aparna then joined Uber as a software engineer on the Marketplace Forecasting team, where she led the development of Uber’s first model lifecycle management system for running ML model computations at scale to power Uber’s dynamic pricing algorithms.
(12:40) Aparna talked about how she became interested in model monitoring while Uber’s model store.
(17:29) Aparna discussed her decision to join the Ph.D. program in Computer Vision at Cornell University, specifically about bias in model, after spending 3 years at Uber.
(23:40) Aparna shared the backstory behind co-founding MonitorML with her brother Eswar and going through the 2019 summer batch of Y-Combinator.
(26:47) Aparna discussed the acquisition of MonitorML by Arize AI, where she’s currently the Chief Product Officer.
(28:41) Aparna unpacked the key insights in her ongoing ML Observability blog series, which argues that model observability is the foundational platform that empowers teams to continually deliver and improve results from the lab to production.
(33:17) Aparna shared her verdict for the ML tooling ecosystem in the upcoming years from her in-depth exploration of ML infrastructure tools covering data preparation, model building, model validation, and model serving.
(37:01) Aparna briefly shared the challenges encountered to get the first cohort of customers for Arize.
(39:23) Aparna went over valuable lessons to attract the right people who are excited about Arize’s mission.
(41:04) Aparna shared her advice for founders who are in the process of finding the right investors for their companies.
(42:24) Aparna reasoned how participating in The Amazing Race was similar to running a startup.
(44:59) Closing segment.
Aparna’s Contact Info
Arize’s Resources
Mentioned Content
Blog Posts
ML Infrastructure Tools for Data Preparation (May 2020)
ML Infrastructure Tools for Model Building (May 2020)
ML Infrastructure Tools — ML Observability (Feb 2021)
The Model’s Shipped — What Could Possibly Go Wrong? (Feb 2021)
People
Rediet Abebe (Assistant Professor of Computer Science at UC Berkeley and Junior Fellow at the Harvard Society of Fellows)
Timnit Gebru (Founder of Black in AI, Ex-Research Scientist at Google)
Serge Belongie (Professor of Computer Science at Cornell and Aparna’s past Ph.D. advisor)
Solon Barocas (Principal Researcher at Microsoft Research and Adjunct Assistant Professor of Information Science at Cornell)
Manish Raghavan (Ph.D. candidate in the Computer Science department at Cornell)
Kate Crawford (Principal Researcher at Microsoft Research and Co-founder/Director of research at NYU’s AI Now Institute)
Book
“The Hard Thing About The Hard Things” (by Ben Horowitz)
New Updates
Since the podcast was recorded, a lot has happened at Arize AI!
Aparna has continued writing the ML observability series: The Playbook to Monitor Your Model’s Performance in Production (March 2021) and Beyond Monitoring: The Rise of Observability (May 2021).
Arize has been recognized in Forbes’s AI 50 2021: Most Promising AI Companies.
Aparna has also contributed to Forbes various articles: from the Chronicles of AI Ethics and Q&A with Ethics researchers, to a list of Women in AI to watch and emerging ML tooling categories.
About the show
Datacast features long-form, in-depth conversations with practitioners and researchers in the data community to walk through their professional journeys and unpack the lessons learned along the way. I invite guests coming from a wide range of career paths — from scientists and analysts to founders and investors — to analyze the case for using data in the real world and extract their mental models (“the WHY”) behind their pursuits. Hopefully, these conversations can serve as valuable tools for early-stage data professionals as they navigate their own careers in the exciting data universe.
Datacast is produced and edited by James Le. Get in touch with feedback or guest suggestions by emailing khanhle.1013@gmail.com.
Subscribe by searching for Datacast wherever you get podcasts or click one of the links below:
If you’re new, see the podcast homepage for the most recent episodes to listen to, or browse the full guest list.