This week, Tesla’s Director of Artificial Intelligence, Andrej Karpathy, was interviewed on The Robot Brains podcast. We’ve summarized the key parts of the hour long interview where the AI director shares his insights into AI and what it’s like to work with The Technoking.
Prior to joining Tesla, Andre was in education and research with focus on artificial intelligence and deep learning. He lectured at Stanford giving courses in deep learning, which was the first of its kind. He also obtained his Ph.D. there before joining OpenAI as a research scientist reading and writing papers.
JOINING TESLA
So, how did Elon Musk and Andrej Karpathy cross paths?
Andrej says:
“I was definitely getting a little bit restless at that time [at OpenAI]. I felt like these algorithms are extremely powerful and can really move the needle on some very, incredibly important problems in society. I wanted to take a more active role in doing that. I was looking at different opportunities, start-ups and things like that.
And then one thing that kind of happened on the side is because OpenAI, at the time under the umbrella of Elon organizations, we were interacting with people at Tesla and I was kind of consulting a little bit for some of the problems on Autopilot.
Elon reached out and he asked me, “Hey, you've been sort of consulting for the team. Do you actually want to join and lead the computer vision team here and help get this car to drive? And so, he caught me at the correct time. This has been an incredibly impactful opportunity. And I love the company. And of course, I love Elon and everything that he's doing. And so, I would say that, again, was a moment when stars aligned for me.”
WORKING WITH ELON MUSK
As a senior manager who works closely with The Technoking, what is Musk really like as a boss?
Andrej says:
“He's a double edged sword in terms of working with him because he wants the future yesterday. And he will push people and he will inject a lot of energy and he wants it to happen quickly. And you have to be of a certain, I think, attitude to really tolerate that over long periods of time. But he surrounds himself with people who get energy out of that and who also want the future to happen quicker. Those people really thrive at Tesla.”
WHAT IS DEEP LEARNING?
It is a highly complex subject, but Andrej used the example of image recognition to explain it as best he could.
“Images are made up to a computer of a large number of pixels, and each pixel just tells you the brightness of the red, green, and blue channel at that point. And so you have a large array of numbers and you have to go from that to, hey, it's a cat or a dog. Typical conventional software is written by a person, a programmer, writing a series of instructions to go from the input to the output. So, in this case, you want someone to write a program for how you combine these millions of pixel values that define if it’s a cat or a dog.
Turns out no one can write this program. It's a very complicated program because there's a huge amount of variability in what a cat or dog can look like. So, we get deep learning. Deep learning is a different class of programming. We arrange a large data set of possible images and the desired labels should come out from the algorithm. We're measuring the performance of some algorithm. And then roughly what we do is lay out a neural network. It's a bunch of neurons connected to each other with some strengths, and you feed them images and they try to predict what's in them. And so that's where the learning comes in. Data labeling is how Tesla, or I suppose anyone doing image recognition establishes that ground truth, which the algorithms can then be refined on.
The neural network looks at the image, gives you a prediction, and then you measure the error. OK, you said this is a cat, but actually this is a dog. And there's a mathematical procedure for tuning the strings so that the neural network adapts itself to you. Deep learning is basically a different software programming paradigm where we specify what we want and then we use sort of mathematics and algorithms to tune the system. As for how Tesla specifically handles this learning process, Tesla collects that data and a variety of different ways. The first one here being very obvious. If there is an intervention when Autopilot is activated, that can be a great source of data for Tesla. Tesla can isolate incidents and events, pull them from the fleet, label them and then retrain the network on them.
What Tesla wants to do and is trying to do is to automate as much of that stuff as possible. That's where Project Dojo comes in, jokingly referred to as Operation Vacation because if it gets working, they can almost go on vacation and the system would make itself much better over time."
Since the first FSD Beta was rolled out, Tesla has been collecting even more data. As time goes by and as the system gets better, the quality of that data is going to improve as well. So, with all those things aligning, it is easy to see how progress is going to be exponential.
