Full episode transcript (beware of typos!) below:
Nick Jikomes
Alex Wiltschko how are you? I'm doing pretty well, Nick, how's it going? It's been a while. Yeah, it's been a while. So Alex and I, for everyone who's listening, we went to graduate school together. We did a PhD in Neuroscience at about the same time, I think you were one year ahead of me. And Alex had some of the coolest work that I saw come out of the labs at the time. And that's a lot of what we'll talk about. But just to start with, can you just give everyone a rundown of who you are, where you're working, where you're living? What do you do? Sure. So I'm, I'm a researcher at Google on a team called Google Brain. I run the digital olfaction group, we might talk about that. But I'm trying to give computers a sense of smell to work, we're kind of inventing the RGB for odor, to organize odor space, eventually to give computers the ability to turn the chemical world to signals to understand it, interpret it, make it searchable.
Alex Wiltschko 3:43
It was a long road to get there. Along the way, did a PhD. As you said Harvard at Harvard Medical School in neurobiology a study with a guy named Professor Bob data. And, you know, he had a diverse set of interests in the lab, and I thought I was going to work on olfaction. And I did not. I worked on animal behavior. We'll talk I guess a little bit about that. And, you know, in the course of that PhD, and the ended up wanting to understand what animals exactly were were doing, and the story of how I arrived specifically at the way that we did that is, is an interesting one, I think. And that ended up requiring some machine learning skills I did not have when I entered graduate school, I'd done some programming before. But I didn't know the first thing about, you know, random forests or neural networks or any of that fancy stuff that people are talking about these days. So I hit a wall in a pretty big way ended up getting a second advisor or mentor I should say a guy named Professor Ryan Adams. And he kind of initiated me into machine learning over a couple year intense period and I work with a guy named Matt Johnson there too. Brian ended up founding a startup around some totally separate technology called Bayesian optimization. The startup was called wet lab. And we worked on that kind of nights and weekends for, I guess, a year and a half. And then we sold that to Twitter. If you have watched Silicon Valley, there's a episode where Jinyang invents an app to identify whether or not you're looking at a hot dog or not a hot dog, then he sells it to Periscope, which is a subdivision of Twitter at the time. And they use it to identify NSFW or not safe for work content. And you can imagine exactly what that might be. Turns out, that's a true story. And it's actually it's two companies that form that storyline nod bits, which preceded us a Twitter and then wet lab. Because what we did was basically work in the sanitation department of the internet of Twitter. And we, we cleaned it up and identified things that people might not want to see. I really do think that that's actually really how the script was written, because one of the advisors to the show is Deke Costello, who used to be the head of Twitter. So that's just a little fun tidbit there, and then move to Google, after about a year and a half of kind of doing, you know, pretty intense industrial machine learning work on the internet, and did some computery stuff at Google for a bit and then ultimately ended up back doing biology just kind of on a lark, and grew and built the team are working on digital affection over the past two and half years.
Wow. Yes, there's a lot. There's a lot of cool stuff we can talk about with you. I want to definitely circle back to the digital olfaction work you're doing at Google. I assume almost no one listening will know anything about that. I know very little about it. I want to talk. Yeah, I want to talk first about behavior stuff. Sure. So you were doing what we might call computational ethology, you were studying using machine vision, machine learning techniques, and computer science stuff to understand animal behavior. So to sort of paint a picture of the general field of the neuroscience of animal behavior for people, can you just give us a general sense at a high level? What are some of the major questions being pursued in the neuroscience of behavior today? What are neuroscientists thinking about when they study behavior?
So I'll caveat all of it by saying I've been out of that field for, I guess, five or six years, did publish a paper in 2020 on it, but that should have been published five years before. So I wouldn't have considered myself a practitioner, even though it did come out while I was at Google. So the question to me was, and still is, what exactly is behavior? Alright, so like, we've got this, at least for humans, this three pound piece of electric meat in between our ears, and the only reason it's there is to move our body to you know, stuff our faces with food, so we can continue to grow, to find mates and to avoid danger, like the point of the brain is to orchestrate behavior. So if we're going to understand the brain and how it works, we really should understand pretty much the only thing it's there for. And it turns out, we have a primitive understanding, even today of what is behavior, what can we do with our bodies? Is every motion that we make a unique little flower? Never to come again? Or is there some kind of an alphabet of motions of motifs of syllables of behavior that we can emit? By contracting and extending our muscles into we compose all of our behavior from that finite alphabet? I think the jury's still out. Obviously, given the work that I've done, I'm kind of more on the syllable camp, but it's not so clear cut, even even today, even in mice, which are much much simpler organisms than than we are.
Nick Jikomes 8:50
So historically, before, modern times, whenever that whenever that is
Alex Wiltschko 8:55
whatever we define before COVID. Before, before computers, really, before
we had all the tools that we use today, how were scientists studying behavior historically, like how did we actually go about it?
Yeah, there's two, I think, amazing scientists that form the basis of what we would call the field of psychology. And from that birthed the field of neuroscience, and they split and they don't really talk with each other that much anymore. And I felt fortunate to be able to kind of bring those fields in some little piece of dialogue again. Lorenz, who wrote a book called Solomon's ring, which is kind of a personal dialogue of the love of animals and their behavior. And then Tinbergen, who wrote the study of I think animal instincts. Let me just
make study of instinct. Yeah, the study of instinct. I think I haven't, I think I can see it in my own zoom view here. I don't know if
I can pick it up because I have a really, really old copy that's falling apart. So I don't know what the new newer bindings look like. But those are incredible books. And if you care about behavior, if you care about The line between instinct or automatic response and involuntary behavior, I think it's really interesting to understand how deep in common and structured instinctual behavior is in the animal world. So they formed the foundation of that field. And I think they got a Nobel Prize together for that. And I think they're finding was, you know, if I were to summarize it really crudely, it's that there are repetitive behaviors that are unguided by particular stimuli. And you can find it all over the animal kingdom, you can think of it as like displays, like mating displays, where the birds kind of fluff up their feathers. That's one example of it, you know, it's a behavior that's gated by the presence of an external stimulus, you know, usually a female if the male is doing the display, and by internal set of stimuli as well, which is the current hormonal balance or you know, circadian rhythm, location or a seasonal location in the seasonal cycles as well. So both the internal and the external stimuli collaborate to inform the brain, it's time to unlock a basically rapid fire set of stereotyped behaviors in order to achieve some goal. And sometimes those behaviors are scattered and kind of statistical like, um, let's look for food more often than we groom ourselves, right? If you're hungry, and sometimes they're really stereotyped and uninterruptible. So, you know, mating displays are an example of that. Grooming is often like cleaning yourself, that's often uninterruptible mating behaviors are often uninterruptible. And in fact, there's a line of papers from circus trainers that I forget the exact author led me to my Giuliano, you really pointed me towards them. That there are some behaviors that you can train and shape. So you can get, you know, an animal to, you know, sit on, you know, their hind legs for some food, but there's some behaviors, you can never train an animal to do. So you can't train an animal to mate for food. And you also can't interrupt it, you can't shape that behavior, you can't have them stop halfway through for any kind of reward. And so in our brains are a series of programs that if you press start on them, a particular set of behaviors begin and you know, some of them, you can stop some of them branch into a bunch of different possibilities, and some of them are very linear. And so to answer the original question, that's, I think what Lorenz and Tinbergen established before computers or anything, is, by just looking at animals really carefully, for a really long period of time, they began to see this kind of structure and behavior that it wasn't random. It was, in fact, there's a lot of beauty and, and almost determinism to it sometimes.
And so they were really sitting and watching with their own two animals very carefully. And they learned some amazing stuff, as you mentioned, and in some ways, the work that I'm familiar with from you was, was just sort of like a very modern version of what they were doing. It was all about looking very carefully at behavior. So can you start to talk about some of the technology that you developed to look at animal behavior for your PhD?
Or Sure, maybe, I mean, I've got some some visual aids from my time doing this work. Would that be useful? If I five presented some of that here? Yeah, put it up. Let's, let's make sure we can see it well. And so let me hit share screen, you so you need to enable participant screen sharing. Let me know when that's good, and then we'll click to go. Okay, share screen. So here's this I can, I can fullscreen it, I can leave it like this. It's what I like, best. Okay, cool. So this is a mouse in a bucket. It turns out this is actually a really important. This is an important thing that people watch. And the reason why it's important is that as this mouse moves around, sometimes people watch it in large pharma companies, and the mouse has been given a drug that they think could cure a human disease, or make human life better, like alleviate depression, or anxiety, or schizophrenia, or even be a cancer drug. Mice are a bottleneck in drug development. And we need to see if their behavior becomes aberrant, because it's an indication that the drug might cause aberrant neurological effects. And if we want to neurological effect, we need to be able to actually assess that that's occurring. Before we put it in humans, we need to de risk it. This is a really critical step. But we people tend to do it or we're doing it more often when I started my PhD, is they turn the mouse into a dot and they follow where that mouse is. And that's all the mouse is you've collapsed this very rich organization that can you know, whisk and move and run and jump and poop and everything into a dot that's basically got a position and a velocity. Or alternatively, people do what Lorenz Hintonburg ended which is watch the animal very closely right count the number of times it grooms with his right paw or it's left paw count the number of times it rears, all kinds of very granular stuff. That's basically human opinion of what matters to the mouse. So for
Nick Jikomes 15:13
people that have only the audio version will try to explain what we're looking at very carefully. But if you if you are interested in this stuff, check out the video version on YouTube because you actually get to see what we're seeing right now. So Alex literally just showed us a movie of a mouse walking in a bucket. And now we're seeing a view where we're seeing it through through a machine lens, where software has basically just turned the mouse into a dot, and we're just watching the position of this mouse scurry around,
Alex Wiltschko 15:40
and it looks ran, it looks like a little kind of Brownian motion particle just kind of bouncing around the circle. You know, it's it's, you know, there's doesn't look smooth or anything like that. And turns out that if you give a mouse a drug, you can affect how fast this dot moves. And that can actually be a useful indicator whether or not your drug is working or not working or has an undesirable side effects something.
Nick Jikomes 16:02
So if you gave the mouse like an amphetamine, it would probably start moving around faster,
Alex Wiltschko 16:05
it would go around faster. And I'll show you some data that indicates that is indeed the case. So you know what, what I was assigned to do very early in my PhD was watch mice in these kinds of buckets. And we had some specific experiments we were doing, but basically, I was relegated to a dark room eight hours a day, and mice are nocturnal. So we put them on a reverse light dark cycle. So I'd I'd get up, it'd be light for a little bit, I go into a room that was dark, I'd work, you know, handling these mice in a dark room for eight hours, not leaving, it'd be dark. And I felt like we could be doing better. And so really, this this, what I'm about to show you was an invention of necessity, I just thought, Man, I don't need to be in this room. And a.is just not enough information. And so what ended up doing is, is, is going to Best Buy and a bottom Microsoft Kinect, like those things that you can point at you. So you can play Dance Dance Revolution, or whatever. And it tracks your your motion. Turns out inside of that camera, there's a lot of really interesting smarts, there's actually a depth camera that will tell you how far away each pixel is measured in centimeters. It's a depth camera. And so I pointed that at the mouse. And what you're seeing here is the same mouse running through this bucket. But there's a heat map, there's a color map on the mouse where when the mouse's nose goes up, it gets red, and you can see the contour of the spine, you can see that the edges of the mouse which are closer to the ground are in a cooler color. This is a representation of the actual 3d shape of the mouse's body,
Nick Jikomes 17:38
you can tell it's actually a mouse. Now, if you showed this to someone, and you didn't tell this mouse to be like, Oh, it looks like a little mouse running around a little rodent.
Alex Wiltschko 17:44
That's like it's just like psychedelically colored, you know, but turns out that's data, right. And this is something that, for one reason, or the other just hadn't been tried, I think part of it was the Connect was kind of new, I had on the side and hacking on connects with my friends to like build cool art installations and stuff. And so that's kind of where the actual link came to try this out. And, you know, what we did is a little bit more computer vision and machine learning to identify the head of the mouse. And so now what I'm showing is the same mouse running around the color scheme is slightly different, but indicates the same thing. And those a lollipop that's kind of seems like it's glued to the top of the mouse. And the the head of the lollipop is at the head of the mouse. That's just a representation to show that, you know, I've used a computer algorithm to identify where the head of the mouse is and where the tail the mouse is. And then I can for every frame take like a scalpel on the image and cut out the rectangle around where the mouse is. And now I've got like this, this bird's eye view that's always aligned to the mouse's head and its tail, and the spine and along the horizontal axis of the image. And this is a really, really rich source of data for behavior.
Nick Jikomes 18:56
So Xbox Kinect, it's a depth camera, it's looking at a mouse in a bucket, and you can already tell so much more than we could before. So you basically have precise moment to moment knowledge of exactly where the animal is, how fast it's moving, whether or not it's rearing up, or looking down. And is that about it at this point?
Alex Wiltschko 19:16
Yeah, that's about it. I mean, you can think of it as like getting about the same level of like metrics that we put on like football players and basketball players. And we really care about those stats because it indicates their performance and what they're doing and stuff. And so just kind of applied that sabermetrics idea basically to mice to get as much information as we possibly can about what they're doing. And then the question is like, well, what are they doing? So what this is a representation, I'm, I think, I'm gonna see if there's, okay, great, that's what that'll play. So I'm showing the same mouse, but each each. This is kind of a weird representation, but let me walk you through it. So, um, on the left is the mouse and this kind of aligned view where the head is on the right and tails on the left and I'm using In a color heat map to indicate, you know what's higher, and what's lower. Each slice, each vertical slice of this image is one frame one moment. And so instead of two dimensions, I'm just using one dimension to show the mouse's body, I just kind of like flatten the the image basically. And over time, what you can see is that there's these striations, then the mouse transitions from doing one thing to another very abruptly. So one idea of behavior is this is this continuous fluid, kind of mess, you can never tell when one quote, behavior starts one other, quote, behavior ends, but turns out that it's just popping out in the data, that there's these discrete transitions and what an animal does with its body, you can look about, go ahead,
Nick Jikomes 20:44
and, you know, you can tell you have this heat map of the mouse body on the left, and you can tell that the mouse is like walking around, and then all of a sudden, it will stop and liquid salt or something. And then you're showing us this weird looking barcode type image that represents the animal's behavior. And, and for those who can't see it, it's really like, you really see these discrete jumps, where all of a sudden, the behavior is different. It's almost like we're looking at a barcode for the mouse behavior.
Alex Wiltschko 21:12
Yeah, over time. And this, this kind of led us to the question of just like, well, what are in these barcodes? What are in these blocks? Like, is each of these blocks here that you can just see with your plain eye? Is that a unique behavior? And we kind of mentioned this, like, is there just like two behaviors that the animal does? Or are there infinite? Meaning that every time the animal does something, it's totally new? Or is it somewhere in between? So our first step was like, Well, how big are these blocks? How long do these little syllables of behavior actually last, and we looked at it in a bunch of different ways. Turns out, there's about a third of a second. So for mice, which move a lot faster than people, the fundamental timescale of behavior is is faster than one second, which I think is part of the reason why people missed this structure of behavior is because you kind of do need computers, and cameras to see it, it's just too fast to kind of take an even accounting of it. Unless, you know, you've got genius level rhythm or something like that. And then our next question, and I won't get into much into the machine learning stuff. Next question was like, Okay, so there are these syllables of behavior, they seem to last about a third of a second seems to be like a fundamental mode or, or timescale of mouse behavior. How many of these things are they are there, and what do they look like? And so that's kind of where the machine learning part came in. And we built this like, fancy machine learning model. And I won't get into the big acronym of what it means. But the way you can think about it, is it's the same model, not anymore. But in the early days of speech recognition, it's the same model that would take an audio recording of human speech and parse it into its constituent parts into syllables or into words. So we apply that same logic of finding the like, the parts of spoken speech and applied it to the animal's body, as it was kind of speaking to us, at least through a computer vision algorithm. And we were able to find that there's not just two behaviors, and there's not an infinity of behaviors. There's some smaller number, and it depends on how long you watch the animal, the long you watch the animal, the more behaviors you can observe, you can tell really subtle behaviors apart as opposed to clumping them together. And for the average experiment in the neuroscience lab, it's like 6060 syllables, the alphabet of behavior, for the average neuroscientific experiment through this lens has 60 letters or syllables in it.
Gotcha. So in other words, if you've got a mouse in a bucket, you're watching it through a depth camera, and there's approximately 60 Different things that mouse will do. And it will do them discreetly, one at a time for different lengths of time. And that tends to be on the scale of milliseconds, hundreds of milliseconds,
hundreds of milliseconds. So let me give you a concrete example. What's beautiful about this, you can go back to the tape, you can go back to the raw recordings and say what was the mouse of doing. And so here's an example of one syllable boop, I'm just putting a white.on top of the animal, what you're seeing is one of these little kind of blobby mice with a false color heat map representing its shape. Turning right, very briefly, for about a half a second. And when the mouse is in the specific syllable that was identified by this old computer vision system, and putting a white.on the animal so you can see what comes before comes slightly after. And then what I can show you is 40 different mice overlaid there's actually 10 mice and just four instances per mice per mouse. So I can overlay them. So you can see this kind of like synchronized swimming view of the syllable as it's been used multiple times by multiple mice. And if this is in fact, a stereotyped unit of behavior, it should look synchronized. And so when the white dot goes on, all the mice kind of turn to the right In a synchronised way, this is just one of 60 of the syllables and the exact number of areas, but it's about usually about 50, or 60. So these are kind of my, this is like my all time top hit syllables, I love these little behavioral motifs. So here's a kind of a stereotype sway walk that the mouse does when casually crossing the middle of the bucket. Here's a hunting dog pose where the mouse kind of gets up on all fours and then rears its its head a little bit. This is rear in turn, where the mouse gets up on its hind legs, and then puts its nose in the air and then falls to the right, if I remember that correctly. Yeah. And then Run Forrest run this is this is one of my favorites, where the mouse from a kind of almost slow walk just starts really quickly. There's a lot of these. And it turns out that if you count how often the mouse does each of these behaviors, that can form kind of a fingerprint, right? And the same way that if you listen to someone's audio, and you do, there's lots of ways of processing it, but you can do what's called the Fourier transform. And you can figure out what frequencies people tend to talk about and your kind of collapse, it can tend to talk at your kind of collapsing, could be three seconds of speech could be 30 seconds, could be three hours of speech into one small description of is my voice high? Is it low? What are my harmonics? So it turns out, you can do something similar with these syllables, just count how often each of them are used. And you might imagine that, under different circumstances, different syllables might be used differently. So for instance, if I'm playing tennis or running, there's certain motions, I'm going to do with my arm that I'm not going to, I'm not going to make a running motion, when I'm sitting at my desk, taking meetings all day. So you can imagine that the position that you place yourself in the context is going to affect these things. Maybe your genetics will affect these things to how often you use different behaviors, maybe the drugs that you have in your veins, and that are in your brain at any given moment, will also affect these behaviors. So that's kind of the the hypothesis that we were working on is, we did a lot of work that I won't necessarily go into unless you think it's of interest to to your listeners. But you know, the suffice to say like, what we were able to create was this kind of behavioral fingerprint, that summarizes behavior in a really information dense way, in the way that a barcode summarizes, you know, what it's representing, you know, is it when you scan a barcode, you get what exactly the product was, you know, how much it costs, who shipped it, all kinds of things. And then here, this barcode is telling us everything that that mouse has been doing with its body and how frequently.
So you've essentially, you're essentially doing what people like Tinbergen, and the rents are doing on steroids, you're using computers and technology to automatically parse all of the all of the little individual behaviors that mice are doing, when they're just hanging out. Different types of mice do different things based on their genetics, and based on other things. And you can automatically come up with this so called barcode that distinguishes different types of mice based on their behavior. What about, you mentioned earlier? How often, you know, so much of the medicine that we take, as humans is first tested on mice and other animals? So what are the sort of applications for this type of behavioral analysis for things like drug testing.
So this this technology was commercialized in a company called syllable life sciences. And that was later acquired by by a separate company. And they're actually actively putting it to use to try to develop new drugs. And the the idea of what came before is that so we actually talked about this a little bit. So usually, when you give a mouse a drug, you measure something about the mouse as a dot. Right? So how fast is it going? And so on the screen, I'm showing kind of distribution. Let me skip to the next slide here. I might measure how fast the mouse was running on average, or what the distribution of of its speed was. And so you know, here in this red curve is if you know me, giving a group of 10 mice, each dose of methamphetamine and mice turn out to run faster when they've been given a dose of methamphetamine injection. Versus when I give them fluoxetine, which is not as you know, it's kind of the it's not it's not a stimulant. It's so here's an obvious difference, like the only thing that I did to these mice which are otherwise genetically identical, born at the same time, raised in the same way, look identical. The only difference is I, you know, put one drug in one set of mice and put a different drug in another set of mice, and you can tell the difference in that drug You can read out the drug almost by looking at properties of its behavior. This is really crude, though. So what we did is we did a quite a large scale experiment, at least at the time, where we took about five actually, I think end up being up around 600 mice. And for each mouse, we gave it a drug at a particular dose. And we basically calculated this behavioral barcode, as you say, or behavioral fingerprint. And we said, Hey, this looks like nonsense to my eyes. But maybe if we feed this barcode into a computer, we can actually build a machine learning model that just by looking at behavior can tell what drug that mouse was gift. That's the goal. Can we just look at the mouse's body and figure out what's going on inside its brain. And so, you know, the way that we evaluate that is using here, we can kind of go into the details. This is called a confusion matrix. And I'll kind of go over the rows and the columns because it's, it's a really good way to evaluate how well a statistical model or a predictive model is working. So remember, the context is we've we've turned mice into numbers. And we hope that those numbers represent what drug that mouse was given. We're trying to read out the state of the mouse's brain just through its behavior. And so as I mentioned, there's a bunch of different ways to turn mice into numbers. The first is just measure like, hey, where is it in the arena. So what I'm showing on the screen here is called a confusion matrix. And on the columns, and on the rows, there's labels antidepressant, anti psychotic benzodiazepine control, SNRI, SSRI stimulant, those are different classes of drugs, you know, broad classes that can come to as many specific drugs. And the columns have the same labels. And the way to read it is, you know, if you look at one rose is antidepressant, and you have a hot square, at the column that also says antidepressant. What that means is, yeah, I gave the mouse an antidepressant, and my machine learning algorithm, just looking at the numbers that represent that massive behavior was able to tell Yeah, I actually gave it an antidepressant. So on this representation, the only drug class that seems to be working a stimulant, I gave a mouse a stimulant, and my representation of behavior using just position where the mouse is, I can only read out stimulant, I can't read out the rest.
And I imagine So for something like this, you know, if you just took a naive person who had no training in psychiatry or anything, but knew a little something about stimulants and other drugs, they could probably watch people who had been given different drugs until like, okay, maybe that's the guy in the stimulant, because he's
jittery moving around too fast, you know, engaged in stereotypy, with just like repeated kind of short behaviors. Yeah,
I imagine that the computer vision tools that you're using allow for much more subtle parsing of things that we can actually see with our eyes. Do you have examples
of that? Yeah. So if we use our kind of behavioral fingerprint using this algorithm, we get a confusion matrix that tells us that for every drug class that we have in our data set, we can read it out, meaning if you show me 20 minutes of mouse behavior, and I parse it with my system, I can tell you what type of drug that mouse's on. And this is kind of a, this was an astonishing finding to us, because like, the only thing we have access to is the outside of the animals is moving around. We're not doing any chemical assays if it's brain, but yet, we actually can tell what's going on inside of its brain, at least to a limited degree. And this is just for the drug class, I'm not telling you about the drug identity, it turns out that you can make the same confusion matrix, but instead of, you know, stimulant or SNRI, you can actually try to read out the specific identity of the drug that the mouse was given, you can do that with this technique as well. Meaning if you have some set of 15, or 16 drugs like we did in this dataset, and you give a mouse that drug and you don't tell me what drug that is, I can just watch it using my system and tell you exactly what drug that is. And beyond that, you can actually do it down to the dose. So if you give a mouse a drug at a high or a low dose, we can read that out too. And that indicates this is a what we've built is a really powerful representation or summary of behavior that allows us not just to understand how behavior is structured, which is a really kind of interesting, basic question in ethology. But it's useful. We can partially you can portion behavior into its pieces, some of them back up again, figure out, you know, over 20 minutes, what was the mouse doing on average, and then use that to actually say, and this is how the mouse's behavior has been altered. And we've used this methodology in a couple different regimes to understand mouse models of autism, mouse models of OCD. Whether or not a given drug for autism actually reverts, reverts the specific disease phenotype, the specific things that are changed in that disease or not. And this kind of this tool, I think, is still living on in, in the data lab and elsewhere and also an industry and people are actively using it to kind of find new treatments and understand disease more deeply.
You have the video of the Fox urine experiments. Oh, yeah. Wow,
that's a blast from the past. I actually think this one's that. I think that this, this is, I spent a lot of time with this box that this is kind of where it all started. You might actually have been in the room when I first presented this data at the HMS pin seminar. I think I was Yeah, so this is, it's gonna be a video of 10 mice, and I'm just showing the contour of their body. So they just look like little squiggly ovals. And behind them is a heat map of where they're usually spending their time and I built this box over many years. It does something simple, it seems stupid. But it's actually important for understanding how odor changes behavior. What it does is it's got four compartments for equal sized squares that are put together to make a bigger square. And in each of those squares, I can control the odor that the mouth smells in that square, using vacuums and tubes and all kinds of stuff. And what I'm doing is I'm pumping in the smell of Fox, but so it's TMT trimethyl Vaseline, and it's a metabolite that foxes can't help but make as a result of being meat eaters, if I understand correctly, and because all predators and prey are constantly engaged in basically chemical warfare, you know, the predator is trying not to give themselves away, so it was the prey and they're trying to figure out okay, what's the advanced notice I can get? What sniff can I get that tells me there's a fox nearby. TMT is one of those weapons in the mouse's arsenal, where the fox can't help but make it and so the mouse has learned to exploit it to be afraid of it in order to get away from foxes. And so in this video, you can see a bunch of nice fruit moving around.
Nick Jikomes 37:07
Certainly, you said the mouse's learn that but this is an eight, right?
Alex Wiltschko 37:11
It's an eight I should say it learned it over evolutionary time. Yeah. Okay. And it turns out that these mice that we use had been inbred for about a century. And so they're a little bit less afraid of this molecule than they might have originally been. But nonetheless, they still have over 100 years of having never seen a fox in their brain somehow encoded in their genome is a relationship between this molecule smell and desire to run away. So as I play this video, you can see by moving around, there's no Fox smell. And then at a certain point, I will introduce Fox smell in the I think it's the bottom bottom, right? Yeah. And you can see mice aren't spending any time there, they're going to the edge of this little quadrant, and then darting back, some few bold ones are actually traversing it, but most are kind of entering and then leaving. So this is kind of where everything, all this whole line of research started was like CI, I had like a better way to describe fear, basically, rather than just where's the mouse relative to this outer source. And we kind of took it overboard a little bit and building the methodology to to explain that.
So if we come out of the screenshare now, I want to use this as an opportunity to talk a little bit about all faction in the brain a little bit to give people a sense for some of the stuff there because it's really cool. So you were in a lab studying all faction. Can you talk a little bit about why all faction is? I don't want to say special but how is it different from things like vision in terms of why it's interesting are still mysterious in many ways.
It's you know what, more of X paradoxes? No, I don't. So I think it kind of encapsulates why all factions been a difficult nut to crack. So Hans more avec was kind of a philosopher in the 20th century. And he he raised this paradox, which is basically to ask the question, why are some things that are so hard for us so easy for computers? And why are some things that are so easy for us like walking across the room blindfolded and then opening a door, which no robot can do today? Why is it so hard for us to program computers to do those things? And the explanation or rationale that he gives is that the things that are hard for us have come on the scene very, very recently on an evolutionary timescale. And so they're a struggle because we're not yet evolved to do them well, and the things that are super easy. Evolution has been cracking on for hundreds of millions of years, like moving or looking or hearing. Those are trivial for us to tell, you know, a forest fire from the smell of a fruit easy. Turns out that there's no electronic Gnosis that can really do that. that reliably. And the reason I think is that evolution has spent a long, long time solving that problem on behalf of all terrestrial species. And so for us to waltz in and think it's so easy for us to do day to day, and then believe it's going to be easy for us to replicate on a computer. That's the paradox, it's not easy. And so smell is evolutionarily much older than any of our senses. And you can see that inside of the brain, it's, it's got, basically, it's got its own VIP entrance into different parts of the brain into the hippocampus, which is responsible for memory, and into other areas of the brain that are responsible for like fear and things like that. So the sense of smell goes directly from the nose to an area of the brain called the olfactory bulb, which you can think of is kind of like the retina, I guess, of smell. And then from there, it gets to go right to memory, and skip, I'll skip all the other things that are that are occurring inside of the brain. So for instance, every other sense has to go through a small piece of tissue in the brain called the thalamus. And so there's a waystation. So the experience of touching my own skin of my own face actually takes longer to reach certain parts of my brain than sniffing, there's just fewer synapses between the direct sensory experience and its availability to the rest of the brain. And if you just actually go look at the part of the cortex that's responsible for smell, it looks evolutionarily very old, it looks closer to like what happens in lizards than it does in the rest of mammals. There's fewer layers and Piriform cortex, for instance.
So humans, you know, at least when we think about like, dogs, you know, people always talk about how good dog smell is. But in some sense, you can, you can imagine that. It's really more that humans are really bad at smell. Is that true? Have we lost? It's a
myth. I think it's a myth. And there's been some research, I think, really puts the nail in the coffin. There's some great research from a guy named Noam Sobel. And what he has shown is, not only do we have a pretty good, you know, acuity and detecting molecules, I mean, the smells just as an example, the smell of natural gas, which you've probably smelled natural gas has no odor. But we by law, add in a type of molecule called mercaptan. We can smell that in parts per trillion. That's the equivalent of like, one eyedropper droplet in an Olympic squat in an Olympic sized swimming pool. I mean, amazing sensitivity, right between our eyes, this This postage stamp sized piece of tissue, called the olfactory epithelium, which is more sensitive than nearly every laboratory piece of equipment that you can get a hold of. I mean, my cell phone camera is a pretty good camera, and approaches the resolution of my eye. I don't know exactly what the resolution of my eye is. But I have no problem looking at a picture that I took on the iPhone and zooming in and still having more resolution than I can handle on a big screen. But we don't have anything like that for smell. Do humans
have any innate, like hardwired smell associations, the same way that the mouse has the fox urine Association built in? Great question,
I think it's an open question. So this notion of whether or not there are like hormones, or a certain that hormones, pheromones, pheromones? That's right. The question of whether or not there's pheromones for people is kind of an open question. I think that there's not based on the evidence that I've I've seen that there's kind of like a strict academic definition of pheromones, which is that if you smell it, you have to do something, you're kind of like, your brain has received a signal and you now must do something. And I don't think that humans have that. But as to whether or not like, you know, the smell of parmesan cheese is good or bad is definitely cultural. The smell of whether or not kimchi is good or bad is definitely cultural and also can change over your lifetime.
Yeah, and I How, how much I mean, it's probably doesn't have a precise answer. But my instinct My instinct is, my opinion is that a lot of our smell associations are probably learned associations, even the ones that we naively assume are built in. Like, I think if you ask the average person, you know, do roses smell good? They would say, yeah, that's that's maybe built in that's natural, or does sewage smell bad? Well, of course, it's just built in. But do you have a sense for? Do you think most of our smell representations are actually modulated by our experience? And how? How plastic are those? How easily can they be changed?
Yeah, I mean, as you say, there's no precise answer, but you know, when you're very, very young, what smells good and bad, isn't quite formed yet. And so whether or not poop or pee You know, smells bad isn't formed yet, you'll notice that by the behavior of very young children that are kind of insensitive to things that we would think they shouldn't, that you shouldn't put in their mouth, right. And you begin to build these associations over time, either through, you know, reinforcement from adults in your, your home or in your your social group, or through actual experiences, like you ate something that smelled a particular way, and then you got real sick. That's also a very quick way to build odor associations. But then there's something that I'd become interested in that's a little subtler, which is, smells don't occur on their lonesome in the world, right, you'd have to build a very constructed environment in order to miss out on some of the patterns that exist. And, you know, for instance, when you smell a rose, you often smell the grass that it's been planted in, or you smell the dirt that has been planted in as well. And so we're always forming these associations that are kind of linking together in the web, that tell us like, Oh, if I smell this, I'll probably smell this other thing. And along with that, we're building associations to experiences and expectations. Like, if I smell the products of the May yard reaction, there's probably baking bread nearby. There's all these chemical reactions that occur that require certain precursors that are important to us. Right, so the main yard reaction takes sugars and amino acids and forms these nice smelling compounds that are, you know, responsible for baking bread smell for, you know, a searing steak, I mean, really, really nice stuff. That smell itself is not nutritious, but it's really, really indicative that there's nutrition, delicious nutrition nearby. And so we're we are these Association machines in a way, and our sense of smell has learned in ways I don't understand. To build these associations that I smell this, it must mean, something's nearby must mean something useful, something harmful, something, you know, interesting. We kind of construct associations on top of the basic sensory percepts.
One of the things that comes to mind here is, I know that a lot of people in, in my world in the world of cannabis, for example, have trouble doing certain forms, certain sensory studies. So can the cannabis plant produces a lot of volatile compounds, just like all flowering plants do that smell really interesting, a lot of them smell really good, a lot of some of them might not smell so good. But different combinations of these molecules produce different sensory perceptions and people. And there's there's a lot of interest in sort of mapping out what the logic of that is, like, which combinations produce produce, which smells and a lot of people who just try and do this, naively, I think, run into a wall right away, which is you give people a bunch of cannabis strains, or you give them formulations with precise chemical contents that are known. And you find that the results you get when you ask people like to describe it are sort of all over the place. And it turns out that, you know, even if people smell the exact same thing, and even if we assume their percepts are the same, they talk about it differently. They don't have the same vocabulary for discussing it. And so I'm wondering if you could maybe comment on this. And it could be a good segue into your work at Google. If it's so difficult than humans to sort of map the perception to the stimulus. Because people talk about it in such different ways, because of all these other associations they have. How do you think about this problem in the context of some of the the digital factory work that you're doing?
When I was in kindergarten, I had a Crayola crayon box. And there was sex 64 crayons in it, and each of the crayons had a name. And I learned, you know, over the years to associate certain colors with certain names and my vocabulary, I probably can name I don't know, 200 colors. I'm making that up, but not 1000 Probably can't name 1000 I can name more than 10. I've learned word sensory associations for color because I was trained implicitly by that Crayola crayon box and buy you know, Pantone colors and RGB and all that stuff. Same with sound. You know, I've I saw a violin being played. I heard it. Now I can identify that as a violin. Can I tell it from a viola? No, because I'm not a trained musician. But I can get in the ballpark of stringed instrument. We have no such opportunities in our society today with olfaction. So we don't train to do word odor associations in any really meaningful way. From a young age and you can learn it as an adult. It's pretty hard. I've got some odor training kits next to me. Because my my team at Google, we've kind of we've gone to a perfume training crash course in France and we've kind of been continuously trained at a couple different ventures to understand exactly what it is that we're working on. You can learn it, but you have to practice it and we practice color word associations everyday. Could you pass me the salt? Oh, what is it? What's the blue thing? Yeah. Okay, great. Got it. Thanks. But we don't say Okay, could you pass me that? Could you pass me that? Oh, which one are the one that smells like Lily of the Valley? So what? Like, like mew gay like the lily of the valley that smell? What are you talking about? Yeah, dryer sheet smell the dryer sheet. Okay, got it. Yeah, yeah. But as
you said, it is learnable My understanding is when people do these, you know, perfume training courses or other sensory studies, the way that you do them to actually get results that are meaningful is you first train people to to just use certain words, descriptors. And you say this, this base molecule is the smell. This is what we're referring to as this. You start with a template.
Yeah. 100%. And then you got to train yourself on that template. And so that's part of the work that I've been doing in my group in collaboration with some some academics as well is, can we train people to be reliable at delivering labels of voters? The answer is, yeah, and we'll be publishing that research later this year, along with some interesting validation of that, I'm going to actually find smells nobody's ever smelled before. These molecules nobody's ever smelled before, kind of an interesting needle in the haystack experiment where we used a machine learning model to, to go find interesting molecules. And you know, have more details about that a bit later this year.
Interesting. Can you speak at all about, you know, so the idea is to give computers machines a sense of smell, just like they already have a sense of sight, right? Like we're using right now. Like, we've got cameras, that's, that's the machine version of our eyeball. What does the hardware look like for a machine to detect molecules floating around in the air.
So look up, I'm actually probably sure you've got an ear nose above you. It's a smoke detector. And that's one simple version of an electronic nose, it's only got two notes to it, it can detect usually carbon monoxide and carbon dioxide, some, some can do more. Inside of a camera, there's a couple different technologies that people have historically used to turn photons light into digital signals, voltage changes, or current changes. And so it's not quite the books not quite closed on how to actually turn photons into into digital signals. For, for older, it's even more open. So DARPA, in I think, the late 90s, funded a program to build electronic noses to detect minds, because they were still landmines in former areas of conflict, and kids would step on those minds and they die. And so the, the initial motive of building an electronic nose was to clear minefields because rats can actually smell minds. And so can train dogs. But gee, we'd really like something that wasn't living doing that work. And so that spurred a whole set of innovation. And then it went away, because the funding kind of stopped briefly. And then there was another DARPA initiative to build electronic noses. And so those two, basically competitions or funding spurts, willed into existence a bunch of different ways of turning the chemical world into digital signals. And I think today, there's about a half dozen feasible ways of doing this. They're wildly different in how they work. And in order to kind of describe them, we'd have to get really nitty gritty. But suffice to say, it's way, way, way earlier than cameras are, definitely earlier than microphones are. Because we don't really know how smell works in the first place. We need an odor theory in order to even interpret the signals that come off of these these devices. And then the devices themselves, not case close there. And so what what my group focuses on is what we think is the first step, the first prerequisite, which is, what's the RGB for odor? And is there just three numbers that describe an odor? Or is there 60, or 600? We don't know. We think we've got a first draft of this RGB for odor. And we've published a white paper on it and described a little bit of it. Now we're putting it to use. But we think that odor is not as simple as vision in that there's more than three channels. We think there's 100 or more. And we think that you need this odor space in order to effectively build an electronic notice, because you've got signals coming off of some piece of hardware. What do they even mean? Like, what are they telling you? You need to convert them into an understandable format, like RGB in order to tell what it is that you're smelling? So that's, that's the problem that we're working on right now.
Nick Jikomes 54:48
As much as you can mention, you know, I love that you brought up the smoke detector example because it's this everyday object. I didn't even think about it before you mentioned it. But that is like a really simple, you know, hardware knows where What, you know, once once someone builds a better machine knows, what are some of the major areas of application that we think are out there that would be immediately useful to humans.
Alex Wiltschko 55:12
There's some prosaic ones, which is, you know, detecting manufacturing defects. So many of the COVID vaccines that are being manufactured now are being produced by other living things like little yeasts, or, you know, they're otherwise they're a biological production process, because the antibodies or the, the mRNA signals are there pieces of living things and pieces of life. In the process of making them experienced manufacturers, at least with yeast fermentation, which makes a lot of biological drugs today, they can smell if they're off. And so there's the ability to kind of make sure that we're able to produce the medicines that we need in an effective way by monitoring the manufacturing process. Same with cars, people that are experienced mechanics can smell if something's off, like as your oil burning, or is there something else being burnt? If he had a nose, you could continuously monitor that car as it was driving?
So a lot of quality control applications. And you know, now that I know that I think about it, it's almost obvious because we use our nose a lot of what we use our actual nose for is quality control.
Yeah, like in the fridge, like, Oh, crap, I think I left this for too long. And then you open the Tupperware container, like definitely for too long, and you throw it away, because not because that smell is going to hurt you the smell is harmless, but because the smell is being produced by something that could poison you, namely a bacteria or fungus that's, that's eating the food and will secrete toxins inside of you if you eat it. It and that I think is the other biggest uses is health related. So the the tradition of smelling and tasting things from patients in medicine is extremely old. So the the the name for the technical name for diabetes is diabetes mellitus mellitus means I think honey tasting is the original way to diagnose diabetes is your pee tastes like honey, and your breath tastes smells sweet. And so this tradition of using a really sensitive sensor between our eyes, our nose, to tell something about what's going on inside of a person's body is a very, very old tradition. And we've largely, largely ignored it in modern medicine. I mean, we've got really great diagnostic tools that use vision, and use hearing, or ultrasound. As an extension,
Nick Jikomes 57:31
this could be a really simple, non invasive way to actually do diagnostics.
Alex Wiltschko 57:36
And we know that it will work. If we can build it, it will work. Because there's a whole slew of papers out there each with their own little flaws. But together I think about two very convincing and persuasive evidence that COVID-19 is smellable. There's a recent article out actually showing that bees can be trained to smell COVID-19. Dogs can smell COVID-19 dogs can smell, Parkinson's disease. So Can people actually there's a story of a woman I believe from Ireland, I'm not going to get it right. It's either Ireland or the UK. And she was able to smell Parkinson's disease. And some researchers put it to the test and they said, Hey, here's T shirts from 10 men that either have or don't have Parkinson's disease. So this
Nick Jikomes 58:22
was just some woman walking around claiming that she could detect Parkinson's with her nose.
Alex Wiltschko 58:26
She claimed it because she could smell it on her husband. And then I believe we should check the story. But I believe her husband then got Parkinson's, and so she could smell it. So she did this test. And she was given something like 10 T shirts. And she said Parkinson's, no Parkinson's for all 10 of them. And the researchers graded her. And turns out she got about 90% of it right nine out of 10. And she said, I don't think I got just nine out of 10. Right. I think I got all of them. Right? They said no, actually, this person doesn't have Parkinson's. You said that they do. Oh, wow. They followed up. That guy got Parkinson's.
Nick Jikomes 59:04
Oh, wow. So she was able to detect it before the doctors could? Yeah. So they
Alex Wiltschko 59:09
there was actually follow up studies, I've got to go get these papers, because I was actually talking about this earlier today. And actually tracked on what it was that you smell. Turns out there's this waxy substance your skin secretes called sebum and it's secreted in different parts of your body, but it's secreted on your back as well. And inside of the sebum is a molecule that seems to be more prevalent and people that have or are developing Parkinson's disease and they believe based on that study, that that's what she was actually detecting. That's just one molecule of kind of a bouquet. And so it might be that you need all of these different molecules of different ratios in order to diagnose a disease. But I think that that's the promise is catching diseases early. I mean, I have a personal motivation my my father passed away from brain cancer. When I was 25, and we didn't catch it until he was having seizures, and then he got an MRI, and there was a tumor in his motor cortex the size of a golf ball. And based on those growth rates, we maybe had months, perhaps a year or more of time when we could have been intervening to slow the cancer given more time, or to treat it outright. But in the first place, it's difficult to treat that specific kind of cancer, because it's difficult to catch it early. And you get in this catch 22 of like, well, how can you like for Alzheimer's? How can you treat a disease, when you don't know the habit until they're too far gone? And so I think that's also part of the promise of building better diagnostics, using smells. We know that people smell differently in different disease states, you know, our mouths and our skin are kind of like the exhaust pipes for a car. You know, what's coming out of it doesn't tell us everything what's happening in the engine. But it certainly tells us something.
Yeah. Wow. I mean, that's yeah, that's fascinating. I want to back up. And so you've had this very interesting trajectory, I think. So we've sort of touched on a couple different pieces of it. You're working at Google now. You did a PhD at Harvard, he did this really cool. Machine Vision work? How? When did you first get into computer programming? How old? Were you?
Three, two, I don't know very young. Yeah. I mean, I've always always around computers from a very young age. And everybody growing up always asked me like, Oh, are you going to go, you know, work on computers, you're going to be computer scientist or computer programmer. And I was like, what it's like asking a fish about water. And so I had interests that weren't necessarily just computers. But I, you know, I always was playing with computers, programming computers making things. And so I never thought I'd make a career out of it. And in a way, I still kind of am not like, I'm a biologist that wears a computer scientist costume by day, I guess. And I should, I should mention, you know, this intersection of the life sciences and of computer science. It's like, I've just lived at that slice for, I guess, 20 years now. And I don't really want to live at any other intersection, because there's still so much promise and possibility. And if I may be so bold, that I'd recommend if you care about the intersection to listen to a podcast I do, called theory and practice, which I am the host of that with a fella named Anthony Philip Pacus, who's the Chief Data Officer at the Brode. And this is something that we put out with GV. So he's also a venture partner at alphabets venture capital wink. And so we kind of explore this intersection with really interesting people. So just talk with David ALTSCHULER. Today, actually, that episodes coming out this Thursday, so that kind of the founder of modern genetics and says tickle genetics and stuff. And we've talked with Professor Sir Rory Collins, who started the UK Biobank and I mean, just amazing, amazing people. Um, kind of living in their in their shadow hoping to eventually do something that's kind of worthy of their, of their achievements as well.
So did you just sort of organically start learning and fiddling around with stuff on your own? Or did you have any like, did you take computer courses when you were a kid,
I took one day of C++ at the University of Michigan. And I was just so bored, I walked out, and I just dropped the class. But I just been doing it for a long time just making stuff like you know, some people like to woodwork. And so they get a really good kinesthetic sense with their hands of how to build things, or they like electronics or something, I just gravitated towards computers and have been, you know, working with them and programming on them for a long, long time. In between undergraduate and graduate school, I kind of was part of the iPhone app, gold rush. And so I lived in basically a closet in San Francisco, and made iPhone apps for audio stuff. So I made like an oscilloscope and a Fourier transform app. And I've just always been nerding out making stuff on the computer.
Do you ever like I never, I didn't learn computer programming until way later in my life. And you know, computers, technology or just eating everything. They're so important for just everything that we do in society now. Do you ever think about like, sort of a random question, but like, do you do you think about early childhood education and whether or not something like computer programming should be something that we actually teach in schools, the same way that we teach reading and writing?
I completely think that we should, and it's not so much that we're teaching computer programming is we're teaching a way of thinking that you know, the computer is stupid. It doesn't do anything that you don't tell it to do. Right? It is it is unintelligent, in the definition of the word like it can't infer things. It can't read between the lines. If you say add two plus two, it's never going to infer that you're, you know, trying to count up an inventory, and then figure out, okay, I need to add these other numbers to it as well. It only does what it's told. And sometimes people write sophisticated computer programs that look like they're doing interesting things like playing the game of Go, or Starcraft. But ultimately, those computers are being told what to do in one way or another. And I think, you know, adopting that mindset of like, okay, I want to achieve a task, how can I break it down into its constituent pieces, and specify them so clearly, that they can be done forever and ever and ever, by computer over and over and over again. And I think that way of thinking is gonna become really important. Going forward, I mean, it already is. And I think we'll see a generation of people, I hope that kind of take it for granted, I want them to take it for granted. I want them to, you know, call me an old fart and say, like, I don't want to hear your, your days before computers, you know, could recognize images or smells old man. Like, I want them to assume those things exist and, and do amazing things on top of that.
So you mentioned that, you know, our machines are stupid. A lot of people use the term, the term artificial intelligence is so widespread now that, you know, two people using it are often referring to very different things. So on the one hand, we have these artificially intelligent quote, unquote, computer systems that are able to do things like play the game go, and things that are very sophisticated. And yet, as you've mentioned, they clearly aren't able to do what our own human brains and minds can do. Do you have any thoughts on what some of the essential differences are there? What is it that allows our brains to do some of the things that even our best computer systems can't do yet?
Well, if I could answer the question I had, be looking into that very deeply, and making interesting algorithms. But I think what you highlight is important, the first thing you mentioned, which is artificial intelligence is a term that people don't use precisely because it's not precise term, it just generally means let's try to make computers smarter. It's a field of scientific study, a subfield of AI is machine learning, which does have a precise meaning. So you hear AI and machine learning used interchangeably. Sometimes that's okay. But machine learning is probably the term that you want to use. And that refers to the field of study or practice. We're trying to get computers to solve a task by showing them examples of how the task was solved. And that's like a subtype called supervised learning. But that's, that's most of the applications you'll see today. And so ml, I think, is it's it's a fixture in I think industry, it like really works. And there's this question then of like, Well, are we approaching human intelligence, as you say, like the program that can play Go, can't drive a car, it can now it can play chess, because it's been trained on both go and chess. But it can't. It can't read a sonnet and tell you why it's interesting or beautiful, or what it's referencing. You have to build a separate system for that, and you probably could. And as best as for why our brains are so much more capable. I don't know. I mean, one thing that I'll mention is, you know, we've built these computer programs under radically different constraints, and our brains have evolved, right? So our brains have to be powered by things we put in our mouth is wild, right? Like, it's an incredibly powerful processing machine that can survive under incredibly diverse circumstances, much more than a computer can and can make more of itself. I mean, really incredible constraints, this things under computers are not computers can be powered, you know, by, you know, way more than just food. And so they can they have more kind of raw processing power available to them. And also, you know, if you think of intelligence as an object of study, I don't really know how to define intelligence. But imagine you're studying it simply, you're studying aerodynamics. If you understood aerodynamics, and you built something to be aerodynamic, would you arrive at a bird? Or would you arrive at a plane? We've arrived at planes, which look very different, but still exploit the same fundamental properties of aerodynamics that birds do. Planes just don't hop from tree to tree, because they don't need to, they hop from airport to airport, and there's a whole support staff that loads in and unloads them and learns to fly them and everything. It's a very, very different ecosystem built around our understanding and instantiation of aerodynamics. So I, I think, whether or not we find the answer in our lifetimes is an open question to me. of how our brains work and why they're different than than the kind of simple systems that we've designed so far. But I don't necessarily think that we should expect that our implementation of intelligence should look or behave like ours does.
What do you think about the fact that, you know, so much of our cognition, so you've mentioned earlier in our discussion on behave of behavior that for humans, for mice, no matter what the animal is, no matter how simple or sophisticated it is, the brain is ultimately in some sense there to just move the body around so that the animal survives and ultimately reproduces, and a lot of learning to be coarse grained about it just has to do with sensory motor integration and sensory motor associations. And our computers don't have bodies for the most part. They're sort of static, they don't have to move a part of themselves around the way we do. Do you think that that could be an important piece of, you know, what, what an intelligence can do is whether or not it has a body?
That's a great point. I mean, Bruno Al Sharpton, who's kind of one of my scientific heroes, I think he's a luminary and, and theoretical neuroscience, and also in machine learning. He gave this talk and must have been eight years ago, I forget it was in he was in Toronto, it was a conference called CIFAR. And he basically raised this question, which is like, our robots are really dumb. Like, he showed an example of a little bug called the Jumping Spider. And this Jumping Spider actually has eyes on a swivel, it's got eight eyes like, spiders do, but it's got to, they can actually like look left or right, and it's got a weird retina that's like a patch that's up and down. It can plan, it can look at a 3d maze of little interlocking bars, figure out where the food is, and it can look around, sit, and then make one jump or two jumps to actually get to some food goal. It can do 3d planning. And it has a whole body that it needs to move around and feed. And in that tiny little body, it's able to perform computations that, you know, whole data centers are currently unable to do. Because our computers don't have bodies that they need to solve these problems with, for the most part, I mean, there's robots, of course, and robots and algorithms collaborate to produce, you know, an entity that can move around. The other observation, where I think our this is my personal opinion, where our robotics is, is askew, from how life is solving its problems is and this is an observation from David Cox, formerly of of Harvard neuroscience, I think now at IBM, he made this point to me that's stuck with me, which is, our bodies are dramatically over sensed, and under actuated. Meaning for every little bit of ability that we have to move our bodies, there's many, many times more abilities to sense what's going on around us, where we have way more sensors than we then we strictly need. If you're thinking in terms of like control theory, and you know, engineering, we have way, way, way more sensors than we would need to actually move around. There's something fundamental about that, that I don't think we understand. And I think if we're going to, you know, again, my opinion, if we're going to build intelligent, embodied systems are systems with bodies that can move around to do stuff, I think we need to examine the the miracle that is our sensory abilities. I mean, like, look at your hand, it's waterproof. It consents, hot and cold at a pixel resolution of like less than a millimeter or something like that. It can sense sharp touch vibrational touch, soft touch, it can tell if I'm blowing on it, because it's a combination of change in temperature, and also slight change in pressure. I've got hair, you know, some people don't, I've got a lot of it, that can detect small deflections as well. It's, it's covering something that's moving, it's relatively, I mean, aren't just a dime of skin is a miracle that we cannot produce in any laboratory in the world. And I think that's a humbling fact, but a motivating one as well that what our bodies do, the kind of abilities that we acquire at birth, far outstrip, far outstrip anything that we can engineer. But I just, I mean, it's not even. It's not even close. We are miraculous. And I don't think we should forget that.
Yeah, I've never really thought about it quite that way. But there are, I mean, if you were to reduce this down to bits, you'd be talking about an incredible amount of information that our bodies detect every moment through every sensory channel. And that's really interesting to think about. Do you think I mean, a lot of people have I hear people make statements like, you know, our machine learning algorithms aren't actually smart because we need to give them millions or billions of examples. But when you think about the way you just described, it's almost like our sensory apparatuses are just constantly giving us this massive training data set at all times. So maybe it's actually the other direction. Maybe we just have a more large corpus of inputs.
Yeah, it's interesting. There's a couple ideas in there. And the first complaint is like, they can't be intelligent, because they require all these, they require all these like, you know, labeled data points. There's this notion of, I forget, there's some term for it. But there's this receding wave of intelligence, which is like, Oh, we build a system that can beat a grandmaster in chess, we thought that would be intelligent. But when you open up the hood, it doesn't really look that intelligent. It's a bunch of rules about what to do and how to brute force search the future. Okay, well, I guess I guess chess wasn't an intelligent activity, then you go solve go, which is dramatically more complex, at least in terms of board positions. And these allocate that must have been something intelligent, and you look under the hood, and you can kind of mostly understand how that system works. And so you think, oh, okay, must not be intelligent, maybe go isn't the activity that requires true Intel. And it's always this receding wave. Of course, chess requires intelligence. Of course, go requires intelligence. But you can engineer systems that don't look like what you thought they look like they can solve the problem. So I think that that's, that's, that's an take note, but we are not data efficient. So when we feel something, we're not getting labels for it, right? When we when I touch this, the top of my, you know, water bottle or touching my keyboard, I'm not getting the label water bottle or keyboard. From nothing. Like I have memories. Of course, when I see a horse, you know, and I say horsey, when I'm like a little kid, if my parents say, good job, I've got one label one. And but the thing is, kids can then look at camel and say horsey. And their parents say no, that's a camel, not a horsey. And then they learn they never make the mistake again. There's not really that capability existing to that extent in machine learning today. So it is a valid criticism, and it's an area of active exploration, which is how can we be more efficient with the data that we learn so efficient as to kind of match our own human learning abilities? And we're not there yet.
I want to shift gears a little bit. So you've always been dealing in technology, you've always been interested in computers, you went through the academic route? As did I, and then we both went into the private sector.
And we took off the collar we took off. Well, I want to talk about this. Most days, I get a priest collar, not the constraining code. But yeah, it's the academy that has certain rules and Right, right, leave, it's difficult to come back and all that.
Yeah. And I, you know, people ask me about this all the time, especially people that are still in academia that are thinking about this. And as far as I can tell from the analytics, there's probably a solid number of like, people in the PhD student postdoc world that listen to the podcast. How do you feel about your trajectory in terms of like, do you ever miss the research world? I'm sure you do. In some to some extent, would you ever go back? Or do you feel like you enjoy where you ended up? More? Why, if you had stayed in,
I didn't know how to say it at the time, because leaving was actually really difficult. It felt like I was leaving. I mean, I had an identity built around being, you know, doing a PhD and maybe being a professor. That was a part of me how I saw myself, it was very difficult to leave that behind. I'd be curious to hear what what you felt actually, maybe just how was your experience? Yeah. Partly because you, you went right, from academia into a startup, which is the deep end?
Yes. Yeah. Um, well, yeah, I'll tell that story. But I mean, I'm hearing everything I'm about to say, I can hear echoes of what you just started to say. Which is, it was an identity, it was really an identity crisis. So it was actually really, it was really hard for me to leave. And it probably seemed sudden, even though it wasn't quite as sudden as it seemed, to other people. But I guess the key thing to understand is, when I was in college, I was always interested in science. So and I got right into a lab, a good lab early on, and like, pretty much from the day I walked in that lab, I was like, This is it. Like I am a scientist, I know exactly what my tracking life is. So I have this purpose and identity. Like I sort of knew what my life was gonna be. Yeah. And I mean, that is when people say they're looking for meaning in life. They just want that intrinsic sense of having direction. And so like, I had it really early on, like age 18. And I went all the way through five years of working in the lab in college, and then went right into, you know, really good graduate school programs. And I was like, this is I didn't even think about it. I was like, this is this is my I track. And in retrospect, I didn't sort of realize it, as it was happening fully. But in retrospect, what happened was, I would say, I don't know, around halfway through my PhD, I was, I was feeling that I maybe didn't want to do this anymore, but I didn't actually recognize that. So I had, like, you know, just these sort of weird feelings of anxiety or, like really satisfied with what I'm doing them, but I didn't identify them as I don't want to be doing this as a career. It's also
difficult because in the middle of the PhD, that's a perfectly normal thing to feel because yeah, you're on your own, the whole point of the PhD is to discover something nobody ever knew before. And that doesn't like, make you want to like shake your pants, at least a little bit. Like, you need to, like check your attitude like that should be humbling and anxiety inducing, at least in part.
Yeah. And so like, in some sense, you should feel that knowing that, you know, even if you're going to get past it, especially if you stay on that track. And so I was sort of grappling with that, without really articulating it to myself, because Because of this, you know, it was my identity. And I think just people don't want to, they don't want to have a sense of their identity dissolving, which is more or less, I think, what was happening. So it was, it was tough for me to leave in that sense. And I definitely miss aspects of it. Like I, I'm definitely an sort of academic person, I love just reading and talking about interesting stuff, which is sort of the fundamental motivation for going through that track. But you know, now that I've come out of it, and I can sort of see it from a different perspective, I wouldn't choose to go back into it. That's, I guess, the bottom line, I'm not that, you know, not that working in the private sector is better in every single way. It's not there's trade offs, and it's just different. But I would say, I'm glad I went through it. I'm also glad I left academia, I think a lot of it has to do with I think to be successful scientists in the academic world, one of the toughest things about it, is you have to become good, come good at things you are absolutely not interested. Doing. Like you have to become a very good bureaucrat. Right? If you have if you want to be a successful scientist, and if you love science enough that you can also do that. That is just the nature of the beast, with our current institutions, you know, structured as they are. But I think I eventually just came to realize like, No, I don't want to. I don't want to do that. I just don't want to do that.
Yeah, I mean, I think your experience mirrors mine in a lot of different ways. What? Excuse me, what, um, but I basically run a lab, I didn't think that I would get to this point, I really thought I was leaving all of that behind. And I would never think about raising money, I would never think about writing grants or writing papers, I just want to build cool stuff that lots of people use. And I did that for a while. And it was awesome. And then I couldn't get away. And I needed to work on the question that I was working on, which is like, why the heck do things smell the way that they do? And can we put this inside of the computer? I've been obsessed about this for like 15 years, and I finally got the chance to work on it. And now all of a sudden, I basically run a lab. I mean, that's not my title, but kind of effectively a PIs. I work with amazing people, which is such a privilege. I mean, just like really brilliant, staggeringly brilliant people, which is great. And I raise money. I mean, the way it works at Google is I don't write an r1 to the NIH. But you know, you don't just get free money at a company, you have to, you know, make a good case for it and ask permission and things like that. And it's taking the form of a proposal in some form or the other. And so yeah, I ended up doing all postings just inside of the company, as opposed to academia. And but I came to it on my terms. And I came to it after a five year hiatus. And I don't do I don't ask and try to answer any other questions and the ones that I'm interested in, which isn't just I know how privileged I am. It's just awesome. And I'm happy for it every day. And as for whether I would go back? Maybe, maybe, I mean, you see this trend and industry researchers where sometimes they'll go to a different company where they do industry research. So I work with people that have been through many different companies like DAC, which you probably hadn't heard of, which birthed a lot of amazing researchers that are, you know, at Microsoft today, or Facebook or Google. And so there's a modern industrial research system, you know, and there wasn't in this strength about 20 years ago, but there was in the era of Bell Labs and Xerox PARC so it comes in comes and goes and phase and so maybe, or maybe I'll go back into startups and do that at Um,
Nick Jikomes 1:25:00
I will say like one of the things I should mention, is part of the reason I think I'm like comfortable outside of academia is even though I'm outside of it, I nonetheless figured out a way as part of my actual job to do research. Yes, I'm still doing the recessionary
Alex Wiltschko 1:25:18
curious. That's the point, right, there's just more than one way to be professionally curious. Part of the travesty of academia is they don't, they're beginning to wise up, they have to tell you that because there's so few jobs at the top. So there's many, many more graduate students in their postdoc positions, many, many more postdoc positions, in an extreme way than there are faculty jobs. Yeah. And it's just not responsible as a training program to not at least tell people about alternate careers, quote, alternate, right, as if that's the other way. And, you know, training programs like, you know, where we were at Harvard, towards the end of my time, there, they were beginning to be a little bit more expressive about that, which is a good thing. The travesty, I think you mentioned about becoming a bureaucrat, I have a slightly different view on it, which is, so I'm effectively I do a lot of bureaucratic work. I mean, I've got to go to meetings, and, you know, be on committees and does, you know, hiring decisions and stuff like that, I do a lot of paperwork, basically. Because I care about my problem, and I care about my team, and I want them to succeed. And sometimes that's what you have to do. The difference is, is that I received training at Twitter and Google for how to do that, how to manage how to lean. And as a young professor, I see this in some of my peers, my friends that are, you know, my age slightly older, slightly younger, they are thrust into a position of basically being the CEO of a little startup, where they've got to figure out how to keep the lights on, they've got to manage, they've got to balance their profits and losses, I mean, you have spending money, and there's only so much money you can spend before you're out of it, and then you got to go raise more. And they were taught how to do none of it. Yeah, there's no ramp up. None. And, you know, my, my partner was in the, in the military. And so she went to West Point Academy, the US Military Academy for her undergraduate. And then she did some other training after that, and is now kind of in the business world. But at West Point, they were, their actual training was how to be a leader, and how to lead people and manage efforts in order to achieve a common shared goal. And I didn't even realize that was possible to be trained how to lead at a young age, I thought that was something you kind of fell into and stumbled around, and eventually, you just sucked less at it, after you did a bunch of it. But no, it's a discipline, it's a trade. And in academia, they don't treat it as such, they treat it as something you learned by accident. And the thing that you need to be really good at is if you're a molecular biologist, pipetting. Or if you're a computer scientist writing code, but that ultimately, it's your job to train the next generation, and to lead those people into doing amazing science and becoming the best version of versions of themselves. And I think it's a bit of a travesty in academia, that that's left up to chance that you just some people figure it out. Some people don't. I don't think that's right.
Yeah. You know, it's, it's, it's an interesting, like, set of set of things to think about, like, on the one hand, you know, I stand by most of the things I said, but I should add the caveat that like, you know, when I said, I don't like the bureaucracy, part of it, one of the things, you know, in my 20, and 22 year old and 24 year old brain that was naive was, you know, I thought, I think I thought of myself as the scientist. And part of that was never having to do that other, quote, unquote, others. But you know, eventually I learned that no, like, no matter where you go, you do have to do that other stuff.
It's like, I live in an apartment, but I don't do dishes. Laundry, like, it's all kinds of stuff you got to do, you don't necessarily want to do at any given moment in time for the regular maintenance and furthering of your actual goals. But I definitely thought that too. Yeah, which is why I did laundry. So it's a great
student. I do think one of the fundamental differences between us we think about the set of all interesting things you might do as an academic, and you and you think about the set of all interesting things you might do in the private sector, and there are a lot way more than I realized before, like sort of, like got out. The supply demand mismatch for on the labor side, you know, that you touched on is a big difference, like there simply aren't enough spots for all the postdocs that are talented and doing good work. And that's pretty much not true in the private sector.
Yeah, I totally agree. And I, you know, I think we saw a lot of our classmates end up in what you could generically describe as data science. And that's because at least in our little line of work in neuroscience, we had to analyze I have some data. And turns out that analyzing some data and kind of slaving away at that is super valuable. And we learned kind of, by doing, and in. So like, Delta valuable skill set, and you know, our classmates as, as a cohort ended up actually, you know, being pretty valuable in the workforce, and they should be valued. We should all be valued for the work that we do. And what's nice about the market is that there's a price associated with it, and you get you get paid in dollars in stock, as opposed to, you know, glory.
Yeah, yeah, that's a good way of putting it as opposed to glory. I do want to ask you, before we run out of time, to tell the story. So you've got this interesting technical background, you've got the biology background, you're at Google now. But before I even got to Google, you, you mentioned briefly at the beginning, that some of the work you were doing with some of your computer science colleagues in graduate school, turned into a startup that Twitter acquired, can you can you talk talk us through that story and how that transpired?
Sure. So it was, um, it was initially, a really close friend of mine, Jasper snuke, and his advisor, Ryan Adams, who ended up becoming a mentor of mine. And they invented some really, really cool technology called Bayesian optimization. And they didn't invent it, they refined it. And they figured out that they could tune all kinds of things. That sounds really simple and trivial, but it's being pretty deep. So if you imagine what's an example, she might have an example from your industry, where you need to make a blend of something. And you got to get the knobs just right on the blend in order to really hit the nail on whatever problem you're solving. And, you know, it could be like, the temperatures got to be just right, like, if you're making coffee, for instance, the grind sizes got to be just right, the temperature has got to be just right, the roast time has got to be just right, all these things. And there's a minimum and a maximum for each of those, we call them parameters. And if they're set randomly, you know, you're going to get a bad cup of coffee, maybe undrinkable. But if you said it just right, you'll get something sublime. And so turns out, what they invented was a system to automatically tune these things with feedback, using a minimal number of rounds of experimentation in a way that was like pretty close to optimal, they find the best cup of coffee, as long as what you had to do is do exactly what the algorithm said, which is try these settings, you had to sip the coffee and say this is good on a scale from one to 10, it would take that rating and then give you an ex proposal. And turns out there's all kinds of problems that are like that, an enormous number of problems that are like that. And they commercialize that piece of software. In fact, there's a there's a company that exists that's kind of continuing that tradition right now for industrial applications. So you can imagine like biological fermentation for like beer, for instance, like there's a lot of parameters, like the amount of hops that you add the amount of malt, the timing of it, the temperature, all that stuff that needs to be set perfectly, in order for you to get a really great, you know, class of beer. So we didn't necessarily know what we were going to tune, we just thought we put together a company and it was originally for machine learning Professor types. And myself, I wouldn't have called myself a machine learner at that point. And they kind of brought me in because I had built all these apps before and had programming experience kind of just doing it. And so I helped them build the website, basically, and some of the backend stuff. And what we got really good at is tuning other machine learning algorithms. So it's kind of a meta thing where it's a it was a machine learning system that tuned other machine learning systems. And turned out that's a really big roadblock in the way of industrial application of machine learning. If you come up with some new idea gets a new dataset, you need to tune that algorithm in order to work well at all. And then the time consuming,
Nick Jikomes 1:33:57
anyone can just run some out of the box package but doing it with all the knobs putting, you know, all the best positions is the tough thing.
Alex Wiltschko 1:34:06
Yeah. And the difference of something that's not tuned versus well tuned, could be hundreds of millions of dollars, both in what you're making in terms of your revenue, whatever your revenue model is your company, or in cost savings. If you're tuning your data centers to save money, like that's the thing that we did, and it worked out really, really well. We saved them a lot of money. But what we originally did was we tuned the models that were being used to protect the safety of Twitter users. And on like my first day at Twitter, we were told what it was that we were doing, and it was basically identifying obscene images. And it's not what I expected to get into. But we said okay, this is the big problem when Twitter needs to be safe. It needs to be, you know, scrubbed of things that people don't want to see. And it's still an ongoing struggle on the Twitter platform I should say. And we got right to it. and learned a lot about all sides of humanity in the process. And the the acquiring company that brought us in, was the first there were the first kind of deep learners at Twitter, the first people applying neural networks in a concerted way at Twitter. And so with our powers combined, we basically built a hot dog, no hot dog app for Twitter.
How did you go about? So you know, there's a lot of graduate students and a lot of talented people out there that are building amazing things, but they have no concept of how to commercialize or think about a strategy for doing something with the thing that they built. How did you guys actually go about getting acquired?
We got really lucky. That's the short of it, the slightly longer answer is, we really tried to understand our customer, we didn't ever build a revenue model, we kind of failed in that way. But we really understood our customer, which was other people building machine learning models, they were us, right, we used our system, actually to tune our system. I mean, we ate our own dog food. And it turned out that there was enough of a need. It's in some small pockets of industry for this particular technology that we've got inbound interest and the way that we got inbound interest is, and this was my small contribution to the company. And this wasn't the only way that we got inbound interest. But I made integrations between our system and all the other machine learning systems I could find. So I made it as easy as possible to use our system that ended up getting the notice of somebody that was working at Twitter, they said, oh, there's a company attached to this, oh, there's some machine learning talent me excluded at that time. And then they ended up kind of taking a closer look, you know, doing a bake off between our system and what they had been doing internally, and we blew him out of the water.
And you made it easy for people to find find you guys and use the technology. Yeah,
yeah. And just got lucky. I mean, we didn't. We understood ourselves. And it turns out, there were other people that were like us that needed this, and that had resources to acquire US. But other than that, I think, you know, in, in subsequent startups that I've founded and advised, I mean, that the main thing is just, who's What are you selling? Who's gonna buy it? And for how much? And you don't have to have an answer to that right away. But like, you should definitely be thinking about that. Because if, if nobody's buying it, you don't have a company. And that that was, that was kind of like the wake up call was surprise for me. And then a lot of my colleagues from biology, so if I make something cool, people will want it right? No, 100% no, except in very extenuating circumstances. You need to solve a problem for somebody who has the money to want to have the problem go away. Like I'm hungry, I'm willing to spend $2 on a candy bar, make the problem go away, right need to find a problem, and the people are willing to trade money money for.
So before we end, I'll just ask an open ended question. Any area what what are the areas of like AI or machine learning applications not necessarily related to stuff that you're working on? Or behavior, that you're really optimistic and excited about? That we might see a lot of innovation happen around in the next couple years?
Great question. I mean, I'm working on what I'm working on, because I believe it's a frontier. So I think, you know, giving computers new senses, particularly the sense of smell is huge. It's a huge untapped area. The other area that's exploding is the ability to generate natural language text, you're seeing this in GPT. Three, and this is a system that Google that does something equivalent. Computers used to be able to generate a couple of words at a time, that could fool us. And now I think they can generate whole paragraphs, not multiple paragraphs, so they kind of lose the thread and the plot. Yeah. And so I think that's an area of just immense opportunity is computers interacting with us in a meaningful way using just plain language?
Yeah, I think I'm also fascinated by that area. In fact, the last episode I did was with a guy named Terrence Deacon, who's a professor at Berkeley. And he has some fascinating ideas in his work that you discussed around symbolic representation and the evolution and development of language. And he had some interesting commentary on why or why not machines today can't actually use language the way that we might want them to. So finish in that area. Check it out. That was a fascinating conversation. But yeah, I cannot wait to see what you know, what machines are doing in two or three years linguistically?
I think the frontier that very few people are working on but I think more people should work on in that vein is understanding what is a story? Understanding what is a character and their relationships to each other and a story and the one way of phrasing the problem like, Well, why can't we generate two paragraphs that are coherent as opposed to just one? Because there's no plot. And there's also no data. There's very, very little data on like the story of the structure, the structure of the story. And that's an area of kind of personal fascination of mine. I don't work on it, I hope to someday, but, you know, even just telling, telling, like a fairy tale, something simple, but that actually is as engaging and as fantastical and maybe perhaps laden with meaning and in the lesson, we're not able to do that. And understanding what is the kind of skeletal underlying structure of a of a narrative and a story, I think, is a fascinating frontier as well.
Nick Jikomes 1:40:46
Awesome. Well, Alex, thanks for your time. It's great to talk to you again.
Alex Wiltschko 1:40:49
Good to talk to you too, man.
Nick Jikomes 1:40:50
Yeah, and I hope we talk again soon.
Alex Wiltschko 1:40:53
Thanks for having me. Yeah, be well