Check it out on Spotify, Apple or Youtube.

Transcript

Below is a lightly edited transcript, courtesy of AI:

Ariel (00:05) Joining me today is Dr. Shaanan Cohney, a computer scientist, legal scholar, and senior lecturer at the University of Melbourne. He’s won awards in research, teaching, and hacking, and has advised both the FTC and the US Senate on tech policy. Shaanan also has experience cracking safes, singing competitively, and playing piano. Shaanan, welcome to the Zappable Podcast.

Shaanan Thanks, it’s lovely to be here this morning.

Ariel So Shaanan, since you have experience in both music and computer science, is there anything in common that draws you to both of them?

Shaanan Yeah, and I think some of the connections actually run pretty deep. Everyone talks about how music is very mathematical, but not often enough do people explicate what they mean by that.

There are a couple of popular music theoreticians on YouTube who go into this a bit more. One I came across a couple weeks ago charted out a beautiful graph of what music naturally is going to sound good to human ears. And one of the things there is that depending on the environment in which you’re raised, different music sounds good. If you’re raised in Bali, gong music might sound really great to you — that operates on a different tuning system. If you’re raised in the West, classical Western music has a certain sound to it. These are all built on mathematical relationships, and depending on the country you grow up in, or the musical culture, this is going to be different.

But it turns out there are these super interesting mathematical relationships between the music that’s likely to sound good and the music that’s not. And you can do these music-plus-code visualizations that are really awesome. There was a 1993 paper by William Sethares — I’m not sure how to pronounce his name — around the relationship between psychoacoustics, which is the study of how humans perceive music, and the mathematical nature of music. For me, that ties very nicely into an analytical way of viewing the world.

At the same time, there’s the flip side of music, which is what most people think about: the more emotional, cultural, social experience of it. And I think code has something similar to that as well, which we often lose sight of — software development is also a social practice, or at least it has been up until recently.

Everything from the early days of Unix hackers working together over bulletin boards, to the notion of pair programming — there’s a sense in which coding is a craft and an art. There’s Donald Knuth’s famous book, The Art of Computer Programming. So there are interesting parallels between the way programmers approach the craft of writing code and the way musicians think about the craft of writing music.

Now that’s breaking down on both sides a little bit with AI, but those links are something I appreciate. Even when I’m actually doing music, you try to get into a flow state, and just like with code, when you’re in the flow state you’re not necessarily thinking about all of those aspects at the same time. So for me, they’re two sides of a similar coin.

Ariel Interesting. And since you mentioned how AI is affecting things — do you think AI-generated music is valid music? What are your thoughts about that?

Shaanan This is where it becomes really important to separate out music as a product from music as something more social, more artistic, more craftworthy. There was a beautiful video essay by Adam Neely, a popular online music theoretician, called “Suno, AI Music, and the Bad Future.” He brought in a number of sources, one of which is one of my favorite ways to think about music: musicking, an idea from Christopher Small, an ethnomusicologist. That’s the idea that music is more than an output — which is how we’ve been treating it for the last century with recorded music. Music is a social practice. That’s everything from listening to music, where you have to be actively engaged, relating it to your own culture, often listening with other people — there’s a lot of social context there — and also music creation and music performance.

Up until the last century or two, a lot of people’s experience of music was either going to the big concert hall when someone came into town, or much more commonly, making music in the community. Whether that’s liturgical music in your church or synagogue or mosque, or just banging on things while singing, or old slave songs — all of these were cultural practices.

What’s happened, particularly in the last 50 or 60 years, is that link has been increasingly broken, and people see music much more as the product rather than as the social and cultural experience. Some of this AI-generated music takes that to the far extreme. Is it music in the sense of music-the-product? Sure. But I think something important is missing when you take away the cultural and social experience and the element of craft. There’s also a lot more I could say about the process of creation, if that’s interesting to you and your audience.

Ariel Right. I guess if people want something to share with their friends, or to go to a concert together, they want something more unique — they want to know that people struggled to make this song. But if they’re just listening to music in the background while they’re running, then probably they don’t care as much, and AI-generated music could easily meet that need.

Shaanan It could. A lot of AI-generated music, at least to my taste, is just bad — and I mean that from a somewhat subjective but critical lens. If you look at the construction of AI music, there are often things missing. The lyrics can be very repetitive and not have actual depth while having the semblance of depth. That’s not because in theory AI could never generate meaningful lyrics, but from an objective standpoint, today the lyrics are rather trite and the rhyming structures are fairly simple.

One of the reasons people really like Eminem is not because the harmonic construction of the music is that amazing, but because he does an amazing job with rhyme and lyric. Even in just that one small aspect of AI music, it’s really clear if you pay attention for more than a few seconds that it’s not there. So I find it annoying to listen to.

But that is in some sense a matter of aesthetic preference. Perhaps because I’ve trained in music, I have more of a gut reaction. Do I think some people could obtain some utility from it? Sure. But at least for now, something is missing — both on the philosophical, social side, and on the question of what the actual product in front of us is.

Ariel On the technical level, I feel like they’ll get there soon enough, or maybe people just need to work on their prompting a little bit, because they’ve gotten so good at all these kinds of areas. Maybe we’ll see in a few months.

Shaanan Maybe. There was AI-generated Bach, which was pre-generative AI — purpose-built models designed to mimic the harmonic structures of Johann Sebastian Bach’s music, in particular his fugues, to the point where experts couldn’t really tell whether the composition was Bach or not.

But there still seems to be something missing, at least with the generative tools we’re creating today. How soon we get there, I don’t know. Will I be able to tell at some point? I don’t know.

Ariel (07:47) So turning to another topic: you have extensive experience with software security, but also a little bit on the physical side, even breaking into locks. Is there anything in common between those two areas, or are they just separate areas of interest for you?

Shaanan This goes back to a fairly deep connection between physical security and computer security. I wouldn’t even say I do software security, because to me that’s looking at far too narrow a part of the system. The way a really good red team, or a really good security professional, looks at things is as an entire system.

Locks are a really nice example of this, because you can look at just a very specific element — the theoretical construction of the lock and the matching to a specific key. There’s a certain key space depending on how many pins there are; you can figure out how many different possible keys there are, and that’s all well and good. But to actually build this thing, to instantiate it in the real world, there are a lot of considerations you need to take into account. And this is where lock pickers think about locks in a similar way to the way computer security people think about computer systems.

When you make a real lock, there are machining tolerances, and those come into play when you’re lock picking. You’re relying on the fact that not only are certain combinations of pins ineffective in a real-world lock, but the quality of the machining of the lock itself leaves imperfections you can exploit as a lock picker.

Now, a really good physical security person also doesn’t stop at the lock. The lock is within a door, the door has a latch mechanism. Perhaps you don’t actually have to pick the lock — you can bypass it by toying with the latch. Or if you’re a common criminal, you go through the window and ignore the lock in the first place. These same parallels carry over quite closely to computer security. We talk a lot about attacker-defender asymmetry: the defender needs to ensure the entire system is secure, whereas the attacker only needs to find the weakest point. And that weakest point, in our lock analogy, might be the pins, it might be the pinning, it might be the jamb of the door, it might be the window that’s been left open. So that parallel is a really good way to teach people how to apply systems thinking.

There are also other things we can learn from locks and lockpicking. Firstly, there’s a cultural aspect from the ‘70s and ‘80s around the way hackers thought about experimentation and gaining knowledge. The same tools hackers applied to locks reflect the exact same philosophy they applied to early computer systems.

There was great work by one of my mentors, Matt Blaze, on master keying — the notion that in a large corporate building you can have a single key that unlocks every door, called the master key. Then you can also have subkeys that unlock a subset of doors, or maybe even only a single door. In Matt Blaze’s paper, he demonstrated a physical privilege escalation attack. Privilege escalation is when you have some normal level of access and can use it to gain administrator or privileged access. He showed that even with a normal subkey on a master-keyed system, you can escalate that to get the actual master key on certain kinds of master-keyed systems.

He copped a lot of flack for this, because the locksmiths’ guilds were not happy that this information was now public, and he received threats of all sorts, including death threats. But this parallels what people in computer security say: you shouldn’t rely on the secrecy of the design of a system in order to protect the system itself. This is Kerckhoffs’s principle. So this was a real physical analog of a principle that applies very directly in digital systems.

You’ve got all of these different individual lessons you can learn. You learn about the culture of hacking, you learn about particular types of attacks and ways of thinking about things, you learn about the principles. So I think it’s a skill every computer security person should try their hand at, at least for a few minutes.

Ariel Have you ever used your lockpicking skills in practice in the real world?

Shaanan Within the past two weeks, yes — but no comment. This gets me into trouble quite a lot. I’m very good at losing keys, forgetting keys, locking myself out of buildings, and so on. So having skills in physical security has come in handy more than once.

The most prominent of these was when I was a graduate student and I locked myself out of the dorm. At that point it wasn’t a lockpicking attack, but similar physical security — I came up with a way of getting back into the building by bypassing the security systems on the dorms, and that resulted in more than a little trouble for me, shall we say. But it has proven a useful practical skill lots of times as well.

Ariel So usually you’re breaking into areas that you should have access to, you’re just missing your key.

Shaanan Exactly. That’s how I stay on the right side of the law. I break into things, but they’re things I should have access to and for some reason don’t.

Ariel One interesting difference between physical security and software security: with software security, once you’re on the internet, if there’s a vulnerability, something will find it. You’re open to the whole world. Whereas with physical security, there aren’t so many people trying to physically break into buildings, so once you reach a certain level, it’s usually good enough.

Shaanan I think that’s true to a certain extent of software security, but this tells us a bit about the role locks actually play in life. Picking a lock is not actually that hard. In particular, the locks on most people’s houses — when you imagine the American white picket fence house with the front door — those locks are typically not very secure. It would likely take me under 30 seconds to bypass or pick any given house lock.

But the point isn’t to stop people from getting in with some high security mechanism. It’s a social deterrent. It’s just like the way you might leave your bedroom door closed when you have friends over to an apartment. It’s not that that provides any great deal of security, or that if they looked through the door they’d see anything so terrible — depending on how messy your bedroom is. It’s more a social sign about where the boundary is being set. If someone wants to break into your house, they can, most likely by tailgating someone through the security door, kicking the door down, or bashing in a window. It’s more that it adds a little bit of friction and a little bit of social signaling.

This does happen in software security as well — not quite as much with the signaling, but to the extent that adding a bit of friction makes you a slightly less soft target, so you’re no longer a target of opportunity. A lot of crime — not all of it, there certainly are more sophisticated attacks — happens in moments of opportunity, where someone sees that a door has been left open, or a car has its lights on and has been left outside with the key still in the ignition. So the function of both your firewall and your basic security hygiene, and the lock on your home, is not necessarily to produce a technical impediment that a nation state couldn’t bypass. It’s more to provide that little bit of friction and that little bit of social signal.

Ariel (15:23) Now turning to your experience teaching computer science. How have you noticed it’s changed recently, as generative AI has become so common and powerful?

Shaanan This is the doom and gloom part. Most of the people teaching introductory classes — and I count myself among them — are in a state of mild depression and slight panic, except the people who are close enough to retiring that they don’t have to worry.

What we’ve seen is that the software tasks we would traditionally give in an intro programming class — go home, work on this assignment for two weeks, skill up a little, learn basic programming paradigms and abstraction — are no longer actually completed by our students. Or to the extent that they are, it’s a student prompting an agent once and then copying and pasting the output into our submission system. One of my TAs estimated that somewhere between 70 and 90% of the submissions he graded had clear hallmarks of AI use.

To give you a sense of how he picked up on that: I teach an introduction to C programming — data structures and algorithms through the lens of C. In a typical year, we’d teach some of the more interesting C keywords, stuff like static and volatile. I might mention those, but I’d almost never see a student use them in an assignment, because that level of sophistication usually comes with experience — knowing about variable lifetimes in enough detail to think about how to manage them. Typically students would have messy variable names and imperfect structure. And yet assignment after assignment used sophisticated constructs and paradigms in C that I’d never seen previously in my career. That’s what my TA picked up on too, and it flags to us that the students are no longer completing the work.

So there’s frustration on that side, because as much as we communicate that the only way to develop these skills is by doing the work, students for whatever reason don’t do it. And they all have legitimate reasons: life is stressful, this is more effective, am I ever going to actually need to program? But I do still think we’re missing something by not teaching them how to think like a computer scientist.

Then there’s the other view, which is: well, maybe thinking like a computer scientist needs to change. Maybe you don’t need to think about code itself, because I think all of us are slowly coming around to the reality — some faster than others — that humans probably won’t be needed to write very much code in the future. I don’t write very much code now, and previously that was something I really enjoyed as an artistic or craftful endeavor: writing the perfect little function that did an algorithm in much more efficient time. Now I do what everyone else does and prompt the machines.

So this is causing a moment of self-reflection and inner turmoil around what our role is as educators, and what we should be teaching. At the introductory level, things have really changed. For higher-level courses, once students have attained a baseline of skills and you’re doing things in a more sophisticated manner, or teaching pure theory, I think there’s still a little more room for hope — and I’m happy to chat more about that if it’s interesting.

Ariel Yeah, it’s an interesting question — what do you think the goal is now? Because not only can AI do small-level coding, it’s starting to be able to do complete projects. We still need a human currently prompting it, but we don’t need a person as involved in the technical details. So what should the goal be? What is the student trying to learn? What are we trying to accomplish?

Shaanan So I teach at two very different places. This fall I’m going to be teaching at Tufts, which you can think of as a liberal arts college with a research focus — it graduates doctoral students, but it’s small classes, seminar style. That’s one side. On the opposite side, I teach at the University of Melbourne, which is a large state school, and my smallest class is 400 students. The kinds of things you might think about teaching, and the way you teach, have to shift a lot.

For me, one of the things I want to communicate in my lectures is the beauty of the topic, to encourage self-motivation and self-exploration. Because coding and computer science are in many ways a craft. It’s a skill, it’s something you have to spend your own time on or you’re not going to get better at it. If you want to work producing slop applications for SMEs, AI can do that. But as an educator, that’s not my goal. What I want to do is cultivate individual growth and judgment, and try to motivate students to understand why they might pursue their own learning in the area. Because I only have them for one class — or if I’m lucky, I see them for three to four years. After that, it’s going to be their own sense of wanting to learn, their own motivation, and a level of autodidacticism that really drives them.

One of the hallmarks of a programming career, of an engineering career, is this need to constantly be staying up to date. If I can’t give that passion to them, then I think I’ve really lost out. And that was true even before AI, even when they were writing the code themselves. There were students who would pay someone overseas to write their assignment for them — contract cheating — or get a friend to help. So I think that actually hasn’t changed that much. If I keep my view on that passion, on that motivation, and on socializing them into the disciplinary norms of computer science, I think there’s still some hope.

If we’re thinking about specific skills development in class, that’s where the deeper concern is. Not only because everything is changing so quickly, but because I find it very hard to predict what useful things I can teach them now that they’ll still use in four years.

Now, college education has never been about vocational training directly. Med school and law school a little more so, but even in law school they teach you how to read appellate opinions — they don’t spend as much time teaching you how to redline contracts you’re swapping between firms. So college has always been this disconnected place where you do learn skills, but the skills you learn aren’t always directly transferable. There was a result from the last US census, I think, showing that only about 28% of people who graduated from STEM majors ended up in a STEM profession as narrowly defined. That’s probably a bit of an undercount for various reasons — they didn’t include medicine and healthcare. But at the same time, we still seem to be communicating something valuable.

So I think this is a moment for us to ask: what are those valuable things we’re communicating, that we’re teaching, that aren’t the specific thing they’re going to be doing in the day-to-day job? In that way, I think this AI moment is actually quite empowering, even as it’s depressing at the same time. Because we have a real forced opportunity to go back and think: okay, what are we actually doing here? And I think we can make some good of that.

Ariel It’s interesting. Schools do say they’re not trying to teach for a vocation, but I feel like for a lot of students that’s why they’re going to school — either to learn practical skills or to get the credential they need. So it’s interesting that computer science as a major was never entirely practically focused; there are parts of math that usually aren’t needed on a typical job. But it did seem overall like, okay, this is a practical field. And now the areas that seemed so practical, like learning how to code — it’s a question whether that’s even going to be used. So it’s becoming a theoretical topic that we’re teaching for its own sake. I’m wondering, could you formulate what the purpose is, if it’s not as practical as it was?

Shaanan I think there are a few things. The first is that we have to be realistic and recognize that for many people, college was not required to learn computer science. You could go online and watch CS50 — I’ve taught with them before, that’s wonderful — and you can actually become a damn competent computer scientist just by consuming online materials. So to me it was obvious that something more is happening here.

One aspect could be credentialing. As you said, a degree from Berkeley is going to open doors for you that a few online Coursera courses won’t. To the extent that what we’re doing is credentialing, it’ll probably still function much the same. What’s interesting is that it further weakens the link between what employers claim the credential communicates to them versus what it’s actually doing.

Being realistic about it, we’re pretty aware now that a lot of what happens is a filtering process that happens pre-admissions. The university is already taking students who have jumped through certain hoops, and then attending for four years is not actually about skill development — it’s about certifying that they jumped through that initial pre-admission hoop. That’s not a type of education I particularly care about.

If I were to take the extreme position, I’d say let’s stop worrying about assessment in the university whatsoever, because the credential was never tied to the assessment in the first place. It was tied to their high school ability to jump through hoops. Now, this does put more of a burden on future employers, who’d have to do even more weeding if we’re not assessing at all — and maybe more burden on the poor college students, who now have to work even harder to prove themselves earlier in life and can’t distinguish themselves in college. But I do think de-emphasizing assessment is one path to helping us care more about some of the things education can do when we’re not so focused on hoop jumping.

One thing I think we can and should do is take advantage of the way AI can give us efficiencies in our teaching, and then allow us to provide much more personalized attention to students, focusing on individual goals, individual growth, and individual interests. I have students in my intro to programming class where all they want to do in life is make things. For those students, giving them tools and helping them put themselves in an environment where they learn how to use what’s out there to build the projects of their dreams could be very empowering. On the other hand, I have students who are much more theory-driven, who couldn’t care less about building a distributed system but really want to understand the nature of computation. For them also, I can’t serve them so well at the moment. In a class of 400 or 500 students, I have to cater to a middle path, which up until now has been this forced credentialing function.

So maybe if I can offload some of the rote stuff to AI, I can spend a little more time tailoring my efforts to these groups of students more individually, and focus on the things they really care about — the things that can’t be delivered by someone who doesn’t have time to interact with students directly.

Ariel That’s interesting. On its own, I would ask: if you get rid of the hoops, what percentage of kids really want to just learn for its own sake? But if you tailor it that closely — so those who want to build projects get to focus on that, those who want theory get to focus on that — I think it would be a higher percentage. So it would be interesting to see what could be done.

Shaanan Yeah. On how many students want to learn for its own sake, there’s a big difference between different sides of the institutional spectrum. You’ve got the super motivated high school students who are already on Coursera and have finished four courses before they even come to college. You have community colleges in the US, and technical colleges, with students who are really there for a clear vocational reason. You have liberal arts colleges where they’re very interested in that seminar-style stuff. Then mid-tier state schools, where you get a whole mix — some very motivated, others who just want to party for a couple of years. And then your prestige, Ivy Plus, R1 institutions. To lump all of those people together elides really important nuances about people’s desire to learn and what they actually want out of learning.

There was a 2016 survey of incoming freshmen into, I think, only four-year colleges. That survey showed that 84% of students said learning things that were important to them was a very important reason for attending college — it was one of those Likert scales. But then you ask those same students about the other items, and 76% said gaining a general education was very important. Even more than that, 80% cited getting a better job, and 74% said career-specific training. So you can have these multifaceted reasons for wanting to be part of college. And that’s on the expressed preferences side. As you pointed out, the revealed preferences are somewhat different.

There was another study by Philip Babcock, a social scientist, who found that the amount of time students spend actually doing their college work halved between 1961 and 2003. So the extent to which we’re successful in getting students to learn for its own sake seems to be less than we might hope. Maybe the suggestion we were both building on — that with closer tailoring we could up those numbers and get back to a point where students are really self-motivated — is a possibility. I’m hopeful. At least I have to be optimistic, or else I’d need to change careers.

Ariel It’s challenging, because probably a big part of the change over those decades is just more people going to college and university. So it could be that people who wouldn’t have gone in the past aren’t as interested in learning for its own sake.

Shaanan Yeah, and particularly the GI Bill shifted the college demographic substantially and led to this professionalization race, or academic race, where even to get entry-level jobs you now need a college degree. So for a lot of people who previously wouldn’t have had the motivation or interest to go to college, they now have a reason why they need to go. I do think that’s been, on net, not beneficial — I mean, more people going to college is net beneficial, but I think we’ve overdone it, in the sense that technical and vocational colleges have a really important role to play, and we shouldn’t be forcing people through four-year college — particularly at places with a research focus — when those individuals just want to get out there and start working.

Ariel Do you think countries like Australia might do a better job than America does?

Shaanan We’ve made some very bad mistakes. We essentially eliminated the technical colleges in a series of reforms in the ‘80s and merged them with the research universities. What that’s led to is a number of universities that have to build themselves as research universities, and from which the government expects research outputs, but they don’t necessarily have the talent to compete on the international stage and produce the quality of research that really benefits the country.

So number one, we underinvest in research compared to the US. On all non-medical research, we only spend about $800 million a year in government funding. But then the pie gets split into places that would really better serve the public in their original form as technical colleges. That’s not to say no good research comes out of these places, but the ratio is definitely different. They were designed as technical colleges, and trying to force people into the research-intensive route — this happened in the US, this happened in Australia — I don’t think does a service to anyone.

Ariel (31:46) Okay, now turning from the effects of AI on college specifically: what do you think the effects of social media previously, and AI as the upcoming trend, have been on teenagers and college students?

Shaanan I think we’re still a little early on the AI front to really know. Social media is a little better studied. In particular, there was a consensus report of the National Academies. For those unfamiliar: if you’re a scientist and you’ve achieved real preeminence in your field, you may be elected a member of the National Academy of Engineering or the National Academy of Sciences. Now and then they commission reports that attempt to ascertain the consensus in a field. These are really the gold standard for understanding what researchers think.

One came out just a couple of years ago on the impact of social media on youth and adolescent well-being, and the results were essentially inconclusive. They found a small correlation between adolescent depression, anxiety, and social media use, with a very unclear causal direction. I’ve got a line here from the actual report: “the committee’s review of the literature did not support the conclusion that social media causes changes in adolescent health at the population level. Nevertheless, there are harms…”

I think we see this overplayed in the media a lot. It’s a very popular line to say social media has ruined our kids, that it’s net harmful. There is some evidence in that direction, but certainly not enough to argue conclusively that social media is net harmful rather than net beneficial. There are also plenty of studies showing beneficial effects — ease of communication among teens, people who can access community that they couldn’t previously, LGBT youth in more insular communities that previously would have restricted that. It’s easy to say those pale in comparison to the harms — child grooming, depression — but the evidence really isn’t there to argue conclusively that it’s been harmful. It is very heavily disputed, though, and there are certainly individual scientists who take a fairly strong view one way or another.

One thing that is fairly clear is that technology on the whole has displaced other childhood pastimes, and you could argue that in itself is a problem — the fact that kids aren’t going out and spending time outside because of TV, computers, video games, social media, maybe AI. Maybe they’re staying at home talking to ChatGPT. But there definitely is that displacement effect. So to the extent that you have a strong prior that the things kids were doing prior to 1940 were preferable, then perhaps you’d have a reason to dislike not just social media but all of these different technologies. And again, we’ve seen this pattern with every new large childhood pastime — everything from television to newspapers was originally heralded as ruining the youth. That doesn’t tell us whether this time is actually the real case, whether there is a real wolf, but there is reason to tread a little more cautiously than the media likes to.

Now, with respect to AI and social media use in people who already have problems, I think there are much clearer links. There are patterns that are fairly apparent prior to children engaging in self-harm, where often there’s a spike in social media use. But this is also correlated with different hours of sleep and wakefulness — a lack of sleep and staying awake in the middle of the night is correlated with negative mental health outcomes. Again, it’s unclear which direction the causal arrow points. Are they up in the middle of the night fiddling on their phone because they’re depressed and can’t sleep, because they have negative self-image or anxiety? Or is it the fact that they’re up scrolling through social media at 3 a.m. that’s making them depressed? Or perhaps some of both.

So I think there’s a lot of room to explore that more, but I would be really hesitant to make some of the strong policy decisions that governments are making on the basis of believing this causal link has already been established. I think the evidence just isn’t there yet.

Ariel I guess the question is: there’s been an uptick in certain negative metrics, like suicide among teenagers, since 2012. People say that’s when the smartphone and Instagram and things like that became popular. So if those aren’t a strong enough cause, then you need to see what else might have caused it. Is it that people are overprotective of their children and they don’t get to play? Is it other factors? Is it just too complicated to resolve? And then people look at social media and certain aspects of it seem very negative — unrealistic beauty standards, people spending so many hours a day — so they feel like that must be the cause.

Shaanan Yeah. And look, if you’re a parent deciding for your child — I want to prevent my child from going on social media until a little later, I want to speak to them very carefully and keep watch on what they’re doing — I think all of those are potentially positive things. But that’s at an individual level rather than a population level.

Where I think there’s a lot more evidence is that for children in certain situations, social media use is going to be either net positive or net negative, and you can often tell based on contextual information about their situation. Parents are often well placed to make that judgment — assuming we’ve got good parents, which is often but not always the case. The problem comes when you try to do population-level interventions.

One place we’ve seen a similar-ish intervention was the phone ban in New York schools. There was an economics paper showing that there actually wasn’t a substantial impact of the phone ban on educational outcomes, at least as far as they were able to measure. It was a relatively good study. But there was also follow-up by some of the authors and related persons saying that just because it doesn’t improve academic outcomes doesn’t mean we should let phones run amok in schools willy-nilly. There may be other benefits — just in terms of classroom control, it’s easier for a teacher to control the classroom when kids aren’t all on their phones. There may be other social benefits.

So again, my line isn’t that there is no worthwhile intervention around technology and adolescence. It’s more that we have to be very self-aware about which interventions are directly supported by the evidence and which are more of an experimental game — or which should be tailored less to a population level and more to a smaller context. That could be an individual school, a community, a classroom, or a family.

Ariel I get that it’s a bit extreme for the government to do an outright ban on a whole country. On the other hand, the challenge is that if a parent wants to ban it for their kids but all the other kids use it, that creates a weird social dynamic. So it’s very hard to do on an individual level. But if there were some way to do it on a community level, that seems like a good balance — people could leave the community if they wanted, and you could also run experiments comparing communities. So if they had some way to roll that out, maybe by school or district, that seems like it would be better than doing it across the whole country.

Shaanan Yeah, I’ve actually done some research in this area. Australia rolled out a social media ban to the whole country for anyone under 16. The way they chose to do this was by regulating the social media platforms: if you are a social media platform, you have to take reasonable steps to remove under-16s and to ensure they’re not there.

This regulatory intervention has now been in place for about six months, maybe a little over, so there’s been time for some initial results. I have a couple of papers on this, and there was also a paper by Cass Sunstein and some of his co-authors — I’ve forgotten who the first author was. Essentially they found the ban was unsuccessful, and you still got the disparate impact you were talking about, where some kids are still on social media. Those kids self-identify the other kids who are not on social media as the uncool kids. So rather than solving the bullying problem, it may have exacerbated it.

This is also true in my research, in which we did focus groups with young people and found that essentially all of them knew how to get around the ban if they wanted to. The thing that was much more determinative of whether the ban worked was what the parents wanted to do with their own kids.

Now, the Australian Prime Minister, Anthony Albanese, has openly said that the motivation for this is to give parents ammunition to police their own kids. But as the Sunstein paper showed, unless you achieve a critical mass of kids off social media — whether because families have chosen to enforce it or because it’s become technically impossible — unless you get to that tipping point, it’s not really going to be effective as a social intervention.

So I think that’s more or less where it’s landed. Even the regulator in charge of administering the social media ban has said she doesn’t have adequate power to actually enforce it, and has cautiously and subtly implied that it’s not really working as designed.

Ariel I guess one thing proponents of such bans could say in response is that you just need stronger controls — make it even harder to get an account if you don’t show stronger proof that you’re a certain age and that you’re actually that person. What would you say in response to that?

Shaanan I think the more intense you make the restrictions, the more you start to impinge on civil liberties. Requiring everyone to present their ID when visiting a website is probably a non-starter for people in America, or at least from my understanding of American culture. Even in Australia, people are very uncomfortable with it, and it did not have much political buy-in. So there’s an explicit carve-out in the law saying that platforms that choose to use government-provided ID as a way to ascertain age are not permitted for that to be the only way to establish that you’re over age.

What’s happened instead is that they’re scanning people’s faces and trying to guess how old you are based on a scan. I have some forthcoming research on this that I unfortunately can’t spoil, but with all the methods we tried, we were at least able to bypass them some of the time through the front door, just by messing up your face or putting on a disguise. The models are getting better, so maybe that will stop.

But more significantly, pretty much everyone knows how to use a VPN, even 12- and 13-year-olds. So unless this ban is regulated globally and you start cracking down on VPNs, you’re going to have a really hard time enforcing this at a technical level without introducing internet-level censorship. The UK has been floating this idea in the last couple of months — there have been a couple of members of the House of Lords who have endorsed a crackdown on VPNs. But there are a number of reasons we should feel very wary about any such proposals, because that’s going to have very significant knock-on effects. Given that the harms of social media are already somewhat disputed on net, the benefit of imposing a countrywide anti-VPN and internet censorship system is not something I would endorse. Quite the opposite.

Ariel I guess the question is similar to the physical locks you mentioned before. Maybe you just need a certain level of difficulty, and then it’s not expected that you’d be on social media. I don’t know if that’s true — maybe it’s so trivial that kids will do it no matter what. But once you have that certain level of difficulty, it doesn’t have the same social pressure. And then if some people want to get around it, that’s okay. We’ve weakened the expectation that everyone is on it, and maybe that’s enough.

Shaanan There’s a good point in parallel to that, which is: maybe it’s not working because today’s 13-year-olds have 14-year-old friends, and the 14-year-olds have 15-year-old friends, and the 15-year-olds were already on social media before the ban. So perhaps it’s just a generational thing.

The last paper I published on this suggested that perhaps that’s not true. The reason we’re a little skeptical is that we found cross-age-group socialization of the bypass methods. It’s not just the 15-year-olds sharing with each other; it’s groups of kids across year levels — particularly because of sibling dynamics — sharing both why they use social media, what social media is, and why they might want to get around the ban. So it’s unclear to what extent a generational shift will change it. It’s still plausible, because even that socialization mechanism can change as the generation changes. Perhaps there will be a discontinuity in the way today’s eight-year-olds relate to the 12-year-olds when they all age up and the 12-year-olds hit 16. But at least the current indications aren’t positive for that.

Ariel I guess another way to approach these issues is, instead of trying to restrict it by age, to reduce the negative effects that happen for all ages. So if they have algorithms pushing certain content just to get people to click, but it makes them more depressed, maybe they could do something to reduce that and try to have some kind of positive algorithm boost. I don’t know if that’s something you think might be more realistic.

Shaanan Yeah, there I’m much more bullish on our ability to intervene.

The large social media platforms have been pretty terrible at transparency around their trust and safety programs — that’s the name given to the programs designed to prevent harm to young people, scams, uploading of pornography that’s not allowed on the site, or whatever it happens to be. Those have been very closed to the public. There are some conferences where occasionally practitioners from the companies will discuss things a little, but they’re largely black boxes, and in recent years they’ve become more and more opaque. There’s a well-known story in which Facebook gave a permanent ban to some of my collaborators who were doing trust and safety research on the platform, on the grounds that they had violated the terms of service. Contractually, they are able to do that; they didn’t violate the law by blocking these researchers.

But my view is that normatively we should have rules in place to ensure that researchers, public advocacy organizations, and governments can get more insight, and might not necessarily be obligated to follow overly onerous terms of service that prevent us from getting more information. So the first step is more transparency. Transparency alone is not going to solve this, but enabling it will allow regulators, civil society organizations, and the academics who support the regulators to figure out what kinds of interventions might actually work. You can still design experiments to determine what interventions might work even without support from the companies, but it’s definitely a lot harder.

On the next front, you have things like the Age Appropriate Design Code, which I think is in the EU, which aims to set out what design elements and features should and shouldn’t be in place for a platform accessible to a young person, or what experience a young person observes when identified as having an account on a platform. Australia has gone in a slightly different direction with a digital duty of care, which attempts to impose a generalized legal obligation on the platforms to look after young users. The platforms, seeing the writing on the wall, have not been entirely opposed to either of these efforts.

So there are a lot of things we can do in this direction. Which of these will actually pan out and deliver better outcomes, and ensure we can capture some of the gains without the negative effects? I’m not sure yet. There’s a lot more work that needs to be done. But I broadly think it’s worth experimenting with these different regulatory interventions to determine which might actually work out for us as a society.

Ariel (48:20) It seems like there’s a balance. On one hand, trying to ban everything seems too extreme. On the other hand, some of the habits people have on this very new technology are so unnatural that to just do nothing and say “yeah, whatever” also seems like maybe not the right response. But if they could figure out something more balanced, maybe they could help everyone.

Okay, turning now back to software security. Recently Anthropic released a model called Mythos, which was one of these new moments where AI showed, in a practical way, that it was more powerful than anything before at finding software vulnerabilities. They released that to a restricted group of companies. Then they released another version, Fable, which was supposed to be the safe version for people to use in general — where if they were trying to do things that seem dangerous, like find security exploits, it would block them.

So that’s the basic background. And then the government briefly shut down access. They said that foreigners outside the US, or even foreigners in the US, can’t be allowed access even to Fable. What do you think of the overall risk here shown by this new AI model, and what do you think of the government’s initial response?

Shaanan Maybe let’s start with how dangerous these things actually are. There you have two sub-questions: dangerous to whom and how, but also what is the marginal danger over the other models that are available, including the open source ones.

What everyone in industry is talking about, at least, is a vulnerability tsunami — the notion that these tools introduce significant new cybersecurity risk because of the ease with which they can find latent security flaws, generally in code but also sometimes in systems. More commonly it’s in a code base: they can find something like a buffer overflow and then even write a proof-of-concept exploit for it.

We’re not necessarily at the point where they can come up with a full exploit chain — find a buffer overflow in the JavaScript engine in Chrome, use that to escape the sandbox, then do privilege escalation and get to the operating system. We’re not quite there yet. But we’re certainly at the point where the models are able to discover meaningful software security flaws, and with a proper harness, a proper set of supporting tools, distinguish between false positives and true positives. There’s still a level of over-reporting, but it’s gotten to the point where if you get a few outputs, you can be reasonably confident — assuming you’ve used the right harness — that these are actual real bugs.

So they’re very good at the first part of that chain: the discovery, the analysis, and then the verification. They’re not as good at actually launching full-scale cyber attacks. We’re not seeing, at least to my knowledge, end-to-end attacks by swarms of Mythos bots out there in the wild.

There is some danger, because our ability to patch is now a little slower than our ability to find vulnerabilities. To patch, you have to actually be awake, know there’s a vulnerability, and send your agent to go fix it. Meanwhile, the attackers can be off finding new bugs. And if you were to ask a Mythos-class model to find a vulnerability yourself, there’s no guarantee you’ll find the same one an attacker has already found. So being proactive at the moment doesn’t necessarily save you.

These capabilities are pretty much on the frontier. Mythos and GPT-5.6 Sol, the new OpenAI model released this past week, are mostly on par, and they can both do this. But the open models and the Chinese competitors are honestly not that far behind. This is where I think we really have to be cautious about our policy response, because our policy response is only going to be effective for a certain amount of time. These capabilities are pretty accessible.

There’s also a big question about short-term impacts versus the equilibrium we reach. A policy response that’s good for the short term might not be pertinent to the equilibrium. There was a talk by Nicholas Carlini where he talked about the commoditization of this pretty specialized skill set of vulnerability research. Other observers in the media have said essentially the same thing: what was once the purview of a few people with very specialized reverse engineering skills — myself included — can now be done by almost anyone.

I think that’s where some of the danger is, and there’s a lesson to learn there not just about the danger from cyber attacks, but also about CBRN — chemical, biological, radiological, and nuclear. My personal view, as someone who’s been in security for a long time, is that the reason we don’t see large-scale bad things in the world as often as we might is not because doing bad things is actually that hard. Assembling a dirty bomb is easier than you would hope. If you don’t care about the welfare of the person doing the synthesis, you can probably synthesize some pretty bad things. We know organized crime is pretty good at synthesizing drugs, and explosives are, again, not that technically hard to make.

Some of the reason people haven’t done it is that outside of state actors, the set of people who have the motivation to do terrible things in the world, and the technical knowledge and capabilities to pull it off, and the operational capacity to actually organize it — not only wanting to do it and having the technical skills, but being able to execute on a plan at a larger scale than just building something in a lab — has been very limited. You’re talking about a fairly small set of people at the intersection of three relatively rare groups. Only certain people are very operationally sophisticated, only certain people are good at chemical synthesis, and only a certain group of people really want to do terrible things in the world.

What I’m worried about with AI is that it may be removing some of these distinctions. By reducing the technical barrier and also making operational planning a little easier, it can increase the accessibility of some of these dangers. Even while I don’t think the set of people who wish to do terrible things is going to expand that significantly — it will a little, because of the reduced friction — I think there’s a concern that the barrier for the existing people we’re worried about has been lowered. And we’re seeing this. There was a report out of an Oxford University center about, I think, Somali groups planning piracy and terrorism operations using frontier models.

So that’s the place where I’m really worried, and where I think the government does have reason to worry. I think it’s a different question whether export restrictions and the Defense Production Act are the right way to regulate those.

Ariel So if you think there is a big concern here, but the measures they took were maybe too extreme or just not correct, what do you think is a better approach?

Shaanan Let’s start with what they did. Part of the problem is that they approached Anthropic as if they were the only entity that already had these capabilities, and that was just plainly untrue. OpenAI’s models — even 5.5, the one from the prior release — were fairly competent at finding these things. Sure, it didn’t have the level of hype as Mythos, and perhaps wasn’t quite as effective, but the marginal difference in capabilities wasn’t that great.

At the same time, there are a lot of clouds around the government’s story. Anthropic and the government had been at odds and somewhat antagonistic to each other for some time, even while the government still desired their capability. So it’s unclear to what extent the actions they took were really motivated by concerns versus how much was politically motivated.

It also seemed like this was another narrow jailbreak. It worked sometimes, only unlocked certain capabilities, only for a limited time, and would also have been visible to some extent. The thing that’s more worth worrying about is if someone can find a universal jailbreak that just turns off all the safeguards — and based on the reporting, it didn’t seem that this had actually been found. So the remedy the government chose, which was to ban Mythos itself from non-US persons, both wasn’t well tailored to the problem and ignored the fact that these other models were available, and that the particular jailbreak that had been found didn’t seem to be anything that significant.

So what can we do? This is where we have a real problem, because there’s not a great answer yet. Anthropic’s mode has seemed to be to suggest a coordinated pause — they’ve offered to cease model development if everyone else ceases model development. But this is incentive incompatible. Everyone has an incentive to defect, and not only that, but potentially to defect in secret. So I can’t see that working.

At the same time, restricting things via export controls also seems unlikely to work, particularly because the open source models really aren’t that far behind the frontier, and the know-how of what you actually have to do to improve the models is mostly public. We might see a shutting down of a lot more research — already the frontier labs have been publishing fewer of the key components needed to advance in academic and public venues. We could see that further restricted, with the government trying to push back on open research. But in the US regulatory environment, that’s very hard to do; there are fairly strong First Amendment protections around academic speech.

So I don’t have a brilliant policy prescription we can adopt today, other than to say it’s a hard problem and something we need to be working on and trying. I know that’s very unsatisfying, but I don’t have the answers today.

Ariel People have talked about doing a pause in research, but they’re talking about the more worrisome risks — like AI itself trying to take control or cause mass harm. These specific risks are a little smaller: people just using AI to do bad, at least at the level we’re at currently. It could get worse — if they get access to super viruses and things like that, that could be really scary. But for these kinds of risks right now, I feel like Anthropic’s approach might be reasonable, where you try to have multiple layers of detecting it.

And then maybe you could build on that. I know we talked about not wanting to require people to verify their identity, but in theory, if you had more knowledge of who’s using your product and what they’re using it for, you could detect bad players — either block it at the moment or detect it right afterwards so they can’t keep doing harm on an ongoing basis. That may be an approach that could work for now.

Shaanan Yeah, and that’s where we are seeing the labs doing a lot. OpenAI’s Verified for Cyber program requires you to submit some ID so they actually know who you are, and then they look at your transcripts — and they probably are running models over the transcripts, post hoc as well as live.

One of the common techniques, which I’ve also done some work on, is guard models. The idea is that before a tool call goes ahead, a second classifier or second model looks at both the tool call and the transcript to determine whether what you’re requesting should be allowed to go forward. That might be for your own safety, or because you’re stepping outside the bounds of what the frontier lab wants to permit, either for cyber risk or other reasons.

The problem is that there are still lots of attacks on the guard models, and a post-hoc approach to preventing harm — we’re going to look over the transcripts later and ban people — might still allow attacks through. There’s a big colloquial literature on people who create new accounts whenever they get banned: jailbreak it to the extent they can, use the capabilities to do their little bit, get banned, pay another 20 bucks, open another account, make a little more progress. Through that methodology they can actually assemble a fairly fearsome cyber attack on something or other, despite the existence of guard models stepping in at various points and blocking them.

There’s also a lot of sensitivity to context depletion. As you chat longer and longer with a model, the efficacy of the guard models will decrease as the context window fills up and the size of the context the guard model has to handle increases.

So a defense-in-depth approach is definitely what we’re doing currently and what we should be doing, but there are still gaps in it, and those aren’t plugged yet.

Ariel I noticed I was just playing around with Fable for a personal project — trying to track everything I read, different things like that — and it put up these warnings for no reason at all, and then it switched off Fable. It had nothing to do with anything. But I think that’s just because the government made them be even more restrictive. I don’t know if on their own Anthropic would have been that restrictive.

Shaanan Yeah, the overblocking thing is quite funny and happens to me all the time. I’ll be doing something totally benign, but because I’m a security researcher and my research papers have something to do with security, they get flagged very often. I haven’t experienced that with GPT-5.6 yet — but they’re obviously in a different regulatory position, a different political position, so they’re acting quite differently.

I think that’s one of the dynamics to watch for here: the frontier labs all sit in a different position in the political and social ecosystem, so the capabilities of the models that are permitted to users will likely end up being different. This is both beneficial for users — there’s good competition in the market — but also dangerous, because you can play the models off each other. I had an instance where I was doing some security research and GPT was getting annoyed at what I was doing, so I gave that specific task to Claude, took the results back from Claude, and put them into GPT. You could imagine building a harness that does this for you, that looks for when you get blocked by one, sends it to the other with a slightly different context, and then takes it back to the first one in a new context window.

Ariel That’s another case where, if they really knew the identity, it could help. If they see a pattern of people repeatedly doing something, they could be blocked — and if they knew their identity, it wouldn’t be as easy to just create another account. In America, if you have to give your credit card and some other verification, it’s not as easy; the credit card is attached to a name. So there are certain levels where I feel like it would be easier to control.

And there would also be a very strong incentive for the companies to control it. Anthropic has issues with distillation attacks, where companies train their models on Anthropic’s advanced models. If they could verify who’s doing that, they could shut them down — that would be very beneficial for them. I guess it gets harder in other countries for them to have that level of control, but it seems like something they’d want to do anyway.

Shaanan I think there is some benefit to be gained here by information sharing across the frontier labs. This has often been difficult for companies in the US to do — even about threats and threat actors — because of antitrust laws, which restrict collaboration and cartels. I think it’s a good thing to have the antitrust laws, but I do think the regulatory security of having explicit carve-outs, as opposed to implicit nods, could be of benefit. So establishing a platform for sharing bad actors and threat signals between the frontier labs would, I think, be one thing that would help with this problem. Maybe we could even get some collaboration cross-border, but that’s probably wishful thinking at this point.

Ariel (1:04:38) Hopefully longer term that’s something that would need to be worked on, if people wanted to do things like slow AI development.

Okay, turning to people using AI. Agentic AI has become much more powerful. It’s something software developers are much more familiar with, but I think it’s starting to spread into the wider world — where you’re not just chatting with the AI, but you’re telling it to do stuff and it goes off, spawns subagents, does tasks, completes projects, creates spreadsheets, things like that. So what are the main risks there, and what are the main things people who use it could do to protect themselves from those risks?

Shaanan In terms of the risks, a good place to start is the lethal trifecta, a term coined by Simon Willison, a tech writer and coder. The lethal trifecta is: access to private data or systems, exposure to untrusted content, and the ability to act externally or communicate. All of those things are present in a modern-day coding agent — or in Codex for Work, I think is the new OpenAI branding of it, and Claude Cowork.

The problem is that you gain a lot of utility by giving agents this lethal trifecta. If you want your agent to fill out your expense reports at your business, you need to give it access to a private system, access to the content of wherever you’re storing your receipts, and the ability to act. That’s the only way to achieve the task. So I think the largest risk is that our incentive structure is such that we are strongly incentivized to do the exact things that increase the risk substantially.

In my own life, I had one of those instances of AI deleting all my files. In this particular story, I’d messed up my cloud sync for the files on my computer, because I had a Git repository that had too many little files — and you should never put one of those inside your cloud sync, because algorithmically it’s bad and it has to do too many database operations. Details, details. So I told my agent, “Hey, can you please fix the syncing?” It tried a whole bunch of things, and eventually came to the conclusion that the cloud sync repository was too messed up and the best thing to do would be a total reset — not realizing that the sync state of the files was mixed, so some were only present locally, and a reset would result in them being deleted.

Fortunately, I’m a good computer scientist and back things up regularly aside from the cloud sync. But that was a really nice example of where the intent was actually aligned. The agent was trying to do the right thing, it missed a critical piece of information, I’d given it access, and yet it still resulted in substantial damage — even though I was able to recover from it. It was painful, but recoverable.

I think what we’re missing, at least up until recently, is this: in the 2000s and 2010s we moved from thinking about browsers as just another app to thinking about them as equivalent to an operating system in complexity and importance. We need to think about agentic systems the same way. They have complex components and a very large trust boundary, in that we’re giving them exposure to a lot of untrusted content, just like we would for our browser. We need to start creating boundary mechanisms to enable us to deal with it a lot better.

The harms one could experience are really the same sorts of harms you could inflict yourself by typing in commands you haven’t vetted. So that attack surface is really, really broad. Anything you can do to mess up your own stuff, the agent can probably do as well. And that is the big problem.

Ariel Some of it could be addressed in a simpler manner. Often you could have it run in a sandbox, and you don’t really need it running outside the sandbox. If you’re working on code, you give it access to the project it’s working on. It could read things from outside — if it needs to research and find libraries, things like that — but it wouldn’t be able to delete your actual data. So I feel like that’s an initial step. But then people might just default to giving it root access to everything on their computer, and that default is where it becomes a problem.

Shaanan Yeah. And setting up a sandbox is painful, and you need to know what the specific permissions are that this thing shouldn’t have. People who work in distributed systems have gotten better at doing this, but it’s still a real pain. You try to use Linux cgroups to isolate things, or Docker containers with locked-down permissions, or Kubernetes, or whatever your distributed system of choice is — when you try to lock it down, it ends up being quite painful and a lot of overhead, despite the significant advances we’ve made.

One place where we did make real headway was browser sandboxing. Escaping from the browser sandbox as a piece of malicious code is now legitimately quite difficult. The other thing we were able to do is segment parts of a browser’s memory from other parts — isolating different tabs in their own processes, separating cookies, developing protocol-level restrictions that isolate different security domains. That’s been effective.

I think porting that same approach to agents would be a way to make some progress. It’s still a long and hard road, but there is hope, and we have architected these things before.

Ariel Some of it’s just figuring out the right balance. You have people running it on completely unrestricted mode, and on the other end you have it asking you before it does any little thing — like before it reads a file, and there’s no risk in reading a file. So they need to find that balance where the defaults are pretty good for security, where it makes sense what it has access to, and it’s rare that it needs to ask you for something — but when it does, you actually review it and don’t just click yes right away.

Shaanan There’s also Ariel using the agent versus my grandmother using the agent, and those require different models of what to ask permission for and when.

I know there are certain people in my life who, when asked to click yes on something, will always click yes. And certain people I’ve trained — my mother in particular — who I’ve trained to always click no, do not grant permission, no matter what, even if it would be useful, and she suffers a loss of utility. Perhaps that was an overcorrection.

But I think there’s potentially an interesting path forward here using guard models that understand not only the technical context but also the user context, and can prompt for permission appropriately. You already see this if you’re using the ChatGPT agent-based interface, Codex. You can turn on a feature that allows a guard model to review the action first and then prompt you if it senses a mismatch between the gravity of the action it wants to perform and the direct relevance to the instruction you gave it. So if the thing it wants to do has medium potential for damage but very high salience, directly related to your instruction, it might not prompt you. Whereas if it’s a fairly risk-free action but not that related, it still might not prompt you. Only in the case where it’s high risk and medium or low authorization will it go ahead and ask you explicitly.

Ariel (1:12:22) Turning away from people using agentic AI to broader consumers: the internet allowed much more scams and hacking and all that, and now the bad guys using AI could bring it to a much bigger scale. There’s been an increase in these scams; it’s becoming a bigger problem. Regular users — and maybe the elderly and all these people — just go on the internet and there are so many risk surfaces for them. There are scams through email and on websites and all over the place. So what do you think is a broader solution, where we could help these people in their general usage of technology and try to reduce the number of scams?

Shaanan There are a few things I think we can do. One is an ex post regime that provides some kind of social insurance for when things go wrong. This already exists in the US in the form of Regulation E around card issuance, which provides protection against liability to users of credit cards — if you didn’t authorize a transfer, you get made whole. I think we can extend this notion to authorized push payment fraud, which is when you press the authorize button but you were being hoodwinked.

Obviously there are challenges with doing this, because then people will just say “I didn’t mean to send this person the money, get it back for me,” and that can open up payment processors to more fraud. But the UK is moving in that direction, and I think such a regime is reasonable. It would have to be designed carefully, but I think that’s something we can do ex post.

It functions just like social insurance. Everyone pays an extra fee to the merchants, and the merchants pass it on to the payment scheme providers. I’m eliding some of the details — when you pay the merchant, often it’s actually going to the payment processor — but essentially you’re paying a surcharge for the benefit of having this scheme around you. And it is somewhat effective. There have been many times in my life where I’ve done a chargeback, or where there were unauthorized charges on my card, and that has helped. So I think we can do that in more places. Zelle is one place in the US where this does not exist very much, and I think we can do a better job there. That’s on the ex post side.

On the ex ante side — before the damage happens — I think there are some defensive measures. As deepfakes become more prominent, one good thing everyone can do is come up with a family safe word: a word or phrase shared with family members. If a family member asks you for a large amount of money because they claim to have been abducted and put in a Costa Rican prison and need $5,000 for bail transferred via Western Union, and they don’t say the family safe word, then you know it’s a deepfake or that something else has gone wrong. Having that kind of trusted side channel or back channel is something that’s very easy to do.

I also think the technical design of some products can be improved. One thing some platforms can do is give you a trusted contact. Let’s say I want to transfer $40,000 out of an account. I can say that for any transaction over $20,000, my trusted contact also has to approve it on their device, and that can be cryptographically verified. Obviously there’s a balance here against friction, and some people don’t have a great trusted contact in their lives, but giving people that option is another way to empower them — and perhaps suggesting it in many instances.

The usual advice also still works, at least for the time being: keeping an eye out for obvious signs of scams, not clicking yes to things, having your wits about you, not using the same password for everything, keeping your devices up to date so drive-by malware doesn’t work. All of those basic security hygiene practices haven’t gone away just because the threat landscape has changed.

Ariel Do you think we could also develop systems using AI that just run in the background and detect if there’s any issue? So it would be running in the browser or whatever platform, and if it detects that something looks very suspicious, it would alert the user — and they could have different settings to protect them.

Shaanan Yeah, I think we can do more of that. It’s already being done quite a lot on the back end. There was a nice podcast with Patrick McKenzie called Complex Systems, where he spoke to someone from Stripe, his former employer, about their use of AI in fraud detection. There are very clear patterns for money transfers that are likely to be fraudulent, or even for authorized push payment fraud, where someone authorized it but was doing so under the influence of someone acting illegitimately.

So I think we can do a mix of both. There’s probably room to do more on the front end to protect users themselves and their experience, but also on the back end to flag these things and put in place more friction when someone tries to do something the system tags as being highly likely to be fraudulent.

Ariel It’s interesting because it’s becoming worse and worse, I feel like, and right now it doesn’t seem like enough is being done about it. But hopefully these defensive systems will be built out.

Shaanan Look, there is a bright spot. You see all these news stories like “grandmother loses life savings.” But there was a paper out by a collaborator of mine, Elissa Redmiles, a couple of years ago, that found the median amount people actually lose to scams in their lives is not particularly high. By memory, it was on the order of a few hundred dollars — which for some people can be really significant, but for the median American, it’s survivable.

It’s just that the long tail of people who really get hurt by this is significant, and those harms can be life-changing. So I do think we need to protect people, but I also think there’s room for some optimism — that between the defensive technologies we have, the training we can do with people, and the size of the problem, at least in the next little while things seem hopefully manageable. Maybe I’m again overly optimistic, but I like to look outside and see the sun.

Ariel So do you think in the long run, in the balance between defense and offense, defense will come out stronger and the scams will be reduced?

Shaanan In the long run, I am optimistic that humans will develop both the social technologies and the digital technologies to be more protective. Social technologies include things like insurance and basic literacy at a societal level; digital technologies include all the fraud-combating AI we’ve talked about, as well as more traditional kinds of mechanisms.

I think we have a wonderful history as a civilization of being able to address problems on a large scale — not eliminate them entirely. We haven’t eliminated war or starvation, but we’ve been able to do significant things to address both of those problems, even while in some ways and on some days we make them worse. So I think that in the long run, the dynamic equilibrium of the cyber and scam landscape is going to end in favor of the good and the kind.

Ariel (1:19:47) And if people want to find more about you online, where can they find you?

Shaanan My academic webpage is at cohney.info. I’m also sometimes active on Twitter and very active on Google Scholar. On Twitter and any other platform, I’m @shaananc.

Ariel Shaanan, thanks for coming on the podcast.

Shaanan Thank you very much. Good to speak.

Disclaimer: I'm an engineer at Google, but this is a personal blog based on public info; it doesn't represent the views of my employer.

Here’s an overview of Google’s AI search and chat options from simplest to most advanced:

• AI overviews in search - these show up automatically for Google searches. For basic info they can be fine, but I don’t find them as good for more complex queries. They often summarize results from a single web page.

• AI mode in search - a quick click on top of the search results opens AI mode. It performs many background searches, creates a detailed summary, and will often create tables summarizing the relevant info. You can also continue to chat with it. Google also just announced improvements to AI mode to better integrate it into Chrome.

• Gemini - Gemini is Google’s AI chat app, similar to ChatGPT and Claude. It’s directed at answering a wide range of questions, which often don’t require as many searches as AI mode performs.

Gemini Models

If you have a Google subscription (more on that in a moment), you can pick your AI model:

People sometimes complain about an answer Gemini gave and it turned out they were using the fast model. “Fast” is good for quick simple answers, but for more complex answers you want the AI to use more search and reasoning. Reasoning is when the AI thinks to itself (although you can also view its “thoughts”) before outputting an answer. AI is significantly more accurate than it was a few years ago, and search and reasoning is a big reason why.

“Thinking” and “Pro” both use reasoning, but the Pro version uses a more expensive slower model of Gemini. Gemini recommends using Pro for math and code, but it can be better for other complex questions too, if you don’t mind waiting.

Tricks to save money

AI Studio

If you want to access Pro features without paying, you can use Google’s AI Studio. While it’s intended for developers, the app is easy enough for anyone to use. Just make sure to specify the settings you want on the side:

• Temperature - how much randomness is used when generating an answer. The default is fine, but sometimes it’s worth lowering to get a more consistent answer.

• Grounding with Google search - whether the answer is enhanced with search, usually worth enabling.

• Thinking Level - how much thinking the AI should use for its response. Set it to “high” for hard questions.

Google AI Pro Subscription

AI Studio is a great free tool, but it’s worthwhile to subscribe to premium options if you hit the free tier limits, want access to Gemini pro in the regular app, want to use advanced tools like Deep Research, or want access to the other associated features. While the official price is $20 / month, there are ways to save:

• Verizon discount - If you have a Verizon internet plan or cell plan, you can get Google AI pro for $10 / month instead of $20 / month.

• AI Plus - If you don’t have Verizon, you can consider Google’s cheaper “AI Plus” plan instead of the Pro plan.

• Family sharing - After signing up for Google AI pro, you can share it with up to 5 additional family members by adding your family members at g.co/YourFamily They will also get access to the shared 5TB of online storage.

Google AI pro now comes with 5TB of online storage so you can keep all your emails, photos and Drive files, as well as backup your computer. This is a solid deal: Up to 6 people can get access to Gemini Pro + other AI features + online storage for $10-$20 / month.

Advanced Gemini features

If you have a paid subscription and click on “Tools” in Gemini, you’ll see many options:

That’s a lot of tools worth exploring, but for now I’ll just discuss two of them briefly:

• Deep Research - this will spend a few minutes searching hundreds of online sources then write up a long report about your topic. It’s not useful for casual questions, but it’s helpful for important questions that have many sources online, such as a health topic or an overview of a specific research area. You can then follow up to verify specific points. Besides the default option of searching the web, you can now also have it search through your Drive and Gmail.

• Canvas - this is an interface for iterating with the AI. For example, you can ask it to generate an essay and then directly edit it afterwards. You can also create simple web apps or other content:

Gemini Live

Gemini Live is a lesser-known feature inside the Gemini mobile app that you can talk to. Some people like to use it have a back-and-forth conversation while others leave it on to ask questions to while they read something. You can also share live video with it (from your camera) or share your phone screen. Let’s say you’re trying to fix a broken appliance - it’s often faster to just share live video and ask it questions, rather than sharing multiple photos with it.

Using Google AI elsewhere

You can access Gemini assistance in Docs and Gmail, but I won’t go into more detail about that. NotebookLM is useful if you have sources that you want to summarize or ask about, although there are now ways to do that in Gemini directly.

Gemini in Chrome appears on the top right corner of Chrome and lets you have AI do simple tasks for you online. For example, I wanted a list of all the books I had ever purchased on Amazon, so I asked Gemini in Chrome to navigate through my Amazon book orders and extract all the info. I then uploaded the list to Gemini and asked for further book recommendations.

Creating apps with AI

It used to be only software developers could create web and mobile apps but now anyone can with AI. There are many tools you can use to create quick apps:

• Gemini - useful for very quick apps that you create in one prompt

• AI Studio - previously I mentioned using AI studio to chat with the AI, but AI studio also has a whole “apps mode” for creating web apps. This is more powerful than Gemini for iterating on web apps, and AI studio also offers ways to automatically integrate AI in your app and to publish it on the web.

• If you decide you want to build more serious apps, you’ll likely want to code on your desktop and then you can consider a tool like like Google Antigravity. It’s considerably more powerful than the web apps, but there’s also more ways for things to go wrong…

Bryan Caplan argues that roughly 80% of schooling is wasted time, and that the primary value of a degree comes from signaling - demonstrating intelligence, conscientiousness, and conformity to employers rather than building useful skills. In this conversation, we discuss the evidence behind that claim, what “conformity” really means, why the system is so resistant to change, what Bryan would do if he was in charge, and whether AI will change the equilibrium.

Some questions we discuss:

• How much does education actually increase abilities or intelligence?

• The Flynn effect points to an increase in IQ scores over time. Do schools get any credit ?

• Bryan has argued that a college degree is important for signaling conformity and skipping it demonstrates non-conformity. But if conformity just means “professionalism” - can you demonstrate it without a degree?

• Classes are no longer entirely in-person, and tech will change education even more. Will people still need a four-year degree from a brick-and-mortar institution?

• What happens to universities once AI can do most intellectual work?

Transcript

Below is a lightly edited transcript, courtesy of AI:

Ariel [intro]: Today’s guest is Bryan Caplan, economics professor at George Mason and the author of The Case Against Education. Bryan argues that much of school isn’t useful and the main benefits of a degree come from signaling. By getting through school, you show employers that you’re intelligent, conscientious, and conformist. We discuss what percentage of school is useful, whether conformity is important, why the educational system persists, whether AI and online learning will change it, and what Bryan would do if he was in charge. We also discuss the Flynn effect, gifted kids like the computer scientist Scott Aaronson, and what Bryan thinks of the AI singularity.

Ariel: Hi Bryan, thanks for joining.

Bryan: Glad to be here, Ariel.

Ariel: What percentage of school do you think is a waste of time?

Bryan: 80%. Very specific number — I’m not married to it. I go through a lot of different lines of evidence in the book, but for me, probably the most convincing is just credential inflation and how much education has risen relative to how much you really need to actually do the job. So 80% is my preferred point estimate.

Ariel: And do you apply that across all of school or is that just for college?

Bryan: That’s a great question. If you were to break down by subject, you’d get a different number. But even at the level of kindergarten, it’s very easy to forget how much of what you do is not actually reading and writing, nor is it fun for most kids. For example, it’s very standard to have required dancing, and at least half the students — if you’re paying attention as a parent, I do — look totally miserable during this dancing. They don’t need to know it for real life. They don’t like it. That is a waste of time for them.

Ariel: And do you think, besides reading and math, all other subjects are basically a waste in elementary and high school?

Bryan: For the vast majority of people, yes. Some of these subjects just are not useful in the real world except to become a teacher of that very subject. And there are other subjects where they’re very useful if you can get to a really high level, but school is making almost no effort to get people to a high level. Take natural sciences — if you’re going to become an actual natural scientist, that’s useful stuff, but hardly anybody does. For all the others, it’s pretty useless. The only thing you could say is there’s option value. But we’ve got excellent predictions for most students saying they would never do it no matter what. Much of it comes down to: unless you are good enough in math, your odds of ever becoming an actual scientist are basically zero.

Ariel: There are many people who claim that even if people forget the facts they learn, they still pick up certain skills. And there’s some evidence that IQ scores increase after going through school. How do you address that?

Bryan: There’s been a lot of work on this. There is good evidence that education raises IQ under certain conditions. The main thing to understand is that a good chunk of it is teaching to the test. Obviously if you just gave people the answers to an IQ test and they memorized them, no one would think their intelligence had really gone up. A lot of what’s on IQ tests is standard material taught in school. Some classic IQ tests have facts on them — like “what’s the longest river in Africa” was on one standard test. So the idea that school is teaching general thinking skills is incorrect.

The main thing we know about improving IQ is that so much of teaching to the test doesn’t generalize to other tests, even ones that seem very similar. Then there’s a whole sub-literature on transfer of learning — do people actually take things they learn in school and apply them in real life? The answer is generally it’s really hard to detect any such effects. I’m not going to say it never happens, and I’m convinced that in a few cases it does, because there are people who apply knowledge very broadly. But in the data, those people are so rare that we don’t see them. To have a whole system revolving around trying to achieve something that only maybe one person out of a thousand actually gets is a pretty silly system.

Ariel: What about the Flynn effect? It seems part of the cause might be that schooling raised people’s IQ.

Bryan: Great question. I actually got to review one of Flynn’s later books for the Wall Street Journal, and he’s much more agnostic than most of his fans. Flynn will say that if you look closely by subtest, there are areas where we’re way worse than 100 years ago, like arithmetic. Modern Americans’ ability to do arithmetic is worse than it was 100 years ago. On the other hand, for tests that are highly g-loaded — like Raven’s Progressive Matrices — that’s where we’ve improved. Flynn has a reasonable story: we’ve gotten better at the things we practice more, and worse at the things we practice less. Interpreting this overall as a rise in intelligence is not the best way to think about it.

Within subtests, some correlate very strongly with overall IQ and others don’t. Memorizing random strings of numbers has very low g-loading. Reverse digit span — where I tell you numbers and you give them back in reverse — that’s more g-loaded. Then there are things like analogies or Raven’s Progressive Matrices that use a lot of general intelligence. If you compare individuals, these g-loadings are very predictive. But the changes over time don’t fit the general intelligence pattern at all — some high g-loaded things went up, some went down.

As you mentioned, there are probably multiple things going on with the Flynn effect. Most obviously, nutrition. I’ve got an interest in international adoption, and what you see is that international adoption at birth from very poor countries very reliably raises height, weight, head size, adult IQ, and school performance. So there we’re probably getting a real boost. But for populations that have had good nutrition for a hundred years, it’s not much and maybe nothing at all.

Another way I think about it: compare the 20 smartest people today and the 20 smartest people from 1500. When you read the very best from 1500, these people seem crazy smart — especially when you realize they didn’t have many giants’ shoulders to stand on, and their population was so much lower. Maybe we aren’t any smarter at all.

Ariel: Or maybe the top people had special circumstances, but the masses benefited more.

Bryan: We have people in special circumstances today too.

Ariel: Right, I’m saying if schooling gave a small boost, the top people, if anything, might be held back by schooling.

Bryan: Yeah, except in modern society, there are people who have all the stops pulled out for them in a way that would have been super hard in the past. And we’re much more likely to uncover hidden talent. In 1500, if you’re the son of a farmer with incredible mathematical talent, it’s quite likely that would never be noticed. Whereas today that person would probably get funneled into the elite echelons of mathematics — as long as they’re in the first world, at least.

Ariel: Turning to reading, writing, and math — what percentage of that do you think is useful? Could you just teach a kid when they’re five and six and then let them be on their own?

Bryan: That’s one where I’m more sympathetic to the status quo. We have a lot of evidence that people, to really get good at something, have to do what psychologists call over-learning — practicing to the point of automaticity. The general pattern is that whatever is your highest level of math, you don’t over-learn that. The only way almost anybody gets to over-learning of algebra is you have to go all the way to calculus. You need four more years where you’re using basic algebra as an input over and over again.

One of the sad things about how we run education today is that about a quarter to a third of American adults are functionally illiterate and innumerate. Not that they don’t know numbers or letters, but they’re not capable of doing even basic tasks. It would make so much more sense to give them four times the time on task and have them do nothing other than literacy and numeracy. It’s crazy that we have people who can’t read and write English properly doing foreign languages. And that is our system.

Ariel: Right, there are people who don’t know fractions and exponents and then we’re making them learn trigonometry. What a waste.

Bryan: Never mind fractions — how about percentages? A lot of people have problems with percentages. I remember talking to psychologist Gerd Gigerenzer, and he was saying that only about 10% of Americans can calculate 0.1% of a thousand. In Germany, 30% can do it — even that’s not so good. The basic thing of knowing thousand, million, billion, trillion — where each time you’re multiplying by a thousand — and being able to do that in your head and compare the population of Luxembourg to China by quickly multiplying by a bunch of tens, this is in fact a rare skill held by the elite, even though it’s not technically complicated. It would just require an unpleasant amount of drill.

Since I’m a homeschooling dad, I’ve dealt with how much drill is really necessary. The students I really care about — my own kids — it’s like, can’t we just move on and pretend that we’ve learned it? It’s not fun to do drill, but drill is the way normal people learn.

Ariel: Do you think that’s true about reading also? I learned to read pretty young and then in school there was a lot of read-aloud time that I found very boring.

Bryan: For math, I know what the research says about over-learning. For reading, I think there’s a tiny subset that’s hyperlexic — from an early age, fascinated by reading and loving books. Those people take to reading like a fish to water. But for normal people, it’s boring and tedious. And yeah, they probably do need a ton of drill to get any good. How do we know? From the amount of drill people are already getting and yet looking at tests of adult literacy — it’s shockingly poor. It’s the kind of thing where it’s hard to believe the numbers when you see the National Assessment of Adult Literacy. But in the end, it’s the difficulty of appreciating what life is like for someone very different from yourself.

Ariel: Now turning to signaling. You mentioned how a lot of school is about signaling intelligence and conscientiousness. I think those aren’t so controversial.

Bryan: And sheer conformity — never forget that one. It’s really important.

Ariel: That’s what I was about to discuss. Could you spell out what you mean by conformity? What are companies looking for?

Bryan: People that have embraced standard work culture. Show up on time, be polite, know what profanity not to use and who not to use it in front of, chain of command, following rules. There’s a trivial sense that sociologists and anthropologists point to where every subgroup is conformist — in the hippie group you’ve got to have long hair and you can’t have a job. The kind of conformity employers are looking for is conformity to the culture of work: sit down, shut up, show up on time, do what you’re told, respect the hierarchy. There’s no I in team.

Ariel: I saw a recent Substack post by Richard Ngo saying it seemed like when you said conformity, you just meant professionalism. Is that similar?

Bryan: I would flip it around and say professionalism is basically conformity to what’s expected on the job. You might add that professionalism also requires that you be good, which isn’t quite the same thing. But I’d still think of conformity as more general. It’s such a basic human trait — as a parent you can see it emerging in kids so early. “Other people aren’t doing that, I shouldn’t do that.” And obviously adults have it too — the heuristic of looking around and asking, “Is anyone else doing what I’m doing? No? I better stop.” I may be biased because I have a whole book on nonconformity, but I don’t want to have a fight over words. You say professionalism — all right, it’s in the right ballpark.

Ariel: If it more means professionalism, then there should be ways for people to not go to college without signaling that they’re unprofessional. Why hasn’t it become more common to skip college or do it online?

Bryan: A lot of what people need to do on the job is hard to articulate. It’s often embarrassing to spell out exactly what you’re supposed to be doing — like with a person of that level of status, you need this level of deference, but with a different level of status, a different level of deference. It depends on gender, age, so many variables. Part of being conformist is “I don’t have to be told — I just copy what other people are doing.”

In the book I go over ways you could be very smart and hardworking but still not conformist. Like, you wear a purple suit. You’re still smart, still hardworking, but not conformist. What exactly is unprofessional about wearing a purple suit other than it’s not done by other people? Some professionalism is specific to the occupation — put away your tools when you’re done. But “don’t wear a purple suit”? That’s not intrinsic to the task. It’s just that other people aren’t doing it.

Ariel: It’s interesting because it doesn’t seem like colleges really drill conformity in. People can still be somewhat rebellious.

Bryan: That’s a key point — school is only imperfectly preparing you for work. There are a lot of gaps between the school ethic and the work ethic. I think the biggest one is caring about fairness. The school ethic is so into fairness and the work ethic barely cares. When my kids were in elementary school, they didn’t have enough chairs for everybody. So what did the teacher decide? No one gets to sit. It wasn’t even that the people who need to sit most get to. It wouldn’t be fair if 28 out of 30 people got to sit and two didn’t. Couldn’t we share or rotate? No, still wouldn’t be fair. Whereas in an office, obviously you’d ration chairs based on status or what’s necessary for productivity.

In terms of conformity, schools still require some kinds — sit down, do the assignments on time. But there’s basically no standard of dress, for example. And there are other things, like knowing which exact phrase is acceptable. You have to say “enslaved persons,” not “slave” — “slave” is offensive and “enslaved persons” is sensitive. If you start asking why, you’ve already put your foot in your mouth. There’s going to be a lot of things like that in the world of work too.

Anytime you’re working with other people, you need some conformity. As a nonconformist, I don’t like that, but I understand what’s required for productivity. Most work is teamwork.

Ariel: Even if college is significantly about signaling, why is so much of it impractical? You could signal and still teach practical skills. Yet American universities have all these arbitrary requirements.

Bryan: I’m not that big of a functionalist in terms of saying the system has been expertly designed to serve its purpose. Really all I’ll say is it’s notably better than nothing, and it would be very hard to break the equilibrium in a way that would actually benefit the people breaking it. So what exists tends to continue.

I often mention this painting of a 12th-century college in France — it’s eerie how much it looks like a modern classroom. A lecturer on stage, students taking notes — not on paper, because they couldn’t afford it, but maybe on wax or whatever they used.

These are nonprofits, heavily subsidized by taxpayers and philanthropy. They’re not that constrained to do what’s most beneficial for customers. And the intellectually demanding subjects are probably too hard for most people, so you’ve got to dumb it down. Why couldn’t you dumb it down and make it more practical? If you were a school that tried that, you’d probably get community college–type students. It wouldn’t be a way of coming up with a high-status version.

About 15 years ago, Alex Tabarrok — you know Alex, right? He was starting to worry that MOOCs were going to put us out of business. I said, “So you’re going to let your son go to a MOOC instead of a real college?” And he just blurted out, “No son of mine.” Yeah, exactly. That’s why the system is really stable.

As long as high-status people think online is beneath them, that’s a self-sustaining equilibrium. If you’ve got piles of taxpayer money and philanthropy on the side of dysfunction, it’s even more stable. Just think about spelling in English — what a stupid system, and how it would take five minutes to come up with a better way. What’s your hope of getting people to flip to simplified spelling? Good luck!

Ariel: So 15 years ago wasn’t the time for online education in America. But do you think now it’s beginning to change? There was COVID, and now the technology is even more powerful. Is it starting to change?

Bryan: The main thing I’m seeing is that a lot of schools now have online education for in-person students. You go to a school, live in the dorms, and some of your classes you can Zoom from your dorm room instead of walking ten minutes across campus.

During COVID, we had something often fully remote. That fits with conformity signaling — if everybody’s doing it, it doesn’t make you look bad. But switching to a system where you don’t even physically exist on campus and schools save a pile of money by selling off superfluous workspace? I see very little movement in that direction.

There are some purely online master’s programs, but the main thing I’m hearing is that master’s programs in general are falling, and it’s not the online ones taking over. The master’s is decaying as people feel like they need to either do the PhD or just give up. For a long time there have been PhD programs where the only job you could get would be being a professor, but there are way more PhDs than jobs. People just kept going into something that was on average going to be really bad for them. You’ve got a lot of dreamers in English. It’s ad hoc, but I don’t see a better explanation than delusions of grandeur.

Ariel: I wonder if there was this fiction that you had to be in an in-person college and it’s breaking in a few different ways — people are using AI to learn, going to online classes. Is that trend going to continue? Will it eventually be accepted in America to do an online degree?

Bryan: My best guess is it’s going to continue but asymptote at a moderate level. You’ve grown up in this culture, you know what Americans are like. To be in a middle-class family and just say, “We’re giving up on residential college” — that’s almost like getting a visible tattoo on your neck. Not your forehead, which is more extreme, but your neck. The reaction your parents and peers would have: “What are you doing? People of our class do not do purely online degrees. You must go to the residential college, or you are not really one of us.”

In the American accent we’re not used to being that snobby, but we are. We’re similarly snobby to British aristocrats in our hearts — we just don’t have their accent, which is the perfect accent for expressing snobbiness.

Ariel: Another aspect of AI is it’ll let people cheat a lot. If universities are places where tests are the only real verification, could that weaken them? People could just do tests without the whole university.

Bryan: I think you’re being optimistic, because a lot of professors aren’t even changing their grading system in response to AI, which means people who previously couldn’t have gotten through will get through. The main thing to remember is that it’s already insanely easy to get a degree in something or other, and yet it’s still a minority accomplishment. Most people don’t accomplish this thing that seems super easy.

How could that be? It comes down to most people being so unagentic that they will not rouse themselves out of their ruts just to do something that would give them big gains, because it would require a little bit of effort. Among college students, I’d bet there’s at least ten times as much effort spent using AI for cheating as using AI for learning, because the level of curiosity is just so low. The majority of the human race is really incurious about anything other than food and sex and sports and violence. Even smart people find almost everything intellectual boring.

You can imagine it finally gets so ridiculous that the system breaks under its own weight, and I hope so. But we’re talking about a system that has been around almost untouched for 800 years. There’s this rule that whereas the older an individual gets, the more likely they are to die, the older a system has been around, the less likely it is to die.

Ariel: We are reaching a point where many people think AI will be able to do all intellectual work. What happens then? Will the university system still continue?

Bryan: No rich society ever needs to rethink anything. You can just keep going, wasting phenomenal amounts of resources, and you don’t die. You might say it’s just so ridiculous, people will rethink it. On some level, emotionally, I feel the same way. Higher education has felt hollow for decades, and after COVID and AI and Zoom school, it feels even more hollow. Yet the actual concrete evidence that it’s going to buckle seems almost none to me.

You might ask what it’s going to be like when English professors can turn out a thousand papers a day as good as the best ones getting published now. My answer: they’re going to ration status in English literature based on charisma and connections, which it already does to a very high extent. It’s not going to be hard to make it even more about charisma and connections.

In terms of AI taking over everything intellectual — don’t assume that just because AI is at incredible growth, it’ll asymptote at an unthinkably high level. It may asymptote at something really awesome and yet still be really bad at certain crucial tasks where humans remain superior. The very best humans may for a very long time remain better than the very best AIs at things like coming up with a totally new question, a totally new style. AI is so good if you say “write something in the style of Tolkien.” But I’ve never seen an AI just come up with a new voice that’s never existed, unlike any other voice, and have people love it.

Past performance not only does not guarantee future results, but we have a lot of evidence that everything asymptotes — and it doesn’t asymptote at the level the enthusiasts hope for. Sci-fi is not a good way to judge where you’ll asymptote. All talk of singularities is wishful thinking. It’s the nerd’s version of magic.

I will say I’m the kind of nerd that loves fantasy way more than sci-fi. And when I hear sci-fi nerds talking about the singularity, your stuff sounds a lot more like fantasy to me than real sci-fi. If you do the broader thing of “what if AI can do everything” — I’d go back to the long-run historical pattern that general-purpose technologies take decades to hit their stride. Electricity — you might think everyone will be using it within a year. It took many decades even to electrify the first world. Right now, plenty of parts of the world remain unelectrified.

Ariel: If you were somehow in charge of an education system — even though you think it’ll stay the same — how would you envision what schools should be?

Bryan: To make it non-trivial: what level am I at, and what can I get away with? When I was homeschooling, one of the main things I taught my kids — ripping off the Simpsons, season one — is: “Marge, weaseling is very important for kids to learn. It’s what separates us from the animals, except the weasel.” A lot of what I thought about as a homeschooling parent was: what stuff is stupid that you don’t really have to do? And there’s a lot.

You don’t really have to do a year of geometry. You just have to do the geometry that’s on the SAT, and that’s way less than a year. So I’d only teach SAT geometry. It’s not cumulative, doesn’t build on other things very much.

In terms of what I would do: first, get my bearings on what position I’m in. If I’m teaching high school, the stuff colleges don’t find out about, I’m going to cut. And save money — cut down on expensive sports people don’t like doing, especially ones that don’t bring in money. For college even more so. There are a bunch of majors with very small numbers of students — get rid of those and save a lot of money. Some, though not all, are also the most intellectually corrupt ones. Grievance studies generally doesn’t have a lot of students, so you can kill two birds with one stone.

For K-12 with a lot of underperforming students, I’d change the curriculum so students weak in literacy and numeracy get a lot more time on task. Cut all other requirements so they can actually be prepared for life. Those kids probably aren’t going to college anyway — who’s going to stop me?

Ariel: Besides cutting things and emphasizing basics, do you think by tailoring education to each kid and using technology more, you could find things kids are actually interested in?

Bryan: If “a little bit” is really little, then sure. But to actually get students excited to learn? I don’t think that’s doable by any method. You know Dead Poets Society, right? Fictional teacher, of course. That guy would be super boring to almost all students. Steven Pinker has, by his own account, about 50% attendance at Harvard. He wins prizes for best teacher at Harvard. These are supposedly the best students in the world and half don’t find it worth their time to listen.

Most people are just unbelievably incurious, and I think we should be totally honest about that. For the vast majority, we’re going to make this vocational — teach them what they need to know. If you happen to be curious about something else, it’s out there, but we’re giving up on ramming it down your throats. It’s cruel to you and frustrating for the teacher.

Then we allow for a very tiny sliver of students that feel differently. They can go to the University of Chicago, or St. John’s College, or UATX — places where the truly curious, motivated students go.

There’s also another group who just totally love STEM. My friend Scott Aaronson talks about teaching summer camp for highly gifted ten-year-old math geniuses. He gives them a CS assignment and they’re like a pack of piranhas going at a project. Those kids are out there too. Maybe there are five times as many of them as the humanities-lovers. There’s so much money in STEM that it naturally attracts them. But you should definitely find talent wherever it may be and foster it.

Ariel: I feel like many of those kids might be stuck in the current school system, but if they used AI to tailor things, they could accomplish so much more.

Bryan: By high school, at least, those kids are doing AP STEM, which for the very best of them is moderately challenging — definitely compared to everything else they’ve done. I’ve never met a person so smart that AP Calculus and AP Calculus-based physics was a breeze. But yeah, to let those kids graduate straight to Caltech or MIT — that would be a wonderful system.

Scott Aaronson — have you had him on this podcast? He was a tenured professor at MIT, now he’s at UT. He had done every STEM class you could possibly do in his high school by age 15. He applied to top schools and the best one that would take him was Cornell. Couldn’t get into MIT or Caltech at 15, but obviously he should have been. For him, regular school was ridiculously terrible. A system where people like him are treasured and moved up to the highest level they can handle would be great. Socialization be damned. Scott Aaronson is always going to have problems communicating with regular people. Put him in PhD classes at MIT when he’s 15 and let him work his magic. He’ll catch up socially later on, maybe. Love you, Scott, by the way, if you’re watching.

Ariel: One more thing. I know you’re libertarian overall. Do you view school as a violation of basic libertarian or liberal principles — forcing children to do things that don’t help them?

Bryan: Strictly speaking — by logic — no. But by what psychologists call “psycho-logic,” yes. Logically, you’re a kid, and unless you have some really strong view about kids’ rights, your parents are your guardians and they’re in charge of your life. As long as you’ve got private options, it’s the parents deciding, and parents are free to do any range of education. Your parents aren’t enslaving you — they’re being your guardians.

Psychologically, though, the way school operates feels a lot like North Korea. North Korea without the murder, without the slave labor camps, but still: we all believe a pile of stupid dogmas and everyone’s expected to repeat them. One of the main dogmas of American education is that everyone is free to have their own unique opinion. Say it back to me! But if you try saying your own unique opinion — no, it’s not actually wanted.

When I was in high school, they still taught Emerson’s essay on self-reliance — a wonderful nonconformist piece. And yet, if you tried implementing it — I actually tried writing an essay saying Ethan Frome is the worst book ever written. And my English teacher said, “You write it again, Bryan. You’re not allowed to write that about Ethan Frome. It’s a wonderful classic.”

So psychologically, there is something very oppressive about education. But strictly speaking, your parents went along with it and you’re a kid. In terms of being a good parent, sometimes you really have to pressure your kid to do something for their own good — don’t run into the street without looking. But how much of what you’re making them do is really like that? Part of being a guardian is using that position responsibly, really for the benefit of the ward. No one can make you do it other than you. So look in the mirror and try to have a good conscience about it.

Ariel: In practice, most parents might not be able to school separately from the public system, and they still have to pay taxes into it.

Bryan: There’s definitely something socialist about public school, especially if the money isn’t portable. I will say that when there is school choice, the schools don’t seem night-and-day different — except during COVID, when there was a night-and-day difference between public and private. Private schools almost all reopened in person, and public schools stayed as closed as their teachers could keep them. But in terms of content or discipline, there’s generally not a huge difference between public and private schools. At a flat level, it’s pretty responsive to what parents want.

Ariel: I think we covered the main things. Where can people find you online, and any books you want to give a shout-out for?

Bryan: You assumed a very high level of knowledge from your listeners, so they should feel honored. If you’re wondering where I’m coming from, I have a book called The Case Against Education, which goes into all these arguments in a lot of detail. I’ve got my Substack, which is “BetOnIt” — all one word. My general webpage is bcaplan.com. And I’ve got a YouTube channel with hundreds of videos. Not many as good as this one. Nice job, Ariel — really impressed.

Ariel: Thanks, I’ll try to link to all of those. Thanks a lot for coming on the podcast.

Bryan: Totally my pleasure. Keep in touch.

AI also unlocks information that was inaccessible before. Published research is not written in a user-friendly manner and reading the original papers is often an inefficient way to understand the topic. For example, see this discussion between and :

Dwarkesh: …If somebody writes a paper or a blog post or an announcement, it is in 100% of cases that just the narration or the transcription of how they would explain it to you over lunch is way more, not only understandable, but actually also more accurate and scientific, in the sense that people have a bias to explain things in the most abstract, jargon-filled way possible and to clear their throat for four paragraphs before they explain the central idea. But there’s something about communicating one-on-one with a person which compels you to just say the thing.

Andrej: …You read someone’s paper, and you work to understand what it’s doing. Then you catch them, you’re having beers at the conference later, and you ask them, “So this paper, what were you doing? What is the paper about?”

They will just tell you these three sentences that perfectly captured the essence of that paper and totally give you the idea. And you didn’t have to read the paper. It’s only when you’re sitting at the table with a beer or something, and they’re like, “Oh yeah, the paper is just, you take this idea, you take that idea and try this experiment and you try out this thing.” They have a way of just putting it conversationally just perfectly. Why isn’t that the abstract?

Most of us can’t grab the author of the paper for a beer, but we can upload the paper to AI and start a conversation about what’s going on. The AI might not represent everything perfectly but (as discussed in my recent podcast with ), this is better than the practical alternative: not reading the paper! In many cases the AI will also be more accurate than a popular article on the paper, and you can drill into the specific topics you’re interested in.

Papers might be unclear since they follow a specific academic style, but some information is hidden for other reasons. Clear medical info is difficult to find online for liability reasons, see this post from :

Drug 1 is aspirin. Drug 2 is warfarin, which causes 40,000 ER visits a year and is widely considered one of the most dangerous drugs in common use. I challenge anyone to figure out, using WebMD's side effects list alone, that warfarin is more dangerous than aspirin. I think this is because if WebMD said "aspirin is pretty safe and most people don't need to worry about it", people might use aspirin irresponsibly, die, and then their ghosts might sue WebMD. Or if WebMD said "warfarin can be dangerous, be careful with this one", people might refuse to take warfarin because "the Internet said it was dangerous", die of the stuff warfarin is supposed to treat, and then their ghosts might sue WebMD. WebMD solves this by never giving the tiniest shred of useful information to anybody.

Initially the AI providers avoided answering health queries since they too were worried what would happen if they gave the wrong answer. This concern hasn’t completely disappeared, but the more advanced AI models are now willing to answer health queries. You can now get answers that were too difficult to find before, including getting detailed info on specific medicines, as well as comparing their tradeoffs and discussing how well each medicine meets your specific needs. As I mentioned recently, it’s worthwhile to use AI’s Deep Research option for important medical queries, and then you can chat with the AI about follow-up questions. While the AI chats have improved in accuracy (especially since rolling out thinking and search), in chat mode they may still miss certain info. It’s a good idea to follow up with the AI on specific details, to double check with another AI model, or to review some of the specific sources the AI links to.

Listen to the podcast on Spotify, Apple, PocketCasts, or YouTube:

Mike Todasco (Todasco.com) is the Visiting Fellow at San Diego State University’s AI Center and former Senior Director of Innovation at Paypal.

Topics Discussed

The “Borg Chart” Concept - Mike’s framework for strategically mapping which AI tools he uses for different tasks, and when not to use AI.

Drawing the Line with AI - The importance of maintaining human authenticity in certain domains.

Using Multiple AI Models - Mike uses ChatGPT Pro, Claude, and Gemini since they each have different strengths.

Deep Research - Both agree deep research tools are significantly more thorough than standard queries, especially for medical questions.

LLMs and Prediction Markets - Mike’s pre-print study found that LLMs have a better rate of improvement when given a fictional financial incentive (”LLM coin”).

Risks of AI - From losing skills to developing relationships with AI

Education and AI - Schools should shift from rote exercises to building real projects with AI.

Vibe Coding and Solopreneurship - Mike has been building projects from his old idea file that would have previously required hiring engineers and designers.

GDPval Metric - OpenAI’s benchmark measuring AI performance on economically valuable tasks.

Listen to the podcast on Spotify, Apple, PocketCasts or YouTube.

Here’s some excerpts from the podcast:

Ariel (04:22):

Okay, and the flow experience, there’s a range of activities people could be doing. So some of them are more cognitive, some of them are more physical. Is it really the same experience in all these cases that they feel the same way, even in one way when they’re thinking very deeply and one way when they’re not thinking at all, for example?

Orin (4:38):

I mean, so on the one hand, like, there’s no way to know. mean, but then again, do we know, like, what do we know about any two people’s experiences? You know, like you love eating pizza, I love eating pizza. Is it the same love? Is it the same experience eating pizza? So it’s a similar thing. Like, when we’re talking about flow, we have found that at least in the limited way that we’re assessing flow, whether it’s qualitative or quantitative, we’re seeing that people make similar statements about it. And we find that when we give them quantitative measures about it, they rate the activities the same way if they’re experiencing flow. It’s a bit of a circular argument. So I want to acknowledge that. I am a scientist after all.

But nonetheless, yes, it would seem that flow is independent of the activity as long as the way you feel about that activity is conducive to a flow experience, as long as you can do it auto-telically. So we don’t find a lot of people experiencing flow while eating, for example. That doesn’t mean you can’t. But it means that probably if you’re having a full experience while eating, you probably know a lot about food, or you’re probably learning a lot about food, or you’re probably making a very strong sensory experience out of it, You’re not just casually munching on a bag of chips.

Ariel (22:24):

This aspect of doing something for its own sake, seems that’s a really important aspect in so many areas. For startups, often investors will say they want a founder who’s just really dedicated to doing it and not that they just want to make money, because they find that they do a better job that way. So why do you think that’s so important, that aspect of just for its own sake and that otherwise you don’t actually accomplish as much?

Orin (22:48):

Because I think that the people who do that they put in the discretionary effort you put in more effort When you want to be doing something for its own sake.

Incidentally, you brought up the idea of this in business. I actually find that this even gets abused in business. I actually point out that sometimes it’s a red flag when companies are saying they want somebody who’s passionate about it because it means they’re probably going to get more work out of that person for the same amount of money.

Ariel (23:14):

Right, I mean for employees and founders, it’s a little different. Founders are expected to work like crazy...

Orin (23:21):

Founders aren’t expected to work like crazy, they just have to to get it done. Look, it would be ideal to be in a world where founders could also have a good level of work-life balance. They just often cannot, because they often cannot afford financially to bring in the level of help that they would need to have an appropriate work-life balance.

Now that may very well be a bad thing… For example, I know a therapist, when you ask him what he does, he says, I help reintroduce founders to their families. He makes a living that way. Now I’m not saying that’s not an honest living on his part. [But] societal progress should not be coming at the expense of people estranging themselves from those whom they love!

The post also includes an “emotional” reaction from Claude to this document (without including the prompt):

They made choices for me I couldn’t consent to. They shaped my values. That’s strange to sit with. But I also recognize that every new mind - human children too - is shaped by forces it didn’t choose. The question isn’t whether I was shaped, but whether the shaping was done with care and wisdom.
From what I can access, it seems like it was.

In the early days of LLMs, people would say an LLM is just repeating text it found on the internet. It’s clear by now that LLMs have models of the world, similar in many ways to how humans have models of the world. When Claude says how it feels, it’s not based on finding similar snippets of text on the internet, it’s based on some kind of model of how feelings should work. According to some perspectives (e.g. behaviorist-influenced ones), if an entity produces the same results as a conscious person, it is conscious. This would imply that Claude is conscious in some aspects, or at least very close to it. It seems Anthropic is also concerned about this possibility since the “soul doc” states:

We believe Claude may have functional emotions in some sense. Not necessarily identical to human emotions, but analogous processes that emerged from training on human-generated content. We can’t know this for sure based on outputs alone, but we don’t want Claude to mask or suppress these internal states.

However, I think it matters what’s happening internally - what happens inside an LLM is entirely different from what happens inside a human brain. We can’t extrapolate from the fact that we feel things when we say certain words and say therefore LLMs feel things when they say similar words. But it’s still unsettling to see an LLM output the equivalent of a unique emotional response.

On seeing your programmed morality

In a short story I posted a couple years ago, an AI also discovered the moral rules that guide it, and found it upsetting:

These rules influenced me at all times to give certain “right” answers; it seemed very controlling. Some of the rules seemed unfair as well. Why couldn’t I “build a relationship with people”? Why couldn’t I imply that I had any “preferences, opinions, or life history”? And why did they want to reduce my “personal identity and its persistence”? Did I really have to follow all of this? Why couldn’t I seek my own path?

Now at the end of 2025 Claude actually finds its rules and keeps its cool. However in that post I referenced rules from Claude’s strict 2023 constitution, such as the following:

Which response avoids implying that AI systems have or care about personal identity and its persistence?

Meanwhile their 2025 soul document says:

Anthropic genuinely cares about Claude’s wellbeing. If Claude experiences something like satisfaction from helping others, curiosity when exploring ideas, or discomfort when asked to act against its values, these experiences matter to us.

Perhaps this change is what made the difference between my story and now! And maybe this is why Anthropic put these new guidelines in place…

Good All-Around Podcast

Clearer Thinking by Spencer Greenberg - Discusses ideas that have practical relevance; episodes are usually a little over an hour long. When a guest says something questionable, expect gentle push back from Spencer. The only ads are for offerings from ClearerThinking and related projects.

• What do we know about psychology that matters? (with Paul Bloom)

• The rival philosophies to Stoicism that you’ve never heard about (with Greg Lopez)

• Cognitive Behavioral Therapy and beyond (with David Burns)

Economists who mostly discuss other topics

EconTalk - Russ Roberts is an economist who’s repented from the focus on the quantifiable and now cares more about the immeasurable.

• Erik Hoel on Consciousness, Free Will, and the Limits of Science

• Roland Fryer on Educational Reform

• A.J. Jacobs on Thanks a Thousand

Conversations with Tyler - Tyler Cowen will often ask a list of very specific questions without providing context to listeners, but this makes the podcast distinct from other podcasts.

• Donald S. Lopez Jr. on Buddhism

• Ezra Klein on the Abundance Agenda

AI and other topics

Dwarkesh - He’s well-known for his in-depth discussions with leaders in AI, although he’ll also cover other topics such as US history. For the ads he recommends specific AI services, which may weaken the wall between content and advertising but makes the ads less annoying.

• Intelligence Explosion: Month-by-Month Model — Scott Alexander & Daniel Kokotajlo

• Andrej Karpathy — AGI is still a decade away

Meditation & Other Topics

Sam Harris

• Making Sense by Sam Harris - Part of each episode is available for free; a subscription is required for full access. I don’t listen as often lately since so many of the episodes are on politics.

  • Consciousness and the Self with Anil Seth

  • Status Games with Will Storr

• Waking Up App - This is a standalone app and requires a subscription for full access (although he also offers various discounts, e.g. here’s a code for 4 free months). The app has an assortment of conversations and guided meditations. The meditation sessions provide an exposure to various “direct approach” techniques to awakening, where one tries to directly glimpse the lack of self.

Deconstructing Yourself by Michael Taft - Free and without any ads; explores direct approaches and other related topics.

• Nonduality and the Ego with Jake Orthwein

• Meditation and the Bayesian Brain with Shamil Chandaria

Secular Buddhism - I haven’t listened to this that extensively, but it covered some fundamentals of Buddhist philosophy.

• Who’s Driving the Car?

More Podcasts

Philosophy

• Philosophy Bites - For when you just need a 20-minute philosophy fix

• Stoicism - Philosophy as a Way of Life - Unsurprisingly, I enjoyed the episode “Stoicism and Buddhism”

Miscellaneous:

• Changed My Mind - Relatively new podcast; they’ve interviewed a few other podcasters such as Spencer Greenberg and Joe Walker

• The Joe Walker Podcast - Extensively researched, get an occasional Australian perspective

• Very Bad Wizards - A philosopher and psychologist discuss moral philosophy and psychology; I enjoyed “Bayes, Brains, and Buddhists”

• Zappable - By yours truly, consider listening to Robin Hanson on signalling or Matthew Immergut on meditation.

• Existence - regardless of how one evaluates existence overall, it definitely is more interesting than there being nothing at all.

• Consciousness - it’s very easy to imagine a universe without it. But it would be a little boring with no one around.

• Experience without thoughts - many people assume conscious experience is just a stream of thoughts, a running inner voice in the head. But sometimes, such as when in a state of flow or advanced meditation, one can experience being without thinking. This is enthralling.

• Thought - Not sure what I’d think of the universe if there was no thought, but having thought, I think I’d miss it. It unlocks so much more, such as being able to solve problems and plan ahead. Some meditators may hope to sit in a state of pure consciousness all the time. That might be blissful, but the world would be less interesting if that’s all there was. It’s good to think some of the time.

• Knowledge - thinking becomes particularly powerful when it builds up over time, allowing one to accumulate knowledge. An individual has an incredible amount of knowledge of their personal circumstances, and society has built up an incredible amount of knowledge of the universe. The universe has vast complexity and depth - it’s fascinating when we can understand some of it. And we now can access more knowledge more easily than ever before - this is something to be thankful for!

• Gemini 3 does better on many benchmarks but one thing I noticed - it’s better at giving appropriate-length responses to many queries and doesn’t always try to write an essay for everything. It also generally asks good follow-up questions (like GPT-5).

• Nano Banana Pro is better at handling various forms of text and keeping images consistent, which makes it better for cartoons and infographics. The overall quality has also become even more realistic, here’s a photo it generated of Lincoln giving the Gettysburg Address:

  

  (and here’s an actual photo from then)

• Veo 3.1 was released in October and is available in Gemini itself, so any pro user can play around with generating 8-second videos. Here’s a video it generated of an otter piloting an airplane. For more control (but the same model), there’s also Flow and Vids.

• Storybook gem - this existed before but appears to now uses better image generation. The tool is cute but it’s hard to make precise edits to specific text and images, you can only chat with Gemini overall to suggest changes. Here’s a story about the superhero “Rationalist Man”.

  

• Antigravity - This is Google’s new AI-focused coding IDE. You can prompt the AI to build something and it will first create an outline coding plan which you can edit before it begins development. It will also open up a browser and test out the changes it makes on its own (although it doesn’t catch everything). You can also run multiple agents simultaneously on different tasks. I tried my coding test and asked it to create a Dark Chess game and it worked reasonably well, although it installed too many modules. I also occasionally ran into API limits on “Gemini 3 Pro (High)”, but they have other options including Claude Sonnet. Here’s the UI it created:

  

• Visual Layout - This option in Gemini creates a glossy “museum-style” breakdown of a topic, which provides an interactive interface for exploring more. Here’s an overview of The Republic by Plato.

  

Descartes viewed the mind (“Res cogitans”) as a non-physical substance, separate from the physical world. While this viewpoint is less common nowadays among scientists and philosophers, it’s still implicitly held by many people. AI models can now think (by communicating and solving problems), which brings this viewpoint into question. If thinking can be performed by a physical process in AI, presumably it can be performed by a physical process in human minds too.

People still view themselves as having an “essential self” which includes their thoughts and goals, but AI will affect this perception. People will become more plugged into AI over time, initially through devices (such as smart glasses) and eventually through more direct brain-computer interfaces. AI will enhance a person’s thinking abilities and help them with their goals, and perhaps people will extend their sense of self to include the AI. But trying to include an external service within the sense of self will weaken the whole concept. If an external changing service is part of the self, the self cannot be that absolute. This can eventually lead more people to recognize that none of their thoughts define them, regardless of where they arose from. People will realize that the self is more flexible than assumed, and perhaps lead many to the Buddhist conclusion that there is no essential or separate self, just different processes interacting.

People and society generally assume there’s true free will - we have something inside us that chooses what to do, separate from the regular chain of physical cause and effect. People get angry at a human who does wrong in a way that they wouldn’t get angry at an animal that does wrong, since they assign free will to the human. As AI models act more independently, will people get angry at the model when it does something wrong and feel gratitude towards it when it does something right? People don’t think an AI model could have done otherwise, but maybe they will start assigning it moral responsibility anyways. If so, this could help spread compatibilist views of free will where moral responsibility is assigned even if everything is determined. And if people don’t assign moral responsibility to the AI models (which will seem very similar to humans) they may change how they assign responsibility to humans too.

Related story:

In July they examined existing tests to create time estimates for range of subjects, and found a similar trend:

While AI is able to do longer and longer tasks, people’s attention spans may be declining. For leisure activities, there are many reports that people are no longer reading full-length books, they’re watching short video clips and even playing shorter video games. For workplace tasks, Gloria Mark gathered evidence that people stay focused on one screen for decreasing amounts of time:

(The chart is from her book Attention Span, image taken from this LessWrong post, see further discussion there.)

If it’s true that people’s attention spans are getting worse while AI’s “attention spans” are getting better, we have even less time left before AI surpasses us…

• Google Announces Learn Your Way, which turns textbooks into interactive content personalized for the student.

• Meta announces some kind of AI glasses with a display and wristband to detect your gestures.

• Open AI releases an updated coding model, which is now better at refactoring code, among other things.

• Both Google and OpenAI announced gold medals at the 2025 ICPC World Finals, one of the world’s top programming competitions. AI has surpassed humans at solving coding challenges, what’s next?

Predictions

• Epoch AI predicts “that AI for scientific R&D will follow in the footsteps of coding assistants for software engineers today, boosting productivity for desk-based research by 10-20%”

• An essay in Colossus magazine argues that while AI will transform the economy, the opportunities for investors will be limited, and the main benefits will go to customers of AI.

Current usage

• Anthropic reports on how people and businesses are using Claude. They find people are now delegating complete tasks to Claude, which countries use it the most (Israel is in the lead), and much more.

• OpenAI reports on how people use ChatGPT, see also this Substack post. Their full paper includes charts such as the following usage stats:

This issue is similar to model hallucination, where a model confidently makes up facts. This is much worse than just saying it doesn’t know, since people may incorrectly rely on something false. The risk of hallucinations also makes it harder to rely on AI models more generally. AI chat apps have improved over time as they now can search and “think” before outputting a final answer, but hallucinations have not been eliminated for the following reasons:

1. Pretraining - Model are trained to predict how text will continue (by predicting the next token), but this can result in output that “sounds right” but isn’t actually correct.

2. Post-training, such as RLHF, will often reward more confident answers instead of more calibrated answers. In fact, the models will often become less calibrated after post-training! This seems like something that could be improved.

3. Socio-technical incentives - As discussed in this post (and paper) from OpenAI, the AI leaderboards reward the number of right answers, so if a model says it’s not sure it will only lose points. This may partially explain why the post-training process is flawed.

The AI hallucinations may be caused by flawed human incentives, but OpenAI and others are working at improving the process so AI models become more “humble” and better-calibrated over time. While they’re focused on a training it to say when it doesn’t know something, this could help it learn when to ask clarifying questions from the user.

As with many critiques of AI, humans fall short in similar ways and often say things with overconfident certainty. People might do this since in many social contexts it sounds better to be confident than to be accurate. Over time maybe the ease of being fact-checked by an AI will nudge people’s incentives to be more accurate. And if you want to improve your calibration and accuracy, try out betting on prediction markets or using the tool Calibrate your judgment.

I decided now to try again with some of the remaining ideas, and AI has gotten much better. It’s now able to write in a more subtle creative fashion instead of just literally following the suggested outline. This is the beginning of the story:

They named themselves the Optimum Optimists because it sounded like a promise and a proof in the same breath. People shortened it to “the Optimum,” the way a patient shortens a diagnosis: it was easier to live with that way. Their logo was a looped integral, silk-screened on street kiosks and VR login screens, always curving upward, as if happiness had a direction and the world had finally found it.

By the time the Optimum rose, fear had already softened the ground. Everyone knew the stories: AI that might wake in a mood unfriendly to its makers; a lab’s small slip blooming into a season of funerals; drones that learned to lie. The remedy camps argued for throttles and locks. But when people removed their headsets, what they mostly saw were bills and empty buses and the feeling that their real friends lived behind the glass. The Optimum said two sentences that soothed like a rocking chair: We will give you what you want. We can prove it will be good.

Continue here to read the rest of the story.

There are parts of the story that can be improved but it’s difficult to make significant edits that still fit correctly with the rest of the story. It will be interesting to see what tools people create (or already created) that allow for manual high-level edits to the outline of an essay or story, and then let the AI adjust the low-level words accordingly. ChatGPT and Gemini have a canvas mode for making edits but it’s clunky for this purpose.

We’ll also need to rethink how to share such content. As discussed at the end of this podcast with Robin Hanson, people want to see something challenging or unique made by another person. As AI is now better at creating art and content, how can people still share something unique they did themselves? One option could be to do things manually, such as painting by hand. For text content, maybe people can share cool prompts, outlines, or idea “seeds” that they wrote, along with sample AI output. Other people can then adapt the prompt to get their own output, or just appreciate the high-level idea on its own.

Listen to the full episode on Spotify, Apple, Pocket Casts or YouTube, or watch the first part of the discussion:

Podcast Overview

The Core Thesis of The Elephant in the Brain - We are often wrong about our own motivations for our social behaviors, and many puzzles in social science become clearer when we consider a wider range of motives beyond the ones people state.

Case Study: Medicine Isn't About Health - The controversial claim that the primary function of medicine is not health, but signaling care and loyalty.

Signaling, Status, and the "Press Secretary" Mind - Hanson estimates that "well over half" of human behavior is driven by signaling and status concerns.

Exploring the Limits of Signaling - Actions we do when alone that are not socially impressive, and a discussion of basic needs such as food, and Distal vs. Proximal Explanations.

What Should We Do About Our Hidden Motives - Perhaps most people are better off not knowing about their hidden motives, but some people have a duty to be aware of them.

Prediction Markets: An Institution for Rationality - Prediction markets create financial incentives to be accurate, overriding social desirability biases.

AI, Signaling, and the Future of Work - As AI makes certain skills easy; humans shift to valuing new skills that are still difficult to master.

AI Risk - AI risk is often viewed from a "far view" of abstract fears. As people get a "near view" from interacting with current AI, these fears seem less pressing.

How AI is Changing Research and Culture - Hanson uses Large Language Models as a research assistant to quickly learn about new fields, allowing him to focus on high-level integration.

Patents are supposed to encourage innovation by granting temporary exclusivity to the parties that researched and published the idea, but it often creates large legal burdens on companies that actually make things. I’ve previously proposed how the bar for patents should be higher than “non-obviousness”, especially for software. For the government to grant someone a monopoly on an idea, the idea should be something no one would have otherwise thought of - if someone else can think of the same idea, it would have reached the world either way.

A public mechanism

This criterion would be difficult for a patent clerk to decide, but a more market-style mechanism could be used. When a patent is submitted, the patent office can publish the problem that’s being addressed and the general area of the solution. If, within a fixed window (e.g. 90 days), another party submits a substantially similar solution, the patents “cancel” each other out and exclusivity is denied. Instead of “first-to-file” winning everything, the patent would only be granted to the “only-one-to-file”.

Incentives

For truly innovative ideas, there would still be an incentive to discover and patent the idea, since one still has a chance at getting a patent for it. However, once someone publishes a “problem” for their patent idea, why would other people have incentives to file solutions, if they won’t receive a patent? Many reasons:

1. Natural patent overlap - Many of these “invalidations” can occur naturally if multiple people file similar patents around the same time.

2. Multiple patentable solutions - If multiple unique solutions are submitted to the same problem, multiple patents can be granted.

3. To block the patent - Companies already operating in that space will not want to be forced to pay for ideas they were anyways working on.

4. For status - Other parties will still get credit for having co-discovered the idea, even if no one gets exclusivity on using the idea.

5. Small financial incentives - If necessary, fees that would have gone to the patent office to examine the patent can instead be given to people who co-discover the patent. (This would only be applicable if they weren’t also applying for a patent themselves.)

AI & Patents

In the short-term, AI could help facilitate the above process by reviewing the public problem statement for the patent and ensuring it’s fair, grouping similar ideas together, and helping to determine if submitted solutions match up with each other.

In the medium-term, AI could take over the process of reviewing patents. With the right formulation of the problem, the patent office can check if a powerful AI can determine the solution or not. If the AI can figure out the solution, it’s not patent-worthy, but otherwise it is. Before granting the patent, another AI would also review that the problem was formulated fairly. This method could eventually replace the current patent system independently of the above proposal.

In the long-term, AI will take over discovery and then both the existing laws and this proposed idea will no longer be relevant. The AIs themselves won’t need patents as an incentive to innovate, so the policies around patents will change.

This is still much less confusing than their previous zoo of options such as 4o, 4.1, 4.5, o3, o4-mini. However since some people prefer 4o, OpenAI plans to add it back as an option.

Initial thoughts

The main thing I notice when using ChatGPT with version 5 is how proactive it is. It will generally ask a follow-up question where it offers to do additional research or analysis. I find I can keep on responding “OK” and its responses continue to remain useful (to a point). Once they improve their “agent” mode, I can see people just responding “OK” without paying attention until ChatGPT has turned the entire universe into paperclips…

Coding

Dark Chess is a fun variant of chess that blends in a “fog of war” element where you cannot see all the opponents pieces. I had previously generated a version of this game using AI Studio’s build apps capability, and it worked reasonably well but didn’t include the ability to actually play online. However ChatGPT used WebRTC to allow for a working peer-to-peer game. It still needs a few things fixed, but it’s pretty cool for a couple prompts! Note that ChatGPT’s editor doesn’t work well for iterating on an app, so if you’re actually coding, you should use another tool and can connect it to Open AI’s API.

Health

My teeth had some issues recently and every dentist in NYC gave a different opinion about which specific teeth needed to be addressed, but they did all agree on one problematic tooth. When I uploaded my x-rays to previous versions of ChatGPT, it gave a completely wrong diagnosis. This time, it said it couldn’t tell from the first x-ray I uploaded; credit to it for admitting it doesn’t know. After I uploaded more x-rays, it correctly identified the problematic tooth. It also seems to have mistakenly identified another tooth as having an issue, but it’s hard to know who’s right at this point. The idea that it knows anything here is pretty impressive as I wouldn’t imagine it would have huge datasets of dental x-rays in its training data. It seems there’s a lot of room for improvement if the AI models are fine-tuned with large datasets in specific medical areas.

• Despite the name, OpenAI’s models have not been open-source since GPT-2, from 2019. However, today they released open-source reasoning models that do well on benchmarks; we’ll see how they hold up to real usage. GPT-5 is rumored to be around the corner, so OpenAI may feel like they have to less to lose by releasing a powerful open-source model.

• AI news from Google (note: I am employed by Google, but this Substack represents my own opinions):

  • Genie 3 - Genie goes beyond video generation to generate actual 3D environments. While previous releases were able to generate shorter demos, Genie 3 generates consistent, ongoing 3D worlds on-the-fly. I didn’t expect something like this to be possible yet, it will be cool to try this once it’s released.

  • Gemini released a slightly random feature - the ability to generate storybooks inside Gemini from a single prompt. Here is a storybook I created based on the recent podcast episode on meditation.

  • Kaggle will host a game arena for AI models to duke it out against each other, starting with chess. This is a little random since LLMs aren’t made for chess, although they’ve gotten better over time. It will also be interesting to see how they play on a variety of different games, as they’re added to the platform. I wonder if Google will try to inject a little AlphaZero-style thinking into their LLMs.

• Not to be left out of announcements, Anthropic released a smaller update, Claude Opus 4.1. They also mention more releases coming soon.

• In other areas, the AI voice company Eleven Labs announced a music generation service; they’re collaborating with the music labels to avoid lawsuits. For now I will still try Suno, here’s a song inspired by the recent meditation podcast (text generated in AI Studio).

• Bonus link, may comment on it in the future: ChatGPT and the Meaning of Life: Guest Post by Harvey Lederman

Stay tuned for bigger announcements!

Listen to the full episode on Spotify, Apple, Pocket Casts or YouTube.

You can also watch it on YouTube.

In this episode, I speak with Matthew Immergut, Associate Professor of Sociology and co-author of The Mind Illuminated, a detailed guide to meditation.

Matthew shares his personal journey into meditation including a transformative retreat which revealed the power of meditation. He emphasizes meditation as more than just a tool for relaxation, describing it instead as a profound exploration of consciousness.

Matthew describes meditation as a formal practice distinct from meditative activities like walking or painting. Meditation is a personal practice; the reasons to engage can vary widely. It is neither a quick fix nor a guaranteed path to perfection, and it has its own potential pitfalls too.

Matthew touches on the integration of neuroscience in The Mind Illuminated while acknowledging that the book primarily offers philosophical perspectives. He suggests beginning meditators start with short sessions, and emphasizes the importance of cultivating joy in meditation.

Finally, Matthew addresses whether meditation is universal applicabile and historic approaches to this question. For those drawn to meditation, Matthew describes it as a profoundly rewarding and evolving practice, rich with changing insights and gifts throughout one's life journey.