Locks, Loopholes, and Language Models
Podcast Episode with Computer Scientist and Legal Scholar Shaanan Cohney
In this episode I speak to to Shaanan Cohney about what lockpicking teaches us about software, whether AI-generated music is good, why 90% of his intro programming students are now submitting AI-written code, why the evidence on social media and teen mental health is weaker than the headlines suggest, what happens when vulnerability discovery gets commoditized, and how to stay safe online.
Check it out on Spotify, Apple or Youtube.
Transcript
Below is a lightly edited transcript, courtesy of AI:
Ariel (00:05) Joining me today is Dr. Shaanan Cohney, a computer scientist, legal scholar, and senior lecturer at the University of Melbourne. He’s won awards in research, teaching, and hacking, and has advised both the FTC and the US Senate on tech policy. Shaanan also has experience cracking safes, singing competitively, and playing piano. Shaanan, welcome to the Zappable Podcast.
Shaanan Thanks, it’s lovely to be here this morning.
Ariel So Shaanan, since you have experience in both music and computer science, is there anything in common that draws you to both of them?
Shaanan Yeah, and I think some of the connections actually run pretty deep. Everyone talks about how music is very mathematical, but not often enough do people explicate what they mean by that.
There are a couple of popular music theoreticians on YouTube who go into this a bit more. One I came across a couple weeks ago charted out a beautiful graph of what music naturally is going to sound good to human ears. And one of the things there is that depending on the environment in which you’re raised, different music sounds good. If you’re raised in Bali, gong music might sound really great to you — that operates on a different tuning system. If you’re raised in the West, classical Western music has a certain sound to it. These are all built on mathematical relationships, and depending on the country you grow up in, or the musical culture, this is going to be different.
But it turns out there are these super interesting mathematical relationships between the music that’s likely to sound good and the music that’s not. And you can do these music-plus-code visualizations that are really awesome. There was a 1993 paper by William Sethares — I’m not sure how to pronounce his name — around the relationship between psychoacoustics, which is the study of how humans perceive music, and the mathematical nature of music. For me, that ties very nicely into an analytical way of viewing the world.
At the same time, there’s the flip side of music, which is what most people think about: the more emotional, cultural, social experience of it. And I think code has something similar to that as well, which we often lose sight of — software development is also a social practice, or at least it has been up until recently.
Everything from the early days of Unix hackers working together over bulletin boards, to the notion of pair programming — there’s a sense in which coding is a craft and an art. There’s Donald Knuth’s famous book, The Art of Computer Programming. So there are interesting parallels between the way programmers approach the craft of writing code and the way musicians think about the craft of writing music.
Now that’s breaking down on both sides a little bit with AI, but those links are something I appreciate. Even when I’m actually doing music, you try to get into a flow state, and just like with code, when you’re in the flow state you’re not necessarily thinking about all of those aspects at the same time. So for me, they’re two sides of a similar coin.
Ariel Interesting. And since you mentioned how AI is affecting things — do you think AI-generated music is valid music? What are your thoughts about that?
Shaanan This is where it becomes really important to separate out music as a product from music as something more social, more artistic, more craftworthy. There was a beautiful video essay by Adam Neely, a popular online music theoretician, called “Suno, AI Music, and the Bad Future.” He brought in a number of sources, one of which is one of my favorite ways to think about music: musicking, an idea from Christopher Small, an ethnomusicologist. That’s the idea that music is more than an output — which is how we’ve been treating it for the last century with recorded music. Music is a social practice. That’s everything from listening to music, where you have to be actively engaged, relating it to your own culture, often listening with other people — there’s a lot of social context there — and also music creation and music performance.
Up until the last century or two, a lot of people’s experience of music was either going to the big concert hall when someone came into town, or much more commonly, making music in the community. Whether that’s liturgical music in your church or synagogue or mosque, or just banging on things while singing, or old slave songs — all of these were cultural practices.
What’s happened, particularly in the last 50 or 60 years, is that link has been increasingly broken, and people see music much more as the product rather than as the social and cultural experience. Some of this AI-generated music takes that to the far extreme. Is it music in the sense of music-the-product? Sure. But I think something important is missing when you take away the cultural and social experience and the element of craft. There’s also a lot more I could say about the process of creation, if that’s interesting to you and your audience.
Ariel Right. I guess if people want something to share with their friends, or to go to a concert together, they want something more unique — they want to know that people struggled to make this song. But if they’re just listening to music in the background while they’re running, then probably they don’t care as much, and AI-generated music could easily meet that need.
Shaanan It could. A lot of AI-generated music, at least to my taste, is just bad — and I mean that from a somewhat subjective but critical lens. If you look at the construction of AI music, there are often things missing. The lyrics can be very repetitive and not have actual depth while having the semblance of depth. That’s not because in theory AI could never generate meaningful lyrics, but from an objective standpoint, today the lyrics are rather trite and the rhyming structures are fairly simple.
One of the reasons people really like Eminem is not because the harmonic construction of the music is that amazing, but because he does an amazing job with rhyme and lyric. Even in just that one small aspect of AI music, it’s really clear if you pay attention for more than a few seconds that it’s not there. So I find it annoying to listen to.
But that is in some sense a matter of aesthetic preference. Perhaps because I’ve trained in music, I have more of a gut reaction. Do I think some people could obtain some utility from it? Sure. But at least for now, something is missing — both on the philosophical, social side, and on the question of what the actual product in front of us is.
Ariel On the technical level, I feel like they’ll get there soon enough, or maybe people just need to work on their prompting a little bit, because they’ve gotten so good at all these kinds of areas. Maybe we’ll see in a few months.
Shaanan Maybe. There was AI-generated Bach, which was pre-generative AI — purpose-built models designed to mimic the harmonic structures of Johann Sebastian Bach’s music, in particular his fugues, to the point where experts couldn’t really tell whether the composition was Bach or not.
But there still seems to be something missing, at least with the generative tools we’re creating today. How soon we get there, I don’t know. Will I be able to tell at some point? I don’t know.
Ariel (07:47) So turning to another topic: you have extensive experience with software security, but also a little bit on the physical side, even breaking into locks. Is there anything in common between those two areas, or are they just separate areas of interest for you?
Shaanan This goes back to a fairly deep connection between physical security and computer security. I wouldn’t even say I do software security, because to me that’s looking at far too narrow a part of the system. The way a really good red team, or a really good security professional, looks at things is as an entire system.
Locks are a really nice example of this, because you can look at just a very specific element — the theoretical construction of the lock and the matching to a specific key. There’s a certain key space depending on how many pins there are; you can figure out how many different possible keys there are, and that’s all well and good. But to actually build this thing, to instantiate it in the real world, there are a lot of considerations you need to take into account. And this is where lock pickers think about locks in a similar way to the way computer security people think about computer systems.
When you make a real lock, there are machining tolerances, and those come into play when you’re lock picking. You’re relying on the fact that not only are certain combinations of pins ineffective in a real-world lock, but the quality of the machining of the lock itself leaves imperfections you can exploit as a lock picker.
Now, a really good physical security person also doesn’t stop at the lock. The lock is within a door, the door has a latch mechanism. Perhaps you don’t actually have to pick the lock — you can bypass it by toying with the latch. Or if you’re a common criminal, you go through the window and ignore the lock in the first place. These same parallels carry over quite closely to computer security. We talk a lot about attacker-defender asymmetry: the defender needs to ensure the entire system is secure, whereas the attacker only needs to find the weakest point. And that weakest point, in our lock analogy, might be the pins, it might be the pinning, it might be the jamb of the door, it might be the window that’s been left open. So that parallel is a really good way to teach people how to apply systems thinking.
There are also other things we can learn from locks and lockpicking. Firstly, there’s a cultural aspect from the ‘70s and ‘80s around the way hackers thought about experimentation and gaining knowledge. The same tools hackers applied to locks reflect the exact same philosophy they applied to early computer systems.
There was great work by one of my mentors, Matt Blaze, on master keying — the notion that in a large corporate building you can have a single key that unlocks every door, called the master key. Then you can also have subkeys that unlock a subset of doors, or maybe even only a single door. In Matt Blaze’s paper, he demonstrated a physical privilege escalation attack. Privilege escalation is when you have some normal level of access and can use it to gain administrator or privileged access. He showed that even with a normal subkey on a master-keyed system, you can escalate that to get the actual master key on certain kinds of master-keyed systems.
He copped a lot of flack for this, because the locksmiths’ guilds were not happy that this information was now public, and he received threats of all sorts, including death threats. But this parallels what people in computer security say: you shouldn’t rely on the secrecy of the design of a system in order to protect the system itself. This is Kerckhoffs’s principle. So this was a real physical analog of a principle that applies very directly in digital systems.
You’ve got all of these different individual lessons you can learn. You learn about the culture of hacking, you learn about particular types of attacks and ways of thinking about things, you learn about the principles. So I think it’s a skill every computer security person should try their hand at, at least for a few minutes.
Ariel Have you ever used your lockpicking skills in practice in the real world?
Shaanan Within the past two weeks, yes — but no comment. This gets me into trouble quite a lot. I’m very good at losing keys, forgetting keys, locking myself out of buildings, and so on. So having skills in physical security has come in handy more than once.
The most prominent of these was when I was a graduate student and I locked myself out of the dorm. At that point it wasn’t a lockpicking attack, but similar physical security — I came up with a way of getting back into the building by bypassing the security systems on the dorms, and that resulted in more than a little trouble for me, shall we say. But it has proven a useful practical skill lots of times as well.
Ariel So usually you’re breaking into areas that you should have access to, you’re just missing your key.
Shaanan Exactly. That’s how I stay on the right side of the law. I break into things, but they’re things I should have access to and for some reason don’t.
Ariel One interesting difference between physical security and software security: with software security, once you’re on the internet, if there’s a vulnerability, something will find it. You’re open to the whole world. Whereas with physical security, there aren’t so many people trying to physically break into buildings, so once you reach a certain level, it’s usually good enough.
Shaanan I think that’s true to a certain extent of software security, but this tells us a bit about the role locks actually play in life. Picking a lock is not actually that hard. In particular, the locks on most people’s houses — when you imagine the American white picket fence house with the front door — those locks are typically not very secure. It would likely take me under 30 seconds to bypass or pick any given house lock.
But the point isn’t to stop people from getting in with some high security mechanism. It’s a social deterrent. It’s just like the way you might leave your bedroom door closed when you have friends over to an apartment. It’s not that that provides any great deal of security, or that if they looked through the door they’d see anything so terrible — depending on how messy your bedroom is. It’s more a social sign about where the boundary is being set. If someone wants to break into your house, they can, most likely by tailgating someone through the security door, kicking the door down, or bashing in a window. It’s more that it adds a little bit of friction and a little bit of social signaling.
This does happen in software security as well — not quite as much with the signaling, but to the extent that adding a bit of friction makes you a slightly less soft target, so you’re no longer a target of opportunity. A lot of crime — not all of it, there certainly are more sophisticated attacks — happens in moments of opportunity, where someone sees that a door has been left open, or a car has its lights on and has been left outside with the key still in the ignition. So the function of both your firewall and your basic security hygiene, and the lock on your home, is not necessarily to produce a technical impediment that a nation state couldn’t bypass. It’s more to provide that little bit of friction and that little bit of social signal.
Ariel (15:23) Now turning to your experience teaching computer science. How have you noticed it’s changed recently, as generative AI has become so common and powerful?
Shaanan This is the doom and gloom part. Most of the people teaching introductory classes — and I count myself among them — are in a state of mild depression and slight panic, except the people who are close enough to retiring that they don’t have to worry.
What we’ve seen is that the software tasks we would traditionally give in an intro programming class — go home, work on this assignment for two weeks, skill up a little, learn basic programming paradigms and abstraction — are no longer actually completed by our students. Or to the extent that they are, it’s a student prompting an agent once and then copying and pasting the output into our submission system. One of my TAs estimated that somewhere between 70 and 90% of the submissions he graded had clear hallmarks of AI use.
To give you a sense of how he picked up on that: I teach an introduction to C programming — data structures and algorithms through the lens of C. In a typical year, we’d teach some of the more interesting C keywords, stuff like static and volatile. I might mention those, but I’d almost never see a student use them in an assignment, because that level of sophistication usually comes with experience — knowing about variable lifetimes in enough detail to think about how to manage them. Typically students would have messy variable names and imperfect structure. And yet assignment after assignment used sophisticated constructs and paradigms in C that I’d never seen previously in my career. That’s what my TA picked up on too, and it flags to us that the students are no longer completing the work.
So there’s frustration on that side, because as much as we communicate that the only way to develop these skills is by doing the work, students for whatever reason don’t do it. And they all have legitimate reasons: life is stressful, this is more effective, am I ever going to actually need to program? But I do still think we’re missing something by not teaching them how to think like a computer scientist.
Then there’s the other view, which is: well, maybe thinking like a computer scientist needs to change. Maybe you don’t need to think about code itself, because I think all of us are slowly coming around to the reality — some faster than others — that humans probably won’t be needed to write very much code in the future. I don’t write very much code now, and previously that was something I really enjoyed as an artistic or craftful endeavor: writing the perfect little function that did an algorithm in much more efficient time. Now I do what everyone else does and prompt the machines.
So this is causing a moment of self-reflection and inner turmoil around what our role is as educators, and what we should be teaching. At the introductory level, things have really changed. For higher-level courses, once students have attained a baseline of skills and you’re doing things in a more sophisticated manner, or teaching pure theory, I think there’s still a little more room for hope — and I’m happy to chat more about that if it’s interesting.
Ariel Yeah, it’s an interesting question — what do you think the goal is now? Because not only can AI do small-level coding, it’s starting to be able to do complete projects. We still need a human currently prompting it, but we don’t need a person as involved in the technical details. So what should the goal be? What is the student trying to learn? What are we trying to accomplish?
Shaanan So I teach at two very different places. This fall I’m going to be teaching at Tufts, which you can think of as a liberal arts college with a research focus — it graduates doctoral students, but it’s small classes, seminar style. That’s one side. On the opposite side, I teach at the University of Melbourne, which is a large state school, and my smallest class is 400 students. The kinds of things you might think about teaching, and the way you teach, have to shift a lot.
For me, one of the things I want to communicate in my lectures is the beauty of the topic, to encourage self-motivation and self-exploration. Because coding and computer science are in many ways a craft. It’s a skill, it’s something you have to spend your own time on or you’re not going to get better at it. If you want to work producing slop applications for SMEs, AI can do that. But as an educator, that’s not my goal. What I want to do is cultivate individual growth and judgment, and try to motivate students to understand why they might pursue their own learning in the area. Because I only have them for one class — or if I’m lucky, I see them for three to four years. After that, it’s going to be their own sense of wanting to learn, their own motivation, and a level of autodidacticism that really drives them.
One of the hallmarks of a programming career, of an engineering career, is this need to constantly be staying up to date. If I can’t give that passion to them, then I think I’ve really lost out. And that was true even before AI, even when they were writing the code themselves. There were students who would pay someone overseas to write their assignment for them — contract cheating — or get a friend to help. So I think that actually hasn’t changed that much. If I keep my view on that passion, on that motivation, and on socializing them into the disciplinary norms of computer science, I think there’s still some hope.
If we’re thinking about specific skills development in class, that’s where the deeper concern is. Not only because everything is changing so quickly, but because I find it very hard to predict what useful things I can teach them now that they’ll still use in four years.
Now, college education has never been about vocational training directly. Med school and law school a little more so, but even in law school they teach you how to read appellate opinions — they don’t spend as much time teaching you how to redline contracts you’re swapping between firms. So college has always been this disconnected place where you do learn skills, but the skills you learn aren’t always directly transferable. There was a result from the last US census, I think, showing that only about 28% of people who graduated from STEM majors ended up in a STEM profession as narrowly defined. That’s probably a bit of an undercount for various reasons — they didn’t include medicine and healthcare. But at the same time, we still seem to be communicating something valuable.
So I think this is a moment for us to ask: what are those valuable things we’re communicating, that we’re teaching, that aren’t the specific thing they’re going to be doing in the day-to-day job? In that way, I think this AI moment is actually quite empowering, even as it’s depressing at the same time. Because we have a real forced opportunity to go back and think: okay, what are we actually doing here? And I think we can make some good of that.
Ariel It’s interesting. Schools do say they’re not trying to teach for a vocation, but I feel like for a lot of students that’s why they’re going to school — either to learn practical skills or to get the credential they need. So it’s interesting that computer science as a major was never entirely practically focused; there are parts of math that usually aren’t needed on a typical job. But it did seem overall like, okay, this is a practical field. And now the areas that seemed so practical, like learning how to code — it’s a question whether that’s even going to be used. So it’s becoming a theoretical topic that we’re teaching for its own sake. I’m wondering, could you formulate what the purpose is, if it’s not as practical as it was?
Shaanan I think there are a few things. The first is that we have to be realistic and recognize that for many people, college was not required to learn computer science. You could go online and watch CS50 — I’ve taught with them before, that’s wonderful — and you can actually become a damn competent computer scientist just by consuming online materials. So to me it was obvious that something more is happening here.
One aspect could be credentialing. As you said, a degree from Berkeley is going to open doors for you that a few online Coursera courses won’t. To the extent that what we’re doing is credentialing, it’ll probably still function much the same. What’s interesting is that it further weakens the link between what employers claim the credential communicates to them versus what it’s actually doing.
Being realistic about it, we’re pretty aware now that a lot of what happens is a filtering process that happens pre-admissions. The university is already taking students who have jumped through certain hoops, and then attending for four years is not actually about skill development — it’s about certifying that they jumped through that initial pre-admission hoop. That’s not a type of education I particularly care about.
If I were to take the extreme position, I’d say let’s stop worrying about assessment in the university whatsoever, because the credential was never tied to the assessment in the first place. It was tied to their high school ability to jump through hoops. Now, this does put more of a burden on future employers, who’d have to do even more weeding if we’re not assessing at all — and maybe more burden on the poor college students, who now have to work even harder to prove themselves earlier in life and can’t distinguish themselves in college. But I do think de-emphasizing assessment is one path to helping us care more about some of the things education can do when we’re not so focused on hoop jumping.
One thing I think we can and should do is take advantage of the way AI can give us efficiencies in our teaching, and then allow us to provide much more personalized attention to students, focusing on individual goals, individual growth, and individual interests. I have students in my intro to programming class where all they want to do in life is make things. For those students, giving them tools and helping them put themselves in an environment where they learn how to use what’s out there to build the projects of their dreams could be very empowering. On the other hand, I have students who are much more theory-driven, who couldn’t care less about building a distributed system but really want to understand the nature of computation. For them also, I can’t serve them so well at the moment. In a class of 400 or 500 students, I have to cater to a middle path, which up until now has been this forced credentialing function.
So maybe if I can offload some of the rote stuff to AI, I can spend a little more time tailoring my efforts to these groups of students more individually, and focus on the things they really care about — the things that can’t be delivered by someone who doesn’t have time to interact with students directly.
Ariel That’s interesting. On its own, I would ask: if you get rid of the hoops, what percentage of kids really want to just learn for its own sake? But if you tailor it that closely — so those who want to build projects get to focus on that, those who want theory get to focus on that — I think it would be a higher percentage. So it would be interesting to see what could be done.
Shaanan Yeah. On how many students want to learn for its own sake, there’s a big difference between different sides of the institutional spectrum. You’ve got the super motivated high school students who are already on Coursera and have finished four courses before they even come to college. You have community colleges in the US, and technical colleges, with students who are really there for a clear vocational reason. You have liberal arts colleges where they’re very interested in that seminar-style stuff. Then mid-tier state schools, where you get a whole mix — some very motivated, others who just want to party for a couple of years. And then your prestige, Ivy Plus, R1 institutions. To lump all of those people together elides really important nuances about people’s desire to learn and what they actually want out of learning.
There was a 2016 survey of incoming freshmen into, I think, only four-year colleges. That survey showed that 84% of students said learning things that were important to them was a very important reason for attending college — it was one of those Likert scales. But then you ask those same students about the other items, and 76% said gaining a general education was very important. Even more than that, 80% cited getting a better job, and 74% said career-specific training. So you can have these multifaceted reasons for wanting to be part of college. And that’s on the expressed preferences side. As you pointed out, the revealed preferences are somewhat different.
There was another study by Philip Babcock, a social scientist, who found that the amount of time students spend actually doing their college work halved between 1961 and 2003. So the extent to which we’re successful in getting students to learn for its own sake seems to be less than we might hope. Maybe the suggestion we were both building on — that with closer tailoring we could up those numbers and get back to a point where students are really self-motivated — is a possibility. I’m hopeful. At least I have to be optimistic, or else I’d need to change careers.
Ariel It’s challenging, because probably a big part of the change over those decades is just more people going to college and university. So it could be that people who wouldn’t have gone in the past aren’t as interested in learning for its own sake.
Shaanan Yeah, and particularly the GI Bill shifted the college demographic substantially and led to this professionalization race, or academic race, where even to get entry-level jobs you now need a college degree. So for a lot of people who previously wouldn’t have had the motivation or interest to go to college, they now have a reason why they need to go. I do think that’s been, on net, not beneficial — I mean, more people going to college is net beneficial, but I think we’ve overdone it, in the sense that technical and vocational colleges have a really important role to play, and we shouldn’t be forcing people through four-year college — particularly at places with a research focus — when those individuals just want to get out there and start working.
Ariel Do you think countries like Australia might do a better job than America does?
Shaanan We’ve made some very bad mistakes. We essentially eliminated the technical colleges in a series of reforms in the ‘80s and merged them with the research universities. What that’s led to is a number of universities that have to build themselves as research universities, and from which the government expects research outputs, but they don’t necessarily have the talent to compete on the international stage and produce the quality of research that really benefits the country.
So number one, we underinvest in research compared to the US. On all non-medical research, we only spend about $800 million a year in government funding. But then the pie gets split into places that would really better serve the public in their original form as technical colleges. That’s not to say no good research comes out of these places, but the ratio is definitely different. They were designed as technical colleges, and trying to force people into the research-intensive route — this happened in the US, this happened in Australia — I don’t think does a service to anyone.
Ariel (31:46) Okay, now turning from the effects of AI on college specifically: what do you think the effects of social media previously, and AI as the upcoming trend, have been on teenagers and college students?
Shaanan I think we’re still a little early on the AI front to really know. Social media is a little better studied. In particular, there was a consensus report of the National Academies. For those unfamiliar: if you’re a scientist and you’ve achieved real preeminence in your field, you may be elected a member of the National Academy of Engineering or the National Academy of Sciences. Now and then they commission reports that attempt to ascertain the consensus in a field. These are really the gold standard for understanding what researchers think.
One came out just a couple of years ago on the impact of social media on youth and adolescent well-being, and the results were essentially inconclusive. They found a small correlation between adolescent depression, anxiety, and social media use, with a very unclear causal direction. I’ve got a line here from the actual report: “the committee’s review of the literature did not support the conclusion that social media causes changes in adolescent health at the population level. Nevertheless, there are harms…”
I think we see this overplayed in the media a lot. It’s a very popular line to say social media has ruined our kids, that it’s net harmful. There is some evidence in that direction, but certainly not enough to argue conclusively that social media is net harmful rather than net beneficial. There are also plenty of studies showing beneficial effects — ease of communication among teens, people who can access community that they couldn’t previously, LGBT youth in more insular communities that previously would have restricted that. It’s easy to say those pale in comparison to the harms — child grooming, depression — but the evidence really isn’t there to argue conclusively that it’s been harmful. It is very heavily disputed, though, and there are certainly individual scientists who take a fairly strong view one way or another.
One thing that is fairly clear is that technology on the whole has displaced other childhood pastimes, and you could argue that in itself is a problem — the fact that kids aren’t going out and spending time outside because of TV, computers, video games, social media, maybe AI. Maybe they’re staying at home talking to ChatGPT. But there definitely is that displacement effect. So to the extent that you have a strong prior that the things kids were doing prior to 1940 were preferable, then perhaps you’d have a reason to dislike not just social media but all of these different technologies. And again, we’ve seen this pattern with every new large childhood pastime — everything from television to newspapers was originally heralded as ruining the youth. That doesn’t tell us whether this time is actually the real case, whether there is a real wolf, but there is reason to tread a little more cautiously than the media likes to.
Now, with respect to AI and social media use in people who already have problems, I think there are much clearer links. There are patterns that are fairly apparent prior to children engaging in self-harm, where often there’s a spike in social media use. But this is also correlated with different hours of sleep and wakefulness — a lack of sleep and staying awake in the middle of the night is correlated with negative mental health outcomes. Again, it’s unclear which direction the causal arrow points. Are they up in the middle of the night fiddling on their phone because they’re depressed and can’t sleep, because they have negative self-image or anxiety? Or is it the fact that they’re up scrolling through social media at 3 a.m. that’s making them depressed? Or perhaps some of both.
So I think there’s a lot of room to explore that more, but I would be really hesitant to make some of the strong policy decisions that governments are making on the basis of believing this causal link has already been established. I think the evidence just isn’t there yet.
Ariel I guess the question is: there’s been an uptick in certain negative metrics, like suicide among teenagers, since 2012. People say that’s when the smartphone and Instagram and things like that became popular. So if those aren’t a strong enough cause, then you need to see what else might have caused it. Is it that people are overprotective of their children and they don’t get to play? Is it other factors? Is it just too complicated to resolve? And then people look at social media and certain aspects of it seem very negative — unrealistic beauty standards, people spending so many hours a day — so they feel like that must be the cause.
Shaanan Yeah. And look, if you’re a parent deciding for your child — I want to prevent my child from going on social media until a little later, I want to speak to them very carefully and keep watch on what they’re doing — I think all of those are potentially positive things. But that’s at an individual level rather than a population level.
Where I think there’s a lot more evidence is that for children in certain situations, social media use is going to be either net positive or net negative, and you can often tell based on contextual information about their situation. Parents are often well placed to make that judgment — assuming we’ve got good parents, which is often but not always the case. The problem comes when you try to do population-level interventions.
One place we’ve seen a similar-ish intervention was the phone ban in New York schools. There was an economics paper showing that there actually wasn’t a substantial impact of the phone ban on educational outcomes, at least as far as they were able to measure. It was a relatively good study. But there was also follow-up by some of the authors and related persons saying that just because it doesn’t improve academic outcomes doesn’t mean we should let phones run amok in schools willy-nilly. There may be other benefits — just in terms of classroom control, it’s easier for a teacher to control the classroom when kids aren’t all on their phones. There may be other social benefits.
So again, my line isn’t that there is no worthwhile intervention around technology and adolescence. It’s more that we have to be very self-aware about which interventions are directly supported by the evidence and which are more of an experimental game — or which should be tailored less to a population level and more to a smaller context. That could be an individual school, a community, a classroom, or a family.
Ariel I get that it’s a bit extreme for the government to do an outright ban on a whole country. On the other hand, the challenge is that if a parent wants to ban it for their kids but all the other kids use it, that creates a weird social dynamic. So it’s very hard to do on an individual level. But if there were some way to do it on a community level, that seems like a good balance — people could leave the community if they wanted, and you could also run experiments comparing communities. So if they had some way to roll that out, maybe by school or district, that seems like it would be better than doing it across the whole country.
Shaanan Yeah, I’ve actually done some research in this area. Australia rolled out a social media ban to the whole country for anyone under 16. The way they chose to do this was by regulating the social media platforms: if you are a social media platform, you have to take reasonable steps to remove under-16s and to ensure they’re not there.
This regulatory intervention has now been in place for about six months, maybe a little over, so there’s been time for some initial results. I have a couple of papers on this, and there was also a paper by Cass Sunstein and some of his co-authors — I’ve forgotten who the first author was. Essentially they found the ban was unsuccessful, and you still got the disparate impact you were talking about, where some kids are still on social media. Those kids self-identify the other kids who are not on social media as the uncool kids. So rather than solving the bullying problem, it may have exacerbated it.
This is also true in my research, in which we did focus groups with young people and found that essentially all of them knew how to get around the ban if they wanted to. The thing that was much more determinative of whether the ban worked was what the parents wanted to do with their own kids.
Now, the Australian Prime Minister, Anthony Albanese, has openly said that the motivation for this is to give parents ammunition to police their own kids. But as the Sunstein paper showed, unless you achieve a critical mass of kids off social media — whether because families have chosen to enforce it or because it’s become technically impossible — unless you get to that tipping point, it’s not really going to be effective as a social intervention.
So I think that’s more or less where it’s landed. Even the regulator in charge of administering the social media ban has said she doesn’t have adequate power to actually enforce it, and has cautiously and subtly implied that it’s not really working as designed.
Ariel I guess one thing proponents of such bans could say in response is that you just need stronger controls — make it even harder to get an account if you don’t show stronger proof that you’re a certain age and that you’re actually that person. What would you say in response to that?
Shaanan I think the more intense you make the restrictions, the more you start to impinge on civil liberties. Requiring everyone to present their ID when visiting a website is probably a non-starter for people in America, or at least from my understanding of American culture. Even in Australia, people are very uncomfortable with it, and it did not have much political buy-in. So there’s an explicit carve-out in the law saying that platforms that choose to use government-provided ID as a way to ascertain age are not permitted for that to be the only way to establish that you’re over age.
What’s happened instead is that they’re scanning people’s faces and trying to guess how old you are based on a scan. I have some forthcoming research on this that I unfortunately can’t spoil, but with all the methods we tried, we were at least able to bypass them some of the time through the front door, just by messing up your face or putting on a disguise. The models are getting better, so maybe that will stop.
But more significantly, pretty much everyone knows how to use a VPN, even 12- and 13-year-olds. So unless this ban is regulated globally and you start cracking down on VPNs, you’re going to have a really hard time enforcing this at a technical level without introducing internet-level censorship. The UK has been floating this idea in the last couple of months — there have been a couple of members of the House of Lords who have endorsed a crackdown on VPNs. But there are a number of reasons we should feel very wary about any such proposals, because that’s going to have very significant knock-on effects. Given that the harms of social media are already somewhat disputed on net, the benefit of imposing a countrywide anti-VPN and internet censorship system is not something I would endorse. Quite the opposite.
Ariel I guess the question is similar to the physical locks you mentioned before. Maybe you just need a certain level of difficulty, and then it’s not expected that you’d be on social media. I don’t know if that’s true — maybe it’s so trivial that kids will do it no matter what. But once you have that certain level of difficulty, it doesn’t have the same social pressure. And then if some people want to get around it, that’s okay. We’ve weakened the expectation that everyone is on it, and maybe that’s enough.
Shaanan There’s a good point in parallel to that, which is: maybe it’s not working because today’s 13-year-olds have 14-year-old friends, and the 14-year-olds have 15-year-old friends, and the 15-year-olds were already on social media before the ban. So perhaps it’s just a generational thing.
The last paper I published on this suggested that perhaps that’s not true. The reason we’re a little skeptical is that we found cross-age-group socialization of the bypass methods. It’s not just the 15-year-olds sharing with each other; it’s groups of kids across year levels — particularly because of sibling dynamics — sharing both why they use social media, what social media is, and why they might want to get around the ban. So it’s unclear to what extent a generational shift will change it. It’s still plausible, because even that socialization mechanism can change as the generation changes. Perhaps there will be a discontinuity in the way today’s eight-year-olds relate to the 12-year-olds when they all age up and the 12-year-olds hit 16. But at least the current indications aren’t positive for that.
Ariel I guess another way to approach these issues is, instead of trying to restrict it by age, to reduce the negative effects that happen for all ages. So if they have algorithms pushing certain content just to get people to click, but it makes them more depressed, maybe they could do something to reduce that and try to have some kind of positive algorithm boost. I don’t know if that’s something you think might be more realistic.
Shaanan Yeah, there I’m much more bullish on our ability to intervene.
The large social media platforms have been pretty terrible at transparency around their trust and safety programs — that’s the name given to the programs designed to prevent harm to young people, scams, uploading of pornography that’s not allowed on the site, or whatever it happens to be. Those have been very closed to the public. There are some conferences where occasionally practitioners from the companies will discuss things a little, but they’re largely black boxes, and in recent years they’ve become more and more opaque. There’s a well-known story in which Facebook gave a permanent ban to some of my collaborators who were doing trust and safety research on the platform, on the grounds that they had violated the terms of service. Contractually, they are able to do that; they didn’t violate the law by blocking these researchers.
But my view is that normatively we should have rules in place to ensure that researchers, public advocacy organizations, and governments can get more insight, and might not necessarily be obligated to follow overly onerous terms of service that prevent us from getting more information. So the first step is more transparency. Transparency alone is not going to solve this, but enabling it will allow regulators, civil society organizations, and the academics who support the regulators to figure out what kinds of interventions might actually work. You can still design experiments to determine what interventions might work even without support from the companies, but it’s definitely a lot harder.
On the next front, you have things like the Age Appropriate Design Code, which I think is in the EU, which aims to set out what design elements and features should and shouldn’t be in place for a platform accessible to a young person, or what experience a young person observes when identified as having an account on a platform. Australia has gone in a slightly different direction with a digital duty of care, which attempts to impose a generalized legal obligation on the platforms to look after young users. The platforms, seeing the writing on the wall, have not been entirely opposed to either of these efforts.
So there are a lot of things we can do in this direction. Which of these will actually pan out and deliver better outcomes, and ensure we can capture some of the gains without the negative effects? I’m not sure yet. There’s a lot more work that needs to be done. But I broadly think it’s worth experimenting with these different regulatory interventions to determine which might actually work out for us as a society.
Ariel (48:20) It seems like there’s a balance. On one hand, trying to ban everything seems too extreme. On the other hand, some of the habits people have on this very new technology are so unnatural that to just do nothing and say “yeah, whatever” also seems like maybe not the right response. But if they could figure out something more balanced, maybe they could help everyone.
Okay, turning now back to software security. Recently Anthropic released a model called Mythos, which was one of these new moments where AI showed, in a practical way, that it was more powerful than anything before at finding software vulnerabilities. They released that to a restricted group of companies. Then they released another version, Fable, which was supposed to be the safe version for people to use in general — where if they were trying to do things that seem dangerous, like find security exploits, it would block them.
So that’s the basic background. And then the government briefly shut down access. They said that foreigners outside the US, or even foreigners in the US, can’t be allowed access even to Fable. What do you think of the overall risk here shown by this new AI model, and what do you think of the government’s initial response?
Shaanan Maybe let’s start with how dangerous these things actually are. There you have two sub-questions: dangerous to whom and how, but also what is the marginal danger over the other models that are available, including the open source ones.
What everyone in industry is talking about, at least, is a vulnerability tsunami — the notion that these tools introduce significant new cybersecurity risk because of the ease with which they can find latent security flaws, generally in code but also sometimes in systems. More commonly it’s in a code base: they can find something like a buffer overflow and then even write a proof-of-concept exploit for it.
We’re not necessarily at the point where they can come up with a full exploit chain — find a buffer overflow in the JavaScript engine in Chrome, use that to escape the sandbox, then do privilege escalation and get to the operating system. We’re not quite there yet. But we’re certainly at the point where the models are able to discover meaningful software security flaws, and with a proper harness, a proper set of supporting tools, distinguish between false positives and true positives. There’s still a level of over-reporting, but it’s gotten to the point where if you get a few outputs, you can be reasonably confident — assuming you’ve used the right harness — that these are actual real bugs.
So they’re very good at the first part of that chain: the discovery, the analysis, and then the verification. They’re not as good at actually launching full-scale cyber attacks. We’re not seeing, at least to my knowledge, end-to-end attacks by swarms of Mythos bots out there in the wild.
There is some danger, because our ability to patch is now a little slower than our ability to find vulnerabilities. To patch, you have to actually be awake, know there’s a vulnerability, and send your agent to go fix it. Meanwhile, the attackers can be off finding new bugs. And if you were to ask a Mythos-class model to find a vulnerability yourself, there’s no guarantee you’ll find the same one an attacker has already found. So being proactive at the moment doesn’t necessarily save you.
These capabilities are pretty much on the frontier. Mythos and GPT-5.6 Sol, the new OpenAI model released this past week, are mostly on par, and they can both do this. But the open models and the Chinese competitors are honestly not that far behind. This is where I think we really have to be cautious about our policy response, because our policy response is only going to be effective for a certain amount of time. These capabilities are pretty accessible.
There’s also a big question about short-term impacts versus the equilibrium we reach. A policy response that’s good for the short term might not be pertinent to the equilibrium. There was a talk by Nicholas Carlini where he talked about the commoditization of this pretty specialized skill set of vulnerability research. Other observers in the media have said essentially the same thing: what was once the purview of a few people with very specialized reverse engineering skills — myself included — can now be done by almost anyone.
I think that’s where some of the danger is, and there’s a lesson to learn there not just about the danger from cyber attacks, but also about CBRN — chemical, biological, radiological, and nuclear. My personal view, as someone who’s been in security for a long time, is that the reason we don’t see large-scale bad things in the world as often as we might is not because doing bad things is actually that hard. Assembling a dirty bomb is easier than you would hope. If you don’t care about the welfare of the person doing the synthesis, you can probably synthesize some pretty bad things. We know organized crime is pretty good at synthesizing drugs, and explosives are, again, not that technically hard to make.
Some of the reason people haven’t done it is that outside of state actors, the set of people who have the motivation to do terrible things in the world, and the technical knowledge and capabilities to pull it off, and the operational capacity to actually organize it — not only wanting to do it and having the technical skills, but being able to execute on a plan at a larger scale than just building something in a lab — has been very limited. You’re talking about a fairly small set of people at the intersection of three relatively rare groups. Only certain people are very operationally sophisticated, only certain people are good at chemical synthesis, and only a certain group of people really want to do terrible things in the world.
What I’m worried about with AI is that it may be removing some of these distinctions. By reducing the technical barrier and also making operational planning a little easier, it can increase the accessibility of some of these dangers. Even while I don’t think the set of people who wish to do terrible things is going to expand that significantly — it will a little, because of the reduced friction — I think there’s a concern that the barrier for the existing people we’re worried about has been lowered. And we’re seeing this. There was a report out of an Oxford University center about, I think, Somali groups planning piracy and terrorism operations using frontier models.
So that’s the place where I’m really worried, and where I think the government does have reason to worry. I think it’s a different question whether export restrictions and the Defense Production Act are the right way to regulate those.
Ariel So if you think there is a big concern here, but the measures they took were maybe too extreme or just not correct, what do you think is a better approach?
Shaanan Let’s start with what they did. Part of the problem is that they approached Anthropic as if they were the only entity that already had these capabilities, and that was just plainly untrue. OpenAI’s models — even 5.5, the one from the prior release — were fairly competent at finding these things. Sure, it didn’t have the level of hype as Mythos, and perhaps wasn’t quite as effective, but the marginal difference in capabilities wasn’t that great.
At the same time, there are a lot of clouds around the government’s story. Anthropic and the government had been at odds and somewhat antagonistic to each other for some time, even while the government still desired their capability. So it’s unclear to what extent the actions they took were really motivated by concerns versus how much was politically motivated.
It also seemed like this was another narrow jailbreak. It worked sometimes, only unlocked certain capabilities, only for a limited time, and would also have been visible to some extent. The thing that’s more worth worrying about is if someone can find a universal jailbreak that just turns off all the safeguards — and based on the reporting, it didn’t seem that this had actually been found. So the remedy the government chose, which was to ban Mythos itself from non-US persons, both wasn’t well tailored to the problem and ignored the fact that these other models were available, and that the particular jailbreak that had been found didn’t seem to be anything that significant.
So what can we do? This is where we have a real problem, because there’s not a great answer yet. Anthropic’s mode has seemed to be to suggest a coordinated pause — they’ve offered to cease model development if everyone else ceases model development. But this is incentive incompatible. Everyone has an incentive to defect, and not only that, but potentially to defect in secret. So I can’t see that working.
At the same time, restricting things via export controls also seems unlikely to work, particularly because the open source models really aren’t that far behind the frontier, and the know-how of what you actually have to do to improve the models is mostly public. We might see a shutting down of a lot more research — already the frontier labs have been publishing fewer of the key components needed to advance in academic and public venues. We could see that further restricted, with the government trying to push back on open research. But in the US regulatory environment, that’s very hard to do; there are fairly strong First Amendment protections around academic speech.
So I don’t have a brilliant policy prescription we can adopt today, other than to say it’s a hard problem and something we need to be working on and trying. I know that’s very unsatisfying, but I don’t have the answers today.
Ariel People have talked about doing a pause in research, but they’re talking about the more worrisome risks — like AI itself trying to take control or cause mass harm. These specific risks are a little smaller: people just using AI to do bad, at least at the level we’re at currently. It could get worse — if they get access to super viruses and things like that, that could be really scary. But for these kinds of risks right now, I feel like Anthropic’s approach might be reasonable, where you try to have multiple layers of detecting it.
And then maybe you could build on that. I know we talked about not wanting to require people to verify their identity, but in theory, if you had more knowledge of who’s using your product and what they’re using it for, you could detect bad players — either block it at the moment or detect it right afterwards so they can’t keep doing harm on an ongoing basis. That may be an approach that could work for now.
Shaanan Yeah, and that’s where we are seeing the labs doing a lot. OpenAI’s Verified for Cyber program requires you to submit some ID so they actually know who you are, and then they look at your transcripts — and they probably are running models over the transcripts, post hoc as well as live.
One of the common techniques, which I’ve also done some work on, is guard models. The idea is that before a tool call goes ahead, a second classifier or second model looks at both the tool call and the transcript to determine whether what you’re requesting should be allowed to go forward. That might be for your own safety, or because you’re stepping outside the bounds of what the frontier lab wants to permit, either for cyber risk or other reasons.
The problem is that there are still lots of attacks on the guard models, and a post-hoc approach to preventing harm — we’re going to look over the transcripts later and ban people — might still allow attacks through. There’s a big colloquial literature on people who create new accounts whenever they get banned: jailbreak it to the extent they can, use the capabilities to do their little bit, get banned, pay another 20 bucks, open another account, make a little more progress. Through that methodology they can actually assemble a fairly fearsome cyber attack on something or other, despite the existence of guard models stepping in at various points and blocking them.
There’s also a lot of sensitivity to context depletion. As you chat longer and longer with a model, the efficacy of the guard models will decrease as the context window fills up and the size of the context the guard model has to handle increases.
So a defense-in-depth approach is definitely what we’re doing currently and what we should be doing, but there are still gaps in it, and those aren’t plugged yet.
Ariel I noticed I was just playing around with Fable for a personal project — trying to track everything I read, different things like that — and it put up these warnings for no reason at all, and then it switched off Fable. It had nothing to do with anything. But I think that’s just because the government made them be even more restrictive. I don’t know if on their own Anthropic would have been that restrictive.
Shaanan Yeah, the overblocking thing is quite funny and happens to me all the time. I’ll be doing something totally benign, but because I’m a security researcher and my research papers have something to do with security, they get flagged very often. I haven’t experienced that with GPT-5.6 yet — but they’re obviously in a different regulatory position, a different political position, so they’re acting quite differently.
I think that’s one of the dynamics to watch for here: the frontier labs all sit in a different position in the political and social ecosystem, so the capabilities of the models that are permitted to users will likely end up being different. This is both beneficial for users — there’s good competition in the market — but also dangerous, because you can play the models off each other. I had an instance where I was doing some security research and GPT was getting annoyed at what I was doing, so I gave that specific task to Claude, took the results back from Claude, and put them into GPT. You could imagine building a harness that does this for you, that looks for when you get blocked by one, sends it to the other with a slightly different context, and then takes it back to the first one in a new context window.
Ariel That’s another case where, if they really knew the identity, it could help. If they see a pattern of people repeatedly doing something, they could be blocked — and if they knew their identity, it wouldn’t be as easy to just create another account. In America, if you have to give your credit card and some other verification, it’s not as easy; the credit card is attached to a name. So there are certain levels where I feel like it would be easier to control.
And there would also be a very strong incentive for the companies to control it. Anthropic has issues with distillation attacks, where companies train their models on Anthropic’s advanced models. If they could verify who’s doing that, they could shut them down — that would be very beneficial for them. I guess it gets harder in other countries for them to have that level of control, but it seems like something they’d want to do anyway.
Shaanan I think there is some benefit to be gained here by information sharing across the frontier labs. This has often been difficult for companies in the US to do — even about threats and threat actors — because of antitrust laws, which restrict collaboration and cartels. I think it’s a good thing to have the antitrust laws, but I do think the regulatory security of having explicit carve-outs, as opposed to implicit nods, could be of benefit. So establishing a platform for sharing bad actors and threat signals between the frontier labs would, I think, be one thing that would help with this problem. Maybe we could even get some collaboration cross-border, but that’s probably wishful thinking at this point.
Ariel (1:04:38) Hopefully longer term that’s something that would need to be worked on, if people wanted to do things like slow AI development.
Okay, turning to people using AI. Agentic AI has become much more powerful. It’s something software developers are much more familiar with, but I think it’s starting to spread into the wider world — where you’re not just chatting with the AI, but you’re telling it to do stuff and it goes off, spawns subagents, does tasks, completes projects, creates spreadsheets, things like that. So what are the main risks there, and what are the main things people who use it could do to protect themselves from those risks?
Shaanan In terms of the risks, a good place to start is the lethal trifecta, a term coined by Simon Willison, a tech writer and coder. The lethal trifecta is: access to private data or systems, exposure to untrusted content, and the ability to act externally or communicate. All of those things are present in a modern-day coding agent — or in Codex for Work, I think is the new OpenAI branding of it, and Claude Cowork.
The problem is that you gain a lot of utility by giving agents this lethal trifecta. If you want your agent to fill out your expense reports at your business, you need to give it access to a private system, access to the content of wherever you’re storing your receipts, and the ability to act. That’s the only way to achieve the task. So I think the largest risk is that our incentive structure is such that we are strongly incentivized to do the exact things that increase the risk substantially.
In my own life, I had one of those instances of AI deleting all my files. In this particular story, I’d messed up my cloud sync for the files on my computer, because I had a Git repository that had too many little files — and you should never put one of those inside your cloud sync, because algorithmically it’s bad and it has to do too many database operations. Details, details. So I told my agent, “Hey, can you please fix the syncing?” It tried a whole bunch of things, and eventually came to the conclusion that the cloud sync repository was too messed up and the best thing to do would be a total reset — not realizing that the sync state of the files was mixed, so some were only present locally, and a reset would result in them being deleted.
Fortunately, I’m a good computer scientist and back things up regularly aside from the cloud sync. But that was a really nice example of where the intent was actually aligned. The agent was trying to do the right thing, it missed a critical piece of information, I’d given it access, and yet it still resulted in substantial damage — even though I was able to recover from it. It was painful, but recoverable.
I think what we’re missing, at least up until recently, is this: in the 2000s and 2010s we moved from thinking about browsers as just another app to thinking about them as equivalent to an operating system in complexity and importance. We need to think about agentic systems the same way. They have complex components and a very large trust boundary, in that we’re giving them exposure to a lot of untrusted content, just like we would for our browser. We need to start creating boundary mechanisms to enable us to deal with it a lot better.
The harms one could experience are really the same sorts of harms you could inflict yourself by typing in commands you haven’t vetted. So that attack surface is really, really broad. Anything you can do to mess up your own stuff, the agent can probably do as well. And that is the big problem.
Ariel Some of it could be addressed in a simpler manner. Often you could have it run in a sandbox, and you don’t really need it running outside the sandbox. If you’re working on code, you give it access to the project it’s working on. It could read things from outside — if it needs to research and find libraries, things like that — but it wouldn’t be able to delete your actual data. So I feel like that’s an initial step. But then people might just default to giving it root access to everything on their computer, and that default is where it becomes a problem.
Shaanan Yeah. And setting up a sandbox is painful, and you need to know what the specific permissions are that this thing shouldn’t have. People who work in distributed systems have gotten better at doing this, but it’s still a real pain. You try to use Linux cgroups to isolate things, or Docker containers with locked-down permissions, or Kubernetes, or whatever your distributed system of choice is — when you try to lock it down, it ends up being quite painful and a lot of overhead, despite the significant advances we’ve made.
One place where we did make real headway was browser sandboxing. Escaping from the browser sandbox as a piece of malicious code is now legitimately quite difficult. The other thing we were able to do is segment parts of a browser’s memory from other parts — isolating different tabs in their own processes, separating cookies, developing protocol-level restrictions that isolate different security domains. That’s been effective.
I think porting that same approach to agents would be a way to make some progress. It’s still a long and hard road, but there is hope, and we have architected these things before.
Ariel Some of it’s just figuring out the right balance. You have people running it on completely unrestricted mode, and on the other end you have it asking you before it does any little thing — like before it reads a file, and there’s no risk in reading a file. So they need to find that balance where the defaults are pretty good for security, where it makes sense what it has access to, and it’s rare that it needs to ask you for something — but when it does, you actually review it and don’t just click yes right away.
Shaanan There’s also Ariel using the agent versus my grandmother using the agent, and those require different models of what to ask permission for and when.
I know there are certain people in my life who, when asked to click yes on something, will always click yes. And certain people I’ve trained — my mother in particular — who I’ve trained to always click no, do not grant permission, no matter what, even if it would be useful, and she suffers a loss of utility. Perhaps that was an overcorrection.
But I think there’s potentially an interesting path forward here using guard models that understand not only the technical context but also the user context, and can prompt for permission appropriately. You already see this if you’re using the ChatGPT agent-based interface, Codex. You can turn on a feature that allows a guard model to review the action first and then prompt you if it senses a mismatch between the gravity of the action it wants to perform and the direct relevance to the instruction you gave it. So if the thing it wants to do has medium potential for damage but very high salience, directly related to your instruction, it might not prompt you. Whereas if it’s a fairly risk-free action but not that related, it still might not prompt you. Only in the case where it’s high risk and medium or low authorization will it go ahead and ask you explicitly.
Ariel (1:12:22) Turning away from people using agentic AI to broader consumers: the internet allowed much more scams and hacking and all that, and now the bad guys using AI could bring it to a much bigger scale. There’s been an increase in these scams; it’s becoming a bigger problem. Regular users — and maybe the elderly and all these people — just go on the internet and there are so many risk surfaces for them. There are scams through email and on websites and all over the place. So what do you think is a broader solution, where we could help these people in their general usage of technology and try to reduce the number of scams?
Shaanan There are a few things I think we can do. One is an ex post regime that provides some kind of social insurance for when things go wrong. This already exists in the US in the form of Regulation E around card issuance, which provides protection against liability to users of credit cards — if you didn’t authorize a transfer, you get made whole. I think we can extend this notion to authorized push payment fraud, which is when you press the authorize button but you were being hoodwinked.
Obviously there are challenges with doing this, because then people will just say “I didn’t mean to send this person the money, get it back for me,” and that can open up payment processors to more fraud. But the UK is moving in that direction, and I think such a regime is reasonable. It would have to be designed carefully, but I think that’s something we can do ex post.
It functions just like social insurance. Everyone pays an extra fee to the merchants, and the merchants pass it on to the payment scheme providers. I’m eliding some of the details — when you pay the merchant, often it’s actually going to the payment processor — but essentially you’re paying a surcharge for the benefit of having this scheme around you. And it is somewhat effective. There have been many times in my life where I’ve done a chargeback, or where there were unauthorized charges on my card, and that has helped. So I think we can do that in more places. Zelle is one place in the US where this does not exist very much, and I think we can do a better job there. That’s on the ex post side.
On the ex ante side — before the damage happens — I think there are some defensive measures. As deepfakes become more prominent, one good thing everyone can do is come up with a family safe word: a word or phrase shared with family members. If a family member asks you for a large amount of money because they claim to have been abducted and put in a Costa Rican prison and need $5,000 for bail transferred via Western Union, and they don’t say the family safe word, then you know it’s a deepfake or that something else has gone wrong. Having that kind of trusted side channel or back channel is something that’s very easy to do.
I also think the technical design of some products can be improved. One thing some platforms can do is give you a trusted contact. Let’s say I want to transfer $40,000 out of an account. I can say that for any transaction over $20,000, my trusted contact also has to approve it on their device, and that can be cryptographically verified. Obviously there’s a balance here against friction, and some people don’t have a great trusted contact in their lives, but giving people that option is another way to empower them — and perhaps suggesting it in many instances.
The usual advice also still works, at least for the time being: keeping an eye out for obvious signs of scams, not clicking yes to things, having your wits about you, not using the same password for everything, keeping your devices up to date so drive-by malware doesn’t work. All of those basic security hygiene practices haven’t gone away just because the threat landscape has changed.
Ariel Do you think we could also develop systems using AI that just run in the background and detect if there’s any issue? So it would be running in the browser or whatever platform, and if it detects that something looks very suspicious, it would alert the user — and they could have different settings to protect them.
Shaanan Yeah, I think we can do more of that. It’s already being done quite a lot on the back end. There was a nice podcast with Patrick McKenzie called Complex Systems, where he spoke to someone from Stripe, his former employer, about their use of AI in fraud detection. There are very clear patterns for money transfers that are likely to be fraudulent, or even for authorized push payment fraud, where someone authorized it but was doing so under the influence of someone acting illegitimately.
So I think we can do a mix of both. There’s probably room to do more on the front end to protect users themselves and their experience, but also on the back end to flag these things and put in place more friction when someone tries to do something the system tags as being highly likely to be fraudulent.
Ariel It’s interesting because it’s becoming worse and worse, I feel like, and right now it doesn’t seem like enough is being done about it. But hopefully these defensive systems will be built out.
Shaanan Look, there is a bright spot. You see all these news stories like “grandmother loses life savings.” But there was a paper out by a collaborator of mine, Elissa Redmiles, a couple of years ago, that found the median amount people actually lose to scams in their lives is not particularly high. By memory, it was on the order of a few hundred dollars — which for some people can be really significant, but for the median American, it’s survivable.
It’s just that the long tail of people who really get hurt by this is significant, and those harms can be life-changing. So I do think we need to protect people, but I also think there’s room for some optimism — that between the defensive technologies we have, the training we can do with people, and the size of the problem, at least in the next little while things seem hopefully manageable. Maybe I’m again overly optimistic, but I like to look outside and see the sun.
Ariel So do you think in the long run, in the balance between defense and offense, defense will come out stronger and the scams will be reduced?
Shaanan In the long run, I am optimistic that humans will develop both the social technologies and the digital technologies to be more protective. Social technologies include things like insurance and basic literacy at a societal level; digital technologies include all the fraud-combating AI we’ve talked about, as well as more traditional kinds of mechanisms.
I think we have a wonderful history as a civilization of being able to address problems on a large scale — not eliminate them entirely. We haven’t eliminated war or starvation, but we’ve been able to do significant things to address both of those problems, even while in some ways and on some days we make them worse. So I think that in the long run, the dynamic equilibrium of the cyber and scam landscape is going to end in favor of the good and the kind.
Ariel (1:19:47) And if people want to find more about you online, where can they find you?
Shaanan My academic webpage is at cohney.info. I’m also sometimes active on Twitter and very active on Google Scholar. On Twitter and any other platform, I’m @shaananc.
Ariel Shaanan, thanks for coming on the podcast.
Shaanan Thank you very much. Good to speak.