I got called out by a Stanford learning scientist

This is really a response video to two of Justin Sims videos on studying- a very weak system for learning. You really should be focused on something else. I have a PhD in learning Sciences from a fancy University. I basically study the research on learning for a living. I'm making this video to clear up some misconceptions brings up when it comes to retrieval based study techniques. Everyone. It's Justin here again and I am joined by very special guests. Today we've got Dr Benjamin. Keep here, Benjamin. Could you quickly introduce yourself? Hey, Benjamin, Keith, I make YouTube videos right now. I consult with companies on how to create learning experiences. I have a PhD in learning sciences and Technology design and before that I was a lawyer- very briefly for about three years before I jump ship, but I'm very excited to be here. Awesome, and the reason that we've invited Benjamin on to this is because Benjamin actually made a video in reaction to a couple of the videos that I've made, and I have held off on watching them because I wanted to get a genuine reaction about some of the points here and so I'm like a little bit apprehensive about watching it. But anyway, we'll have a discussion. I will be just kind of cutting to parts of it. So if you haven't watched the full video yet, check the description down below and I would recommend that you watch the full thing, like in its entirety, first, and then maybe come to this and we can see the commentary and reaction over it. Show more

We will be having a relatively higher level, like a more technical dive into some of the points in this video. So if you are more technically minded, that's going to be great for you. You'll love it. But if you are someone that's just wanting that quick hit, the quick tip, not too interested in the science or the, you know, the research aspect of it, that goes on behind the scenes in this video may not be quite down your alley. But anyway I'll leave it to you and without further Ado we'll jump into the video. This is really a response video to two of Justin's songs videos on studying. Show more

One of his videos critiques the effectiveness of study techniques based on retrieval and the other one advocates for study techniques based on effective encoding. What are retrieval and encoding? Encoding is about putting things into your head. So when you first see a bit of information, how do you get that thing into your head? Retrieval is this animation or recalling that you think so. This is my first foray into animations. Out again, Justin is one of the better people talking about study techniques on YouTube. I highly recommend his videos. If you're interested in developing effective study techniques- especially if you're in Circle student or an undergraduate student or a law student or something like this- go watch his videos. I'm making this video to clear up some misconceptions that I think his video brings up when it comes to retrieval based study techniques. Who am I? I have a PhD in learning Sciences from a fancy University. I basically study the research on learning for a living and I help companies to implement that research effectively. If you understand Justin's perspective on encoding and my perspective on retrieval, you're gonna find out they're not that different. Then you are going to have a much better understanding of how to create effective study techniques on your own. Now, just based on what you said there, I'm gonna think now. Show more

When I made that video in the first place, I thought I'm not gonna be able to give the full picture on how everything works, and so I was trying to be strategic in terms of what can I bastardize and what can I not. And, based on what you're talking about there, I'm wondering whether you're talking about the sort of re-encoding aspects of retrieval or the effect that retrieval has on encoding or the encoding effects of Google practice. Yeah, I mean. Show more

Well, I guess you will see. But certainly the part of this is an artifact of like you can only make a video about one thing at a time. Like you cannot tackle every subject and all the complexity that goes on in a single video. It's like man Justin's video from I don't know, maybe a year ago or longer. Man, it was wrong, you know. It's more like oh okay, like if you only watch this video, you might walk away with the wrong side deception. Show more

Yeah, you know, you know, I think, you know, I think I realized that because I talked about the concept and I taught people around the balance between encoding and retrieval and the, the functions I suppose that it has and also how you can sort of think about them separately, but also how you can view them in a very similar lens. But that video was actually the first time I'd done it at, I would say, Mass scale and didn't actually expect that video to receive quite so many views, and based on that video I realized that there are a lot of ways for something to be interpreted. Show more

That was more than what I had anticipated, and so I have, since that video, been, I think, multiple times, been trying to like clarify on those those things, but unfortunately I don't think any of those videos got as much traction as the original video. So I still, to this day, I'm dealing with repercussions of not having thought through the initial explanation properly. I think that the work that you have engaged with on this and that I am currently engaging on this, it is about having conversations, developing conversations, bringing people along with you. Show more

So I feel like it's just a consequence of the work of translating Research into practice and trying to kind of bridge those gaps, which I find to be kind of fascinating and interesting work. But it's just like saying one thing is not enough. It's kind of create a community that's thinking about these issues, that's furthering the conversation, that keeps on kind of correcting misconceptions and moving on, and I mean you have to start from somewhere. One of the things I always think about is that I feel my role is as a research practice Bridge. Show more

You know, I'm not- I'm never going to be- the best researcher and I I'm never going to be the most experienced practitioner, but I am in a position where I can be spending a lot of time in the research and liaising with researchers and then extracting that, and so for me what's really exciting is that both is already interested in talking with me. Like the practitioners really want to talk to me because they want to see a distilled, condensed version of the research that they can actually use and practice. Which I think probably a lot of people don't realize is that a lot of that research is very difficult to really extract into something that is practical and actually probably I'd say 80- 90 of the research has published is not really designed to be extracted into practice. In the first place. It's like this is the groundwork. We're starting and progressing, and so there are errors from extrapolating too early. But then there's also issues with it not being at a point where, like it probably could apply some of these things, at least the principles of it, to adjust your practice and then you can get some experiential data on that. But then the question is: what, in order to know how to do that, unless you just wait for that systematic review to just be, you know, to come out and just tell you what to do is very, very confusing. Show more

And then on the other side, researchers. Most of the researchers I meet and talk to they are very well aware of the fact that they are not practitioners either anymore, or ever have been, and so they don't know what some of the implications are like and that Gap. And I think in this Ideal World there's meant to be continuous stream of information back and forth, but in reality I very rarely. When I do Consulting for organizations and companies there's almost no one that really actually sits in that role and sometimes the person that sits in their role has really no place being in that role. They're usually the person that just expressed interest in it and then they were like, oh, that's your role now and they're like: okay, great, just another another responsibility to add on. So that's an area that, like a position that I see myself as being in, is to kind of bridge together the research and the practice through like a pretty practical lens and again, like you said, like create more normative beliefs. I find it very it's kind of. You know, it's a little odd to think about and I- and I can understand why it's happened, but there is this existing normative belief around what learning is and how learning should be. Show more

That is really sort of odd because it's not really. It's actually not very logical to think about learning as occurring that way, yet it is the way that everyone thinks about intuitively. It's illogical. Yet the normative belief people have- and so there are, I think, not many people trying to spread the message of correcting these misconceptions- is out there. Right. Two quick things before we move on. One: I think what you said before about the research communities and the practice communities is really important, because a lot of times what people don't realize is that the researchers, they're not doing research to help you. That's not why they do research. Show more

They do research to figure out why the brain works the way it does, or what what happens, and then it's just an offshoot of that, of you know, a kind of small subset of that which is like: oh, this is actually practical or what other practical implications of that. And I think it in recent years researchers have, especially in learning Sciences area, have done more of that and more Outreach, more like: hey, wait a second, why are our textbooks designed like this? Why are teachers doing this? Why are students doing that? But that is not the primary purpose that most research goes on. You know, that's not why people are writing papers, that's not how people further their careers really in the research Community. We should make a disclaimer that that's the case for most researchers. Show more

There are some researchers who are focused on fixing the Gap and actually you know- correct me if I'm wrong- this is- this is always the impression that I've had when I read research- is that there are, like you know, tens of thousands of researchers publishing things, but there's not really actually tens of thousands of people leading the charge. It's kind of like for each concept there's like two or three people that are really just pushing the frontier and everyone else is just kind of filling in the the thing that they've pushed through. Show more

And so there are. There are actually a lot of studies that I read and I think: did you really contribute anything with this research, or was it just like you saw a publication Gap and you're like I could get something through the door here, so jumping on a bandwagon, and I guess if you're really optimistic, you could think, well, yeah, it's providing that additional body of evidence to continue to support it, but at the same time I sort of think, well, I just feel like there are more important questions that maybe could have been examined here, other than just this thing and the reason I bring that up- sorry, I I just finished. Show more

I've been rambling for a while- I understand, but the reason that I bring this up- I get so few chances to talk about this, I'm just like this opportunity- the reason I bring this up is that I find it really problematic because, like I said the before- well, like we agreed on- is that in organizations and institutions, even in universities, there's often not a person that is really deep in the research practice bridging, and so there is almost like a bandwagon fallacy effect almost that occurs with this publication bias, where you have all these Publications around a certain concept or a certain branch, that sort of self-validating each other without taking a more critical lens at like. Show more

Well, this entire body like how relevant is that or is that really the direction we should be going in? But it continues to go in that direction because it has been going in that direction and then that, because there's so much research around it, it is now well researched or very evident based, and then that makes its way into mainstream recommendations. The great example I have around this is learning styles, where learning styles just became this enormous movement and fed, it became like essentially ubiquitous, Universal everywhere, like you'd be an idiot if you didn't know about it. And then we've been. We've been for decades now trying to just say like we need to stop implementing this, but to this day there are studies that are coming out that say, oh, you know, we're going to examine the effect of learning styles on this thing, and in the, the literature, like in the background, they say learning styles is contentious. Show more

However, it is one of the most well researched- blah, blah, blah and it's like it's well researched and we prove that it's not really like a thing. Yet the research continues to public. So, anyway, I think it became quite become quite dangerous when we look at research through practical lens, which I think is the trend. Research is no longer in isolation, like people are wanting to extract more from it to inform their, their policies of decision making or recommendations. Anyway, that was my little Spiel. Okay, I still have one other thing to say is about something you said earlier, but I want to continue on this train of thought, because sometimes, obviously, people say something is really well researched or this is a finding that is well established and it is true, but oftentimes when you're reading something, it's kind of like the. Show more

You know the person that's protesting too much, you know like where they're like this is one of the most established findings in X, Y and Z thing and that immediately makes me skeptical because I'm like if, if you are like trying to say that so hard, then what's really going on? So so that oftentimes it gets me to at least check like is this legit or not? And as you bring up with the learning styles, like a lot of sometimes people, if you begin a research study with a faulty presumption, then that kind of the whole study itself doesn't really count for much. Show more

And this is true like there is a tremendous amount. I mean you said there are tens of, you know tens of thousands of researchers and and I think the dive. You know what you said about kind of the diversity of people who are investigating learning and the diversity of perspectives that they enter into, not just ideologies but also research perspectives. You know there are like more cognitive science people. There are more educational psychologist people. There are more like kind of anthropological approaches. There are neuroscientists there and and kind of they all bring their own perspective but also their own methods, their own, their own way of looking at evidence, their own values about which kinds of evidence are most important and- and this is I mean to to step outside for a second. If if you're really interested in this thing, like learning is one example of this. This is something that happens in every field and research, just because something has a paper behind it, or this is one of the things. Sometimes people ask me for citations in my, in my videos, and I put I generally am fairly like I think citations are important, but the problem is there's this behavior of like I need a citation and then, once the citation's there, okay, I feel good. Well, you can find a citation for almost anything. There's just a lot of bad research out there. Like bad, like just wrong. Show more

They don't measure learning outcomes very well, they don't have a well-designed study, they reach conclusions like way out of whack with kind of what the, the study design is supposed to do, and so there's kind of no way getting around the fact that you have to Grapple with how research is done or someone has to. You know, like someone in this research practice role, if Learners dig into this enough and they want some of the primary resource, like they have to start thinking about this on a certain level, like there's a. There's a level of you need to be able to examine and synthesize from the research to really understand what it's saying one of the. Show more

I completely agree with you. You know, one of the things that I found was that the people that are always asking for references and citations for everything are the people that even if you give it to them is going to be pointless, and the people that don't ask for it, they know how to look it up themselves and so, like, I completely agree, and sometimes there are certain conclusions or statements that are supported from, like the amount of citations you'd have, like you know, primary sources you have to read to understand where that perspective came from, is like not, you know, viable. Show more

And so what for that video? You know the, I think, the space travel one. Like people asking for citations, I put it off for so long because I didn't know how to communicate the citations in a way that was productive, and so I like, crammed in a literature review and just published, like a very, very simple some you know, something as concise as possible to just put out there and, of course, like, very few people actually read it. Show more

And then I took the references in, but the references were like 500 lines of just like references and it's just like. Well, you asked for it, but I don't think you really knew what you were asking for. Like, just listing references without any context is more just kind of symbolic because, as you say, I like the proportion of people who are going to actually like pull up a couple of these are more like reading out of curiosity, because it's not a research community and even in the research Community, people cite references all the time to things that they have not deeply read, because a research piece comes to symbolize an idea and they would just want to say, hey, remember this idea, yeah, it's here, or they have to cite some. Show more

You know someone's going to review their paper and they need to cite that person because they know that they need to get it through a review or whatever. Citations are good, but they are- they're not universally just positive, yeah, and, and I think people have this faith in just the fact that if it was published, it must be super legit. Show more

But you know, the peer review process is not bulletproof, right, and it's important, it's good, it's important, but it is not not. It's not a foolproof thing, and some journals have a reputation for having a peer review process. That is questionable, to say the least sometimes. And the number of times where I've read an article and it's given me a reference, I'm like oh, that's a really important statement, like I really want to see where that conclusion came from. And I'll look at the reference and I'll read the original article and I'm like I'm pretty sure this article did not say that. Let's be honest here, a lot of researchers, when they're citing things like what you're not really meant to, but in reality a lot of the time they're just reading the abstract as well. Show more

You know they're just reading the abstract and they'd be like, yeah, this kind of fits and they just sort of Chuck it in there, okay, and it's like I'm hoping that no one really catches me out. One of the things that my- you know- academic mentor and supervisor told me, which really flipped the switch for me, she said: when you are first writing and first publishing, you want to have so many references because you see it as like a statement of your rigor. Show more

But as you become more and more certain and confident and experienced with writing and Publishing, the number of references that you really include reduce to only the references that you know are important and necessary to just get your point across. And there's no point having 700 references, because no one's going to read dividend references and it doesn't matter if you have a certain point and next to it you've got like seven lines of like just reference names, unless you're doing like a systematic review and you have to actually specify it. But just the way that people view, I think, references is they see it as almost like this mystical thing, you know, like this thing that just exists as long as it's labeled to it. Show more

It has this like like stamp of approval, like it's valid 100, true, actually. One last point on that. And then- and then I want to get back to back to this one other thing. But there's research and I again I have to actually find, find the citation. But you know, if you look at papers that are retracted. So there's a website called retraction watch. It looks at papers that are retracted and even- and they're doing good work, they're doing important work, but it is amazing like nothing happens after a paper is retracted- literally nothing. I have a video on my, on my channel. Show more

It's. It's. No one watches it, it's. It has nothing to do with learning, but it is basically about like how I Hoodwinked Myself by, you know, going down this rabbit hole and like thinking, oh, this guy, this guy is behavioral Economist, famous dude, lots of big studies, and then there are just so many problems with you know. I mean he was published for for years and years, I mean for probably 30 years. He had a huge career and then they, they kind of quietly, he quietly left his university after there were some problems raised with with his papers. But the problems are like hey, look at your sample size, look at your mean, you can't get this meme from this size without having some kind of outrageous things going on. And so it's like well, and, and then it becomes even harder to interpret these, these papers, because it's like I do think there's some values still, like I think some of his ideas are still well supported from some other, some other papers as well. But how much am I being influenced by these papers that are now kind of under a pretty high degree of Suspicion? Yeah, a very similar things occurred in the inquiry-based learning space. Show more

I feel like, you know, inquiry-based learning has not a lot of like high quality research. Like methodology is often extremely, very I mean it's. There's even studies on the level of variation, of execution, of inquiry-based learning. And now you've got these meta-analyzes saying: well, it doesn't really seem to have much of an effect, in some cases can even be detrimental, but it's like, well, you know, are the principles or the theories behind it, like actually fairly valid and are components of it supported in it? In itself, like it is true, but it's just that the research quality is so variable- it's- you know, so not you know, there's no standard for it- that it's developing this bad rip. There's also this question of like: how hard is it to do a task, how hard is it to like? Show more

I mean this is going to come up in the video in a little bit- but like flash cards, flash cards are a really easy thing to do and, in the same way, like there are some other techniques like, like doing free recall techniques or doing other things that it's like these are easy to do and they are easy to test. If you are going to design, say, a project-based learning scheme or an inquiry based kind of kind of thing, the number of decisions that you have to make, that all have to be going well, is there's a lot, and there's a lot of areas where you could screw up, and so that makes doing the research in that area a little bit more challenging. I think the last thing I wanted to say- sorry, is when you're talking about these naive ideas and intuitions about how we learn and how illogical they are. Show more

There's this idea- he focuses on physics education, decessa, who has come up with this idea of knowledge in pieces, which is basically like our, our naive intuitions about physics, which we get through various interactions in the physical world and maybe early interactions with other people. You know, they're kind of like little pieces that are separate, that when we go to answer a question or think about something we pull together to kind of come up with some kind of coherent thing. But individually they are internally, can be at least internally inconsistent. The same thing is going on with learning, where if you look at like, let's say like, learning styles is really inconsistent with the kind of retention chart where the like you learn 20 of what you see, or thirty percent of what you see, 20 of what you hear, whatever, all the, all that kind of stuff and so a lot of these kind of core ideas, like if you start thinking about them for more than 10 seconds, if you, if you don't just accept them. Then you see that these are just, it's just a mishmash of different stuff, foreign. Show more

This is not a good sign for the conciseness of this video. Yeah, I'm sorry, no, no, but I love it. And look, I'm here's. What I'm thinking, right, like you know people that are watching this video- is that I said it in advance, we're going to go a little bit deeper. Show more

On the surface, not your cup of tea, that's fine, that's all good. You know, feel free to move along. But if this is, this is a type of discussion that honestly, like it's very hard to access this kind of insight about things you know researchers don't like talking about this. Practitioners often don't even really know about what goes on in some of the nuances here and, at the end of the day, like individual Learners impacted and I feel like knowing why certain people have certain beliefs and knowing how people can be misled and understanding that gives more credibility to the alternative ideas. It's like here are the reasons why you could have been misnatal, you might have these misconceptions and things, and just spreading light on that I think if even just you know, like a few thousand people watch this video, I think that's a few thousand more people that can maybe spread their message along. So for me, I think this is it's good. I think it's doing a service. Anyway, let's just jump back into this. Show more

Yeah, let's start with some of the basics here, which Justin goes over really well. Here's how this works. Some of this information that initially came in through sensory Pathways goes into our working memory and then from our working memory where you're going to be spending a lot of your time when you're manipulating the information. But it's not a good place to hold on to it, because the working memory will also forget information relatively quickly, in the span of sort of seconds to minutes. I think if I were to do this video again, I probably wouldn't have separated sensory memory and working memory, because I feel like it added additional confusion and I would have just stuck with the idea of information. Show more

And this is the way that I teach it now is. I just say there's information in and it goes into your brain and then it's processed in some way that leads to it being encoded in your long-term memory. I mean because I felt like it opened up a can of worms to talk about the idea of sensory memory versus working memory and then sort of differential levels of Decay and also the debate around what, what is working memory? Even, is that even a thing? And you know what is the relationship. So I just felt it was, yeah, quite messy. Yeah, I mean what I do is- yeah, I mean I don't include the sensory, and you notice, I started your clip, I think, after you had already talked about the sensory memory. Yeah, I mean, I, I think I just do working memory and long-term memory and and leave it at that. Um, the reason I included it was because I wanted people to understand that, you know, intention alone does produce a beneficial effect on retention, but that that effect is relatively short-lived and it's not particularly scalable or sustainable, and so that's why I decided to do it. But in retrospect, the idea is just quite complicated. To try to communicate, I mean, I think it's also important to know that, like what there's, there's a- I mean- at least I think I'm correct in saying this- but there is a difference in like what you perceive is not necessarily what you are working with in your, in your memory, right. Like you can, you can imagine things that are not being perceived right and and you might be kind of staring off into space- but working with something in your working memory. So it's kind of, I mean, I do think it's, it's fine, it's, I think it's a reasonable, a reasonable thing to do. But yeah, I agree, like like just from a explanation standpoint things as well. But it will forget things much more slowly. It will forget things in the span of hours to even months and it depends on how strongly it was encoded in this process. Here my little tiny addition is just to say that memories in the long-term memory system do not just kind of disappear. They don't go away like they do in your working memory. So if it's stored in long-term memory, the main problem is how to get access to it again. How can the brain retrieve and find the memory again? Show more

So what's Justin's B for the active recall? Active recall and space repetition is inherently. This is the biggest regret that I have. When I talked about the idea of active recall and space repetition, I I didn't realize it initially and there was a bit of sunk cost fallacy because when I reviewed the video I was like I really should have added more about this. Show more

The biggest regret I have is not clearly differentiating active recall versus Space repetition and the usage of active recall or recall based techniques and the usage of space repetition separately, and because I grouped active recall and space repetition together- because it is often grouped together in in popular mainstream sort of things, like I to this day, I continue to get people saying, like, what's the issue that I have with active recall? Show more

And it's like, oh crap, like that's not really the point. Actually, the point I had was when you are using and relying on, you know, spaced repetition and active recall is your primary form of revision, which is operating at a low order, like exclusively, and then that is essentially like your entire system. This is that. That's that's probably the biggest problem that I have with with my own video. Anyway, I don't know what you're going to say about this, but I will just. Show more

I'll just continue actually here just for Justin to critique his own video here. It's just it's been a long time since I watched this and now that I'm looking at like, now that I've looked at this again, after having spent the last year kind of answering questions and refining and clarifying again, like there are certain things that I've taught so many times and I never realized that it could be problematic when, like until I taught it to 100 000 people and then I realized, oh crap, like actually there's this whole permutation branch that I just didn't happen to encounter because my sample size was only, like you know, like a few hundred students that I taught this to before, so, anyway, very repetitive in fact. That's kind of the whole reason it works is that it's repetitive. A lot of the students that are using space repetition active recall based systems, will find that it's not actually working as well as it seems like it should be working. Show more

It's not giving them the results that they've kind of been promised, that they expected to get. It can be extremely monotonous and Incredibly tedious, very, very time consuming and, in fact, actually pretty demoralizing if you're not getting the results. He basically says active recall is okay but not that helpful and all it's doing is really helping you to move these forgetting curves kind of longer and longer and longer, but that you really should be focused on something else. It seems like that contradicts what I say in this video. Free recall is one of the most powerful and basic study strategies there is, but it it doesn't. The thing that he is criticizing is actually not the thing that I am advocating for. Let's dig into the language he is using, because I think this is really, really important. The technique that he's criticizing is very common technique that a lot of people are using. It's easy to use, very easy techniques to learn. Pretty much anyone can pick it up. It doesn't have any real learning curve. He mentions flash cards a number of times. Flash cards, cards- I can use this term- revision, or we're biased to talk about the recall: repetitions: revise this information, repeat this information. Repetition and revision. Now what he's talking about is called a spaced retrieval system. Determine literature. Show more

No, well, yeah, I, I, actually I, I did it and I was like I think no, so space doesn't SRS stand for spaced retrieval system? And then space retrieval, space repetition system, maybe? So this is the yeah. So here's one of the things I realized: space retrieval is the proper term. It is, it's retrieval practice, it's retrieval, it's- you know, the word is retrieval- like that's ubiquitous in the mainstream space. People just know it a space repetition. I don't know where that originated from, but people understand the space repetition. So I call it space repetition. When I talk about it in public, right, but in if I, if I do writing about it or if I'm publishing about it, or if I'm talking to people that know about the science bite. I call it space retrieval. So you're right, it is actually space retrievals. Yeah, I was kind of confused when I was editing and I was like, wait, did I maybe I just made a mistake? No, no, yeah, you were right. It's funny how I mean I mean this. The more that we go along this video we'll see. But, like this is another aspect of how research gets translated into practice is the language that people use, and sometimes I mean there are anything that you can think happen will happen. Happen. You know, like the, a word that's used in the research Community to mean one thing can mean something else in in the practice Community, or or various. I mean there are various research communities, various practice communities, a thing that that you know words that get used in the in the research Community sometimes never actually make it out to the. You know you, you can have two different words for the same exact thing, or, yeah, I, I mean everything that that can go wrong does go wrong. And I do think that paying attention, especially if people are, you know, I I don't know if listeners, our viewers here, are going to like like our reading research, but but paying attention to the language that people use, because it's really easy to get confused and to think that they mean something different or the same when they don't. Yeah, the like the number of times- and especially early on, before I realized that there are lots of synonyms- I go down this path and I'd be like, oh, there's not really a lot of research on this and I spend a lot of time and then I'll just stumble across, you know, one connected article that uses a different term and I search for that term and I was like, oh okay, yeah, like here it is. Show more

That is, you know the number of times that that has happened, the number of dozens of hundreds of hours I probably would have saved on searching for things if I just knew to scope the keywords properly before I started. I guess this is why research skills are important. Show more

But anyway, retrieval practice sounds almost the same, doesn't it? What he is talking about are apps like super memo, Anki Quizlet, memorize, all of these flash card apps, and there's a million of them out there. So he is critiquing a very specific form of retrieval practice and he does not not talking about things like free recall or other more effective retrieval based techniques. I also think flash cards, I mean, I don't know, I made that assumption terribly. Show more

No, you're absolutely right. No, you're completely right. So in my program, actually, one of the first things I teach people is how to do spacing correctly and different forms of retrieval using interleaving principles, and that's one of the first things I teach, because the idea is that encoding requires a more complex series of cognitive skills. Like you say, there's lots of micro decisions that have to go well in order to be able to do the first pass encoding at a level that is high, high quality, and so, as a result- especially if you're doing it- a higher order- it takes time to build that skill to the point where it's really going to make a decent impact, and so during that entire time, just the the net at the bottom of the cliff is going to be Gap finding and testing from these different angles and actually being able to do the retrieval practice. Show more

So what I? One of the first things I get people to do- is understand how they can do effective spaced interleaved retriever, called sis. And if you can do effective space interleaved retrieval, for most people that just fills the gaps and then you've got a great, steady, relatively error-free platform that you can then start building on to slowly work on optimizing all those higher order learning strategies. Show more

So that's completely right. And and again, that's one of the things that I kind of regret is not really clarifying the difference between that, because it created some issues, because I started creating videos later that talked about different, you know more effective- forms of retrieve. When people like, well, aren't you kind of against this and it's like I'm not, but I can also see why you'd think that and so, yeah, but at this point I'm not going to re-record that video, no. So, and the reason is that they focus on isolated, decontextualized. Show more

They do not encourage synthesis or knowledge reorganization inside the brain. They do not encourage you to compare or contrast things with one another. They do not encourage you to apply things in new context. So, for all these reasons, flashcards are a very weak system for learning. Let's pay attention to these ideas, shall we? I think the most effective studiers either do not use flashcards or use them very sparingly. So what does Justin recommend that people do instead of using flashcards? He wants people to encode more effectively. He thinks people are not very effective at remembering information when they first hear it. They're not trying to understand the information deeply enough and they're kind of revving their engines a bit because they're so focused on getting the information into their flash card app that they're not actually paying attention to the information in the first place. Now, it's interesting that he uses this term- desirable difficulty. I actually are things that make. Show more

There's a point that after some people were commenting on the video and I, I I had re-watched- I can't remember if if it was after you contacted me or before, but I had rewatched portions of your video again and and you see something in there that I wish I would have included in this video, which is that you know you can't retrieve something that's not there. And I noticed, I mean, if you pay attention to this when you, when you're learning- because I mean I've used flash cards in the past and I still like, if I was going to learn a language, I might use some flashcards in the beginning. But I've had this experience before where if you're on the flash card app and you put it in and you miss it that first round, and then it's like with Anki or whatever you, it'll, it'll prompt you again in I don't know 10 seconds or one minute, depending on your settings, or 30 seconds or whatever, and it's like: and then you're like, okay, probably again you know. And it's like: well, did you do anything like? The question is, why did you not know that? And just being prompted again 30 seconds later is not really going to materially change what you know. Show more

If all you look at, oh, oh, yeah, this is supposed to be that. Okay, that's it. So that's the only other extra bit of kind of what you might call encoding that you did is like: oh, yeah, this is supposed to be. You know, X is supposed to be y. Okay, I'll remember that next time. And- and I think that that that is an important point that, yeah, I wish I would have emphasized in this video, or Russia would have excerpted, because I think, um, yeah, I mean, it goes along with what you are saying, goes along with what I am saying. But, yeah, there's another video that I made around. Show more

You know, the same is exactly the same thing. That happens, but with people doing practice questions and practice getting you look at the answer sheet. They'll look at the answer sheet and be like, oh, I knew that. And then, well, if you knew that, you would have gotten it right. You know the fact that you didn't get it right, and a lot of people say, oh this, I just made a silly mistake. Well, it's like it's a silly mistake if it happens once, but if you're making silly mistakes all the time, it probably means that they're not silly mistakes. And so people confuse their ability to understand and answer with their ability to retrieve the answer. And one of the dangers is that you know, if you're not confident with your answer to begin with, that probably already indicates that there's a gap for you to explore. You know whether you got it right or not. Even if you got it right, you might have just gotten lucky that single time. And one of the things with flashcards that I think that happens a lot because people don't really see the difference between queued versus free recall. And so what happens is they: they just build pattern recognition for that very specific cue of that flash card. They'll read the first three words, remember what that flash card was, and they'll be able to record the thing without really having to do any of the actual type of retrieval process that you would need to do if assessed in any other context. And so all you'd have to do is simply change the wording of that initial card, and now, suddenly, you can't record that information anymore and so there's, and it creates this illusion of confidence with it and people can be like very, very confident. They're like, oh, I'm smashing through my flashcards like it's only taking me two seconds to do a single flashcard. It's like you can't even read the question and think about it in two seconds. If you're doing it in two seconds, it means that you're not thinking about it at all, like that's not useful at all unless you're tested on that very flash card. You know. So it creates this illusion anyway. Yeah, for sure. Yeah, learning harder in the short term but result in longer, lasting, durable, more flexible learning. Show more

The classic examples of desirable difficulties are space practice, interleaved practice, contextual variation and testing. So space practice is when you space your practice sessions out or space to your studying sessions out. Interlude practice is when you mix up problem types that you are working on or category types that you're working on. Contextual variation is where you see ideas in different contexts and are forced to say, apply them in different contexts, or very the situations that you're practicing something in testing. That's just, that's just testing. There are still some things we don't know about why these techniques work, but the research consensus is that part of what makes all of these techniques work is retrieval, with spacing. You have to use retrieval because you've forgotten what you were learning and you need to remember it again so that strengthens the retrieval path. So here's a question that I have see is: is it that the the the commonality between them is the retrieval or is it is the commonality between the degenerative effect, generative effect of learning on them? And the reason I ask that is because there are some studies that are done on the idea of self-explanations and it's kind of had a research dearest over the last, I don't know like there were a few studies on a, you know like quite a long time ago, and there hasn't really been too much going on the last sort of seven years or so, I'm not sure why, because I thought it was very promising. But you know they looked at the idea of you know how can you facilitate self-explanations? And this was from an educator approach. Show more

You know, being able to do a guided self self explanation versus being able to do a spontaneous self-explanation, and the idea was that when you do a spontaneous self-explanation you are able to have more generative effects of learning and therefore your learning quality is going to be better and it's, it's going to be stronger and that seems to be arguably there as a level of retrieval that's happening already, like straight off the bat and the, and that you're taking information and you're immediately trying to retrieve it and synthesize it at a higher level on the spot to be able to create your self-explanation. But when people think about retrieval, especially retrieval as retrieval practice rather than retrieval as a cognitive phenomenon in a way or process, that's not usually how people are thinking about retrieval. People are not normally talking about retrieval as in like, literally consuming, immediately, thinking about it and manipulating that. So I- yeah, I have this question mark. So is it the retrieval that is the thing or is it the fact that they are engaging in this generative cognitive persons that? But yeah, I, I think I think one thing again, always coming back to the language issue, is: what do we mean by retrieval versus encoding? Show more

And so, like someone had asked me that question a little while ago in a comment, and you know, one way of thinking about it is to think about it like: I mean this is, this is kind of How It's operationalized a lot in the research studies, and obviously you can speak up if you disagree with me, but but from what I've read is that the a lot of times encoding is when you have the material right there with you and you are looking at the material right like, and retrieval is when you don't have the material with you, like that's a lot of times in these experiments. That's, that's one of the key differences from a practical perspective. Show more

You can also think about, you know, when you know- it almost sounds like to me when you talk and when I I've seen seen some of your videos- that you know, encoding is, you know, equating with this idea of elaboration or or kind of with not necessarily the practice of elaboration, but kind of these kind of higher level restructuring of your memories, right, like, like. It's kind of like in a flash card context, you're kind of putting something in there and it's just sitting there right and and as you said, you're, you're basically training your brain to retrieve that at a particular given cue or given context, to retrieve it if. Show more

If it is kind of quote-unquote encoded properly in your long-term memory, then then it is. It is connected and integrated with lots of other information and Concepts and ideas and and contexts at a higher level, and so that is what is enabling you to use the information effectively. But but it is. It is. I do think that retrieval is- if you look at the research I'm forgetting and why people forget things- this is one of Robert bjork's phrases- right, retrieval is a memory modifier, right? Everything like encoding. I mean differential attention, retrieving things, reorganizing things, how you initially read something where your focus is like, all of these things affect your memory for something. I do think maybe we can dig deeper into this at some point and both go read some stuff and come back and and talk about this. But but I do think that on the whole, retrieval is at least unimportant underlying process that is making these techniques effective. I, I, yeah, I completely agree. Show more

When I talk about encoding, I usually talk about in terms of the quality of encoding, and so you know when I, when I talk about encoding, it's the way that I'm viewing it and this is the way that I have understood it. But actually correct me if I'm wrong and that I've misunderstood it is that encoding is actually just any process that is taking information and shunting it into your long-term memory, whether that's still high quality or not. And if, if that's, you know, sort of decayed very quickly, then it probably indicates that it was too done to a lower level of strength and- and this is kind of where sort of constructivist models would come in, which is contentious in itself, but that if you have encoded it and there's the structure in which it sits in from an actual point of view, is that of a higher knowledge, higher order knowledge structure, something that's more integrated or relational, that correlates with having a like longer, more meaningful retention of the information, and so you could say that the processes that led to that type of knowledge structure would be considered like of a higher quality or which correlates with the higher order learning and higher order thinking, that kind of host space. And so when I talk about encoding, it's like: well, you could encode it really anyway, it's not that encoding is hard, it's just that encoding to a high quality is difficult. Show more

And when I say high quality encoding, I am generally referring to higher order thinking leading to higher order learning outcomes. That generates higher order knowledge structures. And when I say higher order learning, I am referring to Bloom's revised taxonomy there as guidance and, in terms of knowledge structures, I am referring to solo taxonomy as guidance, even though I know that's not really how they were designed to use, but I think it's still relevant to use that way. And, and the other important Point here with as relationship to learning, is that you want your memory to change, like we think, like in the eyewitness scenario, or like we're like, oh no, like I have this, I have this memory with about you know this argument with my spouse, or something, and like I want to be able to remember it accurately or something. Show more

But with learning, it is not a bug, it is a feature. You want to be transforming your learning because you do not know when something is going to be and you have some idea, but but you do not know this necessarily when an idea is going to be useful. Yeah, again, conditionally. Continual knowledge Parts: yeah, yeah, you, you want to be able to be like: oh wait, does this, does this apply in this context? Okay, now, now that I've used it here, well, I'm gonna like that helps me to kind of see that that this idea is deeper than I thought or more more widely applicable than I thought, or what have you. So it's not. It's not bad to modify the memory. It's. It's a good thing to that the memory is modified. Obviously you don't want to remember things that are false. Calibration, the feedback, the kind of. I mean that's also a part of the learning process. Like you don't want to over generalize, you don't want to under generalize, yeah, and, and some of the most important concepts for learning require you to be constantly modifying and retrieving it to be able to understand it to the level of depth that you need to to really see the right perspective. And that's the whole field of threshold Concepts, research, which I don't think I want to get into right now. But yeah, so you know it's, it's a very important feature. Yeah, it's not like you said, it's, it's not a, it's not a bug, it's a feature for leaving some of the things, same things are going on. Show more

You have to activate different Association networks alternately as you are looking at different kinds of problems with contextual variation. Again, you are kind of departing from a practice and then coming back to the practice again. So it involves some some retrieval, some recall, some memory, and the same is true of testing, and I have a bunch of videos on these getting more into detail in the description. So, if flashcards are not effective, what is effective retrieval practice? Well, this is where Justin Sung's video on encoding comes in. As soon as possible. We want to just jump to higher order learning, and that is something like applying, but more so things like analyzing and evaluating the ideas. It's not just about having one idea and then just understanding that idea of really, really well, and focusing on understanding that idea. It's about taking that idea and then looking- oh, I just realizing- I never realized this before- in the background here I'm actually mining ethereum because this video is playing. I just realized that, yeah, this is just me mining etherein the background- oh, my God, I didn't realize that- to another idea and comparing and contrasting between them- but not just one or two, but multiple different ideas- and seeing the relationships between them. And then the next step, evaluating. This is about not just comparing and contrasting the ideas, but it's about figuring out how we can judge them, how we can prioritize them. In order to do correct encoding, we always need to try to relate the information to each other. Yeah, it should be higher order, not correct. So it requires synthesizing Concepts, reorganizing Concepts, applying ideas to new contexts and comparing ideas to each other sound familiar. This advice doesn't just apply to encoding. It applies to retrieval as well. If your retrieval practice does not have these things. Show more

It's just not going to be that effective. Spaced repetition apps get the spacing part, but they miss out on all this other stuff and that's why they're not terribly effective. When you're thinking about retrieval, one of the questions you can ask yourself is: what is an effective way of testing my skill? And this is where a lot of the research on memory bleeds into the research on deliberate practice. This I have to. Show more

I just have to just yell out at the moment just to sing with the software that's marketed for learning. Just because it has marketed a certain way doesn't mean that it's actually designed to work. Like a lot of these software companies do not have Educational Learning expertise. It's just a series of software developers that had some random ideas about how they should do learning and they decided that they could hire someone that knows equally little about Mark, about learning, to do the marketing for them, and now they just sell it as a learning software. Show more

But you know, like it's just so. Yeah, when you were talking before about having about some companies not really having someone in this like research practice role, and you know, when I was in grad school, you know, look looking for jobs and I, you know I mean I was like looking for edtech kind of kind of jobs and it is amazing to me how many job openings there are for software developers and edtech companies and how few jobs there are for actually people who have some expertise in, in learning and instruction. Like I'm not saying this is true across the board, but it is amazing to me. Yeah, how many companies, yeah, and. But the thing is like then they have these like cult followings and it's like they're not on your team. Show more

This company is not looking out for you. Like you do not need to be loyal to them. If they stop, if you stop paying the money, they're not gonna care about you. You know, like so I'm, I'm gonna go out and just say that it is actually, I think, across the board I've been working with edtech companies, software companies around- you know b2c for Learners, as well as for Learning and Development and training programs, e-learning, design, LMS design. Show more

I can think of maybe three companies total where they actually had someone on the team that actually knew an amount that I think is ethically responsible for the, the product that they're designing. A lot of people have place, people that have the position, but they know shockingly little about the research beyond what like even just a typical YouTube, like you know, like a student would know from watching a few YouTube videos. Show more

It's terrible. And also I have- and I won't name names because I probably get sued for it, but I've had people that have contacted me ever since. You know, like my, my personal brand, has become a little bit larger. People- it's like software companies- have been contacting me. I get a contact from, maybe from a software company every month and they're saying, like can you, you know consult or whatever, or you know, be involved in some capacity? So when, when I, when I talk to these companies and I pitch to them and I just say, like I can work with you. However, if I have a recommendation, it's going to be based on what's going to be best for the learner. Like, yes, I'm going to have commercial viability in mind because on my other half, I do see myself as an entrepreneur. So I do understand the idea of having to run a business and I believe that you can have an ethical private education Business Without, like, compromising on the quality. Show more

But when I say that I've only ever had one company agree, that is to say that, out of principle, these companies aren't willing to change their product if it's going to hurt their bottom line or their Top Line, like they're not going to be willing to make changes to for the benefit, they're not even willing to hear it out, they're not even wanting to do the consultation if that is the risk that it involves. I mean, that tells you a lot about their values. The problem is is that everything is done the wrong way around. Show more

So rather than- rather than like, understand the need, really dig deep into the research and kind of create a product that fits a student's need or populations need that, that the kind of they already have, they start out with a product that maybe was developed for some other reason, that then they are. They're trying to find a fit, they're trying to find something to do with this product so that they can make money. And so then it gets slipped in to say that the learner space, or sometimes a language learning space or or whatever, and it's- and- and in some cases it's like this- isn't the absolute worst thing you can do. But but the product like one, you're selling it. I mean, you're way overselling it by saying, well, it's going to do all these things and it's, it's really not a complete system at all. And the other thing is that, like, as you say, if you would, you know you, you could change the product to make it more effective for students. But that requires actual work, right to change the product and your whole like. Show more

If you change the product like your whole, um, the idea that you have been selling- now I mean this is- this is something that I talk about some of my other videos- right to to sell a product. In a lot of these cases in the learner learning space, you have to sell ideas right, like, like. You have to sell that spaced repetition systems actually work, if, if that is what you're selling, right, you have to sell bring the brain training programs are going to improve your brain, if that's what your program is, and so it's the idea that that you are selling. And so if you change the product to suit the, the Learners kind of more effectively, then that whole marketing Spiel is gone, that whole like, like. And you have to admit like that wasn't the right thing to do. But but rather than, yeah, rather than come at it initially and kind of grow as you learn more stuff, which is what I would want to see companies do- it's kind of like they have a specific product, they have a specific brand, they position themselves in a specific way and they're not going to change. Yeah, there's no accountability for it. Show more

Who's going to punish them? No, there, there is not, and this is something that that, that is, you know, it comes up, man, I'm making a video on speed reading right now and it's like seeing people talk about speed reading it makes me so angry because this is just, it's just a lie in the sense that, like, if you are a good reader, you can't read effectively Beyond a certain speed and to some degree, the whole point of reading is to digest the material and understand it deeply. And but, but people, you know, users- will try to use it and they'll be like, oh, hey, like, yeah, now I'm reading x, x amount fast, you know, with, with quote unquote, No loss in comprehension. Show more

It's like you don't even know what comprehending something would mean. Yeah, like, like there is no test for comprehension that you're giving yourself and and you would have no idea if you did lose any comprehension, because you have to choose right, like there's no control group for you. So, anyhow, yeah, I've actually got a video on, like, the idea of reading as well and the you know, people are often asking me to make a video on speed reading and things like that, and the line is always: that's not where your rate limiter is. Show more

I mean, unless you, unless you really are struggling with the language, you know, I think to a certain point. You know, being able to read faster has has good Returns on it, but at a certain point- and that point is usually very low, like, I'd say, anything beyond 150 words per minute- you're getting very diminishing returns on being able to, actually for high level text, I mean, it's different. If you're reading something very easy, then sure you get more returns out of it, but at a certain point the limiter is certainly just the ability to process the information in your brain and do something with it, rather than the ability to take in information which is just you know. So these people that are like, oh, I read this, I read this book, and like you know one day and it's like cool, well, I mean, it's kind of meaningless, but you can just pat yourself in the back, I guess, because you've fulfilled a word count, but yeah, yeah, but people understand it easily. They know, like, speed reading, oh, I know what that means and so, yeah, anyway, it's really simple, but everyone forgets it. Show more

Generally speaking, you want to practice the skills that you want to get better at. If you're studying a language, having a conversation with someone in the target language that you are studying involves retrieval. That is, a retrieval based practice. Most of the stuff we learn in school is about applying information. It's about understanding things. It's about flexibly using Concepts in new situations. If the retrieval practice that you're doing doesn't involve these things, you got problems. It's like playing t-ball all your life and expecting that you are going to suddenly get good at baseball. Now there's not a lot of studies that directly compare, say, encoding methods to retrieval methods. This, however, is a really good example and just to show you the results here, this is two kinds of different tests. Show more

We've got concept mapping, which is a version of elaborative encoding, and we've got retrieval practice, which is a version of I had. So this is quite a well, I don't know if it's actually so heavily cited, but it's I. I feel like it's a well-known study because, like you say, it is one of the rare ones that that look at a form of of encoding compared to the retrieval and the thing, the problem that I always had with this and the findings from it, which I think that the author, the authors, kind of mentioned, but they don't really like make it as big of a thing as that. Um, the actual method that's used and the ability to ensure the accuracy of that method at a cognitive level which is, realistically speaking, almost impossible to actually standardize without having very significant rigorous training and examination, to you know, tested competency beforehand, the- the effect is going to be, like, you know, massive, like the, the difference is going to be massive. So one of the things that I say- and this is the reason why I always teach retrieval practice first with people that untrained, if you're starting from a baseline where you've got relatively low metacognition and insight and relatively low level of either inherent deep processing, just innate ability because of your genetics or random upbringing or whatever it is, or you're not very familiar with higher order learning or higher order thinking, you always start with retrieval because the learning curve for doing retrieval practice correctly is pretty low. Like even higher order retrieval practice, techniques are pretty easy to do in terms of the technique. Now it might be difficult to be able to do it successfully. For example, a very high order retrieval practice that requires you to, you know, synthesize, you know, highly integrative pieces of information. You might struggle with actually doing that. But the tech, the actual method of engaging in that practice, you do it correctly. The difficulty would be and that you just actually can't retrieve the knowledge, and that's kind of like the whole point, right, just to figure that out. Show more

Okay, yeah, so that that's good. But the actual technique- Fidelity- I- I guess some people would say Germain load- is relatively like good there. However, with something like correct encoding principles and correct encoding methods- number one, there's not as much research by far- and also the, the number of sort of micro decisions that need to be made correctly and the amount of self-regulation that is required to execute on that is much, much more varied. And so, like we're finding that there are some students in our program that can get it in, let's say, you know, two or three months, but that's two or three months of like pretty, pretty rigorous, calibrated training. Some people will struggle with that for over a year in the same process, you know, due to so many different variables. So the thing that I, the issue that I have with this paper is that I think it does underestimate the complexity of getting the encoding thing right in the first place. I mean, this is something that comes up in learning and teaching in general and a lot of other research areas too. Show more

But is you know, there's there's a lot of different things that you could call an encoding approach or a lot of different approaches to elaboration and for the. You know, in most cases, if you are running a single kind of study, you, as you say, you have to kind of collapse all of these decisions to choose kind of one thing. Show more

Even if they're, even if there's like 100 decision points, right, you've got to make all those decision points and say, okay, well, we're testing this thing. And I think I mean you mentioned- well, you mentioned it in in your video about how it's like. You know, flash cards are kind of better than doing absolutely nothing, but are they the best thing possible? No, and in the same, in the same way, researchers kind of had this choice about: do you, do you try to test something that people actually do or do you try to test kind of the best version of this thing that you can think of? And and this comes up with, say, like, like, in some of the research on like Active Learning, where you know any departure from like, straight up lecturing to people pretty much improves their learning. Well, like, like, if there's any kind of questioning, any kind of interaction, any kind of problem solving, any kind of thing, like as long as you don't completely mess it up, but but it makes and- and that's a comparison, that is like, well, everyone does lectures, even like online courses, they're just more video lectures, everything is video lectures these days. And so it's like, well, do you compare it to the things that people actually do and do you try to push the envelope that way? Or do you say, well, you know, let's, let's make a really meaningful comparison, which, which might, might be here, like, actually, I got, you know, one of your, I think one of your students came in and was like, yeah, like that's not how Justin does encoding, so so that you know, you know, if you did it the way that Justin does it, then that then the differences would would be much smaller. Or would you know, you would see, you would see how strong the encoding is. And it's like, yeah, it's, it's entirely possible. I think that's like you know, it's a choice that this researcher makes and, and this is like the. This is the thing where I think and correct me if I'm wrong, but I think like distinguishes you and me and maybe some of the other people from who are research oriented. So it's like the answer to this isn't and I just throw this research away, it doesn't matter, the answer is more research. Show more

Right, like, like it is a like, like. It's like, oh, okay, if we, you know when, when you're standing on kind of the corners of of practice and research and it's like, well, hey, like this is why we identify research gaps. This is why people do more research is because it's like, hey, they found this in this study, but I don't think that's that. That's quite right. They didn't use a super effective, you know, concept mapping or mind mapping technique and like, let's test a different technique and and see if, see if we could. You know you could keep all the protocols the same and basically, and test a different technique and and you know, personally I think that would be an interesting contribution to the research. And one interesting thing that I feel is that the the effects of retrieval practice essentially always effective. Show more

Show more

The effectiveness of it scales with the quality of the knowledge structure that you've got access to. So if you have a very weak knowledge knowledge structure that's fairly isolated, then your ability to actually do sort of the higher order types of retrieval practice sort of methods is somewhat restricted. Like you can still do it, but you'd probably have to do it like multiple times to really be able to kind of push through each time. And if you, it's very easy to get out of the scope of, I guess, the maximbenefit that you get for that time. So if you only have, like, let's say, one hour to do a certain type of retrieval practice and you want to do something that's higher order, but your knowledge structure is so weak that as soon as you start thinking that way- it's like I just don't know and you've just never thought about it that way- then yes, you're able to start kind of, you know, reinforcing that learning and supplementing and strengthening it and it will become stronger. But then a lot of that time is spent on just creating a knowledge structure that you probably could have created to begin with without having to spend that additional hour just to figure out where those gaps are. And if you do the exact same thing again, let's say a week later, you'll find even more gaps. So you would push it up, but it requires more frequent and higher volumes of that retrieval, whereas if you're able to kind of push out a reasonably good while integrated knowledge structure on the first bat, then when you do your retrieval practice you're able to hone in on those, those smaller issues, much faster and you're able to consolidate at a wider level. And it means that the full range of sort of higher order all the way through the lower order retrieval practice methods is like very high yield and meaningful and you're able to find those smaller gaps rather than just filling major foundational weaknesses. And I think this is exactly what happens in real practice where people are doing practice exams or or even not even doing practice exams properly. Show more

They'll set an exam and during the exam that is the first time they've ever tried to think about it in a certain way and they leave the exam thinking, man, I actually learned a lot from that. I just don't feel like that's a good feeling to have after finishing that. I don't think you should finish an exam being like man- I learned so much about this topic, you know, like you'd hope that that had already been done, like you pushed yourself beyond that before the exam, but that actually that's a very common experience a lot of students have, I remember, doing. I was in taking math classes in undergrad and I I was very lazy and so I wouldn't do the homework, and then the exams were kind of like: oh, this is really challenging, this is really good, you know, because I I don't, you know, like I don't really know what I'm doing, and so I have to kind of like figure things out during the exam, which that is not a good thing to do. Show more

If you want to get high grades and really deeply understand something. You, you would like to be taking those exams before the real exam. You know, um, I think the you know. One point that may be dovetails with what you're saying is that the the longer you wait- and this is something that in this particular paper was not, you know, it was basically like study the material, then immediately do the thing right, which you know. I'm not even sure if, uh, if, say you would- I mean, I don't know if you call what they, what they call retrieval practice. Retrieval practice, like would you call it encoding- if you immediately read something and then you close the book and you sit and you try to try to remember what you read or kind of try to create some structure out of what you read, like that, yeah, like, but but there's also this, this time lag, that's not being manipulated here. Show more

That is like if you were- yeah, obviously, and- and this is where the kind of research on spacing comes up- is like if you let things go really really far and then you try to retrieve something that you barely understood very well and you forgot most of, that's not going to be a very effective way of you know of of kind of yeah, it's like yeah, you can recognize the gaps, and then you go back again and then you do it all again. There is a kind of Sweet Spot, there are. There are sweet spots kind of along the journey, yeah, and that that immediate retrieval thing is kind of- I think thankfully we don't really see it as much in the more recent research like retrieval practice spacing. Show more

Nowadays I think the trend is that people are trying to space things out in a more realistic way, like they are looking at slightly longer periods of time and there's less of those lab studies where you're being tested on it like one minute later versus one and a half minutes later, and it's like you know, we're phasing out of that thing, which I think is is good to see. Show more

Anyway, let's let's jump back into this. You don't have to choose between encoding and retrieval. Show more

You can do both. Encoding is really important and you want to do that effectively. Retrieval is also really important and you want to do that effectively. And for both of these activities, you want to be focused on the higher level thinking skills that Justin is talking about in his video. Go check out Justin's Channel. Learning is, legitimately, really really complicated and you can see more videos on all the topics I talked about in the description below. See you next time, awesome. Well, I had to pull that quote because I was like my favorite, my favorite quote from your video, which is just like: learning is complicated. Learning is a lot more complicated than people think. Yeah, a lot more complicated than people think. Show more

My favorite, I mean, I do, you know, I do like Instagram live and Tick Tock live sort of things, and someone's always like: what's the best learning technique? It's like: hmm, how do I answer that question? But anyway, you know this has been a great discussion. Benjamin, thanks for being on here and having this, you know, very deep, comprehensive, but for some people, especially people that are watching my channel and part of your channel as well, have the mind and the interest to want to explore these topics a little bit, a little bit further. So you know, I think it's a valuable discussion to have. I like the fact that we've got this and we've got it and we're able to push it out there for all of you that are watching this right now. Make sure to check the description below, check out Benjamin's Channel, check out the other videos, make sure to subscribe. Show more

You know one of the few people that I think are taking that real critical approach to to education responsibly. So I think definitely worth checking out. But thanks everyone for watching. If you've got any questions about any of the concerts that we've talked about today, just check them in the comments below and we'll have a look through them. But otherwise, thanks for tuning in and we'll see you next time. Foreign [Music]. Show more