Journal of the American College of Surgeons • July 2021 • Make No Assumptions, Give Lots of Tests

Disclosures outside the scope of this work: Dr Farley is as paid consultant to the American College of Surgeons and FundamentalVR.

Presented virtually at the Western Surgical Association 128th Scientific Session, 2020.

Correspondence address: David R Farley, MD, FACS, Mayo Clinic, 200 First Street, SW, Rochester, MN 55905. email: farley.david@mayo.edu

I greatly appreciate the honor and opportunity to offer this Presidential Address to my colleagues in the Western Surgical Association (WSA). I have so many wonderful people who need to be thanked for helping me get the most out of my life: initially my parents, for teaching me about unconditional love, the value of effort, and the importance of repetition; my brothers, Steve and Mike, for either pulling or pushing me to competitive success—they were great teammates and opponents in a constant effort to seek excellence in athleticism and academics; my wife, Cathy, the love of my life and the woman who loves me for who I am. She is an extraordinary teacher and educator herself, and I have leaned on her heavily in my efforts in surgical education. I am eternally grateful she has put up with this surgeon/workaholic and stuck with me for 32 wonderful and chaotic years. Thank you, Cathy. My wife and I were blessed with 3 amazing children 20+ years ago, and more recently, with a stellar daughter-in-law and an absolutely lovable granddaughter. The love and support of my family has been critical to my professional success.

My academic career was propelled forward by hundreds of surgical colleagues. Three big thank yous go to my uncle Harrison, a retired general surgeon, to past WSA president, Dr Jon van Heerden, and to Dr Mike Sarr. All 3 mentors took an amazing and heartfelt interest in me as a person and assisted greatly in helping me find meaning in my craft and my place in academic surgery. Thank you, gentlemen. Other surgical colleagues, well recognized by this audience, were similarly helpful in making me a better surgeon and a better educator. Thank you. I am indebted to my extraordinary secretaries, clinic nurses, program coordinators, and a research team that made me look better than I really was. I was blessed with multiple operating room crews that were extraordinarily patient when my trainee and I struggled to repair the hernia, complete the anastomosis, or simply close the skin. All of these scrub nurses and surgical assistants provided a learning environment that was paramount to my success and that of my trainees. A special thank you to the WSA, the members and guests in attendance tonight, to Dr Sherry Wren for a wonderful introduction, to the officers, committees, and a truly passionate executive council that made this job a pleasurable one. A big shout out to our administrative team, LP etc. They have been efficient, effective, and caring members of our WSA family. And lastly, thank you to the 300þ interns who evaluated more than 25,000 patients and operated by my side for 25 years. Having them come back 4 years later to scrub in with me as chief residents was among the highlights of my career: interns really can improve!

But learning surgery continues to get harder: more information to learn, more surgical techniques to master, and far more complex patients to care for. All this under time constraints makes being efficient and effective about learning critical to the future of surgery. Teaching surgery has not gotten easier, with issues of resident duty hours, trainee autonomy, HIPAA regulations, patient complexity, and carving out time to teach. Too many surgical staff, myself included, find it hard to be dedicated teachers each day and challenge our trainees to learn. But challenge them we must!

I have titled this Presidential Address, “Make No Assumptions. Give Lots of Tests.” The objectives of this talk are to remember and incorporate 2 tenets that I believe will make for better and more effective surgical education:

2.Testing trainees enhances memory creation and retrieval, making learning more effective.

As a young staff surgeon and general surgery program director at the Mayo Clinic in the mid-1990s, I had an “Aha!” educational moment: I worked with a solid trainee performing a thyroidectomy as a first case on M-W-F of that first week. I assisted the intern in doing much of all 3 cases. On Monday he had no idea how to ligate in continuity and divide the middle thyroid vein. “It is an important technique,” I said. The intern nodded and responded, “I like it.” On Wednesday we again got to the stage of mobilizing the thyroid lobe out of the neck, and I asked my trainee, “How are we going to take care of the middle thyroid vein?” The intern responded, “I am not sure.” I then proceeded to help him ligate in continuity: verbalizing how to grasp the vein, dissect posterior to it, and pass two 3-0 silks. We each tied our own silk ligature and he divided the vein and trimmed the 3-0 silks. “Oh, now I see.” On Friday, same operation, same blank look when we came to the middle thyroid vein. He had no concept of ligation in continuity. After helping him do it for the third time that week, I proceeded to give him some politely worded, but negative, feedback, which included me saying he didn’t remember any of the 5 steps I taught him about ligation in continuity. I closed with, “Do you understand my feedback to you?” He nodded his head and said:

While I wanted to scream, I burst out laughing and realized I had a long way to go in surgical education.

Humbling indeed. I was making assumptions that because his hands were working pretty well and he was nodding a lot, that he was learning the technique and incorporating it into his skill set. Actually, NONE of my teaching was turned into learning. I made at least 4 bad assumptions that week:

1.Good teaching leads to good learning. (Or poor teaching leads to poor learning.)

No. Just because someone is teaching does not mean others are learning. There is absolutely no guarantee that good teaching leads to good learning; same with poor teaching—that may or may not lead to poor learning. Learning something, anything—that must be done by the trainee. Only the trainee can learn, meaning changing what lies within their mind. They must be engaged in an experience, code it into their brain, and then be capable of bringing out that information or skill when they desire. Teaching is useful, but learning is incumbent on the student. Learning is something students must do.¹ More about this later, but it is clear I did not get my intern to do it.

Sometimes true, but evidence exists that experts often teach over the heads of their learners. I did this to my intern. Experts have information and skills filed away in an orderly fashion in their brains; novice learners have no such platform from which to build knowledge and skill. Inexperienced learners in a stressful environment (operating room, ICU, emergency room, etc) may be overwhelmed and learn little to nothing. A great teacher is not afraid to question learners frequently, using learner feedback (either correct or incorrect responses) to tailor ongoing teaching to assist the learner. I failed my intern.

Bad assumption. When learners ask an “off the wall” question or respond with an answer that seemingly makes no sense, they are giving the instructor clear feedback that learning is not occurring or registering. Funny questions or poor responses should be immediate triggers for teachers to re-evaluate the learning points and try again to offer something understandable and useful. I missed these obvious cues with my intern.

The assumptions that baby boomers, generation X, generation Y, or generation Z are somehow inherently less driven, committed, intelligent, or talented than previous generations are bad assumptions. I was thinking of this when my intern failed at ligation in continuity. Big mistake on my part. In my opinion, in 2020 this assumption of modern learners is wrong, and I believe it is detrimental to our learners’ psyche and detrimental to us as surgeon educators. Millennial learners are bright, talented people who were brought up in a different culture, but when turned on to surgery, they are every bit as resilient, resourceful, and committed to their education as the baby boomer crowd. No less than future US President John Adams made a comment about American youth in the 1760s being less driven and motivated than their parents.² I believe John Adams was speaking of young men and women who eventually fought in the Revolutionary War. I think we can all agree that our forefathers and for—mothers were plenty driven and motivated, and they stepped up when challenged. I think gen Y, gen Z, and ongoing generations will do just fine as well if we respect them, teach them, and really engage them. Let’s agree to avoid making assumptions when it comes to learners and surgical education.

Humans are born with more than 2 billion neurons. While we may lose a few million of them over the course of a lifetime, the brain improves function over time by adding new connections with each experience, pruning unwanted connections, and ramping up speed with myelin coating of axons. Indeed, at age 16, we have enough neural connections—upwards of 15 trillion pathways—and enough storage capacity to record literally every second of our lives from birth to death. The storage capacity of our brains for memory is virtually unlimited.³ The ability to retrieve information from the brain is not. This difficulty in memory retrieval is the crux of the problem for all learners, and especially surgical learners in 2020. There are thousands of facts, concepts, and skills to encode and store in the brain and then organize, consolidate, and process so that the information is readily available. This metamorphosis takes time and effort. A lot of effort.

On the upside, the human brain is a learning machine. Although our brains’ architecture and gross structure are determined genetically, the fine neural networks are shaped by experience and can be substantially modified.⁴ Increased experience and numerous repetitions build up more neural connections, stronger and faster pathways, and more easily retrieved information for that subject or skill. The thickness of the myelin-coated pathways correlates with increased practice; such myelin deposition improves the strength and speed of the electrical/chemical signals and, as a result, improves function and performance.⁵

Over the past 40 years, significant advancements have occurred in understanding how humans learn and how to improve our abilities to imbed knowledge and skill within our brains so that retrieval is efficient and effective. In 1979, when I was a freshman at the University of Wisconsin, I was a good student. I went to class. I took notes. I read the assigned reading. I highlighted the key facts. I turned my homework in on time. And like many of you, I crammed for each final exam: re-reading my notes and highlights to prepare me for the exam. Cramming for the English literature, organic chemistry, and Russian history exams—what a mistake! While I got As in the classes, within weeks I remembered nothing of the material. Zilch! Educators, psychologists, and researchers have made great strides over the past 4 decades to better understand the actual science of learning. I wish they could have taught me in 1979 about the weakness of “cramming,” or so-called “blocked,” or massed learning. They would have told me it is effective for short-term use, but it is abysmal for long-term retention. In our line of work, teaching and training surgical learners for 30-year careers, there is no place for cramming. What else does the science of learning tell us in 2020?

Learning is a process. It involves changing knowledge, skills, beliefs, or attitudes. But as surgical educators, we must understand, that learning is not something done to students. On the contrary, it must be something students do themselves.^1,6,7 That fact bears repeating: Learning is something that trainees do themselves.

Learning always builds on previous knowledge.^3,6 Unless there is some change in neural connections, no learning can occur.⁷ Teachers can impart knowledge and information, but only learners can create or change their neural pathways. Neurons form new connections with one another with every new experience we have. The more times the pathway is then used or tested, the stronger the connection becomes, and therefore, better retention, retrieval, and performance.⁸ Periodic retrieval of any learning helps strengthen connections to the memory and the cues to recalling it. In fact, the ease of retrieving a new memory, after a delay of a few days, is a good indicator of learning.⁹ The amazing thing with our brains is that as new knowledge is added to and connects with previous knowledge, each additional bit of information or skill “sticks” better to the growing neural connections.¹⁰ The more we learn, the better we learn. There is virtually no limit to how much learning we can remember, as long as we relate it to what we already know.¹¹

This is true in surgery: offer a new, unusual, or trivial fact to a master surgeon and she will easily incorporate it into her mind and use it efficiently; a surgical intern has no such “safety net” of neural connections and the miniscule fact falls through and fails to “stick,” unless novice learners are offered further teaching, repetitions and retrieval practice. The more we learn, the more possible connections we create for further learning. Importantly, for better memory creation, Roediger and Karpicke¹² admonish that “to measure any type of learning, you have to administer a test.” Testing is a good thing. Some other study habits are not!

Repetition has been called the Mother of Learning. Indeed, for centuries, teachers, coaches, and surgeons have encouraged their students to practice their multiplication tables, practice their free throws, and tie thousands of square knots, respectively. Seeing, hearing, tasting, smelling, and feeling the subject or skill multiple times is beneficial. to a point. What students are often advised to do is often plain wrong, and mere repetition by itself does not necessarily enhance learning. By learning, I mean something that is usable for the long haul; something the trainee can rely on for a career.¹³

As I did 40 years ago, at least 80% of college students (and our surgical trainees) today re-read material to learn. There are 3 reasons to avoid this learning strategy: it is time consuming, it does NOT result in durable memory, and it unwittingly creates self-deception.¹⁴ Re-reading text, scanning notes, or reading highlighted passages takes little effort. The brain can race through the material and the learner is deceived into believing he or she knows the subject matter. The brain is making no effort to learn the material. Despite hours of scanning or highlighting, facts are NOT placed in long-term memory. Karpicke and colleagues¹⁵ note that “the amount of time of study is no measure of mastery.” Repetition by itself does not lead to long-term memory, and not all practice is alike.¹⁶ Similarly, any learning that is achieved during a solitary, massed practice session is transitory and invariably is lost over the next several days.¹⁷ There are, however, an assortment of learning strategies that do lead to longterm retention of knowledge and skill acquisition.

Psychologists Robert and Elizabeth Bjork¹⁸ coined the term, “Desirable Difficulties” for enhancing learning. They suggest that the harder one tries to remember something, the more likely it will be retained and retrievable for use. The more effort you make to learn something, the better you will retain it and retrieve it. Simply reading or reviewing notes or reading highlighted material is effortless for the brain, and therefore, does not lead to retention or become retrievable. Extensive research shows that when learning is harder or more effortful, it’s stronger and lasts longer.¹⁸ For instance, a study of students learning about the circulatory system found that the group that read a passage twice (minimal brain effort) on the subject fared worse on testing than a group that read the passage once and then explained to themselves what each sentence meant (self-explanation = more effort). The self-explain group scored 32% better.¹⁹ Although both groups studied for the same amount of time, effortful retrieval made for stronger learning in the self-explain group.

To be desirable, a difficulty must be something learners can overcome through increased effort.¹⁸ For instance, slightly blurring some words in a text to read would be a desirable difficulty, readers would have to slowdown, squint, and truly concentrate to recognize and digest the words and their meaning; blurring them beyond recognition would make this an undesirable difficulty. There are numerous ways to allow or make learners bring forth more effort to learning. As surgeons, I bet you can think of many. Let me expand on a couple that will resonate with creating better young surgeons to follow in your footsteps.

Benedict Carey²⁰ offers that nothing in learning science comes close in terms of immediate, significant, and reliable improvements to learning as distributive learning. Spreading out learning sessions over 2 or 3 or more separate opportunities takes extra time and effort, but this “spaced learning” facilitates retention and long-term retrieval.

As an example, Moulton and colleagues²¹ split a group of 38 surgical residents into 2 groups: half were taught microsurgery (vascular anastomosis on rat aortas) over 4 sequential weekly sessions, while the other group learned in a 1 day-long session. Total training time was equal for both groups. Waiting 1 month after each group’s last teaching session, all 38 trainees were tested on their microsurgery vascular anastomosis skills. When it counted for the test, the distributive learning group was faster, had fewer wasted motions, and had greater success with the anastomosis than the massed learning group.²¹ Whenever possible, learning should be spaced out over time; multiple shorter learning sessions will lead to better retention than 1 longer learning experience; allowing students to step back after a first session and actually forget some material is useful. With second and third sessions, the effort it takes to retrieve the lessons of the first session creates stronger memories and eventually allows easier retrieval. Distributive learning feels harder and requires concentration over multiple sessions, but the final outcome is better, and more durable, long-term memory.

Novice learners need objective feedback to speed their transition toward competency. Both positive and negative feedback are useful, but surgeons need to understand there are several ways to offer this information to trainees, and some work better than others. For starters, delaying the feedback briefly produces better long-term learning than immediate feedback. Brown and colleagues²² note that in motor learning, trial and error with delayed feedback is a more awkward, but more effective, way of acquiring skill than trial and correction through immediate feedback. “Immediate feedback is like the training wheels on a bicycle: the learner quickly comes to depend on the continued presence of the correction.”²² While hovering over a young learner (say, an intern working on ligating in continuity the middle thyroid vein) in the operating room leads to better patient safety, surgeon educators will serve their surgical trainees best in the cadaver or simulation labs by allowing them to make mistakes and delay feedback. Allowing learners to ponder what is working, what is failing, and how they might change their technique is called “reflection”; such thinking helps create lasting memories. Following that, learner effort with feedback from a savvy surgeon with to and fro discussion leads to memories that are stronger and easier to retrieve in the future.

When learners commit errors and are given corrective feedback, the errors are not learned.²³ Allowing students the opportunity to practice and take risks, fail and get feedback, and repeat the effort, without their grades or evaluation or confidence suffering, is a powerful learning opportunity.²⁴ Indeed, Thomas Edison joked, “I’ve not failed. I’ve just found 10,000 ways that don’t work.” Ericsson was similarly high on learners failing, because “failure becomes an essential experience on the path to mastery.”²⁵ In our operating rooms, ICUs, and emergency rooms, we need our trainees to be at their best to offer exceptional patient outcomes, but anywhere in controlled teaching/learning sessions, encouraging trial and error effort, accepting mistakes, and learning to correct them creates deep memories. If the student’s performance did not meet expectations, try again. What matters is that one learns from the effort.²⁴

In our own wonderful, but imperfect surgical training program, we use bi-annual Objective Structured Clinical Examination (OSCE)-type assessments of our surgical residents. The fall exam is identical to the spring exam with about 12 knowledge (anatomy, reading CT scans, etc) and skill (cricothyrotomy, chest tube insertion, GI anastomosis, etc) stations, appropriate for each postgraduate year level. Our expectation is that learners will meet performance minimums in the spring. For those not meeting the standard, they are tutored and offered feedback within practice sessions, with the expectation of retaking that station’s exam as many times as they like until they achieve superior marks (well above the minimum standard).²⁶ Every single general surgery resident in the last decade eventually met or exceeded all standards set before them. Springtime results were consistently 20% to 50% better than fall exam scores. When general surgery residents offer effort, they clearly do learn.

Repetitions, reflection, with forgetting, and subsequent retrieval practice help the brain to better organize and consolidate the information. Experts package such material in “chunks” and eventually gain easy ability to offer knowledge and skills when a topic is offered. Say “Zollinger-Ellison Syndrome” to a master endocrine surgeon and she suddenly can spit out information as though she has a surgical atlas, a surgical history book, and dozens of personal experiences with gastrinomas playing on a teleprompter in front of her. She kind of does. She has a vast mental representation of what she knows on the subject. With her well-organized mind and extensive neural pathways, she really does see, feel, and hear virtually everything she knows about gastrinomas. Hours of deliberate practice combined with memory consolidation, organization, and reorganizing her neural pathways lead to efficiency and intelligence and proficiency.

While most of us really do not enjoy being test—deit often is hard work—the learning advantages are impressive: tests strengthen memory, tests make subsequent retrieval easier, tests clarify what you know, tests tell you what you don’t know, and tests are objective. Even better, tests need not be high stakes exams, they need not be lengthy, and they can be administered by anyone and are often best if given by learners themselves. Let’s take a closer look at tests and testing and why surgeons should be testing others and themselves.

The Science of Successful Learning²⁷ suggests “testing” can be as simple as reading material and then closing the book or covering the page and trying to explain or recite what was just read. Rather than simply moving on or even re-reading the material, making the extra effort to retrieve the knowledge is a “test,” or a retrieval challenge, and represents more effort on the part of the student. In effect, retrievaleor testing—interrupts forgetting; the sheer act of attempting to recall the facts, skill, thoughts, or knowledge strengthens and modifies the memory and creates a better neural network that will be easier to access the next time the information is sought. Ideally, waiting a couple of minutes to recall the material works better than waiting just a few seconds.²⁷ The more effortful the retrieval, the stronger the benefit. Studies have shown that practicing retrieval (ie testing) makes learning stick far better than re-exposure to the original material does.²⁸

Although it is unclear physiologically exactly how testing or these simple challenges enhance memory and retrieval, the brain is thought to be more focused and makes more effort to find the material when placed in a testing situation; tests are memory exercises.²⁹ Studies consistently show better performance when study time involves testing; if the material is important, it needs to be practiced (tested), and practiced (retested) again.³⁰ Using a test only for evaluation misses out on the true value of the learning experience. I doubt an Army drill sergeant would forego using calisthenics (ie push-ups) after his new recruits can do 20 push-ups. The test (doing 20 push-ups) is just another learning experience on the road to superior performance (maybe 100 push-ups). The value of testing in a formative surgical training program should not be minimized—testing is the best “studying” our trainees can do. Surgical staff must not be shy about testing, quizzing, or challenging trainees; ultimately, learners need to take up this mantra and challenge themselves as well.

For surgeons, retrieval practice or testing will be most effective and efficient if it reflects what surgeons will be doing with the knowledge later on.³¹ Viewing anatomic drawings of the neck, reading about surgical technique for thyroidectomy, self-quizzing on the thyroid gland’s blood supply, and trying to dictate an operative report would serve most trainees well before assisting with such an operation. With such low stakes, self-quizzing should be embraced. Self-testing is one of the strongest study techniques there is.³² It forces you to choose or generate the right answer and gives you immediate feedback.

Students who don’t quiz themselves (and most do not) tend to overestimate how well they have mastered the subject.³³ Surgical program directors see this each winter, when American Board of Surgery In-Training Examination (ABSITE) results are returned and they meet with several of their stunned residents who truly believe they “studied hard for the exam.” “Bombing the exam” typically occurs when learners have unknowingly deceived themselves that they know the material. Re-reading surgical texts, reading highlighted notes, and even re-taking a bank of quiz questions are not effortful attempts at memory acquisition. It is too easy for our brains to scan notes or highlights, and we become deceived into believing we know the material. Stopping to reflect on what was just read, or attempting to recite what was just seen, or looking at the quiz question and deciding why “A” was the right answer and why options B, C, and D were wrong are serious efforts—truly testing—that will lead to better retention and create retrievable information. Simplistically, testing has proven itself to be superior to additional study.³⁴

Dynamic testing is used by performers everywhere. Dynamic testing is nothing more than taking a test of some kind, dedicating to improve by using reflection, practice, spaced learning, and other strategies, and then re-testing.³⁵ Concert pianists do this as routine; same for professional athletes; indeed, anyone in a competitive environment (4th grade soccer teams, high school debate teams, or little Susie in a 4H club striving for a blue ribbon) typically plays a game or competes (testing!), and then goes back to practice and improves for the next competition. The hard knocks of real life are often a form of dynamic testing if we have the drive to get back up when we fall.

In a surgical world, with real live patients being cared for, setting aside practice time is difficult. All the more reason our learners need to get into a dynamic mindset, and between patient care opportunities, they must reflect, self-quiz, and practice what they can. Using cadaveric or simulation opportunities in a distributive manner would be ideal. We created a vending machine for our surgical trainees where they can check out low cost models and surgical tools to work on at home and practice wound closure, anastomoses, and assorted other skills in between times of clinical responsibility. Including access to operative video and a scoring rubric with step-by-step instructions helps make the educational effort more powerful and dynamic. This effort—trainees practicing and testing themselves—has improved surgical performance.^26,36,37

Weekly morbidity and mortality (M & M) conferences interspersed with ongoing patient care could be a form of dynamic testing. Program directors and surgeon educators will serve their trainees well by spending just a minute or 2 of M & M, basic science, or chief conferences, quizzing the group about the pearls offered last week or last month. If that key message last week was really critical to learn last week, it is all the more important this week to refresh and mandate memory retrieval of that key fact or pearl.

Summating the insights on testing: The more students practice retrieval (testing), the better they learn. Frequency matters.³⁸ The answer to illusion and misjudgment in learners is to replace subjective experience with a set of objective gauges (testing) outside ourselves.³⁹ Making frequent use of testing and retrieval practice to verify what you really do know vs what you think you know is imperative for surgical residents. If testing is only for measurement, we miss the value of the extra workout (like a push-up contest). Simply put, testing is “retrieval practice.”³⁴

Professor K. Anders Ericsson created a body of work that showed to become masterful at playing chess, playing the violin, or flying an airplane, it takes roughly 10,000 hours of serious practice, repetitions, and refinement of technique. Ericsson believed there were 3 key factors to help create expertise and generate those who become capable of masterful performance: the learner must be driven, the learner must undertake deliberate practice, and the learner must be given voluminous feedback. While repetition and practice are vital to advancing from novice to master levels, the repetitions must be effortful. Repetitions that are easy rarely lead to meaningful learning or a change in performance. Really substantial learning and improvement is hard work. Repeat: generating usable learning is hard work.

The recent passing of Professor Ericsson should remind us in 2020 to refocus our efforts on training young surgeons. His landmark work on how humans become master performers should remind us that “practice, not innate ability, is the critical factor in determining performance.”⁴⁰ Mastery requires both the possession of ready knowledge and the conceptual understanding of how to use it.⁴¹ This takes effort. I am convinced that if surgeon educators avoid assumptions with teaching and learning, and offer testing to their trainees and encourage them to self-test, effortful learners will carry the torch of surgery masterfully.

Make NO assumptions of surgical learners. Learning is hard work. Teachers must respect, but challenge, learners to enhance their memory creation and retrieval. Giving lots of tests—either by staff, peers, or self—requires more effort, but is far more efficient and effective study time. Testing lets all parties know what the learner knows and does not know; from there, the education and effort can be tailored and directed to make learning more likely.

¹ Mayer R. The promise of educational psychology, Vol 2: Teaching for meaningful learning. Upper Saddle River, NJ: Merrill Prentice Hall; 2002.

³ Brown P, Roediger H, McDaniel M.Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:100.

⁴ Bruer JT. Neural connections: Some you use, some you lose. Phi Delta Kappan 1999;81:4.

⁶ Ambrose S, Bridges MW, Lovett MC, et al. How Learning Works. Hoboken, NJ: John Wiley & Sons, Inc; 2010:3.

⁷ Zull J.The art of changing the brain: Enriching the practice of teaching by exploring the biology of learning. Sterling, VA: Stylus; 2002:112.

⁸ Lang J.Cheating lessons: Learning from academic dishonesty. Cambridge, MA: Harvard University Press; 2014:114.

⁹ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:126.

¹⁰ Ambrose S, Bridges MW, Lovett MC, et al. How Learning Works. Hoboken, NJ: John Wiley & Sons, Inc; 2010:15.

¹¹ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:76.

¹² Roediger H III, Karpicke JD. The power of testing memory: basic science research and implications for educational practice. Perspect Psycholog Sci 2006;1:181—210.

¹³ Roediger H III, Karpicke JD. Test-enhanced learning: Taking memory tests improves long-term retention. Psycholog Sci 2006;17:249—255.

¹⁴ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:12.

¹⁵ Karpicke JD, Butler AC, Roediger HL. Metacognitive strategies in student learn: Do students practice retrieval when they study on their own? Memory 2010;17:471—479.

¹⁶ Tulving E.Subjective organization and the effects of repetition in multi-trial free recall learning. J Verbal Learn Verbal Behav 1966;5:193—197.

¹⁷ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:10.

¹⁸ Bjork RA, Bjork EL. A new theory of disuse and an old theory of stimulus fluctuation. In: Healy A, Kossly S, Shiffrin R, eds.From Learning Processes to Cognitive Processes: Essays in Honor of William K Estes, Vol w.Hillsdale, NJ: Erlbaum; 1992:35—67.

¹⁹ Chi MTH, DeLeeuw N, Chiu MH, LaVancher C.Eliciting self-explanation improves understanding. Cognit Sci 1994; 18:439—477.

²¹ Moulton CAE, Duborwski A, McRae H, et al. Teaching surgical skills: What kind of practice makes perfect? Ann Surg 2006;244:400—409.

²² Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:40.

²³ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:91.

²⁵ Ericsson KA. The influence of experience and deliberate practice on the development of superior expert performance. In: Charness N, Feltovich PJ, Hoffman R, eds. The Cambridge Handbook of Expertise and Expert Performance. Ericcson KA. Cambridge: Cambridge University Press; 2006: 683—704. Chapter 38.

²⁶ Gas BL, Buckarma EH, Mohan M, et al. Objective assessment of general surgery residents followed by remediation. J Surg Educ 2016;73:e71—e76.

²⁷ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:20.

²⁸ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:29.

³⁰ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:125.

³¹ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:56.

³² Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:44.

³³ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:17.

³⁵ Sternberg R, Grigorenko E. Dynamic Testing: The Nature and Measurement of Learning Potential. Cambridge: Cambridge University Press; 2002.

³⁶ Al Jamal Y, Buckarma E, Ruparel R, et al. Cadaveric dissection vs homemade model: what is the best way to teach endoscopic totally extraperitoneal inguinal hernia repair? J Surg Educ 2018;75:787—791.

³⁷ Pandian TK, Buckarma EH, Mohan M, et al. At home preresidency preparation for general surgery internship: a pilot study. J Surg Educ 2017;74:952—957.

³⁹ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014: 124.

⁴⁰ Ericcson KA, Ward P. Capturing the naturally occurring superior performance of experts in the laboratory: Toward a science of expert and exceptional performance. Curr Direct Psycholog Sci 2007;16:346—350.

⁴¹ Brown P, Roediger H, McDaniel M, Make It Stick. The Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014:18.