In this episode of Hidden Brain, Shankar Vedantam explores the relationship between obsession, secrecy, and identity through the story of Riley Shepherd, who spent decades creating an encyclopedia of American folk music. Shepherd's massive scholarly achievement came at tremendous personal cost—financial ruin, family abandonment, and lifelong instability. Folklorist Steve Winick and Shepherd's daughter Stasha reflect on how society judges single-minded pursuits differently depending on their outcomes, and what gets lost in the shadow of obsession.
The episode also examines the widespread practice of keeping secrets and its psychological toll. Harvard psychologist Leslie John discusses why people instinctively focus on the risks of revealing secrets rather than the costs of concealing them, and how secrecy contributes to anxiety, depression, and damaged relationships. Through personal stories about hidden identity, religious beliefs, and family history, Vedantam and John explore how disclosure—when approached thoughtfully—can strengthen trust and ease the burden of silence.

Sign up for Shortform to access the whole episode summary along with additional materials like counterarguments and context.
Riley Shepherd's life story reveals the complex relationship between genius and folly through his decades-long obsession: creating an encyclopedia of American folk music. His daughter Stasha recalls discovering the massive scale of the project—boxes upon boxes of indexes and handwritten notes.
Folklorist Steve Winick explains that Shepherd's fixation is familiar to scholars, particularly folk song researchers who dedicate themselves to cataloging the past. However, Shepherd gave his entire life to this pursuit, often forgoing basic needs. Winick and host Shankar Vedantam note how society judges such obsessions differently depending on outcomes: we celebrate monumental works like Hamlet, but apparent failures seem tragic.
Despite having only a fifth-grade education and a reputation for cons, Shepherd painstakingly indexed over 43,000 versions of folk songs into 4,000 categories over nearly twenty years—entirely without computers. The result was 55 volumes of meticulously catalogued work, unprecedented in American folk music studies. Winick describes it as a scholarly feat typically requiring a team of archivists and years of funding, yet it languished in obscurity as computers were changing how such work was valued.
This monumental effort came at tremendous cost. Financial desperation drove Shepherd to borrow and swindle money, write pornography under pseudonyms, and constantly evade creditors. Stasha recalls a chaotic childhood of frequent moves and witnessing her father prioritize his manuscripts over basic family belongings. He ultimately abandoned Stasha and her mother when she was 18, and the family bonds frayed under the weight of his obsession.
The encyclopedia sat untouched for years until scholars like Steve Winick acknowledged its genuine scholarly merit. Stasha, having spent years doubting its value due to her father's compulsive lying, only gradually realized the magnitude of his achievement. The encyclopedia was eventually digitized and made available to the public, standing as both testament and epitaph to the power, beauty, and cost of obsession.
Shankar Vedantam and Leslie John explore how secrecy is both widespread and silently destructive, and why relief from sharing is often disproportionate to the distress secrets cause.
Vedantam shares that 80% of people lie to their doctors, a striking statistic from Harvard psychologist Leslie John. John explains that people instinctively focus on the risks of revealing secrets rather than the risks of keeping them. Holding secrets is associated with depression, anxiety, stress, rumination, and deteriorating relationships—even minor withheld truths can foster significant emotional distress.
John notes that the burden of a secret often grows larger the longer it's kept private, exacerbated by rumination and imagination. She cites the "spotlight effect," where people vastly overestimate how much others notice their embarrassing moments or secrets, causing unnecessary suffering.
A listener named Claire felt unable to look her friend's parents in the eye for nine years after secretly urinating on their property near a security camera. When she finally confessed, both her parents and her friend barely reacted. Vedantam and John emphasize that the distress caused by secrets is often wildly disproportionate to others' actual reactions. Much of the distress comes from the act of secrecy itself, while immense relief often follows revelation.
Emma, assigned male at birth, realizes by age four or five that she wishes she had been born a girl. She carries this secret in isolation for almost 40 years, never fully disclosing these feelings even to therapists. In 2014, with her wife's encouragement, she confronts her fear and reveals her transgender identity. After beginning her transition in 2017, Emma expresses "no regrets," finding significant relief in living authentically. Leslie John notes that identity-based secrets carry a unique form of self-betrayal, preventing genuine connection.
Philip describes concealing his shift from evangelical Christianity to agnosticism within a strict faith community where revealing his beliefs would risk losing his wife, family, and friends. He endures profound psychological suffering until he begins confiding in a childhood friend, which provides much-needed emotional relief. However, this breach of trust contributes to his marriage's collapse and divorce in 2024. With perspective, Philip most regrets how long he concealed his beliefs.
Leslie John emphasizes that revealing core identity is not a one-time confession but an ongoing interactive dialogue. Selective disclosure—telling trusted individuals rather than everyone—can enable authentic self-expression and foster genuine intimacy, easing the psychological toll of secrecy while allowing for meaningful relationships.
Leslie John argues that sharing sensitive information is one of the most powerful ways to build trust. She notes her own tendency to withhold much more than she shares—for every thought expressed, four go unsaid. These hidden truths, often minor, prevent partners from understanding our state of mind and adjusting accordingly.
John cautions that people sometimes reveal secrets primarily to alleviate their own guilt, transferring the emotional burden onto their partner. She encourages reflection on one's motives: Is the disclosure intended to strengthen the relationship or simply gain relief? Disclosures motivated by mutual growth benefit both parties, while self-serving revelations may inadvertently harm the listener.
John emphasizes "disclosure flexibility"—the ability to adapt openness according to the situation and relationship. Vedantam notes that we don't need to know everything about someone to foster companionship, and tolerance for selective disclosure encourages others to share when ready. Both agree that embracing the complexity of disclosure with empathy and context-sensitive judgment strengthens trust and intimacy.
Hanno discovers late in his grandfather's life that his grandfather served as a Nazi officer in forced labor service during the Holocaust. Although not responsible for his grandfather's crimes, Hanno struggles with inherited shame. Leslie John points out that without the chance to converse with his grandfather, Hanno is denied opportunities for closure and understanding, leaving him with incomplete and troubling information.
Shankar Vedantam observes that such secrets are sometimes collectively upheld through social pressure and unspoken norms. Leslie John explains that silence is enforced not by explicit agreement but by tacit, mutual discomfort—if nobody initiates conversation, everyone remains silent.
Research shows Holocaust survivors who process experiences through dialogue achieve better long-term psychological outcomes compared to those who remain silent. John emphasizes that developing deeper understanding through conversation is crucial for psychological integration. Silence about trauma traps later generations in mystery and unprocessed burden, while structured dialogue allows individuals and families to address inherited burdens and mitigate isolation and anxiety.
1-Page Summary
The story of Riley Shepherd captures the perilous and profound terrain of obsession—its creative glory and its searing personal toll. As a musician turned con man and relentless compiler, Shepherd spent decades on a single, staggering endeavor: an encyclopedia of American folk music. His life and work reveal the tangled relationship between genius and folly, the hidden triumphs within apparent failure, and the unbearable costs of single-minded devotion.
Riley Shepherd’s legacy pivots on his all-consuming project, which, at first glance, seemed quixotic if not entirely foolish to those closest to him. His daughter Stasha recalls learning about his major artistic pursuit not as a song, but a massive writing project: The Encyclopedia of Folk Music. The scale was epic—boxes upon boxes of indexes, volumes, and handwritten notes, the fruits of untold hours of labor.
To folklorist Steve Winick, Shepherd’s fixation is familiar. All scholars, especially folk song researchers, experience a pull toward obsessive completeness, dedicating themselves to cataloging and preserving the past. Yet, unlike most, Shepherd gave his entire life to this pursuit, often forgoing basic needs to continue his work. Winick and host Shankar Vedantam note how society judges such obsessions: when they yield monumental works like Hamlet or the iPhone, we celebrate them; when they stall in obscurity, they appear as tragic folly. The true nature of such quests often doesn’t reveal itself until long after the pursuit is over.
Beneath the shadows of Shepherd’s documented deceit lies a formidable achievement. Shepherd—a man with only a fifth-grade education, notorious for cons and fabrications—painstakingly indexed over 43,000 versions of folk songs into 4,000 clearly defined categories over nearly twenty years. He sourced rare and out-of-print books to ensure rigor, cross-referencing titles, published versions, languages, and history, a task he accomplished entirely without computers. Nearly every song entry included musical notation, historical notes, and comprehensive source lists.
The scale was unparalleled, described by researchers as an unprecedented individual work: 55 volumes, much of it meticulously hand-catalogued. Winick notes that as late as the 1970s, no project of this ambition had ever been attempted in American folk music studies. Riley's system allowed researchers to trace the evolution of a single song across cultures and eras, as with the sea shanty "Holloway Joe," where he lists variants, sources, and notations.
Shepherd’s encyclopedia was a scholarly feat requiring resources and effort that would typically demand a team of archivists and years of funding. Yet, it languished in obscurity, boxed away as computers were changing how such work was shared and valued.
This monumental effort came at a high cost. Shepherd's focus on his encyclopedia overshadowed everything else. Financial desperation drove him to dubious schemes: borrowing and swindling money from friends and acquaintances, writing pornography under pseudonyms, and constantly evading creditors—at times pretending to be another person to avoid calls.
His daughter Stasha recalls a chaotic childhood: frequent moves, fleeing homes to avoid unpaid rent, and witnessing her father prioritize the careful transport of his encyclopedia manuscripts over basic family belongings. The family’s poverty was mitigated only by Shepherd’s endless promises that the encyclopedia would someday provide wealth and recognition. Family bonds frayed under shepherd’s obsession; he ultimately abandoned St ...
Obsession: Power, Beauty, and Cost
Secrets shape our psychological lives in profound, often detrimental ways. Shankar Vedantam and Leslie John explore how secrecy is both widespread and silently destructive, and why the relief from sharing is often disproportionate to the distress secrets cause us.
Vedantam shares a striking statistic from Leslie John, a Harvard psychologist: 80% of people lie to their doctors. This finding is surprising because the context of healthcare assumes honesty in pursuit of our own well-being. Yet secrecy, even when self-defeating, is a default human behavior, powered by fears of embarrassment, conflict, rejection, or loss of status. Leslie John explains that when confronted with disclosure dilemmas, people instinctively focus on the risks of revealing their secret rather than the risks of keeping it. This bias toward silence prevails despite significant psychological costs.
Holding onto secrets is associated with a host of negative outcomes. Studies show secrecy correlates with depression, anxiety, physiological stress markers, rumination, and deteriorating relationships. These effects are not limited to egregious secrets; even minor withheld truths can foster significant emotional distress and undermine well-being.
The burden of a secret often grows larger the longer it is kept private, exacerbated by rumination and imagination. John recounts lying to a student to prevent him from joining a trip and then feeling terribly guilty, ruminating over the lie for years. Vedantam points out that, from an outsider’s perspective, the secret may seem trivial, but for the keeper, it grows into an exaggerated weight. The longer a secret is held, the more it is revisited and amplified in the mind, compounding itself through imagined negative consequences and unlikely interpretations.
Leslie John cites the “spotlight effect,” a psychological phenomenon in which people vastly overestimate how much others notice or care about their embarrassing moments or secrets. This cognitive bias causes unnecessary suffering, as individuals believe they are the center of attention much more than they truly are.
A powerful illustration of secrecy’s toll comes from a listener named Claire. For nine years, Claire felt unable to look her friend’s parents in the eye after secretly urinating on their property and seeing a security camera pointing at her mid-act. She avoided sharing the story due to intense guilt and shame, imagining negative judgments from both her parents and her friend’s ...
Secrets: Why We Hide and What It Costs
Emma, born in 1956 and assigned male at birth, realizes by age four or five that she wishes she had been born a girl. Throughout elementary school, she privately fantasizes about life as a girl but believes this secret is so deep and stigmatized that she must never reveal it to anyone. Despite seeing therapists for depression throughout adulthood, Emma never fully discloses these core feelings, carrying the burden in isolation for almost 40 years.
In early 2014, with encouragement from her wife, Emma returns to therapy. This time, she confronts her fear and slowly reveals her transgender identity, a process that takes weeks and is frightening. In 2017, she begins her transition. Now living authentically as a woman, Emma expresses "no regrets" and finds significant relief and satisfaction in her decision to come out. Her experience shows how hiding a core identity imposes psychological, physiological, and social costs, while disclosure—even if only to selected loved ones—brings real relief and enables authentic self-expression.
Leslie John notes that identity-based secrets, compared to secrets about one's actions, carry a unique form of self-betrayal and inauthenticity. When these secrets involve stigmatized aspects integral to one's selfhood, they prevent genuine connection unless at least some close individuals are let in. Authenticity, she argues, is critical for being truly known.
Philip describes his secret shift from evangelical Christianity to agnosticism, complicated by a faith community with strict rules regarding apostasy and excommunication. Revealing his true beliefs would have put him at risk of losing his wife, family, friends, and community. So, he conceals his views, enduring "all the side effects" Leslie John references, including profound psychological, physiological, and social suffering. The impossibility of living between his authentic self and the community's expectations inflicts lasting damage.
Eventually, Philip begins privately confiding in a childhood friend. The experience of being able to express himself honestly and receive nonjudgmental acceptance provides him much-needed emotional relief. However, since the confidant is a woman his age, this act breaks trust with his wife and violates their emotional commitments. Thi ...
Identity and Self-Disclosure: Authenticity Versus Concealment
Leslie John highlights that being known for who you are and feeling comfortable revealing your authentic self is deeply rewarding. Sharing sensitive information, she argues, can be one of the most powerful ways to build trust—it demonstrates confidence, invites genuine connection, and reassures others that intimacy is valuable. John cites research and personal experience to underscore that disclosure creates opportunities for true intimacy; for instance, telling a partner about feeling unwell allows them to adapt and provide needed support, a gesture impossible if those concerns remain unspoken.
John also notes her own tendency, documented through a personal tally, to withhold much more than she shares. Keeping track of expressed and unexpressed thoughts in daily life, she finds that for every thought shared, four go unsaid. These hidden truths, often minor or trivial, add up—preventing partners from understanding our state of mind and adjusting their behavior accordingly. John’s experience highlights that even seemingly small unspoken concerns can hinder genuine connection.
However, John cautions that not all sharing is equally beneficial. Sometimes, people reveal secrets or transgressions primarily to alleviate their own guilt or shame, transferring the emotional burden onto their partner rather than fostering mutual understanding or growth. She encourages careful reflection on one’s motives: Is the disclosure intended to strengthen the relationship, or simply to gain relief from uncomfortable feelings? In instances where sharing only serves the discloser, it may be more appropriate to seek self-understanding with the help of a therapist instead of imposing on a loved one. Disclosures that are motivated by a desire for mutual growth and understanding tend to benefit both parties, while self-serving revelations may inadvertently harm the listener.
John emphasizes the value of "disclosure flexibility"—the ability to adapt openness according to the situation and relationship. Effective communicators exhib ...
Trust and Intimacy: The Paradox of Protective Silence
Hanno, from Germany, discovers late in his grandfather’s life that his grandfather served as an officer in the Nazi forced labor service during the Holocaust and may have participated in the ethnic cleansing of Jews. This shocking secret, kept hidden for generations, burdens Hanno with a profound sense of inherited responsibility and shame for actions he did not commit. Although he is not responsible for his grandfather's crimes, Hanno struggles with how these moral failures of his ancestors extend across generations, imposing emotional weight on descendants.
Hanno’s experience is compounded by the fact that he did not have the chance to speak in depth with his grandfather about these actions. Leslie John points out that when descendants cannot converse with those who lived through or perpetrated darkness, they are denied opportunities for closure and understanding. Without hearing the perpetrator’s side — their perspective, motives, or reflections — descendants like Hanno are left with incomplete and troubling information. The lack of conversation closes the path to therapeutic sense-making, preventing descendants from metabolizing difficult family truths.
Shankar Vedantam observes that such secrets are sometimes not just personal or familial, but collectively upheld by entire groups or societies through social pressure and unspoken norms. Leslie John explains that the persistence of these secrets comes from the assumption that others do not wish to discuss them, so everyone remains silent. If nobody initiates conversation, silence is enforced not by explicit agreement but by tacit, mutual discomfort.
This phenomenon is not limited to dark family secrets but extends to group dynamics in ordinary settings. For example, John shares how voicing frustration at a stalled meeting brought collective relief and bonding, showing that one brave voice can break enforced silence and foster connection.
Research shows that Holoc ...
Collective Family Secrets: Inherited Burden and Generational Impact
Download the Shortform Chrome extension for your browser
