Category Archives: “Facial Expressions”

Catching Liars

August 2014

From a distance of twenty feet, without your knowledge or consent, the surveillance cameras, microphones and physiological sensors apply computer algorithms to process your gait, facial expression, gestures, voice, posture, heart rate, blood pressure and skin temperature — determining within seconds if you are lying or telling the truth.

“Diogenes,” the lie catching machine in the sci-fi screenplay a friend and I wrote twenty years ago, made no mistakes.  And its mere existence changed the way people interacted: Bargainers would no longer agree to meet in person, juries became superfluous, salesmen were forced to be honest in their claims, and suspicions of marital infidelity were resolved in an instant, for a fee.

In real life, we aren’t quite there yet, but our recent research on truth and falsehood is getting closer than I ever expected it would. I’ve spent nearly forty years studying how changes in demeanor (face, body and voice) might betray a lie. I focused on serious lies in which life, freedom, reputation, or the continuation of an important relationship were at stake, rather than the white lies of everyday life such as politeness, flattery or exaggeration. The original impetus was to help doctors evaluate whether psychiatric inpatients claiming to no longer feel depressed were lying so that they could commit suicide free of the hospital’s supervision. Responding to interest from law enforcement and intelligence agencies, my associate and I (the late professor Maureen O’Sullivan) extended our focus to lies about taking money, or false claims about a strongly held political opinion. Since 9/11 our experiments have focused on the lies told by political extremists, in hope that our results will have relevance to anti-terrorism. We set the rewards for success and the punishment for failure as high as ethics committees permit.

We have not found the modern equivalent of Pinocchio’s nose, nothing in face, body, voice, speech or physiology that is unique to lying, never present when someone is worried, thoughtful, cautious, perplexed or nervous. While some still pursue that goal (and a few claim to have found it), I doubt that such a silver bullet exists.  Instead our measurements of facial muscular movement, gestures, voice and speech uncover what I call hot spots, signs that something is amiss; that the full story is not being told.

The typical hot spot is a momentary conflict between the words spoken and the sound of the voice, the gesture, or the facial expression. Just as important are very brief micro expressions that can flash across a person’s face in 1/25 of a second. A micro looks exactly like a normal facial expression except it is so fast that most people don’t see it. It always is a sign of a concealed emotion, sometimes deliberately concealed sometimes-repressed emotion. Just as important is a slight edge in the voice that doesn’t fit calm words.

In our experiments in which there are only two possibilities — someone is either lying or telling the truth — hot spots allow high accuracy in distinguishing one from the other. In real life hot spots occur for many reasons, such as remembering an argument at breakfast with your spouse, worrying about missing a flight, or annoyance at the screening process at an airport. Lying to conceal malicious intent or past wrongdoing is only one, and not the most frequent reason that hot spots occur where terrorists might lurk.

Nevertheless learning how to identify hot spots can be useful in figuring out where to probe further in an interview, or who among the millions each day who wait on line in our airports to ask a few questions about the purpose of their trip. We are training law enforcement and national security officials, here and in England to identify hot spots, emphasizing that they are not signs of lying, only that something is amiss. People can learn to recognize the micro expressions in an hour, and the benefit lasts. We don’t know how long it takes to learn to recognize the many other hot spots, who learns the most and the least, and when a refresher course is needed. Paul Ekman International is a company that provides four day courses on evaluatating truthfulness and a 3 day course on emotional skills, primarily to the corporate sphere in many countries.

Another line of active research is trying to develop the modern equivalent of our sci-fi Diogenes lie-catching machine. The hardware and software that identifies hot spots in real time isn’t ready for prime time now, but it is progressing. Soon automated hot spot detectors could evaluate facial expressions and bodily physiology instantly, from a distance. Before it is deployed as a substitute for a highly trained human observer it is essential to determine whether it is as accurate as such an observer, and if it were to be used as an aid rather than a substitute, that it doesn’t distract and lower observer performance.

The ACLU has complained that hot spots aren’t shown just by terrorists, but also lead to the apprehension of wanted felons, illegal immigrants, or smugglers. I am afraid there is no terrorist-specific hot spot, but I am not personally convinced it is bad to identify others breaking the law. Another criticism is the possibility that members of minority groups, especially those whose names or appearance suggest they are Arabs, may be more uncomfortable in places such as airports, and for that reason may show up more often as suspicious. That is possible, and even warning about it during training may not be sufficient to avoid such mistakes. However, I favor deploying an independent organization to check periodically on the performance of those doing hot spot detection, to make certain they are not slipping into racial profiling.

Another concern is what happens to the information picked up by an automatic hot spot detector.  Suppose the heart rate and blood pressure readings strongly suggest someone is on the verge of a heart attack. After a warning that an emergency room might be a smarter destination that an airplane, what will happen to this private medical information? Regulations need to be developed to insure that it is erased rather than secretly sold to employers and insurance companies.

People around the world are already using and teaching these new approaches to identifying people who might intend harm. There is no putting this Genie back in the bottle. The issue is how to use these new methods wisely, cautiously, and with safeguards for privacy and civil liberties.

Paul Ekman, a retired professor of psychology at UC San Francisco, has been studying facial expressions and gestures for more than 40 years. He is the author of many books. His most recent book Moving Towards Global Compassion is available as an e-book at www.paulekman.com.

Updated from an article by Ekman for the Washington Post in 2006.

To receive updates on news regarding facial recognition technology,  Like and Follow the Paul Ekman Group on Facebook and Twitter.

#AskEkman: How do I become a facial expression expert?

July 2014

The most popular questions we receive at the Paul Ekman Group are questions relating to which courses and universities are best equipped to promote a career in becoming an expert in facial expressions and emotion. The answer to this question depends on what level of education you are seeking, and what topic interests you. To help you on your journey, we’ve put together a series of short answers from Dr. Paul Ekman.

If you are in high school and are interested in emotion:

Apply to a college or university at which there is an expert on emotion and/or facial expression. The Psychology Department at UC Berkeley has four faculty members who work on emotion, one of whom studies facial expression. The University of Wisconsin has a program on emotion and compassion, with emphasis on neuroscience substrates. If you are really ambitious, here is our list of over 250 major contributors to the field of emotion.

If you are an undergraduate, interested in emotion and/or facial expressions:

I assume you are not already at a college where there is an emphasis on emotion, so you’re going to have to do a lot of reading. If facial expression is your major interest, check the work of Professor Dacher Keltner, UC Berkeley. If it’s the physiology that drives or underlies emotion, try Professor Robert Levenson, UC Berkeley and Professor Richard Davidson, University of Wisconsin.

If you are seeking a graduate school to obtain a Ph.D.:

We recommend the same two universities above, UC Berkeley and the University of Wisconsin. But there are many other choices; emotion is a popular topic these days!

If you are high school student or undergraduate interested in deception:

Unfortunately, there isn’t much to recommend. Mark Frank in the Communications department at University of Buffalo, and Steve Porter at University of British Columbia, Okenagan Campus, both do research on the behavioral clues relevant to lying. Porter has challenged some of my work, but I respect his work. Frank is continuing many of the studies and approach I initiated.

If you are not currently a student:

PEG has established an international network of Licensed Delivery Centers (LDCs) through Paul Ekman International which deliver courses in Emotional Skills and Competencies (ESaC) and Evaluating Truthfulness and Credibility (ETaC). These cater to a wide range of professionals in their public courses and corporate programs. Please contact Paul Ekman International for further information, dates, and locations near you.

We wish you much success in your studies! For more information and helpful links, please visit our FAQ page.

To have your questions answered by Dr. Paul Ekman, submit them by using the hashtag #AskEkman on Twitter.

 

 

 

 

 

 

 

 

 

 

Is Snowden Lying?

June 2014

Many readers have asked whether Snowden was lying in his recent NBC-TV interview, knowing I have worked for the government and corporations spotting lies by how someone behaves. When I attempt to evaluate truthfulness I need to be the one asking the questions, able to ask follow up questions, allowed as many hours as I need, and the person I am interviewing must not have had time to prepare or be coached. Even if these requirements were to be met, I maintain a strict policy: I never evaluate anyone involved in litigation. In our judicial system it is the responsibility of the jury not an expert to determine truthfulness, difficult as that often is when only yes/no questions can be asked, and there is plenty of time for answers to be prepared. So, under these circumstances, I simply cannot comment on Snowden’s veracity. Still, some of Snowden’s claims merit consideration.

Snowden claims longer and more serious employment than was previously revealed by the government, and many attempts to be a whistle-blower, which, he says, were met with resistance by the NSA. The next day after the NBCTV interview the NSA denied these claims, as did Senator Diane Feinstein, head of the Senate committee that oversees intelligence capabilities. We don’t know who to believe.

Snowden claims he would not be granted a fair, open trial with access to all the charges and witnesses against him if he were to return to the United States. That is probably true, because a public trial, or even a closed trial in which all the information against him were to be revealed, might help our enemies if they were to find out. This leaves Snowden in limbo, the resolution of which has not been suggested by anyone.

Snowden also said that our mobile phones can be turned on by the intelligence agencies of the industrialized world, without our knowledge or consent, to listen to what we are saying. No one has denied this claim! When I was a Fulbright lecturer at Leningrad State University in 1979, people whose homes I visited would immediately put their telephones in the bathroom, convinced that otherwise the KGB would hear what we said. Do we need to take the same precaution against the NSA?

No one is claiming Snowden forged the NSA documents he stole, which revealed previously unknown threats to privacy. I would like to see an impartial judicial authority, perhaps an international one, review those revelations, charged with suggesting regulations of whose privacy can be invaded without notice or consent. They might also consider how to resolve the question of whether there is any way for Snowden to get a fair trial, and if not how should the issue of whether he should be punished be resolved.

We need public discussion of the tradeoffs involved if and when privacy is invaded, to be certain the public knows what is being done, if not in every specific instance, then in general. If what I am suggesting is not feasible there must be some way to change where this matter presently stands – in a state of confusion about charges and countercharges.

 

For more on Privacy Invasion, read “Who Should Know How You Are Feeling?” by Paul Ekman featured on his blog, Face It!

Who Should Know How You Are Feeling?

May 2014

Where you go on the Internet, where you travel on city streets, that and more is all up for grabs

Google? NSA? Walmart? It soon may be possible for them to track your emotions in addition to your whereabouts without your knowledge or consent. No regulations from the government to prevent massive surveillance are on the horizon. Already companies are selling software that identifies your name just by looking at your face. The New York Times Business section on Sunday May 18th ran a front page article about the threat of privacy invasion now that software programs identify not only who you are by using hidden cameras to scan your face but then also link to your Facebook posts and other sources of information about you which you didn’t know you were providing.

The next step in privacy invasion

The Times article didn’t mention the next step in privacy invasion, just around the corner – new computer programs that can link your identity to how you are feeling at the moment. What was your emotional reaction when you read the latest news about … Edward Snowden, Kim Kardashian, or President Obama? How did you feel about the latest packaging of Tide, or Trojans? Which TV ad for which politician got the biggest emotional kick from you?

Automated computer recognition of moment-to-moment emotions, even attempts to hide emotions, is nearly here. Crude or incomplete versions are already in use, and better ones will soon be available. I know about all of this having developed the first comprehensive offline tool for measuring facial expressions, and my participation in a company that is developing automated emotion recognition based on my work.

People wearing Google Glass may know not only who you are but also how you are feeling

Despite Google’s attempt to prevent this use of their product, something similar will become available from a copycat manufacturer that reveals the same information. The real threat to privacy comes from the surveillance cameras that are nearly everywhere, when they are able to know our most personal feelings about whatever we are looking at, or the emotions that we are feeling as we remember something from our past, recent or distant. These automated “emotion sensors” won’t know what triggered our emotions, just what emotions we are feeling.

The clever use of this invasive technology will attempt to provoke us by triggering engineered emotional events, without prior consent for having our emotions triggered, or having our emotions read, or having our emotions linked to our identity and everything that is already stored about us (what we buy, where we go, who we live with, and who knows what else).

Can our privacy be rescued from invasion?

Can our identity be hidden, the emotions shown on our face not read?

Joseph J. Atick, one of the pioneers of identity recognition, was quoted in the  N.Y. Times as saying that companies using face recognition to identify us should post notices they are doing so; they should seek permission from a consumer before identifying them, and use that information only for the purposes the consumer has consented to. Those are very modest protections, in my judgment, but still opposed by some of the companies selling the recognition software.

More protection is needed when it comes to reading our emotions

Our consent has to be obtained by more than a public notice that it is being done, and more than clicking on a website’s fine print. A specific, explicit consent to have our emotions monitored by a particular company for a specific purpose, that is time limited, probably to one- or two-hour period, should be obtained. Reading children’s emotions should require parental consent for each occasion it is done. And when a government agency such as the NSA reads our emotions, some regulation, better than ones currently in place, need to be established to protect our privacy.

I am a face scientist, not an attorney, data wonk or legislator. I hope this blog will alert all those responsible for the protection of the public to the urgent need for establishing privacy guidelines.

Afterthought:

Brian Williams of NBC News talks with former NSA contractor Edward Snowden about the global impact and debate sparked by his revelations. Read more about this exclusive here.

Spotting Poker Bluffs

May 2014

The game has changed now that TV broadcasts people playing poker.

In the old days not a word was spoken, and that tradition continues today in some venues. But lies had to be spotted; bluffs called. I learned about this from winners of the International Poker Tournament held each year in Las Vegas. It costs $15,000 to enter the fray, and then, after two weeks of serious gaming, the winner walks away with a million bucks.

Two of the winners, in different years, sought my advice knowing that I am an expert in spotting liars. I told them I had not played poker since junior high, and had never watched poker being played. They thought that didn’t matter. It turned out they were wrong.

You can’t win without bluffing, but you can’t win if you can’t spot the bluffers.

The key to winning, each said, was spotting bluffs. Once you know they have a bad hand, if you have a good one, you keep raising the stakes causing the bluffers to lose a lot of money when they are called. I asked them how they did it, given what each said about what happens during a game. Everyone wears large dark glasses, blocking most of the face. Not a word is spoken. That worried me since I had found that detecting lies using my methods got easier the more words that were spoken. What did they rely on if they couldn’t see most of the face and no clues from the words or sound of the voice?

Cards are picked up and laid down, cards are examined, and a movement of the cards on the table signals a wish for another card. They had learned the differences in those few movements to detect when someone is bluffing. But they were not going to tell me how they did it. They wanted me to tell them anything additional they could use to spot the bluffer.

Are poker winners wizards of deception detection?

I tested each of them by showing them a videotape I had made in which you saw 30 people lying or telling the truth. Some were lying about whether they were watching a film of gruesome surgery or beautiful flowers. Some were lying about a strongly held opinion about the justification of capital punishment. And others lied about whether they had taken money which was not theirs. I had shown this test to nearly 15,000 people in every profession you can think of. Over 95% of the people I tested did not much better than chance. But would these poker winners be in the 5%, who I called the wizards of deception detection? No; they were in with the 95%! They were highly skilled in interpreting a very special vocabulary of movements in this silent game of poker they played. Those movements did not occur when people were interviewed.

If people talk while they play their hand, if they claim to have a great hand when they don’t, then what I rely on to catch liars – very brief (micro) and very small (mini) facial expressions, gestural slips, voice changes, and so forth — will probably work. But the stakes have to be high: there has to be a lot to lose or gain. And there has to be conversation during and about the game. For the moment, the silent game of poker remains a mystery to most of us.  Mum’s the word.  And it’s also the best how-to tip for playing poker and getting away with bluffs.

Authorized Lying and Uncovering Deceptions

No one objects to trying to spot poker bluffers; in a sense it is authorized by the rules of the game. Catching liars is also authorized when interrogators question criminal suspects. Although told they have to answer truthfully, confessing crimes they committed, interrogators expect suspects will lie if they think they can get away with it. Although torture is not allowed, interrogators can trick the suspect to get at the truth. Our supreme court has upheld convictions based on confessions obtained by lying to the suspect – for example, claiming another suspect has already confessed, or their fingerprints were on the gun, when neither is true. (Incidentally, that is not allowed in England and most other European countries).

Medical patients are not always truthful, sometimes concealing problems they are embarrassed about having, and often lying about whether they really did use all the medications that were prescribed. No one objects when the doctors or nurses uncover such deceptions, but unlike the interrogator-suspect context, here it is done to help the person engaging in concealment.

Where does this leave us on lie catching?

Whether we should expect to be misled and feel authorized to catch liars is less certain in other arenas. Should we expect someone bent on seduction to be truthful about how many previous partners they have had, or how much undying love they feel? Will our friends tell us about our unwelcome or unattractive mannerisms? Oftenit is not clear what to expect and whether lie catching is needed or justifiable. That is the ambiguity we live with about truth and lies.

Visit our Training Tools page to learn more about how you can train yourself to recognize micro and subtle expressions and improve your ability to ‘read’ others.

Darwin’s Claim of Universals in Facial Expression Not Challenged

March 2014
 
Paul Ekman, Emeritus Professor, University of California, San Francisco
Dacher Keltner, Professor, University of California, Berkeley
 

Lisa Feldman-Barrett’s recent contribution (New York Times, February 28, 2014) seeks to undermine the science showing universality in the interpretation of facial expressions. In her eyes, recent evidence “challenges[ing] the theory, attributed to Charles Darwin, that facial movements might be evolved behaviors for expressing emotion.” Such a disagreement really belongs in exchanges of findings and theory in a scientific journal, evaluated by colleagues as evidence accumulates, not the public press. This is not the first time that Feldman-Barrett publicized her views in the press. We didn’t respond then, but feel compelled to do so now so that the public is not misled, and is apprised of the broader, Darwin-inspired science of emotional expression many scientists are working on today.

First, let’s get the science right. Darwin never claimed in his great book The Expression of Emotions in Man and Animals (1872) that all facial expressions are universal, only a specific set of expressions that he had observed and studied. Nearly one-hundred years later Silvan Tomkins helped Ekman and Carrol Izard refine and add to Darwin’s list. In the late sixties, Izard and Ekman in separate studies each showed photographs from Tomkins’ own collection, to people in various literate cultures, Western and Non-Western. They found strong cross cultural agreement in the labeling of those expressions. Ekman closed the loophole that observing mass media might account for cross cultural agreement by studying people in a Stone Age culture in New Guinea who had seen few if any outsiders and no media portrayals of emotion. These preliterate people also recognized the same emotions when shown the Darwin-Tomkins set. The capacity for humans in radically different cultures to label facial expressions with terms from a list of emotion terms has replicated nearly 200 hundred times.

Feldman Barrett is right to ask whether individuals in radically different cultures provide similar interpretations of facial expressions if allowed to describe the expressions on their own terms, rather than a list of emotion terms. Haidt and Keltner did such a study comparing the free responses to the Darwin-Tomkins set of expressions and some other expressions, with people in rural India and the U.S. Once again the findings of universality were clear cut, and evidence of universality in the expression of embarrassment was also found. The evidence on the judgment of the Darwin-Tomkins facial expressions is robust; so we suppose is Feldman-Barrett’s evidence for the expressions not covered in the Darwin-Tomkins set. She has missed that point, not understanding the difference between unselected and theoretically selected facial expressions.

Feldman-Barrett also ignores two other very powerful data sets that don’t involve showing portrayals of facial expressions to people. Instead what people actually do, spontaneous facial expressions, is measured in numerous, different emotional contexts. Ekman and Friesen published what might be the first such study comparing the spontaneous facial expressions shown by Japanese and American subjects in a private and public setting, finding universal facial expressions – the Darwin-Tomkins set—in private, and different expressions in public. Since then over a hundred studies have been published measuring spontaneous facial expressions, enough to justify two volumes reprinting the articles of dozens of scientists by Oxford University Press.

Another large body of research has established different patterns of physiology – in bodily changes generated by Autonomic Nervous System (ANS) activity and in brain activity – coinciding with the appearance of the Darwin-Tomkins set of facial expressions. Separate, well-replicated studies, have also shown that voluntarily generating the Darwin-Tomkins set of facial expressions produced distinct changes in ANS and brain activity! Still other studies have related the Darwin-Tomkins set of expressions to distinct responses, including cortisol, oxytocin, dopamine, and the cytokine response that is part of the immune system. This work, ignored in Feldman-Barrett’s critiques, suggests that facial expressions not only are informative about individuals’ feelings, but patterns of neurophysiological activation in their bodies. Darwin emphasized the importance of some universal facial expressions in establishing the unity of mankind, challenging the racist assertions of his time that Europeans had descended from a more advanced progenitor that Africans. Those findings and the conclusion that all human beings have a shared set of facial expressions remains unchallenged.

Are there universal facial expressions?

Screen Shot 2014-04-02 at 3.40.08 PMTake our simple test to learn more about facial expressions of emotion.