Deepfake Lips Are Coming to Dubbed Films

As reported on Gizmodo, a London-based company is trying to advance the state of the art in dubbing films. They don't have a solution for how to keep voice actors from sounding like they are standing in a sound booth. But they have a plan for how to match up the lips (and jaw and muscle movements) to the dubbed voices using an AI based recreation of actor's lower (or entire) faces, complete with tongues and teeth as needed.

From the article: "What Flawless is promising to do with its TrueSync software is use the same tools responsible for deepfake videos to manipulate and adjust an actor’s face in a film so that the movements of their mouths, and in turn the muscles in their faces, more closely match how they’d move were the original performance given in the language a foreign audience is hearing. So even though an actor shot a film in English, to a moviegoer in Berlin watching the film dubbed in German, it would appear as if all of the actors were actually speaking German."

The teaser video produced by Flawless somewhat disingenuously only shows examples of English films with English actors being deep-faked into speaking non-English languages (a poor test since we are not familiar with the lip movements for other languages). A better test would be to show foreign films with actors deep-faked into speaking English, since those would be lip-movements that we recognize more naturally. Perhaps they are not quite ready for that. Looks like they are looking for investors.

In any case it is a remarkable and sensible evolution of dubbing. Here is their current demo:

 

Fascinating.

I see a problem here; there may be some languages that aren't as concise as English. That means you either need a translation that may sacrifice script accuracy for how long does it take the deepfaked actor to say it and not be longer than the actor's expression, or you need to get really far out with the deepfaking and then you can't hold true to the "performance preserved" aspect of things.

Simple solution: learn to read subtitles.

 

Farewell reality!

Of course it comes out of Airstrip One, and we can only hope it will be used to further diminish the remaining cult of Goldstein - oh how I hate that man!

 

As a moviegoer from Berlin, who grew up with dubbed films, I can tell you that the lip movements are no problem for the overwhelming majority of viewers here. If you are used to it, you don't really notice it. German dubbing also seems to be of very high quality and made an art of using words that are close to the original lip movements. I don't really see a market here for a tool like this. It may be different in English-speaking countries, where people are much less used to dubbed content.

 

I find German and English tend to match up quite well surprisingly. Italian...no way...

I do prefer undubbed. I hate the dead acoustics of dubbing. Netflix tries hard with its modern dramas and movies but you know...I can read.

Again Italian stuff is hard with subtitles though. Seems the translation into English misses a lot of subtleties. Romantic/latin to Germanic I guess.

 

Sounds like a Flaming Lips cover band...
The Deepfake Lips!

 

This is looking more like the film The Congress everyday (recommended if you haven't seen it).
And isn't it a waste of money? I mean, processing a whole film just to match actors lips to the dubbing? Just, why bother?

 

Rather watch with subtitles, to be honest (inaccurate though they can be...)

 

I prefer subtitles when the acoustics and performances (or even the tone - voice too high or too low for a body of that size) don’t match, which is most of the time.

But some of the streaming services place the subtitles on the picture rather than in the black bar areas that are often available, which is just as bad as having heads in the way in a movie theatre.

I do find it distracting when the lips don’t match. But I notice that the samples they chose to demo are all scenes where the actors are perfectly stationary. So they clearly don’t have the technology yet, to do what they promise to do.

 

This is already a problem for dubs, where the rewritten dubs are a compromise based on trying to shorten or lengthen the translation to match the number of seconds available and also trying to hit as many mouth movements as possible. The faked lips will at least eliminate one of those two problems.

 

I prefer Bad Lip Reading!

 

That's rather annoying.

Since most movies are widescreen 2:35.1, they should be placed always in the bottom black bars, but no! 9 times out of 10, the subtitles are on the picture.

 

I really can’t watch dubbed pictures, unless they are animated movies. I prefer the actors real voice. With subtitles, you learn foreign words also.

Somehow I find this deep fake stuff dire and disturbing.

Maybe, in a few years, nobody will know what the fuss was all about.

 

I think this is serious "Turner Movie Color" country here. Plus I'm sure actor/directors/unions/guilds with clout will kick up dust if it gathers momentum.

 

This could also be used for censoring/amending language in English. I remember a scene in My Cousin Vinnie on TV where Joe Pesci and Marisa Tomei were visibly mouthing the word "****" at each other, whatever the soundtrack said.

 

Dialogue can be 're-written' to be correct, too.

Let the revisionism begin.

 

"Do you see what happens when you find a stranger in the Alps?"

 

Invasion of the Buddy Snatchers.

 

the future possibilities may be amazing...I'm looking forward to the outcome.

 

Deepfake Lips Are Coming... ? Oh yes!, I remember that film! ..

 

for me, dubbing programmes/ films always makes the acting seem worse somehow, the actors doing the translation usually seem to be overcompensating . much prefer original audio track with subtitles.

 

Just because you can doesn't mean you always should. Sure, the technology is neat, but even those examples looked off. They should do a test with an actor that is fluent in two languages (e.g. English and French), film the scene twice, and then compare it to the English version "converted" to French to the one filmed in French. Then you'd see a difference. I wonder if this has been done actually. Do any productions ever film their actors saying the lines in different languages?

But with this computer enhanced process you'd still have issues of atmosphere, hand movements, and the general good it is for the brain to hear the original language. When the first Harry Potter book was released, in the US it was translated (regionalized/localized?) from UK to US English. How dumb was that? Talk about overcompensating!

A lot is missed out when doing these kinds of conversions. Nuance, feel and other intangibles.

 

Yes! They can now remake Mr. Ed in color!

Oh, Whill-buurrrrr...