Is motion smoothing smoothing the way?
Modern TVs include all manner of processes that improve – or at least try to improve – the picture beyond the quality provided by the original signal. Here, Stephen Dawson explains one of these: motion smoothing. And he explains why it shouldn’t be overused.
Some of the things that a modern TV can do are stunning in their complexity. That some of these things are done imperfectly shouldn’t be a surprise. The surprise is that they can be done at all.
Which brings us to motion smoothing. What is it? Why is it done?
It was in the late 1920s and early 1930s that the frame rate for movies settled down. And it was sound that made it so. Before sound, cameras and projectors could be cranked at various rates. But when sound came in, sync was required so the film had to run at a fixed speed.
That speed came to be 24 frames per second (fps). And with very few exceptions, that has remained the standard even now, 90 years later. If movie making were to be first invented today, with today’s technology in hand, it probably would have been something like 120fps, perhaps as low at 60fps. That’s because 24fps brings with it several problems.
The main one is that in many circumstances humans can perceive the jumps between frames. Remember, motion is presented by objects appearing in different places on frames in sequence. Smooth motion relies on a physiological property called ‘persistence of vision’. Your vision works by means of an electro-chemical reaction to light falling on the cells of the retinas in your eyes. These take time to respond, and time to refresh for new images. So an image is held for some milliseconds.
At 24fps, the time from the start of one frame to the next is nearly 42ms. Your persistence of vision does not last that long. In the early days, cinema gained the nickname ‘the flicks’ because of the flickering of the image. That was addressed in part by displaying each frame twice in sequence, reducing the length of the black ‘space’ between each frame.
Nonetheless, you can see moving objects on the screen jump from position to position, rather than moving smoothly. We call this ‘judder’.
Well, you can sometimes. Cinematographers were all too aware of this – after all, they are people who pay close attention to how things look on screen. So of course they developed techniques to reduce the problem. The main technique is to use a slow shutter speed in their cameras.
Most of the time still photographers use a fast shutter speed, hoping to capture sharp detail in their images even if the subject is moving. But on movie film, ‘judder’ is far more perceptible, intrusive, when the moving objects are sharp. Reduce sharpness and you reduce judder.
A slower shutter speed smears the object being filmed, and the smearing is concentrated in the direction of movement. Artfully applied, judder can be almost eliminated. It can certainly be reduced to the point that most people don’t notice it most of the time.
For an example, play any Pixar animated movie. These are gloriously rendered by computers. If Pixar wanted the images to be sharp, they would be sharp. Indeed, computer generating sharp-edged images is easier than fuzzy ones. But pause your Pixar movie on a scene where a character is moving at a fair clip across the screen and you’ll see that the character is smeared in the direction of travel, as though it had been photographed with a slow shutter speed. Pixar programmed its computers to do this, so as to provide smooth motion.
Well, most of the time. In the opening scene of A Bug’s Life the ‘camera’ pans up from a mud flat. This wasn’t processed in that manner for some reason, and it judders – jump, jump, jump – up the screen.
Even with film photography, sometimes circumstances make the use of a sufficiently slow shutter speed impossible. One of my go-to clips is at 19:55 (on the Blu-ray) into the 1969 movie Easy Rider. A low budget production, it was shot on 16mm, a seemingly fast film stock. In this scene Dennis Hopper and Peter Fonda are riding in the far distance along a road threading in front of colourful mountains. The sunlight is bright. My guess is that with the equipment at hand and light levels involved, the image couldn’t be stopped down enough to allow a slow-enough shutter speed. (I guess they didn’t have a neutral density filter to hand, either.) In any case, the mountains jump, jump, jump horizontally, juddering across the screen.
Unless your display does something about it.
So, what can a display do?
What it can do – what most premium TVs, and many home theatre projectors in fact do – is double or more the number of frames. But unlike film cinema projectors of old, they don’t just repeat frames. They create new ones.
The first time I saw this in action was with the mud-flat scene I mentioned above. It was the late 1990s and Philips had just launched a TV (CRT, of course) with what it called ‘Digital Natural Motion’. This set the pattern for all that followed.
TVs were all analogue in those days, but such models would take the incoming analogue signal (whether from external inputs, or the TV tuner) and digitise it to facilitate image processing. What Digital Natural Motion did was take in two image frames, then create a new frame to display in between them. That new frame was an interpolated image based on the frame before and the frame after. If a person was running across the scene, such that they were one third of the way across in one frame, and two thirds of the way across in the second frame, the new frame would show them halfway between those two positions.
I remember being stunned at this technology. This was real-time processing, generating a new four hundred thousand pixel frame every 24th or 25th (for PAL) of a second. It was astonishing. And it worked.
You could flick DNM on and watch the judder disappear and the motion become silky smooth.
Digital technology has come a long way since then. Modern versions of this do exactly the same thing, but more of it and better. Rather than the 0.415 megapixels of standard definition TV and DVD, these days it works on 8.3 megapixel UltraHD video, and can generate two, three or four frames between each real one. That early Philips system was fine for an 80cm CRT TV, but not so much for a 165cm Ultra High Definition TV.
But right from the start there were problems with motion smoothing, and these problems remain. One is distortion. The system has to identify objects in a frame, and correlate them with objects in the following frame so that it could work out where to place them in between those frames. It doesn’t merely average the images. That would produce a largely random mess.
But sometimes it isn’t all that easy to make out details on images, so there’d sometimes be unexpected movement of objects that weren’t supposed to be moving. At various times I’ve seen the rivets on a metal bridge jump into and out of existence, fine latticework flicker weirdly, and a host of similar problems. Over the years, the processes have become better, but those problems can still arise, particularly when the system is turned up too ‘high’.
Another common distortion is what I call ‘heat haze’ distortion. That’s most commonly seen when a coloured object is moving in front of something of a different colour, say a brown object in front of a green background. That can sometimes result in a kind of swirling in the background around the edges of the foreground object. It looks like the object is extremely hot, creating eddies in the surrounding air. Again, most systems can control this so long as they’re not turned up too high.
The final problem was how motion smoothing frequently made film look like it was shot on a video camera in a studio. That was for two reasons. One was the mere fact of eliminating the near-subliminal judder inherent in film shown at 24 or 25 fps makes it look like the motion smoothness inherent in the 50fps of in-studio video.
The second was that there’s stuff in film frames that can’t be correlated. In particular, most film has ‘noise’ in the form of grain. That is inherently random. In the early days – and for many years thereafter – motion smoothing systems seemed to deal with that stuff by filtering it out. It may have merely been that it naturally disappeared, or was at least weakened, by appearing only in the original frames, not in the intermediate generated ones.
Modern systems seem to reduce or eliminate this issue of grain removal. The most likely technique is by them identifying the grain and holding it over into newly generated frames. That doesn’t generate any kind of ‘grain judder’ problems because the grain isn’t moving, just appearing randomly.
For many years Samsung TVs, for example, had their motion smoothing systems turned up to awfully high levels, resulting in all those problems, resulting in a glossy, highly artificial look.
A final problem concerns processing time. All that processing must take time. There’s naturally a near-42 millisecond delay before the creation of an intermediate frame can commence because the system as to await the arrival of the second incoming frame. In practice, TVs typically delayed the video by something around 120 milliseconds – thus the need for an audio delay facility in home theatre receivers. I saw one that went over 200 milliseconds – a fifth of a second.
Modern TVs often have several levels of motion smoothing. They come with different names, as do the different levels. In general the problems can be avoided by choosing the ‘Low’ or ‘Standard’ settings. Certainly don’t choose ‘High’.
Many systems also have a ‘Custom’ or ‘Manual’ mode.
In general, lower settings generate one intermediate frame, and higher levels create more. The best way to choose is to use a scene with familiar judder. I use Chapter 21, at 50:05 of the Blu-ray disc of the 1993 film, The Fugitive. This shows a flyover of some tall buildings in Chicago. The top of the tallest building in particular jumps a huge amount with each frame. The lowest level of motion smoothing – which generates one intermediate frame – still leaves judder visible. But it’s clearly faster, and clearly less intrusive. And with most systems, it doesn’t result in visible distortion in the subsequent scene. Look for heat haze distortion around Harrison Ford’s head as he passes in front of a bridge strut.
Modern motion smoothing systems also try to ‘de-blur’ the image – remove the motion smearing that cinematographers have intentionally put into the image. I guess that’s on the theory that it’s no longer needed, given the judder removal.
Me? I leave that off if possible. I haven’t seen any real advantage from it, so the less processing the better.