I got on a Spring Festival travel train that distorted time and space?!

1

Strange things happened on the train

A while ago, after reminiscing about my hometown, I boarded the train back home. The sun was rising over the rolling hills, and the shadows of the trees were flying by as my hometown faded away. I picked up my phone and took some photos of the scenery along the way.

Picture 1 is so beautiful that I decided to name it "Dead Tree, Setting Sun, and Wanderer"

But...wait? The nearby electric pole was a bit weird, shaped like a bunch of soft noodles, and the "High Voltage Danger" sign on it was twisted abnormally. The editor, who didn't believe in evil, took a few more photos out the window.

Figure 2 Some strange photos

The guardrails, electric towers, and street lights in the photo are all oddly crooked, and the four pillars of the electric tower are all crooked in different ways. Not only does he look like a drunk who is unsteady on his feet, he's also twisting his body around spiritedly.

What's going on? In fact, all of this is the fault of the imaging sensor. Most consumer-grade mobile phones and cameras use a rolling shutter, which exports light signals into electrical signals line by line. After scanning in this way, this distortion will occur.

Figure 3 Rolling shutter | Source: Wikipedia[1]

Similar to the "Blue Line Challenge" that was popular a while ago, the APP slows down the scanning line that reads the signal, allowing netizens to use their imagination to create various distorted photos using the scanning line. Some netizens even use it to paint.

Figure 4 Blue Line Challenge: Use a mask to draw a costume along the scan line

| Image source: TikTok @C&D

This aroused the editor's interest, and he immediately used a camera to reproduce the effect when he returned.

つ(*▽*)つ

٩(*▽*)٩

Figure 5: The word "Fu", the whiteboard, and the whiteboard eraser under high-speed shaking

Jelly-like elasticity

You can also try it yourself. Here is a small shooting tip: in professional mode, try to shorten the exposure time. I used 1/200s. Correspondingly, you should increase the ISO sensitivity and illuminate the subject as much as possible. If the shutter time is too long, motion blur will appear.

つ(*////▽////*)つ

٩(*\\\\▽\\\\*)٩

Figure 6 Yes, it is blurry like this

So in today’s article we will talk about this distorted “filter” – the jello effect.

2

Imaging Sensors

First, let's talk about the basic principles of imaging sensors in photography. After light passes through the lens group and reaches the imaging sensor, it is recorded and frozen by the sensor through chemical or electrical means.

In the ancient times of photography, cameras used photosensitive film as a recorder of light. After the silver halide (mostly silver bromide) on the film is exposed to light and developed, it will be reduced to silver. Because the fine silver powder is black, after washing away the excess silver halide with a fixer, a negative film with black and white reversed is obtained. To reverse the black and white back to the original image, it is only necessary to expose the negative film again. In this way, the image at the exposure stage is recorded chemically.

Now, cameras have entered the electronic age, and the mainstream imaging sensors have become CCD and CMOS. Their working processes include four steps: photoelectric conversion, charge collection, charge transfer, and charge measurement.

CCD (charge-coupled device), charge-coupled device. When photons are incident on the photodiode, photoelectrons are stimulated. The electrons of each pixel are collected together to form a charge packet, which is then transferred from the pixels in the previous row to the pixels in the next row in turn, and finally transferred to the measurement circuit to output their respective electrical signals.

Figure 7 Schematic diagram of CCD structure | Source: Edmund [2]

CMOS (complementary metal-oxide-semiconductor) is a complementary metal oxide semiconductor. Unlike CCD, each pixel of CMOS integrates analog circuits. Therefore, the reading operation that requires a whole column to complete collectively in CCD is completed by a single pixel in CMOS.

Figure 8 Schematic diagram of CMOS structure | Image source: Edmund [2]

From this we can see that the readout uniformity of CCD is better, after all, they share the same readout circuit. However, the tens of millions of pixels on CMOS have their own minds, and the amplifiers of each pixel cannot be guaranteed to be completely consistent, so the yield rate of early CMOS is indeed not high. In addition, the aperture ratio of CCD is larger, which allows more light to enter and obtain a higher signal-to-noise ratio. In comparison, each pixel of CMOS has to occupy a part of the space for the readout circuit, which makes the light receiving area of ​​a single pixel not as good as that of CCD.

But CMOS also has its advantages. CMOS consumes less power because its reading circuit only needs to amplify the electrical signal of a single pixel, while CCD needs to amplify the signal of an entire column of pixels. In addition, CMOS is more stable. If a pixel of a CCD is damaged, the transmission channel of the entire column of pixels will be broken, the signal will be shut down, the image will disappear, and the entire column of necrotic black spots will occupy the high ground.

Cao Cao: What a coincidence, I thought so too.

It is precisely these two points that have put CCD on the shelf, and it is mostly used only in scientific research and medical scenarios with high precision requirements, while civilian cameras are basically CMOS.

3

Rolling shutter and global shutter

Due to the working principle described above, most consumer-grade CMOS use rolling shutter. As the name implies, rolling shutter reads the light signal on the pixel array row by row like a rolling door. The global shutter reads the light signal on all pixel arrays at the same time.

Figure 9 Rolling shutter vs global shutter | Source: Wikipedia[1]

When reading electrical signals, CCD needs to vertically transfer the charge of the previous row to the next row. During the transfer process, these pixels are used to process the transmission process and cannot continue to be sensitive to light, otherwise it will affect the image quality. Therefore, it is easy for CCD to implement a global shutter so that all pixels are exposed at the same time. In contrast, it is much more difficult for CMOS to use a global shutter. It requires additional transistors and the design of a separate synchronization circuit, which increases the cost and is not conducive to market competition.

The consequence of rolling shutter, in addition to distorting moving objects like jelly, is the change of exposure in a single photo. Imagine that when a beam of lightning has not yet struck, the shutter is reading the upper part, and when the lightning strikes and illuminates the sky, the shutter starts to read the lower part, so the photo taken will appear dark on the top and bright on the bottom.

Figure 10 Lightning captured by CMOS | Source: Wikipedia[1]

This phenomenon can also be reproduced in the laboratory by flashing a high-brightness light source rapidly.

Rolling shutter flicker

Global shutter flicker

Figure 11 Two types of shutter flicker | Source: bilibili@陈大陈[3]

4

Turn the camera around

Previously, the camera moved in a direction perpendicular to the scanning direction. What would happen if we rotated the camera so that the camera moved in a direction parallel to the scanning direction? In episode 15 of the documentary “Think Like a Crow” [4], the host conducted an experiment on this issue.

Figure 12 An interesting experiment

| Image source: Think Like a Crow [4]

You can see that as the phone rotates 90 degrees, the train becomes wider, tilted, narrower, and tilted in another direction. If you understand the principle mentioned above, you might as well stop and think about what is going on.

Figure 13 Jello effect in different directions

| Image source: Think Like a Crow [4]

Answer:

The mobile phone camera used in this experiment scans and reads from right to left as shown in Figure ①, which is the same direction as the train's movement. The time it takes for a single window to pass through the scanning line is longer, making the train appear wider. Figure ③ is the opposite.

In Figure ②, the scan line sweeps from top to bottom, and the image below is more inclined to the leftward movement of the train. In Figure ④, the scan line sweeps from bottom to top, and the image above is more inclined to the leftward movement of the train.

5

Jello Effect Behavior Awards

At this point, everyone is very clear about the mechanism of the jello effect. I didn't expect that behind the advanced CMOS, there is such a small defect, so that the global shutter that was very natural in the film era has become a luxury nowadays. However, everything has two sides, just like the waning moon has its own unique arc beauty. Next, let's appreciate some of the unique visual effects brought by the jello effect.

5.1

Guitar Strings

Figure 14 The jello effect of guitar strings

| Image source: bilibili@Fun Science[5]

The vibration frequency of guitar strings is in the range of 100-1000Hz. When the exposure time is close to this level, it will take several guitar string oscillation cycles to scan a single frame, so an "oscilloscope" effect will appear. In order to see the original movement of the guitar string, a high-speed camera with an exposure time as short as 1/20000s must be used to capture the image on the left side of Figure 15. Of course, this phenomenon can be eliminated by rotating the camera 90°, as shown in Figure 16.

Figure 15 Comparison of the jello effect of guitar strings

| Image source: youtube@SmarterEveryDay[7]

Figure 16 The guitar string has almost no jello effect after rotation

| Image source: bilibili@Fun Science[5]

5.2

Audio

Figure 17 The jello effect of the sound system | Source: bilibili@子岚[8]

Similar to guitar strings, when the main frequency of the sound played by the speaker is close to the frame rate of the video, the eardrum will change from chaotic vibration to magical twisting.

5.3

propeller

Figure 18 Propeller side view

| Image source: youtube@SmarterEveryDay[7]

Figure 19 Propeller front view

| Image source: youtube@SmarterEveryDay[7]

I hope that these examples can inspire everyone to pay attention to physical phenomena in life. Behind seemingly strange phenomena, there are always people who are interested. I look forward to your better ideas in the comment section. I wish you all the best when you return to work~

References

[1] Rolling shutter - Wikipedia

[2] Imaging Electronics 101: Understanding Camera Sensors for Machine Vision Applications (edmundoptics.com)

[3] The difference between CCD and CMOS_bilibili

[4] Think Like a Crow_Episode 15_bilibili

[5] 【Fun Science】Season 1, Episode 19 - Understand the principle of the jello effect in three minutes, and you can also take cool movies with your phone_bilibili

[6] [Hardcore Science] The Secret of Cats' Running - Time Domain Sampling_bilibili

[7] Why Do Cameras Do This? | Rolling Shutter Explained - Smarter Every Day 172 - YouTube

[8] Headphone lovers, stay away! Super low frequency test song! Jello effect test song! _bilibili

Editor: Muyu

Source: Institute of Physics, Chinese Academy of Sciences