The mystery of dark energy: Is the force behind the universe's expansion variable?

Author: Duan Yuechu and Huang Xianghong

There are many fascinating and unsolved mysteries in the vast universe, and dark energy is undoubtedly one of the most thought-provoking. Scientists have always believed that dark energy is a constant force in the universe, as reliable as the progress of time. However, new research this spring brought a shocking discovery that dark energy does not seem to be consistent. This result is like a boulder thrown into the calm lake of science, stirring up waves.

Imagine we are sitting at the center of a firework that has just exploded. As the firework explodes, the sparks scatter, some coalescing into fiery filaments, others fading quickly into darkness. After a while, only smoke remains, just like the evolution of the universe. While the expansion of the firework is driven by chemical reactions, the expansion of the universe is driven by the energy of empty space itself. From our vantage point, the universe appears to be expanding in all directions, and the speed is accelerating.

This spring, a big news broke in the scientific community: There might be something wrong with dark energy, the mysterious force thought to be accelerating the expansion of the universe. Data from two well-known dark energy surveys, the Dark Energy Survey (DES) and the Dark Energy Spectroscopic Instrument (DESI), and a third existing component combined to reveal this strange phenomenon. The DES measures distant supernovae, the DESI experiment measures galaxies and sound waves from the early universe, and the third component measures the cosmic microwave background (CMB). When the data were combined and analyzed, the results were jaw-dropping.

The DESI telescope, located on Kitt Peak in Arizona, measured the positions of millions of galaxies that existed about 120 billion years ago. Astronomers compared these observed galaxy positions with the expected galaxy positions based on dark energy predictions, but found no expected spread. When cosmologists combined the DESI galaxies, supernovae from DES, and the cosmic microwave background, an even bigger surprise emerged: The realistic map of the universe began to diverge from the theoretical models.

What does this discovery mean? If the new results are true, then dark energy, which has long been thought to be constant, is variable after all. This is undoubtedly a huge shock to cosmology. "If this is true, it will be a big deal," said Licia Verde, a theoretical cosmologist at the Institute of Cosmic Sciences at the University of Barcelona in Spain. But she also emphasized that, as always, such an extraordinary assertion requires extraordinary evidence to support it.

Since its discovery, dark energy has been viewed as an unchanging force in the universe. Princeton University cosmologist Paul J. Steinhardt noted that the discovery can only be confirmed as real if the results stand up to scrutiny. At present, everything is still too early.

To understand the potential impact of this discovery, we need to go back to the time of Einstein. When Einstein formulated his theory of general relativity, he assumed that the universe was uniform and stationary. However, at the time, we didn't even know that other galaxies existed, and evidence showed that stars were not evenly distributed. In Einstein's equations, gravity and uniformity were incompatible, and gravity would lead to instability. If gravity dominated in a curved universe, then everything in the universe should have clumped into one giant clump, but this was not the case. So Einstein hypothesized that there was some cosmic force that counteracted gravity, which he called the "cosmological constant" and described by the Greek letter Lambda.

But in 1929, Edwin Hubble discovered that the universe was not static, but was expanding. Einstein abandoned this constant reactionary force, calling it his "greatest blunder." However, in 1998, research by Adam G. Rees, Saul Perlmutter, and Brian Schmidt found that supernovae that exploded when the universe was young were dimmer than expected, meaning that the universe was expanding outward faster and faster, presumably because of an omnipresent and constant force. The cosmological constant was thus resurrected, and scientists believed that the force behind it came from energy in empty space, called vacuum energy or dark energy.

Since the late 1990s, all observations seem to confirm this scenario. Lambda is now at the heart of the standard cosmological model, which combines dark energy with the gravity of a large amount of invisible "cold dark matter" (CDM), known as the Lambda-CDM model. In this model, about 68% of the universe is made up of dark energy, 27% is dark matter, and the remaining 5% is everything we can see and measure, including galaxies, stars, and even ourselves. Surveys like DESI aim to measure dark energy with precision to gain insight into its peculiarities.

However, not everyone is satisfied with the Lambda-CDM model. Steinhardt said: "This seems to be a very strange series of things, and the only advantage is that it is described by a number. But this does not mean that we should believe it." If it turns out that dark energy is time-varying, then this will bring many possibilities to cosmology.

Since the DESI findings were merged with the 3D map in April, economists have been getting busier. So far, no theory has been able to replace Lambda with other non-constant cosmic forces. Even before the new discovery, some cosmologists were inclined to look for alternatives to constant dark energy, both because the idea was too exotic—other known forces are not constant but vary with time, pressure, and other factors—and because Lambda seemed absurd to introduce into other physical theories. "In quantum theory, if you calculate the energy of empty space, you don't get a sensible answer; you get infinity," said Joshua A. Freeman, a cosmologist at the University of Chicago. "That's one reason people look for alternatives, because we don't understand why it would be valuable."

Theorists have proposed several ideas for a new dark energy, most of which involve a fluid-like energy field similar to the Higgs field that gives normal particles their mass. These proposed dark energy fields are often called "essences," named after the classical fifth element first imagined in ancient times. There are many ways that this energy field (also called a scalar field) can work, and it can produce the results we observe, such as galaxies being thrown away from each other and the universe appearing to expand, but the difference is that this cosmic force is temporary rather than permanent.

Some theorists favor the classical theory because they have studied a potential, earlier version of a scalar field called cosmic inflation. This field would have influenced the universe immediately after the Big Bang, driving it to expand exponentially before eventually settling down and continuing to accelerate at a slower rate. Rees believes that the scalar field behind modern dark energy is like "little inflation." As long as there is energy in the physical space of the universe, the universe will accelerate. This dark energy field is very weak, about 30 orders of magnitude lighter than the Higgs field, and is temporary, just like cosmic inflation.

One popular version, first proposed by Freeman and his collaborators in 1995, is called “thawed” or “slowly rolling” dark energy. Its effect on the universe is similar to that of the cosmological constant, says Jessie Muir, a theoretical cosmologist at the Perimeter Institute. “It manifests itself as empty space having some kind of intrinsic energy density, but because it changes in later universes, there may be some deviations in later universes,” she explains.

Imagine a ball rolling down the side of a hill toward a shallow, U-shaped valley. Without friction, the ball would roll to the opposite side and then wiggle back and forth. The field potential represented by the ball describes how easy it is to move the field potential relative to the density, or expansion, of the universe. It’s also a way to understand the Higgs field, which may have gone through some changes in its early days before the universe reached its current state. A similar but heavier field could have caused the universe to expand. If dark energy operates the same way, Freeman said, there’s precedent.

“In the only other example of acceleration that we know of (the expansion of the universe), we know it’s not dark energy, which is a constant. It’s something else,” he says. “I’ve always felt that we need to keep an open mind about what’s driving the current accelerated expansion of the universe.” In Freeman’s version, a ball is placed on the side of a mountain at the beginning of the universe. The ball gets stuck at first because the universe is too dense for it to roll fast. As the universe expands and matter dilutes, the ball starts to roll. This is called the “thawed” dark energy model because the ball thaws and starts moving. “It’s just like dark energy, but dark energy has a different effect on the expansion of the universe than if it were completely constant,” Freeman says. “And it could look exactly like what the recent data from DESI, the Dark Energy Survey, and the CMB show.”

In addition to the theories mentioned above, theorists are testing ideas such as the "Big Bounce", a cyclical universe model in which the Big Bang happens again and again, and variants of general relativity - models in which gravity behaves differently in the very early universe or on different scales, among other possibilities.

To reveal dark energy, or whatever other force is driving the universe’s expansion, a new generation of observatories is on the way. The Euclid space telescope, operated by the European Space Agency, launched last year and will work until 2030, mapping nearly a third of the sky and cataloging dark matter and dark energy. NASA’s future Nancy Grace Roman Space Telescope will survey more than a billion galaxies, delving deeper into dark energy. The DES survey will run until 2026.

For Lambda to fall, cosmologists need to reach a five-sigma confidence level, meaning there's only about a one-in-a-million chance that the findings are an error or the result of random chance. So far, results from DESI, DES, and the cosmic microwave background from the Planck satellite suggest a three-sigma probability, or about a 0.3 percent chance that something like this happened by chance. While that sounds like strong evidence, three-sigma results can fail under close scrutiny, so five-sigma findings are essential for true scientific discovery. DESI is still continuing its work, but the team has had a year to work through the Milky Way data, and Verde says her colleagues are hitting their stride.

Jessie Muir, who also studies general relativity and tests of gravity at different scales, thinks the current state of the universe will provide the best clues. If dark energy is a fluid-like energy field like essence, then models will predict some type of relationship between how the universe expands over time and how cosmic structure clumps. Cosmologists can look for correlations between expansion and growth, such as the formation of galaxy clusters, to understand essence and gravity outside of general relativity.

Even Verde, who is working on the DESI analysis, is skeptical that Lambda-CDM will be overturned. “I’m really conservative, but based on this, would I personally be willing to throw persistent dark energy out the window — not yet,” she said. “Now we need to keep observing and understand it better.”

Rees said many cosmologists are watching Lambda closely but have yet to make a final call on it. Steinhardt thinks systematic errors could play a role in the new discovery, especially when three different types of data are combined to draw a comprehensive conclusion. "Everyone is doing their best, but you should approach it with a skeptical heart," he said.

If Lambda somehow survives, it will be a very boring ending—and a philosophically challenging one. The future of the universe will be cold, empty, distant, and serene. The expansion will accelerate forever until the atoms themselves are stretched so thin that their centers can no longer hold up, and they collapse.

But Freeman says the future may be brighter than that. “These hints from DESI and DES tell us to keep going,” he adds.

Scientific exploration is endless, and the study of dark energy is still ongoing. Every new discovery, every question and verification pushes us closer to the truth of the universe. In this vast universe, there are still countless mysteries waiting for us to uncover.

The discovery of dark energy has had profound and multifaceted effects on our understanding of the evolution of the universe:

1. It changes our understanding of the expansion of the universe. Before the discovery of dark energy, people generally believed that the expansion of the universe would gradually slow down due to the effect of gravity. However, the existence of dark energy makes the expansion of the universe appear to be accelerating, which completely subverts previous expectations and makes us realize that the evolution of the universe may be more complex and dynamic than imagined.

2. The discovery of dark energy has forced us to re-examine the composition of the universe. In the past, we mainly focused on visible matter and dark matter, but the discovery of dark energy means that there is a dominant component in the universe that we know very little about. This forces us to reconstruct our understanding of the distribution of matter and energy in the universe and how they interact to shape the evolution of the universe.

3. The nature and changes of dark energy have a decisive influence on the future fate of the universe. If dark energy continues to increase, the universe will expand at a faster rate, causing the distance between galaxies to become increasingly farther, and may eventually form a cold, empty and isolated universe. On the other hand, if the nature of dark energy changes or is proven to be not constant, then the evolutionary path of the universe may be different, which provides us with new possibilities and challenges for thinking about the ultimate destination of the universe.

4. The study of dark energy has also promoted the development of theoretical physics. In order to explain the origin and nature of dark energy, scientists have proposed various theoretical models and hypotheses, which have prompted physics to conduct deeper exploration at the micro and macro levels, and are expected to promote the birth of new physical theories.

In short, the discovery of dark energy has greatly expanded our understanding of the evolution of the universe, while also raising many challenging questions and directions for future scientific research.