New software based on Artificial Intelligence helps to interpret complex data

Experimental data is often not only highly dimensional, but also noisy and full of artefacts. This makes it difficult to interpret the data. Now a team at HZB has designed software that uses self-learning neural networks to compress the data in a smart way and reconstruct a low-noise version in the next step. This enables to recognise correlations that would otherwise not be discernible. The software has now been successfully used in photon diagnostics at the FLASH free electron laser at DESY. But it is suitable for very different applications in science.

More is not always better, but sometimes a problem. With highly complex data, which have many dimensions due to their numerous parameters, correlations are often no longer recognisable. Especially since experimentally obtained data are additionally disturbed and noisy due to influences that cannot be controlled.

Helping humans to interpret the data

Now, new software based on artificial intelligence methods can help: It is a special class of neural networks (NN) that experts call "disentangled variational autoencoder network (β-VAE)". Put simply, the first NN takes care of compressing the data, while the second NN subsequently reconstructs the data. "In the process, the two NNs are trained so that the compressed form can be interpreted by humans," explains Dr Gregor Hartmann. The physicist and data scientist supervises the Joint Lab on Artificial Intelligence Methods at HZB, which is run by HZB together with the University of Kassel.

Extracting core principles without prior knowledge

Google Deepmind had already proposed to use β-VAEs in 2017. Many experts assumed that the application in the real world would be challenging, as non-linear components are difficult to disentangle. "After several years of learning how the NNs learn, it finally worked," says Hartmann. β-VAEs are able to extract the underlying core principle from data without prior knowledge.

Photon energy of FLASH determined

In the study now published, the group used the software to determine the photon energy of FLASH from single-shot photoelectron spectra. "We succeeded in extracting this information from noisy electron time-of-flight data, and much better than with conventional analysis methods," says Hartmann. Even data with detector-specific artefacts can be cleaned up this way.

A powerful tool for different problems

"The method is really good when it comes to impaired data," Hartmann emphasises. The programme is even able to reconstruct tiny signals that were not visible in the raw data. Such networks can help uncover unexpected physical effects or correlations in large experimental data sets. "AI-based intelligent data compression is a very powerful tool, not only in photon science," says Hartmann.

Now plug and play

In total, Hartmann and his team spent three years developing the software. "But now, it is more or less plug and play. We hope that soon many colleagues will come with their data and we can support them."

arö

  • Copy link

You might also be interested in

  • Self assembling monolayer can improve lead-free perovskite solar cells too
    Science Highlight
    04.08.2025
    Self assembling monolayer can improve lead-free perovskite solar cells too
    Tin perovskite solar cells are not only non-toxic, but also potentially more stable than lead-containing perovskite solar cells. However, they are also significantly less efficient. Now, an international team has succeeded in reducing losses in the lower contact layer of tin perovskite solar cells: The scienstists identified chemical compounds that self-assemble into a molecular layer that fits very well with the lattice structure of tin perovskites. On this monolayer, tin perovskite with excellent optoelectronic quality can be grown, which increases the performance of the solar cell.
  • Scrolls from Buddhist shrine virtually unrolled at BESSY II
    Science Highlight
    23.07.2025
    Scrolls from Buddhist shrine virtually unrolled at BESSY II
    The Mongolian collection of the Ethnological Museum of the National Museums in Berlin contains a unique Gungervaa shrine. Among the objects found inside were three tiny scrolls, wrapped in silk. Using 3D X-ray tomography, a team at HZB was able to create a digital copy of one of the scrolls. With a mathematical method the scroll could be virtually unrolled to reveal the scripture on the strip. This method is also used in battery research.
  • Long-term test shows: Efficiency of perovskite cells varies with the season
    Science Highlight
    21.07.2025
    Long-term test shows: Efficiency of perovskite cells varies with the season
    Scientists at HZB run a long-term experiment on the roof of a building at the Adlershof campus. They expose a wide variety of solar cells to the weather conditions, recording their performance over a period of years. These include perovskite solar cells, a new photovoltaic material offering high efficiency and low manufacturing costs. Dr Carolin Ulbrich and Dr Mark Khenkin evaluated four years of data and presented their findings in Advanced Energy Materials. This is the longest series of measurements on perovskite cells in outdoor use to date. The scientists found that standard perovskite solar cells perform very well during the summer months, even over several years, but decline in efficiency during the darker months.