Evaluating Generative Models for Galaxy Images

Generative models are a type of computer program that can create new images based on what they have learned from existing images. These models have great potential in many scientific areas, especially in astrophysics, where understanding galaxy images can lead to new discoveries. The challenge lies in evaluating how well these models perform, especially in terms of realistic details.

In this work, we propose that using galaxy images helps in developing better image generation models. By applying physical rules and relationships that govern galaxies, we can create a more reliable way to measure how well these models are doing beyond just using human opinion.

Galaxies change and evolve over billions of years. This evolution follows physical laws that, while generally straightforward, can be tough for computer models to capture accurately. To address this, we built two types of generative models, one called a Conditional Denoising Diffusion Probabilistic Model (DDPM) and another called a conditional variational autoencoder (CVAE). Both models aim to create realistic images of galaxies based on their age, which we measure using redshift.

Our study is one of the first to look at how well these models do when measured against physical properties of galaxies, not just human assessments. We found that while both the DDPM and CVAE models produced realistic images based on human judgment, the physics-based metrics we used were better at revealing the unique strengths and weaknesses of each model. Overall, the DDPM model outperformed the CVAE when looking at these physics-based measures.

The intersection of large data sets and machine learning in fields such as particle physics and genomics has led to significant advancements in analyzing complex information. In astrophysics, we deal with a lot of image data, which is complex and contains many features. Machine learning techniques can help scientists make sense of this data, which includes various types of images and measurements gathered from telescopes.

Generative models have been effective in producing images that people consider "realistic." Some common methods include variational autoencoders, generative adversarial networks, and denoising diffusion probabilistic models. Most evaluation methods have relied heavily on human judges who can spot issues in generated images, such as missing parts or unusual shapes. While humans are skilled at noticing these problems, checking millions of generated images by hand is not practical.

To address this, metrics like the Inception Score and Frechet Inception Distance were developed to provide a numerical score that aligns with human judgment. The Inception Score focuses on how diverse the generated images are and whether they represent clear objects. Meanwhile, the Frechet Inception Distance improves upon this by comparing generated images with real ones in a more structured way.

With generative models becoming capable of creating images that humans can't tell apart from real ones, reliance on human judgment becomes less useful. Thus, we argue that we need more physics-based metrics, which can provide a stronger foundation for evaluating how well models generate galaxy images.

To successfully produce galaxy images, a model needs to capture various features such as their shapes, sizes, brightness, and how these factors evolve over time. The most critical aspect affecting a galaxy's appearance is its redshift. This not only indicates how far the galaxy is from Earth but also how long ago the light we see was emitted.

Human perception metrics are still important, but they can miss key scientific details. For instance, the size distribution of galaxies at a given redshift should follow a certain pattern, as their appearance is related to their age. By quantifying these features, we can evaluate the quality of generated images using established astronomical tools.

In our work, we created both the CVAE and DDPM to generate galaxy images based on their Redshifts. We trained these models using a dataset of thousands of galaxies with redshifts ranging from low to high. Our goal was to create metrics that are tied to the physical properties of galaxies, adding to the existing human-based metrics like the Inception Score and the Frechet Inception Distance.

We wanted to see how well our models could reproduce important physical properties of galaxies based on redshift. Our findings indicate that while both models generate visually striking galaxies, the DDPM does a better job of matching physics-based characteristics, especially at higher redshifts.

#Related Studies

Recent studies show that astronomers have started using generative models to assess their ability to create galaxy images with strong visual quality. One notable approach involved using Conditional Variational Autoencoders and conditional generative adversarial networks to simulate images from well-known galaxy surveys.

Research from recent years has demonstrated that these models can generate useful galaxy images by conditioning them on various parameters, such as brightness and size. However, these studies mainly focused on visual quality rather than the physical relationships at play. Our work expands this area by examining how the appearance of galaxies changes over time and how generative models can account for this change.

Some studies aimed to simulate galaxy images using advanced techniques but did not prioritize the relationships between generated features and galaxy ages. Our approach differs by focusing on models that can accurately recreate galaxies based on their physical properties.

We build on these previous efforts by ensuring our machine learning methods can reproduce galaxies with both accurate physical characteristics and high visual quality based purely on redshift. We do this without simplifying the dataset, aiming to encourage models to learn the variety in galaxy images.

#Evaluation Metrics for Generative Models

Evaluating generated images quantitatively can be tricky since the features in images often contain complex, underlying relationships that do not reveal themselves in simple pixel-based statistics. Past research established metrics to help correlate with human perception, like the Inception Score and Frechet Inception Distance. However, these metrics can fall short for scientific purposes and in cases where humans cannot discern the quality of images.

To fill this gap, we propose using galaxy images as a new form of ground truth in generative models. The structure of galaxies is complex enough to challenge generative models, yet simple enough to be broken down into measurable properties. By assessing models through physics-based metrics, we can better evaluate how well these models replicate real galaxy characteristics.

We developed new metrics that compare the physical properties of generated galaxy images to those of real images. We can measure features in both generated and real images the same way, allowing us to analyze distributions using statistical methods. These properties are selected to represent meaningful aspects of galaxy evolution.

By focusing on the relationship between real and generated galaxy images, we can assess the effectiveness of our models in producing realistic images. This approach allows us to gauge how well the models understand the physics behind galaxy evolution.

#Fitting Individual Galaxies

To analyze galaxy features, we used a standard tool called Source Extractor. This tool locates galaxies in images by identifying areas with higher brightness than the background. We concentrated on three main parameters: isophotal area, ellipticity, and Sersic index.

The isophotal area refers to the number of pixels above a certain brightness threshold, while ellipticity measures the galaxy's shape. The Sersic index gives insight into how light is distributed in a galaxy based on its distance from the center.

#Measuring Galaxy Performance

By analyzing the properties of generated galaxies, we can contrast their distribution with real galaxy properties. For instance, we can calculate how well our models replicate isophotal areas, ellipticities, and Sersic indices across different redshift ranges.

This comparison allows us to determine how closely the generated images mimic real ones. We can quantify these comparisons using Kullback-Leibler divergence, which measures the difference between two probability distributions. The output from Source Extractor gives us the necessary data.

We also introduced the galaxy-fitting loss, which evaluates the irregularities in generated galaxies relative to real ones. This metric assesses how closely the properties align with the original data, allowing us to examine the quality of the generated galaxies.

In addition to the galaxy-fitting loss, we introduced a redshift loss metric to quantify how well the models recover redshift values. By comparing the actual redshift to predictions made by a pre-trained convolutional neural network, we can evaluate how accurately our models generate galaxy images on a redshift scale.

#Generating the Images

We trained our DDPM and CVAE models on a dataset containing galaxies with various redshifts. Both models produced impressive results, generating galaxies that resemble those seen in real images.

Visual inspection showed that both models could create realistic galaxies, but quantitatively, the metrics revealed important differences. The DDPM tended to produce images with fewer visual artifacts and better background properties, while the CVAE had issues with irregularities that were not present in real images.

#Examining the Results

The results of our study indicated that the DDPM generally outperformed the CVAE regarding generating galaxies with realistic physical properties, especially for galaxies at higher redshifts. The analysis of the metrics showed that while both models could generate visually appealing images, the DDPM more closely matched established galaxy characteristics.

Despite the models' successes, both struggled with accurately predicting redshift values, which is crucial for scientific applications. This inability suggests a gap in the models' understanding of the physics involved in galaxy evolution.

#Conclusion

By utilizing galaxy images as a physical ground truth, we can provide additional perspectives on evaluating image generation models. Our research indicates that galaxy metrics can act as a reliable method to assess how effectively generative models can replicate the complexities of galaxies over time.

Both the CVAE and DDPM models produced visually similar galaxies based on human evaluation, but the physics-based metrics highlighted their limitations. The DDPM excelled at capturing physical features of galaxies more consistently at higher redshifts, while the CVAE performed better at capturing lower redshift details.

In summary, while this work addresses the gap in evaluating generative models using new physics-based metrics, future research should aim to enhance these models' abilities to incorporate more complex relationships found in astrophysical data. Such developments may lead to even greater advancements in our understanding of galaxies and their evolution over time.