Blog - Posts in Update

SunPy Development Team wins a NASA Group Achievement Award!

2024-06-19T00:00:00+00:00

We are delighted to announce that the SunPy development team has been awarded with a NASA Group Achievement Award!

The award was given to our team for “developing and continually improving free and reliable open-source software that supports NASA missions and the analysis of NASA’s vast archives of solar physics data”.

This award is a collective achievement, and we share it with everyone who has contributed to SunPy! We look forward to continuing our mission of supporting solar physics research and making impactful contributions to the scientific community.

pyOpenSci and sunpy

2024-01-24T00:00:00+00:00

pyOpenSci is a “diverse community of people interested in building a community of practice around scientific software written in Python”. They are an exciting community working on improving the information around the packaging and tooling used by the scientific Python community as well to provide support for package maintainers. There is a review process to become a pyOpenSci accepted package which comes with two reviews from independent members of the wider community.

With this in mind, the SunPy Project decided to submit the sunpy package to pyOpenSci to ensure that it is aligned with the community that pyOpenSci is building, and increase visibility of SunPy.

The review was started on the pyOpenSci software-submission repository. The entire review process is open, with one editor and two reviewers and the entire exchange between the sunpy maintainers and the reviewers are on that issue. This process was incredibly helpful as it highlighted areas of our documentation which could be improved and some technical choices that were made several years ago but now were redundant or not best practice.

After these issues were fixed, sunpy was accepted into pyOpenSci and we added the badge to the sunpy readme. The plan in future is to submit more SunPy Project packages as well as integrate more closely with pyOpenSci to replace our own affiliated package system. Our hope is that by aligning with pyOpenSci we can help to contribute back to the wider community.

aiapy: A SunPy affiliated package for analyzing data from the Atmospheric Imaging Assembly

2022-01-06T00:00:00+00:00

This blog post was written in a Jupyter notebook. Click here for an interactive version:

[1]:

import aiapy
import astropy
import astropy.time
import astropy.units as u
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sunpy
import sunpy.map
from aiapy.calibrate import (
    correct_degradation,
    degradation,
    estimate_error,
    fetch_spikes,
    register,
    respike,
    update_pointing,
)
from aiapy.calibrate.util import get_correction_table, get_error_table, get_pointing_table
from aiapy.psf import deconvolve, psf
from aiapy.response import Channel
from astropy.coordinates import SkyCoord
from astropy.visualization import ImageNormalize, LogStretch, time_support
from sunpy.net import Fido
from sunpy.net import attrs as a
from sunpy.time import parse_time

# Increases the figure size in this notebook.
mpl.rcParams["savefig.dpi"] = 150
mpl.rcParams["figure.dpi"] = 150

The Atmospheric Imaging Assembly (AIA) instrument onboard the NASA Solar Dynamics Observatory (SDO) spacecraft has observed the full- disk of the Sun, nearly continuously, for the last ten years. It is one of three instruments on SDO, along with the Helioseismic and Magnetic Imager (HMI) and the Extreme Ultraviolet Variability Experiment (EVE). AIA is a narrowband imaging instrument comprised of four separate telescopes that collectively observe the full-disk of the Sun at ten different wavelengths: seven extreme ultraviolet (EUV) wavelengths, two far UV wavelengths, and one visible wavelength.

aiapy is a SunPy-affiliated Python package for analyzing calibrated (level 1) imaging data from AIA. It includes capabilities for aligning images between channels, deconvolving images with the instrument point-spread function (PSF), computing channel sensitivity as a function of wavelength, and correcting images for telescope degradation, among others. These capabilities are divided between the three subpackages of aiapy:

aiapy.calibrate
aiapy.psf
aiapy.response

In this blog post, we will demonstrate the functions in each of these subpackages on a set of example level 1 AIA images.

Before we begin, we make a note of what versions of astropy, sunpy, and aiapy were used in the writing of this blog post.

[2]:

print(f"astropy v{astropy.__version__}")
print(f"sunpy v{sunpy.__version__}")
print(f"aiapy v{aiapy.__version__}")

astropy v7.0.1
sunpy v6.1.1
aiapy v0.10.1

Obtaining AIA Data

First, we will use the Fido client to fetch two AIA images from the VSO: one from the 171 Å channel and one from the 211 Å channel.

[3]:

t_start = parse_time("2017-09-10T20:00:00")
search_results = Fido.search(
    a.Time(t_start, t_start + 12 * u.s),
    a.Instrument.aia,
    a.Wavelength(171 * u.angstrom) | a.Wavelength(211 * u.angstrom),
)
search_results

[3]:

Results from 2 Providers:

1 Results from the VSOClient:

VSOQueryResponseTable length=1

Start Time	End Time	Source	Instrument	Wavelength	Provider	Physobs	Wavetype	Extent Width	Extent Length	Extent Type	Size
				Angstrom							Mibyte
Time	Time	str3	str3	float64[2]	str4	str9	str6	str4	str4	str8	float64
2017-09-10 20:00:09.000	2017-09-10 20:00:10.000	SDO	AIA	171.0 .. 171.0	JSOC	intensity	NARROW	4096	4096	FULLDISK	64.64844

1 Results from the VSOClient:

VSOQueryResponseTable length=1

Start Time	End Time	Source	Instrument	Wavelength	Provider	Physobs	Wavetype	Extent Width	Extent Length	Extent Type	Size
				Angstrom							Mibyte
Time	Time	str3	str3	float64[2]	str4	str9	str6	str4	str4	str8	float64
2017-09-10 20:00:12.000	2017-09-10 20:00:13.000	SDO	AIA	211.0 .. 211.0	JSOC	intensity	NARROW	4096	4096	FULLDISK	64.64844

[4]:

files = Fido.fetch(search_results)

We can then create sunpy.map.Map objects from the downloaded files.

[5]:

m_171, m_211 = sunpy.map.Map(sorted(files))

[6]:

m_171.peek(vmin=0)
m_211.peek(vmin=0)

Note that both the sunpy and aiapy documentation pages include more detailed examples of how to obtain AIA data from the JSOC as well

More complex queries (e.g. searching by exposure time or quality keyword) can be constructed by searching the JSOC directly via the drms package. See the last example listed above for an example of this.

PSF Deconvolution

AIA images are subject to convolution with the instrument PSF due to effects introduced by the filter mesh of the telescope and the CCD, among others. This has the effect of “blurring” the image. The PSF diffraction pattern may also be particularly noticeable during the impulsive phase of a flare where the intensity enhancement is very localized. To remove these artifacts, the PSF must be deconvolved from the level 1 image. This is because the PSF itself is defined on the pixel grid of the CCD.

We’ll calculate the PSF for the 171 Å channel using the psf function in the aiapy.psf subpackage. The PSF model accounts for several different effects, including diffraction from the mesh grating of the filters, charge spreading, and jitter. This implementation of the PSF calculation follows Grigis et al (2012). Further details on how the PSF is calculated can be found in that document. Currently, this only works for \(4096\times4096\) full frame images.

Note: The following cell will be significantly slower (many minutes) if you do not have a NVIDIA GPU and the cupy package installed. The GPU-enabled calculation has been benchmarked at ~5 s on an NVIDIA RTX 4090. Use of the GPU can be explicitly disabled by setting use_gpu=False.

[7]:

psf_171 = psf(m_171.wavelength)

We can then visualize the PSF. The diffraction “arms” extending from the center pixel can often be seen in flare observations due to the intense, small-scale brightening.

[8]:

plt.imshow(psf_171, origin="lower", norm=ImageNormalize(vmax=1e-6, stretch=LogStretch()))
plt.colorbar()

[8]:

<matplotlib.colorbar.Colorbar at 0x70adf80b3ed0>

We can then use the calculated PSF to deconvolve our 171 Å image using the deconvolve function in aiapy.psf. This deconvolution algorithm is an implementation of the well-known Richardson-Lucy algorithm. If the PSF is not explicitly passed in, the PSF for that wavelength will be computed automatically. When deconvolving many images, it is highly recommend to precompute the PSF function for each desired wavelength.

NOTE: The deconvolution of the image can also be optionally accelerated with a NVIDIA GPU and cupy. On a Titan RTX, deconvolution of a full \(4096\times4096\) image has been benchmarked at \(\le1\) s. On a CPU, the equivalent operation should take ~25 s.

[9]:

m_171_deconvolved = deconvolve(m_171, psf=psf_171)

2025-03-03 14:31:50 - sunpy - INFO: Using a GPU via cupy

INFO: Using a GPU via cupy [aiapy.psf.deconvolve]

We can now zoom in on some loop structures that extend off the limb to examine the differences between our image before and after deconvolution.

[10]:

blc = SkyCoord(750, -375, unit="arcsec", frame=m_171.coordinate_frame)
fov = {"width": 400 * u.arcsec, "height": 400 * u.arcsec}
m_171_cutout = m_171.submap(blc, **fov)
m_171_deconvolved_cutout = m_171_deconvolved.submap(blc, **fov)

[11]:

fig = plt.figure(figsize=(7, 3))
ax = fig.add_subplot(121, projection=m_171_cutout)
m_171_cutout.plot(axes=ax, title="Before Deconvolution")
ax = fig.add_subplot(122, projection=m_171_deconvolved_cutout)
m_171_deconvolved_cutout.plot(axes=ax, title="After Deconvolution")
ax.coords[1].set_axislabel(" ")

Note that comparing the images on the left and right, deconvolving with the PSF removes the appearance of the diffraction pattern near the bright looptop. This also brings out more detail in the strands of the post-flare arcade.

We can see this more clearly by plotting the intensity of the original and deconvolved images across the top of the looptop.

[12]:

x = np.linspace(m_171_deconvolved_cutout.dimensions.x.value * 0.55, m_171_deconvolved_cutout.dimensions.x.value * 0.7)
y = 0.59 * m_171_deconvolved_cutout.dimensions.y.value * np.ones(x.shape)
sl = np.s_[np.round(y[0]).astype(int), np.round(x[0]).astype(int) : np.round(x[-1]).astype(int)]

[13]:

fig = plt.figure(figsize=(7, 3))
ax = fig.add_subplot(121, projection=m_171_deconvolved_cutout)
m_171_deconvolved_cutout.plot(axes=ax)
ax.plot(x, y, lw=1)
ax = fig.add_subplot(122)
Tx = sunpy.map.all_coordinates_from_map(m_171_cutout)[sl].Tx
ax.plot(Tx, m_171_cutout.data[sl], label="Original")
ax.plot(Tx, m_171_deconvolved_cutout.data[sl], label="Deconvolved")
ax.set_ylabel(f"Intensity [{m_171_cutout.unit}]")
ax.set_xlabel(r"Helioprojective Longitude [arcsec]")
ax.legend(loc="upper center", ncol=2, frameon=False, bbox_to_anchor=(0.5, 1.15))
plt.tight_layout()

Respiking Level 1 Images

The aiapy.calibrate module contains a number of useful functions, including the ability to “respike” images through the respike function. AIA level 1 images have been corrected for hot-pixels (commonly referred to as “spikes”) using an automated correction algorithm which detects them, removes them, and replaces the “holes” left in the image via interpolation. However, for certain research topics, this automated hot-pixel removal process may result in unwanted removal of bright points which may be physically meaningful. The respike function allows you to revert this removal by putting back all the removed pixel values. This corresponds to the aia_respike.pro IDL procedure as described in the SDO Analysis Guide.

We will create a “respiked” version of our level 1 171 Å image. This function will query the JSOC for the relevant “spikes” file which contains the locations (in pixel coordinates) and associate intensities for each of the hot pixels. The hot pixel values are then replaced in the image at the appropriate locations.

[14]:

m_171_respiked = respike(m_171)

[15]:

fig = plt.figure(figsize=(7, 3))
ax = fig.add_subplot(121, projection=m_171)
m_171.plot(axes=ax)
ax.set_title(f"Despiked (Level {m_171.processing_level:.0f})")
ax = fig.add_subplot(122, projection=m_171_respiked)
m_171_respiked.plot(axes=ax)
ax.set_title(f"Respiked (Level {m_171_respiked.processing_level})")
ax.coords[1].set_axislabel(" ")

Looking at the two images, it is not obvious by visual inspection what the differences are between the despiked and respiked maps. We can use the fetch_spikes functions (which is called automatically by respike) to find the intensity values and the associated pixel coordinates

[16]:

pix_coords, vals = fetch_spikes(m_171)

We can then use the pixel coordinates to find the “despiked” intensity values and plot the distributions of the “despiked” and “respiked” values.

[17]:

vals_despiked = m_171.data[pix_coords.y.value.round().astype(int), pix_coords.x.value.round().astype(int)]

[18]:

plt.hist(vals, bins="scott", log=True, histtype="step", label="Respiked")
plt.hist(vals_despiked, bins="scott", log=True, histtype="step", label="Despiked")
plt.legend()
plt.xlabel(f"Intensity [{m_171.unit.to_string()}]")
plt.ylabel("Number of Pixels")

[18]:

Text(0, 0.5, 'Number of Pixels')

Additionally, we can plot the coordinates of each of our hot pixels on our “respiked” map. We can convert the above pixel coordinates to world coordinates using the .pixel_to_world() method on our level 1 map. Note that we can also return the world coordinates from the fetch_spikes function automatically by passing in the as_coords=True keyword argument.

[19]:

spike_coords = m_171.pixel_to_world(pix_coords.x, pix_coords.y)

[20]:

fig = plt.figure()
ax = fig.add_subplot(111, projection=m_171_respiked)
m_171_respiked.plot(axes=ax)
ax.plot_coord(spike_coords, marker=".", ls=" ", markersize=1)

[20]:

[<matplotlib.lines.Line2D at 0x70ad8c28a0d0>]

For a more detailed example of respiking level 1 images, see the respiking example in the aiapy example gallery.

Transforming Level 1 Images to Level 1.5

One of the most common operations performed on level 1 AIA images is “aiaprep,” commonly performed with the aia_prep.pro procedure in IDL. This prep process, which promotes level 1 images to level 1.5, is a combination of two steps:

update pointing and
image registration.

The first step is accomplished through the update_pointing function in aiapy.calibrate. This function queries the 3-hourly master pointing table stored at the JSOC to update relevant FITS keywords.

[21]:

table = get_pointing_table("JSOC", time_range=[m_171.date - 1 * u.day, m_171.date + 1 * u.day])
m_171_up = update_pointing(m_171, pointing_table=table)

We can verify this step by examining the reference pixel before and after the pointing update.

[22]:

m_171.reference_pixel

[22]:

PixelPair(x=<Quantity 2052.300049 pix>, y=<Quantity 2048.280029 pix>)

[23]:

m_171_up.reference_pixel

[23]:

PixelPair(x=<Quantity 2052.33252 pix>, y=<Quantity 2048.275391 pix>)

The second step is accomplished through register function in aiapy.calibrate. This function performs an affine transform on the level 1 pixel grid to remove the roll angle, scale the image to resolution of 0.6 arcseconds-per-pixel, and shift the center of the image to the center of the Sun. The original intensity values are then reinterpolated onto this transformed grid.

[24]:

m_171_l15 = register(m_171_up)

We can verify that the image has been transformed appropriately by comparing at the scale and rotation_matrix attributes before and after registration. Note that the rotation_matrix is now diagonalized, meaning that the world and pixel grids are aligned.

[25]:

print(m_171_up.scale)
print(m_171_up.rotation_matrix)

SpatialPair(axis1=<Quantity 0.599489 arcsec / pix>, axis2=<Quantity 0.599489 arcsec / pix>)
[[ 9.99999944e-01 -3.35522089e-04]
 [ 3.35522089e-04  9.99999944e-01]]

[26]:

print(m_171_l15.scale)
print(m_171_l15.rotation_matrix)

SpatialPair(axis1=<Quantity 0.6 arcsec / pix>, axis2=<Quantity 0.6 arcsec / pix>)
[[ 1.00000000e+00 -5.90601768e-20]
 [-2.18097419e-20  1.00000000e+00]]

Note that we can also combine these two operations into a “prep” function to imitate aiaprep.pro.

[27]:

def prep(smap):
    table = get_pointing_table("JSOC", time_range=[smap.date - 1 * u.day, smap.date + 1 * u.day])
    return register(update_pointing(smap, pointing_table=table))

[28]:

m_211_l15 = prep(m_211)

Applying this “prep” procedure means that our 171 Å and 211 Å images are now interpolated to the same pixel grid.

[29]:

print(m_211_l15.scale)
print(m_211_l15.rotation_matrix)

SpatialPair(axis1=<Quantity 0.6 arcsec / pix>, axis2=<Quantity 0.6 arcsec / pix>)
[[1.00000000e+00 5.18338069e-20]
 [5.18338069e-20 1.00000000e+00]]

Before performing any operations between two maps from different channels, one should always register each image such that the pixel grids of each image are aligned. For more information on transforming level 1 images to level 1.5, see the following example in the aiapy example gallery:

Registering and aligning level 1 data

Degradation Correction

The performance of the AIA telescope is unfortunately degrading over time, leading to the resulting images becoming increasingly dim. We can correct for this by modeling the degradation over time using the degradation function. This function calculates the degradation factor \(d(t)\) as,

\[d(t) = \frac{A_{eff}(t_{e})}{A_{eff}(t_0)}(1 + p_1\delta t + p_2\delta t^2 + p_3\delta t^3),\]

where the leading coefficient is the ratio between the effective area at some epoch time \(t_e\) and the initial time \(t_0\). The term in the parentheses represents an interpolation from \(t_e\) to the selected time \(t\), where \(\delta t=t-t_e\). \(p_1,p_2,p_3\) as well as \(A_{eff}(t_e)\) and \(A_{eff}(t_0)\) are retrieved from the JSOC (by default) or a user-supplied table.

We can use this function to calculate the degradation of the 211 Å channel from the start of the SDO mission to now.

[30]:

t_begin = parse_time("2010-03-25T00:00:00")
now = astropy.time.Time.now()
time_window = t_begin + np.arange(0, (now - t_begin).to(u.day).value, 7) * u.day

First, we’ll query the degradation calibration table from the JSOC. Doing this ahead of time avoids repeated network calls when we want to repeatedly calculate the degradation.

[31]:

correction_table = get_correction_table("JSOC")

We can then calculate the 211 Å degradation using the above correction table.

[32]:

d_211 = degradation(m_211.wavelength, time_window, correction_table=correction_table)

Note that we can also use the .date attribute on our 211 Å map to compute the degradation at the time of that map

[33]:

d_211_map = degradation(m_211.wavelength, m_211.date, correction_table=correction_table)

The degradation calibration is routinely updated (e.g. using data from sounding rocket flights), with new versions added to the table returned from the JSOC. We can look at older versions of the calibration to see how the predicted degradation has changed using the calibration_version keyword argument. Note that the default version in aiapy v0.10.0 is 10.

And compare the two degradation curves as a function of time over the lifetime of the mission.

[34]:

with time_support(format="jyear"):
    plt.plot(time_window, d_211, label="v10")
    plt.plot(m_211.date, d_211_map, linestyle="", marker="o", color="C0", label=m_211.date.iso)
plt.ylabel("Degradation 211 Å")
plt.legend()

[34]:

<matplotlib.legend.Legend at 0x70ad7e10a210>

Additionally, aiapy.calibrate also includes the correct_degradation function which accepts a sunpy.map.Map, calculates the degradation factor, divides the image data by this degradation factor, and returns a new corrected map.

[35]:

m_211_corrected = correct_degradation(m_211, correction_table=correction_table)

[36]:

fig = plt.figure(figsize=(7, 3))
ax = fig.add_subplot(121, projection=m_211)
m_211.plot(axes=ax, vmin=0, vmax=2.5e3, title="Uncorrected")
ax = fig.add_subplot(122, projection=m_211_corrected)
m_211_corrected.plot(axes=ax, vmin=0, vmax=2.5e3, title="Corrected")
ax.coords[1].set_axislabel(" ")

Computing Uncertainties

One of the newest additions to aiapy is the function for computing uncertainties on AIA intensities. This is done through the estimate_error function in aiapy.calibrate. estimate_error is an exact port of the SSWIDL function aia_bp_estimate_error.pro. This calculation includes errors from the shot noise, read noise, quantization, and onboard compression. The calculation can also optionally include contributions from the photometric calibration and errors in the atomic data.

We can pass in the (unitful) data array attached to our 171 Å map and compute the intensities.

[37]:

error_table = get_error_table("SSW")
errors_171 = estimate_error(m_171_l15.quantity / u.pix, m_171_l15.wavelength, error_table=error_table)

/home/nabil/.micromamba/envs/solar/lib/python3.13/site-packages/astropy/units/quantity.py:658: RuntimeWarning: invalid value encountered in sqrt
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)

To easily visualize the spatial distribution of these uncertainties, we can create a new sunpy.map.Map object from these errors.

[38]:

m_171_errors = sunpy.map.Map(errors_171.value, m_171_l15.meta)

[39]:

m_171_errors.peek(norm=ImageNormalize(vmax=50))

We can also use the include_chianti flag to understand how the presence of atomic data errors from CHIANTI affect the uncertainty.

[40]:

errors_171_chianti = estimate_error(
    m_171_l15.quantity / u.pix, m_171_l15.wavelength, include_chianti=True, error_table=error_table
)

/home/nabil/.micromamba/envs/solar/lib/python3.13/site-packages/astropy/units/quantity.py:658: RuntimeWarning: invalid value encountered in sqrt
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)

Similarly, we can look at how the photometric calibration, using the calibration from EVE, affects the uncertainties.

[41]:

errors_171_eve = estimate_error(
    m_171_l15.quantity / u.pix, m_171_l15.wavelength, include_eve=True, error_table=error_table
)

We can then look at the distribution of these uncertainties with these different combinations of keyword arguments to understand how these different options affect the resulting distribution of errors. Note that the inclusion of the atomic data results in the largest errors because there is an additional 25% (for 171 Å) uncertainty assumed on the input intensities.

[42]:

hist_params = {"bins": np.logspace(0, 4, 50), "histtype": "step", "log": True}
plt.hist(errors_171.value.flatten(), **hist_params, label="Nominal")
plt.hist(errors_171_chianti.value.flatten(), **hist_params, label="CHIANTI")
plt.hist(errors_171_eve.value.flatten(), **hist_params, label="Photometric (EVE)")
plt.xlabel("Uncertainty [ct/pix]")
plt.ylabel("Number of Pixels")
plt.xscale("log")
plt.legend(frameon=False)

[42]:

<matplotlib.legend.Legend at 0x70ad8dd0f9d0>

Exposure Time Normalization

For normalizing images by the exposure time:

[43]:

m_171_norm = m_171_l15 / m_171_l15.exposure_time

This option still properly accounts for the change in units but does not reset the exposure time.

[44]:

print(m_171_norm.unit)

DN / s

[45]:

print(m_171_norm.exposure_time)

2.000169 s

Wavelength Response Functions

Lastly, the aiapy.response subpackage includes the Channel object for computing the wavelength response function of each channel as well as providing various pieces of metadata for each channel.

[46]:

c = Channel(m_211.wavelength)

[47]:

print(c.channel)
print(c.telescope_number)

211.0 Angstrom
2

The wavelength response is a combination of the (wavelength-dependent) effective area and the gain of the telescope such that \(R(\lambda,t) = A_{eff}(\lambda)G(\lambda)\). According to Section 2 of Boerner et al. (2012), the effective area is given by,

\[A_{eff}(\lambda) = A_{geo}R_P(\lambda)R_S(\lambda)T_E(\lambda)T_F(\lambda)D(\lambda)Q(\lambda),\]

where \(A_{geo}\) is the geometrical collecting area, \(R_{P,S}\) are the reflectances of the primary and secondary mirrors, respectively, \(T_{E,F}\) are the transmission efficiency of the entrance and focal-plane filters, respectively, \(D\) is the contaminant transmittance of the optics and \(Q\) is the quantum efficiency of the CCD.

We can compute the wavelength response for the 211 Å channel using the following method.

[48]:

correction_table = get_correction_table("JSOC")
r = c.wavelength_response(correction_table=correction_table)

Each of the individual components of the effective area are also available as properties on the Channel object.

[49]:

print(c.geometrical_collecting_area)
print(c.primary_reflectance)
print(c.focal_plane_filter_efficiency)
print(c.contamination)
print(c.quantum_efficiency)
print(c.gain)

83.0 cm2
[1.0735452e-05 1.0784249e-05 1.0833047e-05 ... 2.8031474e-01 2.8032130e-01
 2.8032786e-01]
[5.95474e-01 5.93567e-01 5.91666e-01 ... 3.08021e-05 3.06914e-05
 3.05812e-05]
[0.96578515 0.96544755 0.9651096  ... 0.4387496  0.4387496  0.4387496 ]
[0.8513821  0.8509894  0.8506473  ... 0.09263941 0.09265867 0.09267791]
[7.42477    7.39518913 7.36584303 ... 0.20628946 0.20626652 0.20624361] DN / ph

The .wavelength_response method also includes several keyword arguments for including the effects of degradation and EVE cross-calibration in the computation of the wavelength response function.

[ ]:

r_time = c.wavelength_response(obstime=m_211.date, correction_table=correction_table)
r_time_eve = c.wavelength_response(obstime=m_211.date, include_eve_correction=True, correction_table=correction_table)

[61]:

plt.plot(c.wavelength, r, label="Uncorrected")
plt.plot(c.wavelength, r_time, label="Degradation")
plt.plot(c.wavelength, r_time_eve, label="Degradation + EVE")
plt.xlim([180, 240])
plt.ylim([0, 1.75])
plt.xlabel(r"$\lambda$ [Å]")
plt.ylabel(rf"$R(\lambda)$ [{r.unit.to_string(format='latex_inline')}]")
plt.legend(loc=2, frameon=False)

[61]:

<matplotlib.legend.Legend at 0x70ad6e166c10>

Note that these wavelength responses, when combined with atomic data (e.g. from CHIANTI), can be used to compute the temperature response functions of each AIA channel using Section 2.5 of Boerner et al. (2012). More information about this subpackage can be found in the Computing wavelength response functions example in the aiapy example gallery.

Resources

In addition to this blog post, there are also a number of resources for learning more about aiapy:

aiapy is under active development and, just as with the core sunpy package and all other affilitated packages, contributions from the community are always welcome. To report a bug, request a feature, or simply ask a question, feel free to open an issue on GitLab.

SunPy Awarded NASA Grant

2021-11-03T00:00:00+00:00

As part of the NASA “Open Source Tools, Frameworks, and Libraries” program, the SunPy proposal “Strengthening the Foundations of the SunPy Ecosystem” has been funded for the next three years.

This proposal was submitted in January 2021, and sought funding for both the core sunpy package and wider goals of the project. After early community input was collected, Albert Shih led the proposal itself, with NASA GSFC as the primary institution. The proposed effort focuses on three main areas:

Improving the technical infrastructure.

This focuses on the tools that the wider project needs to scale up as we adopt more coordinated and affiliated packages. To support multiple active projects with a common development workflow, more tooling and infrastructure is needed, both for maintainers of existing packages and developers new to writing affiliated packages. As part of this work templates for setting up new packages will be improved, documentation will be written, and new automation and monitoring bots will be developed to ease the burden on the SunPy maintainers.
Augmenting science-enabling functionality.

This section aims to improve the tools provided by the core package to better support future challenges in solar physics data. The two main components of this are supporting multi-point observations with improvements to the sunpy.coordinates package, and improving the support for very large datasets in sunpy and sponsored packages by leveraging the scaling and parallelism tools available from Dask.
Providing training and outreach

This section will improve example galleries across both sunpy core and affiliated packages, develop training materials for solar physics graduate students, and perform outreach activities with instrument teams and package developers.

What will the money pay for?

The money awarded to this proposal will pay for:

Albert Shih will manage the project and augment parts of sunpy.coordinates.
Stuart Mumford will work on infrastructure for all sponsored and affiliated packages as discussed in sunpy/sunpy-project#2.
Will Barnes will work on the training/outreach and large-dataset-support components of the project.

Finally

This is a major milestone for SunPy - it is the first significant amount of funding dedicated to directly working on the long term sustainability of the project. This funding will help to significantly grow SunPy, both the core package and the ecosystem of associated packages.

October Release Announcements

2021-10-29T00:00:00+00:00

Today we have a quadruple release announcement for you! It’s been a very busy week in SunPy land getting both the core and ndcube releases over the line; I hope you enjoy them!

sunpy 3.1

The next scheduled release of the sunpy core package actually comes a week early according to our release calendar, but we couldn’t resist bringing you this end of month release party.

The 3.1 release contains many new features and improvements, having merged 145 pull requests from 16 people including 8 new contributors.

Some of the highlights from the What’s New in SunPy 3.1? page are:

Increased in-situ data support
A new limb drawing function
A new WISPR map source
Propagating solar-surface coordinates in time
Convenient reprojection of maps
JSOC keyword filtering with Fido
Arithmetic operations with maps

My personal favourites being the new limb drawing function and the reproject_to method on all maps.

ndcube 2.0

Even more excitingly, for me personally, is the long awaited 2.0 release of ndcube.

ndcube is the sunpy sponsored package for working with multi-dimensional data with an associated WCS object. For an introduction to ndcube and motivation for the 2.0 rework see An introduction to ndcube.

We started working on ndcube 2.0 back in 2019 when we had Yash Sharma do the initial work of porting the code base to use the APE 14 APIs as his Google Summer of Code project. Little did any of us think that it would take over two years from that point to hit the final release.

The road to ndcube 2.0 has been long and arduous; all involved have put in a lot of work and what we have made is something that I am personally very proud of. ndcube 2.0 would not have happened without Dan Ryan, who as the co-maintainer for ndcube has spent many hours on video calls with me designing APIs, and many more hours besides working out problems you would only encounter in one in a million WCS objects! I would also be remiss if I didn’t specifically thank the DKIST data center who have, by paying for my time, funded a lot of the development work on ndcube 2.0, even when the scope increased beyond their direct requirements.

I hope that many packages will start using ndcube 2.0 as the base for their code. There are already many who have helped to shape the release such as the DKIST User Tools, specutils and sunraster as well as others I know are in the works.

For all the gory details of the changes in ndcube 2.0 see the changelog.

pfsspy 1.0

The SunPy-affiliated potential field source surface extrapolation package pfsspy maintained by David Stansby has had a 1.0 release, which includes promises of API stability. You can read about all the changes in the changelog.

aiapy 0.6

The SunPy-affiliated aiapy package, which provides tools for working with SDO/AIA data and is maintained by the AIA instrument team, has a new 0.6 release which includes support for sunpy core 3.1. In addition to a number of bug fixes, v0.6 also includes the ability to calculate uncertainties on AIA images via the estimate_error function. A complete list of changes is available in the changelog.

The What, Why, and How of SunPy Affiliated Packages

2020-08-18T00:00:00+00:00

What are SunPy affiliated packages?

SunPy affiliated packages are well-maintained, open-source software packages that build upon or extend the functionality of the SunPy core library. The main idea behind affiliated packages is to provide additional tools and functionality to the solar physics community that is considered outside the scope of the SunPy core library, but builds upon it. Affiliated packages are aimed to integrate well with the SunPy ecosystem and together with the core library offer a diverse software ecosystem for which to perform solar data analysis. Affiliated packages are reviewed and registered with the SunPy project.

There are two types of affiliated packages - sponsored and non-sponsored. A sponsored affiliated package is an affiliated package that is maintained and overseen by the SunPy project, examples of these include sunpy core and ndcube. Non-sponsored affiliated packages on the other hand remain under the control and maintenance of the original developers, but must meet a set of standards. Examples of non-sponsored affiliated packages would be instrument-specific affiliated packages such as AIApy.

In this blog post we want to outline our new streamline process for becoming an affiliated package and we want to encourage package developers, particularly instrument teams interested in developing a Python package, to submit their package for review for SunPy affiliate status!

Why would a package want to become an affiliated package?

SunPy supports the development of affiliated packages through our community development efforts and by providing our package template. The SunPy project will also ensure that affiliated packages are publicized to encourage community development. For example, affiliated packages are listed on the SunPy website, and SunPy adverstises them at conferences and workshops.

Call to instrument/mission teams!

We invite instrument and mission teams that are interested in developing a Python package to become a SunPy affiliated package. Integrating mission/instrument-specific software within the SunPy ecosystem will be of great benefit to the solar community and will allow for a simplified and unified process for users to perform solar data analysis. For example, a package may consist of instrument-specific calibration routines which would be outside the scope of SunPy core package, but could build upon the functionality provided by core such as the data containers and file readers. Developing a mission/instrument specific package as a SunPy affiliated package will allow the code to live and integrate within the SunPy ecosystem.

Furthermore, for mission/instrument teams that would like to begin developing specific Python software, the SunPy project offers support through our package template and community resources for the package to be ‘incubated’ until it is at the level of affiliate status. We would really like to encourage instrument/mission teams to reach out!

Package template

The SunPy project is currently developing a package template that is designed to help developers create new Python packages within the SunPy ecosystem. The template is still in development, and we are working towards creating more documentation to make it accessible for developers to use. The idea is that this template will provide a simplified way to standardized packaging, testing, and documentation using the same framework as the SunPy core package. The SunPy package template will make it easier for package developers to meet the standards required by SunPy to become an affiliated package!

How does a package become an affiliated package?

To become a SunPy affiliated package, a set of criteria need to be met to ensure that affiliated packages provide useful functionality to the community at a standard of quality similar to the core SunPy package. We now have a new review process for becoming an affiliated package, aimed at being open, approachable and streamlined!

The reviews are performed as an open process through GitHub issues on the https://github.com/sunpy/sunpy.org repository. To submit a package for review to become an affiliated package, an issue should be opened on this repository. We now have an issue template for submitting a package for review to make it easier to provide all the required information and to start a dialogue about the process.

Once an issue is open, the Affiliate Package Liaison will check that the package meets the baseline requirements of an affiliated package. These include checking to see that it is compatible with the SunPy Code of Conduct, has an appropriate licence, provides a Python interface, is on PyPI, and is useful to the solar physics community. Following this, a reviewer independent to the package is assigned and a review is undertaken based on the criteria listed below.

The review criteria that are assessed are listed as follows with the associated gradings:

Functionality Does the package provide functionality that is useful and relevant to the solar physics community?
Integration - Does that package integrate well within the SunPy ecosystem? for example does it make use of all appropriate features in the core library, including the applicable data structures and dependencies?
Documentation - Does the package have satisfactory documentation that is up to the standard of the core sunpy library including a user guide and examples?
Testing - Does the package use a testing suite to verify the code functionality within its codebase? Does it provide integration tests?
Duplication - Does the package duplicate functionality within sunpy core or any of the other affiliated packages or other relevant packages?
Community - Is the package openly developed with active input and engagement from users within the community? Does the package have a code of conduct and does this reflect the same values as the SunPy project?
Development Status - Is the package well maintained? Are contribution responded to by developers? Is the package still under development with large API changes?

These criteria are reviewed on a ‘traffic light’ system and ranked ‘green’, ‘orange’, or ‘red’ based on the submitted package. For full details of how each of criteria are ranked please check out the Review Criteria.

For a submitted package to be accepted to affiliate package status, the package must be ‘green’ in Functionality, and one other category. It must also not list any ‘red’ scores.

If the package in its current state does not pass the criteria, it can be listed as Provisional once it does not list ‘red’ in the Functionality, Duplication or Community criteria. The idea is that the package can be listed as Provisional as it is working towards addressing the criteria for which it did not pass the review in.

We would really like to encourage packages interested in becoming an affiliated SunPy package to please submit an issue and open up a review dialogue. We will have an Affiliated Package Liaison that will help you through each step of this process :). Even if you are unsure about whether you want to submit a package, please feel free to open an issue and informally discuss your package.

Reach out!

If you are a developer of a package that you think fits nicely into the SunPy ecosystem and will of benefit to the solar physics community and want to chat to us about it please reach out! This can be of course regardless of how far along the package is - from concept to maturity! Join us our live chat Matrix room or join in on the SunPy weekly community meetings which occur each Wednesday (see our calendar for the time in your timezone) and are hosted on jitsi.

SunPy awarded a NASA HDEE Grant

2020-06-05T00:00:00+00:00

A NASA Heliophysics Data Environment Enhancement (HDEE) grant has been awarded to a proposal submitted to support SunPy development. The grant was awarded through NASA’s 2019 ROSES grant solicitation.

The grant supports the following development goals:

Creation of a report on the state of the SunPy codebase by analyzing output from code coverage and API inspection tools, etc, identifying areas in the existing codebase that need more coverage, can be consolidated or removed.
Provide the ability to read spectroscopic data into a spectral data object, thereby enabling its later scientific analysis.
Implement a number of heliophysical coordinate systems using the existing SunPy and Astropy-based coordinate system framework.
Create example code snippets that use SunPy and packages from the Python in Heliophysics Community; these examples will be shared via the Python in Heliophysics Community.

The PI of the grant is Jack Ireland, and the co-I of the grant is Andy Terrel (NumFOCUS). The grant has a one year duration. The bulk of the code development will be undertaken by a developer hired through NuMFOCUS. We look forward to supporting the development of SunPy through this grant.

Jack

SunPy Survey

2020-04-27T00:00:00+00:00

The SunPy project is happy to announce the results of the solar physics community survey!

For six months last year, between February and July 2019, the SunPy Project asked the solar physics community to fill out a 13-question survey about software and hardware. Some of the questions were: Do you use software in your research? Have you had formal training in programming? Have you cited software papers in your published research?

We deliberately asked some of the same questions as Momcheva and Tollerud (2015), who informally surveyed 1142 members of the astrophysics community, to compare software preferences between the two communities. A total of 364 community members, across 35 countries, took our survey.

What we found

We found that 99±0.5% of people who responded to our survey use software in their research. Nearly everyone in the solar physics community relies on scientific software. A large fraction (66%) of respondents use Python. Most respondents use both Interactive Data Language (IDL) and Python.

However, students are twice as likely as faculty, staff scientists, and researchers to use Python. This breakdown is pretty different from the astrophysics community, where Momcheva and Tollerud (2015) found that Python is the most popular programming language within every individual career category.

We also found that most respondents (63±4%) haven’t taken any computer science courses at an undergraduate or graduate level. Momcheva and Tollerud (2015) also found that most astrophysicists also lack formal training in software development.

Most of the people who responded to our survey work with observational data taken by ground-based or space-based instruments (82%). And about half (47%) work with numerical simulations. Some work in both sub-disciplines. Although some instruments and numerical models in solar physics generate sizeable data volumes, only a small fraction of respondents use a regional cluster, national cluster, or the commercial cloud for their research.

Finally, we found that 42±3% of respondents regularly cite scientific software. Half the respondents who don’t cite software said they didn’t know how.

Our paper

Please check out our paper! We detailed all of these results, and more, in a paper published on April 20, 2020 (DOI: 10.1007/s11207-020-01622-2). We also included an appendix on how to cite scientific software. You can also take a look at all the raw survey responses (anonymized and randomized), along with the code to analyze these data, at github.com/sunpy/survey.

SunPy Release Schedule and Version Numbering

2019-06-27T00:00:00+00:00

In this blog post I want to outline how, and more over why, we are going to number the SunPy releases and when we are going to release them.

TLDR; we are going to move to a 6 month release cadence, with a Long Term Support (one year of support) release in May and a Short Term Support (6 months of support) in November. We will increment the major version number for each LTS release.

Background

As part of the 1.0 release planning the SunPy developers and the board spent a long time discussing how we want to version SunPy and what we want to use our versioning a release schedule to achieve. We went through two major iterations of this plan, the first can be found in the abandoned PR to the SunPy Enhancement Proposal repository #30, and the second one which was accepted in #40.

Objectives

In moving SunPy out of the 0.y series of releases we knew that our users would be expecting us to make breaking changes less, and for them to be able to rely on SunPy for longer.

We wanted to balance the desire for people to be able to start using SunPy for a project and be able to see that project through without having to constantly adjust to many changes in new releases. On the other hand, we wanted to also allow the development of SunPy to continue without having to delay people contributing improvements to SunPy because of a strict versioning scheme.

We also have to consider that, currently, SunPy core is developed entirely by volunteers, and supporting old versions of the library is a time consuming and sometimes thankless job.

Releases

The solution we landed on is to provide a Long Term Support release that we will support for one year (i.e. until the following LTS release). This gives people who want stability a release schedule where they get bug fixes for a year. While also allowing SunPy to do releases on a six month cycle to get new features out for people to use.

To make it clear which releases are Long Term Support releases, we will increment the major version (the first number) for each LTS release. This means SunPy 1.0 is a LTS release which we will be supporting until the release of 2.0 in May 2020.

The all the non-LTS releases between the LTS releases will increment the minor version number, as there is only one release planned between LTS releases the normal release number will be 1.0, 1.1, 2.0, 2.1 etc.

Deprecations

We still expect, and have plans for, changes to many parts of SunPy. We believe these changes will enable even more people to find SunPy useful in their work.

To enable people to prepare for changes and to not be in for big surprises when updating SunPy we will endeavour to emit a deprecation warning for any breaking changes for one LTS release.

The upshot of this is that, if now after 1.0, we want to change something in a backwards incompatible way we would continue to provide both the old and new ways until after the 2.0 release. The 2.0 release would emit a deprecation warning, to give people who go from LTS to LTS a year to update their code before the next LTS.

Conclusion

I hope this blog post gives you some insight into how we made this decision and why, if you have any comments please don’t hesitate to get in touch using any of the usual ways.

Updates to the website

2017-04-07T00:00:00+00:00

The website has gone some minor updates to incorporate the latest version of Bootstrap and its build dependencies among other items. This work came from Prateek Chanda, Duygu KEŞKEK and Sourav Kumar