Blog - Posts by Kris Stern

GSoC 2019: Project IRISpy 3.2

2019-08-18T00:00:00+00:00

It is my pleasure to report that with my mentors’ assistance as well as my many hours of contributing to the sunpy/irispy GitHub repo, I have been able to reach all four goals original set out by the primary mentor Danny Ryan, which are as follows:

The time-dependent instrument function response code must be rewritten to be more efficient and Python-friendly.
Formal benchmarking between the results it produces and those found using the original IDL code must be performed.
Tests for the Python version must be written.
This software must be incorporated into methods and functions in IRISpy that depend on the instrument response function.

Accordingly, the three outcomes have been accomplished as below:

A function for deriving the time-dependent IRIS response function.
Benchmarking and unit tests so this new software can be reliably maintained.
Updated intensity conversion methods between instrument and physical units that correct for the time observations were taken.

For a product of this effort, please go here , which is to merge the four PR’s already merged into the time_dependent_response branch into the master branch in order to incorporate our work into IRISpy officially.

I would particularly like to take the time to thank every one in the SunPy community for their support of me over the past few months, in particular my project mentors Danny Ryan and Laura Hayes. They have been very helpful in answering my questions and in helping me solve some very thorny IDL questions. The time I spent on GSoC this year is the moments I will not forget!

GSoC 2019: Project IRISpy 3.1

2019-08-04T00:00:00+00:00

Grateful to report that I have eventually solved the issues encountered as stated in blog post 2.2 in the series for my GSoC project… and most important of all with my mentors’ blessings I have successfully passed the 2nd coding phase. Moreover, I am delighted to report that my most significant contribution on GitHub thus far, which is the PR to enable time-dependent effective areas determination of IRISpy, has been merged about 6 days ago. More can be found about this PR at the link here.

Currently I am continuing on my efforts to complete the project by updating the intensity methods between instrument and physical units, in another PR. My goal is after finishing up this PR then move on to completing the testing by tidying up loose-ends left in PR #108 and add to it the required tests. The challenges remain is to determine the test cases used and the best way to keep the tests neat. I have already begun work on the testing front as well, though a former PR has not been opened for it yet.

All in all everything is looking good. I am looking forward to continue working under the supervision of my project mentors further before GSoC 2019 comes to a close in about 2 to 3 weeks’ time. Their expertise and patience have been amazing, and I am grateful for their mentorship.

GSoC 2019: Project IRISpy 2.2

2019-07-21T00:00:00+00:00

To follow up, the problem encountered as reported in edition 2.1 in this series has been resolved with the help of my very helpful and responsive mentors. But some lingering issues remain, which we suspect to be mostly indexing-related, as is apparent from the way the given IDL code is translated into Python. So now all four versions of IRIS response can be used to produce some get_iris_response function output. However, say for example version=4, I have been only able to successfully reproduce 3/4’s of the expected output as generated by the IDL version of the code. This is encouraging, as previously only half was the same as the target output. So the part of the IDL code that I am having some problems with is the following:

; 4. SJI effective areas
if fix(r.version) le 3 then begin
 sz = size(r.coeffs_sji)
 for j=0,sz[3]-1 do begin
  ; calculate pre-launch area from the individual elements
  pl_a = r.geom_area
  for k=0,n_elements(r.index_el_sji[*,j])-1 do $
      pl_a=pl_a*r.elements[r.index_el_sji[k,j]].trans
  ; time dependent response
  rr = fit_iris_xput(tt,r.c_s_time[*,*,j],r.coeffs_sji[*,*,j])
  ; time dependent profiles
  for k=0L,ntt-1 do o[k].area_sji[*,j]=pl_a*rr[k]
 endfor
endif else begin

 for nuv=0,1 do begin
   ; calculate baseline SJI area curves
   asji = r.geom_area
   for k=0,n_elements(r.index_el_sji[*,nuv*2])-1 do $
     asji=asji*r.elements[reform(r.index_el_sji[k,nuv*2:nuv*2+1])].trans

   ; apply time dependent profile shape adjustment to FUV SJI
   if ~nuv then begin
     ; FUV: apply FUV SG "slant", then normalize so that a weighted (2.4:1)
     ;      sum at C II and Si IV gives constant response
     wei = [2.4,1.0]   ; typical solar ratio CII : SiIV
     wav = r.c_f_lambda
     nwv = n_elements(wav)
     wav = [wav[0],(wav[nwv-2]*2.0+wav[nwv-1])/3.0]       ; 2 wvlngts in nm
     ; calculate baseline SG area for scaling purposes
     asg = r.geom_area
     for k=0,n_elements(r.index_el_sg[*,nuv])-1 do $
       asg=asg*r.elements[r.index_el_sg[k,nuv]].trans
     ; SG and SJI areas at wav
     asg2 = interpol(asg,r.lambda,wav)
     asj2 = fltarr(2,2)
     for j=0,1 do asj2[*,j]=interpol(asji[*,j],r.lambda,wav)
     ; calculate the normalized slant function scal, apply to asji
     for k=0L,ntt-1 do begin
       ; best-estimate slant, i.e., eff.area @ wav / baseline SG @ wav
       sca2 = interpol(o[k].area_sg[*,0],o[k].lambda,wav) / asg2
       ; normalize slant so that total(wei*asj2*sca2)/total(wei*asj2)=1
       for j=0,1 do begin
         sca2n = sca2 * total(wei*asj2[*,j])/total(wei*asj2[*,j]*sca2)
         scaln = interpol(sca2n,wav,r.lambda) > 0.0
         o[k].area_sji[*,j] = asji[*,j]*scaln
       endfor
     endfor
   endif else begin
     ; NUV: essentially same calculation as r.version=3
     for k=0L,ntt-1 do o[k].area_sji[*,2:3]=asji
   endelse
 endfor
 for j=0,3 do begin
 ; SJI specific time dependency
   rr = fit_iris_xput(tt,r.c_s_time[*,*,j],r.coeffs_sji[*,*,j])
   for k=0L,ntt-1 do o[k].area_sji[*,j]=o[k].area_sji[*,j]*rr[k]
 endfor
endelse

if keyword_set(angstrom) then o.lambda=o.lambda*10.
return, o

This is the last but most complicated part of the get_iris_response code to compute the SJI effective areas for both the FUV’s and NUV’s. Something of interest to note is that the indexing in Python is the exact reverse of that in IDL. But it will be a lot fun trying to make the current code I have been working on to agree with the above 100%. I have been thinking maybe there is something regarding the IDL interpol function (for interpolation) that I have been translating wrong, as there is no exact equivalents in Python, and the closest thing to it is some two-part scipy.interpolate methods which I can use for the same task. Hopefully I will be able to overcome this almost last hurdle soon, say within a week or two. That would leave me with enough time to complete the project comfortably before the conclusion of GSoC 2019.

GSoC 2019: Project IRISpy 2.1

2019-07-08T00:00:00+00:00

Passed Phase 1 of GSoC. And I have just encountered my first roadblock in this project over the past few days, so now I am working hard on various approach to try and understand some IDL code snippets that does not translate well into Python without some more knowledge of the IDL INTERPOL function. (Source: harrisgeospatial) In IDL, this function takes in a maximum of four arguments: an input vector, the number of points in the result, the abscissa values for the input vector, and abscissa values for the result. In this way, the iris fit can be interpolated onto a new lambda grid readily. However, an exact equivalent does not exist in Python. The closest one we have opted to adopt is SciPy’s interpolate.interp1d function (Source: scipy). This method takes the x-coordinates and y-coordinates of the input data, and interpolate some pattern from this, which means that to project to a grid an extra step is necessary. I have now modified the relevant code snippets of in my PR so that I have a functional line. But to project it to some very specific grid distinct from the input one will take extra checking, since I am iffy about my approach. Currently seeking input from my mentors to sort out the issue. Hopefully this roadblock will be cleared soon.

GSoC 2019: Project IRISpy 1.2

2019-06-23T00:00:00+00:00

Four weeks have passed since the commencement of the Coding Period on May 27th, 2019 (EST). Soon our 1st evaluation will come up, which in itself is an exciting time as we will be judged on whether we will be qualified to go on to the next round. In my opinion I have done most of what have been agreed between me and my mentors. Comparing to my peers my project probably is more manageable, as an IDL version of the code already exists which provides us with a source code from which a Python-translation is written. And, the level of the project is relatively easy, so that the concepts involved are pretty basic. In this blog post, I would like to discuss about a trinity of techniques I have developed as a rookie software developer to deal with issues arising during the implementation of some new function into the broader code base. For the most up-to-date status of the PR please head to link.

Technique 1: Understanding how the new function fits into the code base — First off is to study the code base carefully in order get an idea regarding its architecture. Knowing how the chain of dependencies go can help a lot with implementing a new function. After this, review the code again to gain more insight once this first step is completed.

Technique 2: Check the chain of functions involved in the hierarchy of dependency — This is best done with some keyword search functionality of the editor-of-choice. It pays to be thorough in one’s backwards and forwards double-checking to ensure every dependency in the chain is covered.

Technique 3: Divide-and-Conquer — Dividing the task into smaller steps, in order to help both with writing the new code snippets and the testing. The first task in the process of course would be to actually write the new function to be incorporated. Then, comes the tasks of debugging, testing, and benchmarking. And then, it is to enhance the code using some algorithmic or data-structure hacks to improve its performance. Finally, we worked back up the chain of dependency to add the new elements required, in the case of my PR that would be to implement some time-dependency in the determination of some effective areas covered by the NASA IRIS satellite.

It has been a very blessed journey working with my two very knowledgeable, encouraging and caring mentors thus far. And I look forward to further collaboration with them on this and potentially other projects.

GSoC 2019: Project IRISpy 1.1

2019-06-08T00:00:00+00:00

It has been two weeks since the commencement of the coding phase… I can’t believe it, as it seems like time files. But to sum up the time that passed the following is a brief summary report for what tasks have been completed and what remains:

The GitHub PR opened for the project can be found at link here. This picks up from a previous unfinished PR #102 with link here, while also addresses Issue #27 with link. The ultimate goal of the project is to work on IRISpy further until it is in a state where it becomes ready for an official release. Basically, we would like to add a time-dependent iris_response() function as well as a function fit_iris_xput() to compute the time-dependent effective areas based on some input observation times and coefficients. So far, four out of the five planned stages of implementation has been carried out, and they are as follows:

Triaging
Benchmarking
Bug-fixing
Improving Code Pythonically
Incorporating Time-dependent Fitted Response into Wider IRISpy Code-base

Essentially, the first four of the five steps form the workflow for all similar patterns of package development. The details of the difficulty encountered at each stage are to be outlined in the subsections to follow. §1.1 Triaging

Most of the groundwork has already been laid in PR#102. Therefore the remaining things to do include cleaning up the code, debugging, and rewrite where necessary in order to get both the iris_response() and fit_iris_xput() functions to work without the code throwing errors during testing. This turned out to be relatively smooth-sailing (could be partly due to luck).

§1.2 Benchmarking

Some use cases have been supplied by Dr. Dan Ryan (the primary mentor) which are originated from some IDL software the student has no access to. More details of such data will be given in the appendix. A test function has been written specifically to test fit_iris_xput() to allow us to test and understand the output. This incorporates the six use cases mentioned previously from IDL. Efforts have been coordinated to compare results from the IDL and IRISpy versions to verify they are giving the same answers. The tolerance has been set at 6 decimal places. It has been found that preliminary results from the crude Python code in iris_tools.py before code enhancement to be carried out at the next stage match the IDL ones quite well, which was very encouraging. Most of the time the fall-back tactics have been divide-and-conquer and trial-and-error.

§1.3 Bug-fixing

The aim has been to render the first functional Python version of the code to give the same or effectively the same answers as the IDL version, except perhaps rounding errors. Most common type of bugs encountered arose from differences in syntax, especially from converting between programming languages from IDL to Python. One of the more glaring and easily omitted error encountered stemmed from erroneous indentation. The final bug to be fixed had to do with units used in the variable for time difference t_diff. This has to be presented in years. However, due to the way the packages were designed before and upgrade, some confusion cropped up as it has to be converted from a datetime.timedelta object in days to seconds first before it can be converted into the proper unit. After software upgrade, particularly sunpy, it became easier to perform the same task as the object is now an astropy.time.time one, so that by using astropy’s powerful units and quantities machinery, the conversion from days to years was direct and efficient.

§1.4 Improving Code Pythonically

By this stage the code has been made more Pythonic as well as updated to reflect more recent changes in sunpy practices. Using astropy’s download_file function instead of Requests’ get to download the IRIS .geny files. This adds the cache and timeout functionalities to the download process. Converting the time_obs and time_cal_coeffs into astropy.time.time objects before subtracting to obtain a time difference called t_diff (to be converted into years), which is used to compute the final fit for the effective area(s).

§1.5 Plan for next stage: Incorporating Time-dependent Fitted Response into Wider IRISpy Code-base

To include a call to fit_iris_xput() plus to add a time input for the _get_interpolated_effective_area() function. Add a docstring to the _get_interpolated_effective_area function. To include a time input and pass that on to _get_interpolated_effective_area() further up the chain. This chain includes iris_tools.calculate_photons_per_sec_to_radiance_factor(), iris_tools.convert_or_undo_photons_per_sec_to_radiance(), andIRISSpectrogramCube.convert_to(). Will confirm if there are more cases where this chain leads. To include cases where a fitted effective area is used in the tests of above functions.

Therefore, so far the project is on schedule… hopefully things will stay that way.

GSoC 2019: Project IRISpy 0.2

2019-05-26T00:00:00+00:00

Progress has been better than expected… Within the past few weeks, while continuing work on the main PR (which can be accessed via this link) on enabling a time-dependent effective areas determination in the “iris_tools” file, which contains some IRIS instrument tools, most of the previous difficulties of translating from the IDL programming language to the Python language have been identified and ironed out. In hindsight, for me the hardest part was to comprehend a computing language for which I have no direct access to (as I do not have in possession a valid IDL licence), but at the same time be able to gain enough insight through freely available online documentation to check the rough work put into place earlier in another PR. Through careful and systematic checking, I have been able to move ahead and implement the new function fit_iris_xput() as well as the get_iris_response() function in a pythonic way with some heavy dose of guidance from my mentors Dr. Dan Ryan and Dr. Laura Hayes well ahead of schedule.

For example, instead of converting from seconds to years by brute force with the conversion factor:

yr2sec = np.float64(365.25*24.*3600.)
t_diff = t_obs - t_cal_coeffs
t_diff = t_diff/yr2sec

we opted to use Astropy’s powerful Units and Quantities “machinery” for the conversion instead, in the following manner:

t_diff = t_obs - t_cal_coeffs
t_diff = t_diff.flatten()
t_diff = [x.to(u.year) for x in t_diff]

It has been necessary to flatten the numpy array into an 1D object to allow the list comprehension to be applied.

So the plan ahead would be to incorporate the newly implemented time-dependent fitted response into the wider IRISpy code base.

GSoC 2019: Project IRISpy 0.1

2019-05-15T00:00:00+00:00

The community bonding period of this year’s GSoC is officially under way. The project I am embarking on has to do with adding a new feature to IRISpy, which is a essentially a package that is developed on top of SunPy’s NDCube package, and is written in the popular Python programming language. IRISpy itself (albeit not yet released) provides functionalities to read, manipulate, and visualize data collected with NASA’s IRIS satellite which looks at UV emission from the solar chromosphere. The proposed new feature is a time-dependent instrument response function, which will give scientists far greater power and ability to perform IRIS data analysis in Python, as well as to make new discoveries regarding the energetics and dynamics of the solar chromosphere and transition region than previously allowed. For more information on both the IRIS instrument and IRISpy, please take a look at the SunPy documentations for IRISpy.

As the IRISpy repo is hosted on GitHub, most of the “action” will take place there. (Such that the version control will be performed using Git.) A Pull Request has already been opened in that repo where some preliminary work on the project has already been carried out. And, the required development environment has been set up on my personal computer to develop the new code so that it can be tested locally. For this project, I have opted to use Anaconda as the package management system to create an virtual environment, providing easy access and management to all dependencies via the command line.

The coding part of the IRISpy project is 3-month long, and will kick off on May 28th, 2019 (Hong Kong Standard Time). In the meantime I plan to read up on the background regarding both IRIS and IRISpy, and to study the existing IDL code of the same feature some more before attacking the problem of converting this code from IDL to Python, then benchmarking, and bugfixing, and finally improving the flow and efficiency of the produced Python code. After that, proper documentation will be added to guide the user through the steps of using the new feature. Such documentation will include but not limited to the feature’s docstrings, additions to official SunPy website documentation, and perhaps a brand new Learn Astropy tutorial. After the completion of the project, it is expected that the following tasks will have been accomplished:

An improved function for deriving the time-dependent IRIS response function written entirely in Python.
Reliably maintaining this new software in terms of benchmarking and unit tests.
The new software must be incorporated into the existing methods and functions in IRISpy that depend on the instrument response function.

The IRISpy project is on a topic that is not directly related to my current PhD research (which is mostly on late-stage stellar evolution focusing on planetary nebulae), but will supplement it by expanding my general knowledge of solar physics. And, I expect this undertaking to be a challenge I can grow with, especially by helping me developing good time management skills.

I am looking forward to a great Google Summer of Code ahead.