Blog - Posted in 2013

ESA Summer of Code

2013-09-02T00:00:00+00:00

I am happy to announce that for the third year running SunPy has got a summer student from ESA’s SOCIS program. This year the student is Tomas Meszaros. Tomas’ project this summer will be working on creating a new core data type for SunPy, the HyperMap. Which will be designed to hold ND data with at least one spatial dimension and any combination of other axes such as Wavelength, Temperature or Time. This data type will allow us to support data from instruments such as Hinode EIS and the newly launched IRIS satellite, as well as a multitude of high resolution ground based instruments. Tomas will be documenting his progress on his blog: https://examon.wordpress.com/ as well as less regular more general interest posts on this site.

Stay tuned.

Announcing SunPy 0.3

2013-08-30T00:00:00+00:00

It gives me great pleasure to announce the release of a new version of SunPy. This version has been rather too long in the making, but is here at last!

SunPy 0.3 is now available through PyPI and via GitHub. The biggest change in 0.3 is a shift away from our Map and Spectra datatypes inheriting numpy.ndarray to having their array in a data attribute. This was done to make development of these objects easier and more flexible and also to improve our compatibility with Astropy. In the process of doing this the map submodule has undergone a massive refactor to streamline the creation and inheritance structure of the module.

Below I highlight some of the major changes, the full change log can be found here.

The biggest change to the Map API is the deprecation of the make_map function. It has been replaced by the new sunpy.Map factory which is much more intelligent and able to have custom map sources external to the SunPy library register with it, which is handy if you are developing a custom Map source. Along with this change the old top-level map class Map is now called GenericMap and is to be created using the Map factory under normal circumstances.
MapCube and CompositeMap have also seen some improvements, including the implementation of draw_limb and draw_grid for both datatypes and a new animation based plot() method for MapCube.
To facilitate the changes to map there have also been a lot of improvements to the io submodule, including the ability to read all the HDUs from a FITS file and the addition of a extract_waveunit function that checks the header for common ways to encode the wavelength unit.
There has also been a big cleanup of the various top level and submodule namespaces to make imports simpler. Most of this has not changed the user facing API, however there are not nicer ways to import submodules, like sunpy.util.util is now the same as sunpy.util.
The spectra module has been refactored so it also has a data attribute and it’s plotting API is now consistent with that of Map and LightCurve

SunPy 0.3 consists of 9 months of work from 15 people and over 300 commits to the git repository. The people who have contributed to this release are (in commit order):

Stuart Mumford
Russell Hewett
Florian Mayer
Steven Christe
Albert Shih
Simon Liedtke
Ankit Angrawal
Jack Ireland
Matt Bates
Nabil Freij
Keith Hughitt
David Perez-Suarez
Tomas Meszaros
Benjamin Mampaey
Andrew Leonard

On behalf of all the SunPy developers, we hope you enjoy 0.3!

Experiments in animating plots and saving movies in SunPy 0.3

2013-08-29T00:00:00+00:00

Just over a year ago this post described a simple method for saving a movie of SunPy maps. Since then, SunPy and matplotlib have moved on, and I’d like to describe an updated method for animating SunPy maps, and saving the results as an mp4 file.

First off, all these experiments were conducted on Ubuntu 12.04. The code below is based on this StackOverflow question and answer (where would we be without StackOverflow???? - thanks!), and some googling around concerning Ubuntu and ffmpeg.

So, to begin, I fired up ipython. I tried the StackOverflow code (including the correction in the answer) and got stuck in an ipython error loop. The solution - upgrading to ipython 1.0.0. Trying again, the code crashed because I did not have ffmpeg installed. Matplotlib looks for movie writers it can use, and since I had specified one that was not present, it crashed. You can find which movie writers are present with animation.writers.list() (having imported the animation module as below). I put ffmpeg capabilities on my system through sudo apt-get install libav-tools.

So, the StackOverflow code worked! It was quite simple to adapt it plotting SunPy 0.3 maps. I want to read in a bunch of time-ordered AIA FITS files, and make a movie of them. A very typical use case for a solar physicist. Here is what I managed to get to work.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import os
import sunpy

# Set up formatting for the movie files
Writer = animation.writers['ffmpeg']
writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)

# Get some data
imagedir = '/home/ireland/Data/AIA_Data/'
filenames = sorted(os.listdir(imagedir))

fig = plt.figure("An SDO movie")
img = []
for x in filenames:
print("Processing %s" % x)
im = sunpy.Map(os.path.join(imagedir, x))
img.append([im.plot()])

ani = animation.ArtistAnimation(fig, img, interval=20, blit=True,repeat_delay=0)
ani.save('output_movie.mp4', writer=writer)

The code creates a SunPy map, and then creates a list of plots of those maps for the animator to work with. The animator then saves the animation as a movie using the ffmpeg writer. Works quite well. The resulting movie is here. As you can see the title above each image remains the same, whereas in each individual frame the titles differ (according to the observation time). That needs fixing, and it seems it is possible.

Animating a set of plots is also quite easy. The code is almost the same as above. Just comment out the line matplotlib.use('Agg') and replace the line ani.save(…) with plt.show(). A matplotlib window appears, and the animation plays in it. The interactive zoom feature of the matplotlib window also works, and that is pretty cool.

I have not yet worked out how to animate a set of plots, and then save it as an mp4 movie. These simple examples are enough I think, however, to get you going with animating your own plots. And by plots, I mean any kind of plots, not just movies of 2-d image data. The matplotlib site has examples of how to animate line plots, for example.

Obviously it would be extremely useful to have the animation and movie saving capability baked in to the mapcube functionality of SunPy. The code above is just an experiment, so if you have any improvements, or whole new and better methods, please let me know leave a comment below. Thanks again to SunPy, StackOverflow and matplotlib!

SunPy at SciPy 2013

2013-07-21T00:00:00+00:00

This year I was lucky enough to be able to attend the annual Scientific Python conference (SciPy 2013) in Austin, Texas. This was very kindly supported by a sponsorship from the conference organisers and sponsors.

SciPy is a gathering of people from all over the globe who all use Python for some scientific purpose. People there ranged from astronomers to meteorologists. There were academics and people from industry or people who just write python in their spare time. This diversity in people’s backgrounds and interest lead to the most diverse, interesting and friendly conference I have ever attended.

The conference this year consisted of two days of tutorials, two days of talks and two days of sprints. The tutorials covered different levels and topics ranging from introductions to IPython, matplotlib and scikit-learn to advanced talks on Cython and NumPy. Like with all the sessions at SciPy there are very good quality videos available online of the tutorials.

The talks were in three tracks this year, one on reproducible science, one focusing on machine learning and a general track. In addition to this there were also a set of “domain specific mini-symposia”, covering topics such as astronomy and medical applications.

Attending SciPy as a member of the SunPy development team was also a fantastic opportunity to raise awareness of our project and talk to interested people. I gave a presentation on SunPy and it’s development in the “Astronomy and Astrophysics mini-symposium”, the video of the talk is available .

The last two days of the conference were full of very productive sprints, where you sit in a room with like minded people and work on code. During the sprints, and the rest of the conference, I spent some time working on many projects including yt, astropy, ginga and scikit-image.

Ginga is a fantastic FITS file viewer written in pure Python, with surprising rendering speed and extensibility, it has a lot of potential to form the basis of a SunPy GUI. Ginga makes use of Astropy, which is an excellent community effort to unite Python tools for astronomy. One of the main benefits of attending SciPy was to increase the cooperation between Astropy and SunPy, there is a lot of work regarding coordinate handling as well as the migration of the PyFITS library into Astropy, which means that SunPy will be making good use of Astropy in the future.

On the last day I spent the afternoon with the scikit-image team, and had a very productive day working with them to implement a routine that warps an image to compensate for the rotation of the Sun. This is based on various parts of SunPy to calculate the coordinates and the rotation, and then uses scikit-image to warp the solar disk. I have written up this work and it can be found here.

One other project I spent some time with was the yt project. yt is a visualisation and analysis library for astrophysical simulations. My PhD work is numerical solar physics and this was very interesting to me. yt has it’s own very powerful volume rendering engine as well as a very nice API to do powerful slicing and parallel computation and rendering. I would recommended people who might be interested take a look!

The Database class

2013-06-19T00:00:00+00:00

Note: I have also a private blog which covers more advanced topics. The next post there will be about implementing custom caches and custom commands.

The class Database is the central class of the new database package. It is used to connect to a database and to manipulate it (adding new entries, editing existing entries, removing entries). Later on, it will also be possible to search for entries by specific criteria (I think about using the sunpy.net.vso.attrs module for querying the database).

Connecting, adding entries, getting all entries and checking for existence

To connect to a database, you create a new instance of the class Database. Its __init__ method receives only one positional argument: the URL. You may either pass a string of the form dialect+driver://user:password@host/dbname[?key=value..] or an instance of the class sqlalchemy.engine.url.URL. The special string ‘sqlite:///:memory:’ means that an in-memory SQLite database is used (which is especially handy in interactive session for trying things out). See sqlalchemy.create_engine for more information about the syntax of the passed string.

The method create_tables is mandatory for working with a new database. Missing tables are not automatically created because “explicit is better than implicit”.

Each instance of the class DatabaseEntry represents one row in the database. To add a new entry to the database, simply use the add method of a database object and pass it the entry you want to add. To check if it exists in the database, you can use the in operator. The number of entries can be accessed by using the len() function on any database object. It is also supported to iterate over a database object to get all entries. As a side-effect, using the list() function on a database object gives you a list of all entries.

from sunpy.database import Database, DatabaseEntry
>>> database = Database('sqlite:///:memory:')
>>> database.create_tables()
>>> entry1 = DatabaseEntry()
>>> entry2 = DatabaseEntry()
>>> database.add(entry1)
>>> database.add(entry2)
>>> len(database)
2
>>> assert entry1 in database
>>> assert entry2 in database
>>> pprint(list(database))
[<DatabaseEntry(id 1, data provider None, fileid None)>,
 <DatabaseEntry(id 2, data provider None, fileid None)>]

When are changes committed to the database?

There are two answers to this question: Changes may either be committed explicitly or implicitly. The explicit way is easy: any number of database manipulations is flushed by calling the commit method, i.e. database.commit() if the name of the Database instance is called database.

The implicit way is a bit harder to understand: Changes are implicitly committed directly before a command reads from the database.

Each of the following commands is committed as soon as a query to the database is issued:

add
edit
remove
star
unstar
undo / redo

By “query” I mean any function, method or operator that reads from the database. These are in particular:

list(database)
len(database)
database.get_entry_by_id
database.get_starred
the in operator
iterating over a database

You don’t have to remember each of these functions. Just keep in mind that changes to the database are committed as soon as any operation is performed to read from the database. Later on, more methods and features will be added to read from the database, so that generic rule is more useful than learning a list of operations by heart.

Undoing and redoing actions (also: removing entries)

The actions add, edit, remove, star, and unstar can be undone and redone using the methods undo or redo, respectively. Both of these methods receive an optional argument to specify the number of commands that should be undone or redone (the default is 1, i.e. undo or redo only one command). If the given number of commands cannot be reverted (e.g. if only 2 actions have been undone, it is not possible to redo 5 actions), an exception is raised.

The remove method is as simple as the add method: you pass an entry and it gets removed.

>>> database.remove(entry1)
>>> database.remove(entry2)
>>> list(database)
[]
>>> database.undo(2)
>>> pprint(list(database))
[<DatabaseEntry(id 1, data provider None, fileid None)>,
 <DatabaseEntry(id 2, data provider None, fileid None)>]
>>> database.redo()
>>> list(database)
[<DatabaseEntry(id 2, data provider None, fileid None)>]

Editing entries: (un-)starring entries and custom edits

The database package brings the new concept of starring entries. This is just to mark certain entries, it is not a ranking or a custom label (though while I’m writing this, I think that could be a good idea. I should discuss it with the SunPy devs). To star an entry, call the method star and pass the entry to be starred. The method unstar works accordingly. If you try to mark an entry as starred although it already is, an exception is raised. An exception is also raised if it is attempted to unstar an entry that is not starred. This “verbose” behaviour can be turned off by setting the optional keyword argument ignore_already_starred (or ignore_already_unstarred for the unstar method) to True. The entry method makes it possible to change a specific value of an entry. The first argument is the entry to be changed and all following arguments must be keyword arguments where the key represents the column name in the database and the value represents the new value.

>>> database.star(entry2)
>>> database.get_starred()
<generator object <genexpr> at 0xb54d734>
>>> list(database.get_starred())
[<DatabaseEntry(id 2, data provider None, fileid None)>]
>>> database.unstar(entry2)
>>> list(database.get_starred())
[]
>>> database.edit(entry2, id=42, starred=True)
>>> list(database.get_starred())
[<DatabaseEntry(id 42, data provider None, fileid None)>]

Caching

The database may be used as a cache. In fact, a cache is always used, but the default size is float('inf'), meaning infinite size. There are different possible types of caches and they mainly differ in the way what items they remove if the cache has reached the full size and another item is added. The default cache is an LRU (Least Recently Used) cache. This one removes the item where the access time is the oldest. There is also one other builtin cache, the LFU (Least Frequently Used) cache. This one removes the entry where the number of accesses is the lowest. It is also possible to add custom cache algorithms, this will be covered in one of the next posts in my private blog.

The cache type is the second argument, defaulting to sunpy.database.caching.LRUCache. The cache size of a database is specified by passing the keyword argument cache_size in the __init__ method. To get an entry by its unique ID, you use the method get_entry_by_id. In the following example, you can see (at least I hope you do ^^) that the entries #1 and #3 have been accessed once, whereas #2 has not been accessed at all. So you could imagine that its last accessed time is minus infinity and therefore it gets removed when entry #4 is added to the database.

>>> from pprint import pprint
>>> database = Database('sqlite:///:memory:', cache_size=3)
>>> database.create_tables()
>>> entry1 = DatabaseEntry()
>>> entry2 = DatabaseEntry()
>>> entry3 = DatabaseEntry()
>>> entry4 = DatabaseEntry()
>>> database.add(entry1)
>>> database.add(entry2)
>>> database.add(entry3)
>>> pprint(list(database))
[<DatabaseEntry(id 1, data provider None, fileid None)>,
 <DatabaseEntry(id 2, data provider None, fileid None)>,
 <DatabaseEntry(id 3, data provider None, fileid None)>]
>>> database.get_entry_by_id(1)
<DatabaseEntry(id 1, data provider None, fileid None)>
>>> database.get_entry_by_id(3)
<DatabaseEntry(id 3, data provider None, fileid None)>
>>> database.add(entry4)
>>> pprint(list(database))
[<DatabaseEntry(id 1, data provider None, fileid None)>,
 <DatabaseEntry(id 3, data provider None, fileid None)>,
 <DatabaseEntry(id 4, data provider None, fileid None)>]

What are the next plans?

The next big plans are writing actual tables (the current DatabaseEntry class can be seen as a dummy model for now), support querying and adding a connection to the VSO interface so that downloaded data gets automatically added to the database.

Gsoc Community Bonding

2013-05-28T00:00:00+00:00

From our last post you may remember that SunPy is participating in GSOC-2013 under the PSF <https://wiki.python.org/moin/SummerOfCode/2013>`_ (!= Point Spread Function. Yesterday, Google announced the accepted candidates for their summer of code. If you look the list you will find between all these students that there are two whose projects is to work with SunPy.

Simon Liedtke (a.k.a. derdon) will develop a database for downloaded solar data, this will allow SunPy users to interact with the data in a more efficient way, avoiding to have to download again and again the files that went lost in our hard-disk. This tool will be smart enough that will let you know if you query data to VSO that you have already downloaded.

Michael Malocha (a.k.a. mjm159) is going to work on a better integration of HEK and HELIO in SunPy. This will include querying for event to the available catalogues, but also other capabilities like overploting the events’ positions/contours into solar images.

They will start to work in their projects in just two weeks. You will be able to follow their adventures, feelings, ideas, etc. in their personal blogs (Michael’s , Simon’s ). But they will also give us a bi-weekly update on their progress in here! Keep an eye on them because we want your inputs to develop the best tools we can!

Meanwhile, we are in the Community Bonding Period. In this two weeks, Michael and Simon will have to interact with us, to know how we work, to read documents needed for their projects, so on and so on… On the mentor’s side we are going to work in a couple of wiki-documents to give them a better insight on how we would like to use their implemented tools at the end of the summer. Check the wiki and feel free to contribute!

This summer is going to be AMAZING for SunPy, so don’t doubt to join us for the fun!

Google Summer of Code

2013-04-09T00:00:00+00:00

SunPy is participating in the Google Summer of Code 2013 under the umbrella of the Python Software Foundation .

We have participated successfully in the ESA summer of code in space project which is similar to GSoC for the last two years. Both our students and our project have gained a lot from the experience. A list of project ideas and application info is on our GitHub Wiki .

If you are interested in talking to us about projects, applications or if you are just bored pop into our IRC channel #SunPy on irc.freenode.net or drop us a line on our `mailing list .

Enjoy your summer of code!