The Dark Energy Survey Won’t be using iPhoto
These last couple of weeks have seen me writing code to ingest the images the DES camera will take when it goes online into our database.
The numbers are staggering. Each night, we’ll produce 300GB of data for a total of about 165 terabytes by the end of the survey, and that’s just the raw data that come off the ccd’s.
Once these raw data are processed into science images, we’ll have around 1.7 petabytes of processed data sets. If we reproces at the end of each season and keep everything, the total DES data set will reach 7 petabytes.
How is it that we have so much data? In order to characterize the nature of dark energy, we need lots of pictures that cover a lot of the sky in a number of wavelengths.
The DES camera has 62 2k by 4k ccd’s arranged in a hexagonal array that collectively make a 520 megapixel camera. The camera will take 100 - 400 sec exposures each night in four wavelengths.
The camera will read out each exposure from the 62 ccd’s and send them to us for processing and storage. How many we get after a night’s observing will depend on the weather and other factors, but we anticipate 300GB each night we do this.
And the DES project has reserved roughly 30% of the observing nights over five years on the 4 meter Blanco telescope at the Cerro Tololo Inter-American Observatory in South America.
This is the future of astronomy, lots and lots of image data that fill up petabyte scale supercomputing facilities. Organizing these data so that science can be done is no small task, and I’m thrilled to be on the forefront of some of the technologies that will serve this to scientists.
One thing hit me hard the other day: we were discussing image compression and which method we would use. During the discussion someone asked, “Why do we need to do ANY compression?”
It’s actually not so crazy an idea. Disk space is cheaper than CPU time, at least on the scale we’re operating on, why waste cpu cycles compressing and decompressing when we can just serve uncompressed images?
Another big sticking point is that many people who view astronomy images in their scientific format, known as fits, can’t seem to agree on a compression scheme across the various viewers out there. Some viewers can use gzip compression for example, but not RICE. Others can’t support compression at all.
Personally, I think we’ll be compressing images for a long time, but we’ll be smarter about it. I have been a big proponent of wavelet compression in astronomy images for a long time. If I could somehow get others to think about it, who knows?
Anyway, just a little glimpse into my world these days.
Related Stories
POSTED IN: cosmology, telescopes
0 opinions for The Dark Energy Survey Won’t be using iPhoto
No one has left a comment yet. You know what this means, right? You could be first!
Have an opinion? Leave a comment: