When Big Data Become Astronomical Data

As per the eScience Institute, USA,  What limits astronomy today is no longer obtaining or storing data, it is the question of how do we interact with and analyze Petabyte scale data repositories. Serial access to data sets via SQL queries will not scale to the size of the science questions we wish to address. We are reaching a stage where the data are much richer than the analyses we apply to them.

Please understand that data analysis in astronomy in the coming times will involve petabytes of data every day, this data would be very rich and we need to stack the imagery data, sound data and benchmark daily to notice changes among many other things.

One of the ways scientists are trying to solve this scalable data processing challenge is by using a software called MapReduce alongwith Hadoop.

MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.

Read Victor Eijkhout's answer to What is the difference between parallel computing, MapReduce and Message Passing Interface(MPI) on Quora

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

To Top