The term “Big Data” is thrown around so much lately that we tend to
gloss over its significance. Some folks brush the trend off as nothing
new; a natural evolution under Moore’s law.
As OSHEAN looks to 100+Gbps networks, I have become intrigued by the
applications that are now emerging—applications which utilize massive
data sets and ultra-fast networks to enable entirely new areas of
science.
Data sets have become markedly complex. In response,
Data Mining has emerged as a new branch of IT, tasked with developing
the tool sets necessary to explore and correlate the massive amounts of
data being collected in given research areas. It is interesting to
think about the evolution of these tool sets that can be launched
against these data sets to uncover trends and relationships even when
the researcher doesn’t know what s/he is looking for. I was always
taught the scientific method where one started out with a hypothesis,
collected data, ran test cases, and drew conclusions. Now, with
Exploratory Data Analysis (EDA) techniques, we can start with the data
and launch tools to generate the hypothesis. Wilder still, there has
been an explosion in our abilities to use visualization tools to help
understand and represent massive data sets and outcomes. The
possibilities pouring from these new capabilities are mind-boggling.
With every technological leap of this sort, we are also confronted with
moral dilemmas and this one is no exception as we face ethical questions
surrounding privacy, security, governance, and ownership of data.
I
was a huge fan of the Science Fiction writer Isaac Asimov and read with
vigor his seminal work “The Foundation Trilogy” when I was in high
school (I reread it about 10 years later). One of the central themes in
this work was the idea that well into the future, mankind would have so
much data describing human behavior that we would have the ability to
write programs that would predict future outcomes at a societal level.
Asimov, to his credit, tackled the ethical dilemmas facing the holders
of governance to a populace that had developed a branch of science
called Psychohistory—which produced quantitative capabilities to predict
macro-level trends in societal evolution that ultimately included the
prediction of the demise of its own galactic empire. Holders of this
knowledge were then faced with the moral conundrum of how to handle this
data and what to do to manipulate a different outcome. Sound like any
headlines you have read lately?