I've been doing some thinking about BIG DATA recently. If you're not familiar with the concept the idea is that we are now at a time when technology is not confined to taking a "random sample" for study.
In the old paradigm it was impossible to know and analyse all the data in one topic, example customer preferences, so we asked a randomised or stratified sample. Now we have sufficient computing power to not just ask the customer, but to track their data, triangulate with other data sources and arrive at an answer which is based not on what the customers said but in what they did.
Making this computopia a reality is a matter of fusing certain existing systems and technologies together to fit the purpose.
And there is the issue.
All our current paradigm thinking is based on a model of science; asking a question to get an answer takes the form of a hypothesis. "What is the impact of neuropathy on life expectancies of diabetic patients?" The parameters create a defined subset, a population, a sample and controls out the variables, creating a baseline against which our sample group is compared.
Taking a big data approach is an alien one. Casting the data net widely across a population produces a population image, one in which the paintbrush consists of intelligent analytical algorithms and the appearance of the picture, like most modern art is a matter of interpretation.
The role of the artist could be defined as selection of the paints and canvas, with the inherent properties of the paint, it's colour or texture interacting with the brush to produce the final image.
Examples of biggish data include school results, ofsted and family income comparisons, biggish because the picture created used limited colours and a couple of brushes on a small canvas.
Big Data is a big leap on and in the case of school results would include additions of social media, travel patterns, benefits data, health records etc. More colours to create a more vibrant picture, one which may better suit the complex adaptive systems of our lives and when studied, like a work of art, small details may attract our eye, requiring the microscope of standard scientific method.
Given this fundamental change in analysis paradigm Big Data may find it hard to gain traction with those steeped in scientific method. Those who value complexity and chaos may feel equally uneasy, but ultimately until the practicalities of access and the mechanics of governance are the first hurdles for big data, with the dissemination challenges of acceptance and adoption further away.
Fad or Future? - to early to say.