Beyond Data Mining:: knowledge mining
Thursday, 10 January, 2013 Leave a comment
We live in the digital age. The flow of data is truly impressive. Statistics inform us that each year we generate more data than the past generations did in decades. But the problem is that Information Technology (IT) has concentrated on simply automating old ways of thinking, creating bottlenecks and problems we didn’t even imagine, and not really inventing new processes or approaches. There is little innovation going on in the IT arena. But one thing is certain, we are drowning in data and we’re thirsty for knowledge…
Turn data into structure
Structure is the overture to knowledge. But what is knowledge? What is a “body of knowledge”? Setting aside ontological hair-splitting we could say that a body of knowledge is equivalent to a structured and dynamic set of inter-related rules. The rules can be crisp or fuzzy or both. But the key here is structure. Structure is the skeleton upon which a certain body of knowledge can be further expanded, refined, modified (this is why we say “dynamic”). One could say that structure forms the basis of a model or of a theory. Today there exist many ways of extracting structure from data. Statistics is one way. Building models based on data is another. But because building models and mis-handling of statistics has contributed to the destruction of a big chunk of our economy, we have invented a new method of identifying structure in data – a model-free method, which if free of statistics and building models. A method which is “natural” and un-biased.