Identifying Hidden Structure (in Data) and Computing Knowledge

Extract taken from Ontonix Corporate website. Full article here.


We live in the digital age. True.  The flow of data is impressive. Statistics inform us that each year we generate more data than the past generations did in decades. But the problem is that Information Technology (IT) has concentrated on simply automating old ways of thinking, creating bottlenecks and problems we didn’t even imagine, and not really inventing new processes or approaches. There is little innovation going on in the IT arena. But one thing is certain, we are drowning in data. The problem is not simply storage (disk space is cheap). The big deal is how to extract workable knowledge out of all this data.

But what is knowledge? What is a “body of knowledge”? Setting aside ontological hair-splitting, we could say that a body of knowledge is equivalent to a structured and dynamic set of inter-related rules. The rules can be crisp or fuzzy or both. But the key here is structure. Structure is the skeleton upon which a certain body of knowledge can be further expanded, refined, modified (this is why we say “dynamic”). One could say that structure forms the basis of a model or of a theory. Today, there exist many ways of extracting structure from data. Statistics is one way. Building models based on data is another. But because building models and mis-handling of statistics has contributed to the destruction of a big chunk of our economy, we have invented a new method of identifying structure in data – a model-free method, which is free of statistics and building models. A method which is “natural” and un-biased…

2 Responses to Identifying Hidden Structure (in Data) and Computing Knowledge

  1. David, cool picture. In fact … all knowledge is model-free and unbiased because it is uniquely hidden in each human brain. Yes, we can use encoded knowledge (i.e. such blog posts) to train a brain to recognize such patterns, but mechanistic application of rules to some structure is not knowledge it is mechanics. Mechanics (i.e. BPM) aren’t usable in complex adaptive systems to achieve outcomes.

    The photo is a great example because we only recognize what we know! There could be many more such patterns hidden in there. Just structure is not knowledge, just as rules aren’t knowledge as it takes the human interpretation of patterns with an emotional decision weighting to make it usable. Also data and their statistical interpretations are quite useless observations in terms of actionable knowledge. Structure is no more than a contextual interpretation. But yes, it can be helpful to make such structure visible to the human observer, but if it is model-free and unbiased it is also useless. How would it increase human knowledge?

    As Peter Drucker said: ‘Knowledge is between two ears only.’

  2. Hi Max: good to hear from you again! It is a great picture. Your comment prompted me to revisit a blog item from 2010 that may be of further interest:

    I, wholeheartedly, agree that we are “limited” by our knowledge. However, just to be clear, we are not talking about “mechanistic application of rules to some structure…” we analyse data to establish where “relationships”, based upon the quality/frequency of information flow between each individual piece of data and all others, exist.

    If we want to focus on complexity within a system, we need to express it in terms of structure and variation (uncertainty/entropy/chaos). The structure in the system is simply the relationships that get formed over time between the objects in the system. For any system, these relationships will not be stable over time, due to the energy in the system. As the second law of thermodynamics states, entropy (energy) in a system will increase over time.

    According to our definition complexity is a function of structure and entropy [C = f (structure, entropy)] in the information, measured in “bits”. The data is verification of system functions. Using Shannon’s Entropy as the basis, we can measure entropy as a function of variation in the object relationships.

    Avoiding the “trap” of fitting patterns to knowledge (apophenia) is, vital. In recognition of the wisdom of La Place, infinite knowledge is not/will not be ours but, if, with the application of a “simple” rule, it is possible to gain insight from which we can build new knowledge that enables us to move beyond the limitations of what we thought we knew…that has been so damaging.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s