We are concerned with the general problem of concept mining - discovering useful associations, relationships, and groupings in large collections of data. Mathematical transformation algorithms have proven effective at reducing the content of multilingual, unstructured data into a vector that describes the content. Such methods are particularly desirable in fields undergoing information explosions, such as network traffic analysis, bioinformatics, and the intelligence community. In response, concept mining methodology is being extended to improve performance and permit hardware implementation - traditional methods are not sufficiently scalable. Hardware-accelerated systems have proven effective at automatically classifying such content when topics are known in advance. Our complete system builds on our past work in this area, presented in the Aerospace 2005 and 2006 conferences, where we described a novel algorithmic approach for extracting semantic content from unstructured text document streams. However, there is an additional need within the intelligence community to cluster related sets of content without advance training. To allow this function to happen at high speed, we have implemented a system that hierarchically clusters streaming content. The method, streaming hierarchical partitioning, is designed to be implemented in hardware and handle extremely high ingestion rates. As new documents are ingested, they are dynamically organized into a hierarchy, which has a fixed maximal size. Once this limit is reached, documents must consequently be excreted at a rate equaling their ingestion.
Streaming Hierarchical Clustering for Concept Mining
2007-03-01
1064710 byte
Conference paper
Electronic Resource
English
Dominant Sets and Hierarchical Clustering
British Library Conference Proceedings | 2003
|Dominant sets and hierarchical clustering
IEEE | 2003
|HIERARCHICAL CLUSTERING OF AIRCRAFT SURFACE TRAJECTORIES
TIBKAT | 2020
|Clustering Rules: A Comparison of Partitioning and Hierarchical Clustering Algorithms
British Library Online Contents | 2006
|