We are concerned with the general problem of concept mining - discovering useful associations, relationships, and groupings in large collections of data. Mathematical transformation algorithms have proven effective at reducing the content of multilingual, unstructured data into a vector that describes the content. Such methods are particularly desirable in fields undergoing information explosions, such as network traffic analysis, bioinformatics, and the intelligence community. In response, concept mining methodology is being extended to improve performance and permit hardware implementation - traditional methods are not sufficiently scalable. Hardware-accelerated systems have proven effective at automatically classifying such content when topics are known in advance. Our complete system builds on our past work in this area, presented in the Aerospace 2005 and 2006 conferences, where we described a novel algorithmic approach for extracting semantic content from unstructured text document streams. However, there is an additional need within the intelligence community to cluster related sets of content without advance training. To allow this function to happen at high speed, we have implemented a system that hierarchically clusters streaming content. The method, streaming hierarchical partitioning, is designed to be implemented in hardware and handle extremely high ingestion rates. As new documents are ingested, they are dynamically organized into a hierarchy, which has a fixed maximal size. Once this limit is reached, documents must consequently be excreted at a rate equaling their ingestion.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Streaming Hierarchical Clustering for Concept Mining


    Contributors:


    Publication date :

    2007-03-01


    Size :

    1064710 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Dominant Sets and Hierarchical Clustering

    Pavan, M. / Pelillo, M. / IEEE | British Library Conference Proceedings | 2003


    Dominant sets and hierarchical clustering

    Pavan, / Pelillo, | IEEE | 2003


    HIERARCHICAL CLUSTERING OF AIRCRAFT SURFACE TRAJECTORIES

    Churchill, Andrew / Bloem, Michael | TIBKAT | 2020


    IDENTIFYING WAREHOUSE LOCATION USING HIERARCHICAL CLUSTERING

    Sebastjan ŠKERLIČ / Robert MUHA | DOAJ | 2016

    Free access

    Clustering Rules: A Comparison of Partitioning and Hierarchical Clustering Algorithms

    Reynolds, A. P. / Richards, G. / Iglesia, B. et al. | British Library Online Contents | 2006