_Technology

The first step in knowledge extraction is to identify all of the entities and structural data within a set of documents: the people, places, concepts, numbers, sentiment and quotes. A series of custom classifiers extract and resolve those entities and store them in a knowledge base. We then identify relationships between pairs of entities using unsupervised methodologies. Every piece of data that we capture retains its provenance, giving us full transparency on the decisions made by downstream algorithms.

We construct models of reality based on streams of millions of documents. By de-duplicating and reconciling statements made by multiple observers, we create an ensemble version of the corpus. For any given event, there can be thousands of varying descriptions, from the people involved to the tiniest details. Taking a multi-document approach allows us to capture this variation as signal rather than noise. The multi-document approach improves performance metrics of the structuring engine compared with single document approaches.

This engine looks for evidence of real-world events based on a set of documents. It analyzes a set of structured data extracted from the documents. It is then able to cluster together entity relationships as a function of time. The result is a time-directed graph of inferred real-world events from any given corpus.

Information is best understood in context with all the other information around it. The context engine can be used to analyze any claim, fact or assertion and identify any supporting evidence or any contradictions and return these to the user to better contextualize the information. On a larger scale, the context engine allows us to connect together events based on an inferred chain of probable causality. This allows us to see how a set of events is connected and evolves through time, and to additionally enable us to identify the origin and spread of information over time.

Differences between sets of information can be meaningful. These differences can be detected at multiple levels of resolution: sentence, document, and corpus. At the sentence level, a change in a single key word in a regulatory filing can be surfaced. At the document level, by diffing on structural data such as entities and factual claims, the engine can detect consensus and contradiction. Applying the diff engine across languages allows us to see events that are being covered by one country and not another.

The most efficient means of communicating a complex analysis is through the combination of natural language narrative and graphics: a story. Our engines generate millions of statistical observations about entities and their relationships. We use a Bayesian model of surprise to rank these observations. The story emerges from a massive reduction in the dimensionality of these data and text generation via extractive and abstractive summarization. We are able to handle English, Chinese, and Russian as both input and output, with more languages on the way.