Table of Content

• Forge.AI Overview
  • Forge.AI Platform Abstract
  • Connecting with the Forge.AI Platform
• Schema
  • Documents
    • DocumentSource
    • DocumentMetadata
    • SimilarDocuments
  • Sections
    • SectionMetadata
    • SourceTexts
    • Sentences
  • Entities
    • EntityComponents
    • EntityAssertions
    • EntityProperties
    • EntityLocation
  • DocumentTopics
    • DocumentTopicWords
  • DocumentTopicModelVectors
  • TopicModelVectors
  • Relationships
    • RelationshipPredicateAssertions
    • RelationshipSupportingSentences
• Sample Queries
  • Number of documents collected by sites over the past week.
  • Documents mentioning a company of interest.
    • Knowing the company’s URI
    • Using the company’s CIK
  • Documents mentioning a company of interest within the past 7 days.
    • Number of documents per day mentioning a company of interest within the past 7 days.
  • Documents mentioning a company of interest from a specific source.
  • Documents mentioning 2 companies of interest.
  • Documents classified with a topic of interest.
  • Relationships for a person of interest.
  • Documents that cluster (in one dimension of the LDA space) around a company of interest.
  • Documents that cluster (in the LDA vector space) around a company of interest.
  • List all 8-K filings for Apple (CIK=3571) in 2018Q3
  • List all organizations mentioned in a Risk Factors section (1A) of a 10-K filing for Apple (CIK=3571)
• Useful links

Forge.AI Overview

Forge.AI is a cloud-based data and knowledge platform that powers the analytical, model driven organization. We create real time structured knowledge and data feeds out of the chaotic ocean of unstructured data that flows across the globe and throughout your organization. Our customers can explore, discover, monitor and model the knowledge and information captured in that unstructured data in fundamentally new and powerful ways enabling the rapid delivery of high-impact, data driven models that create a competitive advantage.

Forge.AI amplifies the quantity of data science your organization can conduct. We eliminate the time, expense and technology challenges associated with trying to create and sustain a data pipeline that incorporates the at scale transformation of unstructured data into algorithmically relevant structured data. We fundamentally accelerate the speed at which your data scientist can, explore hypothesis, iterate on and deploy models that deliver business value.

Forge.AI creates a new paradigm in terms of the quality and sophistication of data science your organization can conduct. Historically, data science conducted with unstructured data has been “shallow,” meaning model inputs tend to be simple (e.g. word counts) and model outputs paint with a broad brush (e.g. traditional sentiment analysis). Forge.AI brings unparalleled fidelity to the identification, extraction, resolution and semantic framing of the interaction between the entities, events and relationships expressed in unstructured data. This fidelity enables a new level of sophistication and time series modeling and analysis within your organization. See more. See faster. Predict better.

Forge.AI Platform Abstract

Forge.AI collects publicly available information in real time and transforms it into a resource that: (i) is suitable for a wide assortment of machine learning and artificial intelligence applications, enhancing data science environments; (ii) improves automated decision making and reasoning processes and (iii) enhances internal knowledge bases. The data resources created by Forge.AI are made available to our customers in either a streaming format or addressable through a SQL enabled cloud data warehouse (Snowflake).

Our customers simply “plug into” the Forge.AI data environment and immediately, transformed and enriched data is available to our customers.

To give a sense as to what is happening “under the hood”: a brief abstract of the Forge.AI processing platform is detailed below:

Collection: Our collection environment operates in real time 7x24x365. Source information is collected from tens of thousands of national, local, and industry specific news sites, organizations web sites, different US and international sites (e.g. the SEC), social media, and private feeds and / or feeds that you provide directly to us. Specially designed algorithms continually learn and optimize the process by which we collect source information. We abide by all politeness clause and other requirements in our collection process. If there is ever a source of data we are not collecting that is relevant to you we will work with you and that data source to incorporate into our ecosystem.

Perceive: Forge.AI natural language understanding environment extracts all relevant entities, events and relationship. In addition, it extracts a host of meta information and generates AI informed “calculated data” derived from source documents directly and/or in conjunction with our knowledge base.

Reason and Enrich: Forge.AI’s semantic, temporal and probabilistic knowledge base works in concert with our natural language understanding technologies to resolve all relevant data. Furthermore, the knowledge base enables contextual awareness for enhanced analytics such as dependency analysis etc.

The Forge.AI knowledge base is where knowledge that is extracted and learned by Forge.AI in its processing of information persists. The knowledge base is a dynamic and ever-changing environment that learns and grows every day as the world changes.

The knowledge base is also where Forge.AI maintains the various ontologies we support. An overview of the primary ontologies is listed below. It is worth noting that Forge.AI adds to the ontologies we support both organically and at the request of customers:

• Domain Specific Ontologies
  • Corporate Structure
  • People and Corporate Relationships
  • Business Concepts
  • Financial Concepts
  • Marketing Concepts
  • Risk (Operational Risk, Credit Risk, Market Risk)
  • Science (Physics, Chemistry, Biology, etc.)
  • Military and National Security
  • Events
  • Custom Ontologies / Ontologies as Requested

Data Output and Syndication: Data is syndicated to our customer environment in one of two ways.

• Option One: Customers connect with our real time cloud-based data warehouse. This data document which follows, describes how the data is managed in that cloud data warehouse. It is intended to orient the customer to nature of the data managed in the data warehouse, its structure, and how to query the data and integrate it into your operations.
• Option Two: Customers connect with our real-time streaming data environment. This environment is conceptually similar in structure our cloud-based data warehouse; however the representation is in XML or JSON and the delivery mechanism uses HTTPS 2.0 Push. Separate documentation is available which describes this environment.

The format of the data that is syndicated to our customers is extensively described below. There are three “classes” of data extracted and generated by Forge.AI:

• Direct data that corresponds directly with a published document (e.g. news story, SEC filing, tweet, etc.);
• Calculated information that is derived from a published document, such as an automated LDA topic vectors representing the source information;
• Aggregate information derived by looking at a large collection of the corpus of processed data.

Connecting with the Forge.AI Platform

Connecting with the Forge.AI environment is simple, and an abstract is provided below. You simply choose to connect to our cloud-based data warehouse, or you choose to connect to our push API.

Schema

Diagram

Documents

Notes:

1. docDateTime is NULL if no document date could be identified for the document.

Example:

DocumentSource

The channel the document was collected through.

Example:

DocumentMetadata

Relevant HTTP header information acquired during document collection.

Example:

SimilarDocuments

A collection of references to documents that are closely identical to this document.

Notes:

1. similarDocId may not be present in the Documents table.

Example:

Sections

The collection of sections constituting the document.

Example:

SectionMetadata

Any metadata associated with a section represented as name-value pairs.

Example:

SourceTexts

If the section is a text (T) section then this table will contain the source text.

Notes:

1. title and text may be NULL because of licensing constraints.

Example:

Sentences

The set of extracted sentences from the text section.

Notes:

1. text may be NULL because of licensing constraints.

Example:

Entities

The collection of entities discovered in the source document. An entity is named object or domain concept.

Example:

EntityComponents

The components are the constituent entities that collectively constitute the complex entity.

EntityAssertions

The extracted entities are disambiguated/resolved using the ForgeAI semantic knowledge base.

Example:

EntityProperties

Entity properties intrinsic to the entity. Examples include “latitude=42.0016”, “CIK=320193”, etc.

Example:

EntityLocation

The entity label’s location into the document.

Example:

DocumentTopics

Example:

DocumentTopicWords

Example:

DocumentTopicModelVectors

Collection of document score topic model vectors that can be used for assessing document similarity, change detection, etc.

Comparisons should not be made between topic vectors generated using different topic models. Applications may demand multiple topic models drawn from different corpora. The set of topic vectors allows for associating multiple topic vector calculations with a text section.

Example:

TopicModelVectors

Section level topic model vector.

Example:

Relationships

A semantic association based on the ForgeAI relationship ontology between two entities in the document.

Example:

RelationshipPredicateAssertions

RelationshipSupportingSentences

Information identifying the locations in the document supporting the relationship assertion.

Example:

Sample Queries

Number of documents collected by sites over the past week.

SELECT ANY_VALUE(name) AS name,
       REGEXP_REPLACE(URL, 'https?:\/\/(www\.)?([^\/]+).*', '\\2') AS site,
       COUNT(*) AS cnt
FROM Articles.Documents
JOIN Articles.DocumentSource USING (docId)
WHERE url LIKE 'http%'
  AND collectDatetime >= DATEADD(DAY, -7, CURRENT_DATE())
GROUP BY site
ORDER BY cnt DESC

Documents mentioning a company of interest.

Knowing the company’s URI

SELECT DISTINCT docId, url, title
FROM Articles.Documents
JOIN Articles.EntityAssertions USING (docId)
WHERE assertedClass = 'http://forge.ai/company/apple_inc'

Using the company’s CIK

SELECT DISTINCT docId, url, title
FROM Articles.Documents
JOIN Articles.EntityProperties USING (docId)
WHERE name = 'cik' AND value = '320193'

Documents mentioning a company of interest within the past 7 days.

SELECT DISTINCT docId, url, title
FROM Articles.Documents
JOIN Articles.EntityAssertions USING (docId)
WHERE assertedClass = 'http://forge.ai/company/apple_inc'
  AND docDatetime >= DATEADD(DAY, -7, CURRENT_DATE())

Number of documents per day mentioning a company of interest within the past 7 days.

SELECT TO_DATE(docDateTime) AS date, COUNT(*) AS cnt
FROM Articles.Documents
JOIN Articles.EntityAssertions USING (docId)
WHERE assertedClass = 'http://forge.ai/company/apple_inc'
  AND docDatetime >= DATEADD(DAY, -7, CURRENT_DATE())
GROUP BY 1

Documents mentioning a company of interest from a specific source.

SELECT DISTINCT docId, url, title
FROM Articles.Documents
JOIN Articles.DocumentSource USING (docId)
JOIN Articles.EntityAssertions USING (docId)
WHERE assertedClass = 'http://forge.ai/company/apple_inc'
  AND name = 'CNN'

Documents mentioning 2 companies of interest.

SELECT DISTINCT docId, url, title
FROM Articles.Documents
JOIN Articles.EntityAssertions a1 USING (docId)
JOIN Articles.EntityAssertions a2 USING (docId)
WHERE a1.assertedClass = 'http://forge.ai/company/apple_inc'
  AND a2.assertedClass = 'http://forge.ai/company/alphabet_inc.'

Or

SELECT DISTINCT d.docId, url, tile
FROM Articles.Documents d
JOIN Articles.EntityAssertions a1 USING (docId)
JOIN Articles.Entities e2 USING (docId)
JOIN Articles.EntityAssertions a2 ON (a2.docId = e2.docId AND a2.entityId = e2.entityId)
WHERE a1.assertedClass = 'http://forge.ai/company/apple_inc'
  AND e2.label = 'WallaMe'
  AND a2.assertedClass = 'http://forge.ai/entity#Organization'

Documents classified with a topic of interest.

SELECT DISTINCT docId, url
FROM Articles.Documents
JOIN Articles.DocumentTopics USING (docId)
WHERE name = 'bankruptcy';

Relationships for a person of interest.

SELECT LOWER(predicateLabel), LOWER(objectLabel), MAX(confidence) AS confidence, COUNT(*) AS COUNT
FROM Articles.Relationships
WHERE LOWER(subjectLabel) = 'tim cook'
GROUP BY 1,2
ORDER BY 4 DESC

Documents that cluster (in one dimension of the LDA space) around a company of interest.

WITH LDA AS (
  SELECT *
  FROM Articles.DocumentTopicModelVectors
  WHERE modelVersion = 'VanillaLda.model-identifier-979b20fdc5691083320d4a61c515e64e0b53ecf9-a6728cf46f462fc4e38b53e8420a386dbe5d75ad'
), Apple AS (
  SELECT dimensionindex, AVG(dimensionvalue) AS dimensionvalue
  FROM LDA
  WHERE EXISTS (SELECT 1
                FROM Articles.EntityAssertions a
                WHERE a.docId = LDA.docID
                  AND assertedClass = 'http://forge.ai/company/apple_inc')
  GROUP BY 1
  ORDER BY 2 DESC
  LIMIT 1)
  
SELECT docid, url, title
FROM LDA
JOIN Apple ON (Apple.dimensionindex = LDA.dimensionindex AND ABS(Apple.dimensionvalue - LDA.dimensionvalue) < 0.005) 
JOIN Articles.Documents d USING (docId)
WHERE NOT EXISTS (SELECT 1
                  FROM Articles.EntityAssertions a
                  WHERE a.docId = d.docId
                    AND assertedClass = 'http://forge.ai/company/apple_inc')

Documents that cluster (in the LDA vector space) around a company of interest.

WITH LDA AS (
  SELECT *
  FROM Articles.DocumentTopicModelVectors
  WHERE modelVersion = 'VanillaLda.model-identifier-979b20fdc5691083320d4a61c515e64e0b53ecf9-a6728cf46f462fc4e38b53e8420a386dbe5d75ad'
), Apple AS (
  SELECT dimensionindex, AVG(dimensionvalue) AS dimensionvalue
  FROM LDA
  WHERE EXISTS (SELECT 1
                FROM Articles.EntityAssertions a
                WHERE a.docId = LDA.docId
                  AND assertedClass = 'http://forge.ai/company/apple_inc')
  GROUP BY 1)

SELECT docid, url, title, SUM(SQUARE(NVL(lda.dimensionvalue, 0) - NVL(Apple.dimensionvalue, 0))) AS d
FROM LDA
FULL OUTER JOIN Apple USING (dimensionindex)
WHERE NOT EXISTS (SELECT 1
                  FROM Articles.EntityAssertions a
                  WHERE a.docId = LDA.docId
                    AND assertedClass = 'http://forge.ai/company/apple_inc')
GROUP BY 1
HAVING d < 0.01;

List all 8-K filings for Apple (CIK=3571) in 2018Q3

WITH meta AS (
  SELECT docid,
         MAX(CASE WHEN name = 'forgeai:sec-filing-type' AND value LIKE '8-%' THEN SUBSTR(value, 1, 3)
                  WHEN name = 'forgeai:sec-filing-type'                      THEN SUBSTR(value, 1, 4) END) AS filing,
         MAX(CASE WHEN name = 'forgeai:sec-filing-year' THEN value::INT END) AS year,
         MAX(CASE WHEN name = 'forgeai:sec-filing-qtr' THEN value::INT END) AS quarter,
         MAX(CASE WHEN name = 'forgeai:company-cik' THEN value::INT END) AS cik
  FROM SEC.DocumentMetadata
  GROUP BY 1
)

SELECT docid, title
FROM SEC.Documents
JOIN meta USING (docid)
WHERE filing = '8-K'
  AND year = 2018
  AND quarter = 3
  AND cik = 3571;

List all organizations mentioned in a Risk Factors section (1A) of a 10-K filing for Apple (CIK=3571)

SELECT smd.docid, url, label
FROM SEC.SectionMetadata smd
JOIN SEC.Entities USING (docid)
JOIN SEC.EntityAssertions USING (docid, entityid)
JOIN SEC.EntityLocations USING (docid, entityid)
JOIN SEC.Sentences USING (docid, sectionid, sentenceid)
JOIN SEC.Documents USING (docid)
WHERE name = 'forgeai:sec-section-type'
  AND value = '1A'
  AND EXISTS (SELECT 1
              FROM SEC.DocumentMetadata d
              WHERE d.docid = smd.docid
                AND name = 'forgeai:company-cik'
                AND value::INT = 3571)
  AND assertedClass = 'http://forge.ai/entity#Organization';

Useful links

• Forge.AI Reference
• Snowflake Documentation
  • Connecting to Snowflake