Data Model & Storage¶

This document explains the data model and storage architecture used by OWASP Amass. It covers the Open Asset Model (OAM) specification that defines asset types and relationships, the three-tier storage architecture (graph database, cache, and work queue), and how assets flow through the system from discovery to persistent storage.

Data Model Foundation¶

Amass uses the Open Asset Model (OAM) as its core data specification. OAM is a standardized schema for representing cybersecurity assets and their relationships in a graph structure. Each discovered piece of information (domain, IP address, organization, service, etc.) is represented as an OAM asset type.

OAM Asset Types¶

The system supports 20 distinct asset types, each implementing the oam.Asset interface:

graph TB
    subgraph "Network Assets"
        FQDN["FQDN<br/>oamdns.FQDN"]
        IPAddress["IPAddress<br/>network.IPAddress"]
        Netblock["Netblock<br/>network.Netblock"]
        AS["AutonomousSystem<br/>network.AutonomousSystem"]
    end

    subgraph "Organization Assets"
        Org["Organization<br/>org.Organization"]
        Person["Person<br/>people.Person"]
        Account["Account<br/>account.Account"]
    end

    subgraph "Infrastructure Assets"
        Service["Service<br/>platform.Service"]
        Product["Product<br/>platform.Product"]
        Release["ProductRelease<br/>platform.ProductRelease"]
        TLS["TLSCertificate<br/>oamcert.TLSCertificate"]
    end

    subgraph "Registration Assets"
        DomainRec["DomainRecord<br/>oamreg.DomainRecord"]
        IPNetRec["IPNetRecord<br/>oamreg.IPNetRecord"]
        AutnumRec["AutnumRecord<br/>oamreg.AutnumRecord"]
    end

    subgraph "Contact Assets"
        Contact["ContactRecord<br/>contact.ContactRecord"]
        Location["Location<br/>contact.Location"]
        Phone["Phone<br/>contact.Phone"]
    end

    subgraph "Other Assets"
        URL["URL<br/>url.URL"]
        File["File<br/>file.File"]
        Transfer["FundsTransfer<br/>financial.FundsTransfer"]
        ID["Identifier<br/>general.Identifier"]
    end

Entity Wrapper Pattern¶

OAM assets are not stored directly. Instead, they are wrapped in a dbt.Entity object that provides metadata and tracking:

graph LR
    Asset["oam.Asset<br/>(FQDN, IPAddress, etc.)"]
    Entity["dbt.Entity"]
    ID["ID: string<br/>(unique identifier)"]
    CreatedAt["CreatedAt: time.Time"]

    Entity --> Asset
    Entity --> ID
    Entity --> CreatedAt

    Event["et.Event"] --> Entity
    Cache["cache.Cache"] --> Entity
    Queue["SessionQueue"] --> Entity

The Event structure carries entities through the system:

Three-Tier Storage Architecture¶

Amass maintains three distinct storage layers, each serving a specific purpose in the data lifecycle:

graph TB
    subgraph "Discovery Layer"
        Plugins["Discovery Plugins"]
        Event["et.Event<br/>with dbt.Entity"]
    end

    subgraph "Tier 1: Work Queue"
        QueueDB["SQLite Queue DB<br/>queue.db"]
        Element["Element Table<br/>ID, Type, EntityID, Processed"]
    end

    subgraph "Tier 2: Session Cache"
        CacheDB["SQLite Cache DB<br/>cache.db"]
        TempStorage["Temporary Storage<br/>During Enumeration"]
    end

    subgraph "Tier 3: Graph Database"
        GraphDB["Persistent Database"]
        SQLite["SQLite<br/>assetdb.db"]
        Postgres["PostgreSQL"]
        Neo4j["Neo4j"]
    end

    Plugins --> Event
    Event --> QueueDB
    QueueDB --> Element

    Event --> CacheDB
    CacheDB --> TempStorage

    CacheDB -->|"Flush on interval"| GraphDB
    GraphDB --> SQLite
    GraphDB --> Postgres
    GraphDB --> Neo4j

    style QueueDB fill:#fff5e6
    style CacheDB fill:#e6f5ff
    style GraphDB fill:#e6ffe6

Tier 1: Work Queue Database¶

The work queue is a SQLite database (queue.db) that tracks which assets need processing. It implements a priority-based FIFO queue for each asset type.

Database Schema:

erDiagram
    Element {
        uint64 ID PK
        time CreatedAt
        time UpdatedAt
        string Type "Asset type (FQDN, IPAddress, etc.)"
        string EntityID "Unique entity identifier"
        bool Processed "Processing status"
    }

Indexes: - idx_created_at — Orders by creation time (FIFO) - idx_etype — Filters by asset type - idx_entity_id — Ensures uniqueness - idx_processed — Filters unprocessed items

Key Operations:

sequenceDiagram
    participant Dispatcher
    participant QueueDB
    participant Cache
    participant Pipeline

    Dispatcher->>QueueDB: Next(FQDN, 100)
    QueueDB-->>Dispatcher: [entity_id_1, entity_id_2, ...]

    loop For each entity_id
        Dispatcher->>Cache: FindEntityById(id)
        Cache-->>Dispatcher: dbt.Entity
        Dispatcher->>Pipeline: Append(Event)
    end

    Pipeline->>Dispatcher: Processing complete
    Dispatcher->>QueueDB: Processed(entity_id)

Tier 2: Session Cache¶

The session cache is a temporary SQLite database (cache.db) created per session. It uses the cache.Cache abstraction from the asset-db library.

Cache Lifecycle:

graph LR
    Create["Session Created"]
    TmpDir["Temporary Directory<br/>/tmp/session-{UUID}"]
    CacheFile["cache.db created"]
    Flush["Periodic Flush<br/>Every 1 minute"]
    GraphDB["Graph Database"]
    Cleanup["Session Terminated<br/>Cache deleted"]

    Create --> TmpDir
    TmpDir --> CacheFile
    CacheFile --> Flush
    Flush --> GraphDB
    CacheFile --> Cleanup

Tier 3: Persistent Graph Database¶

The persistent graph database is the single source of truth for all discovered assets. Amass supports three database backends:

graph TB
    Config["Config.GraphDBs<br/>[]*Database"]

    subgraph "SQLite (Default)"
        SQLiteDSN["DSN: assetdb.db?_pragma=busy_timeout(30000)&_pragma=journal_mode(WAL)"]
        SQLiteFile["File: {outputdir}/assetdb.db"]
    end

    subgraph "PostgreSQL"
        PgDSN["DSN: host={host} port={port} user={user} password={password} dbname={dbname}"]
        PgConn["Connection String"]
    end

    subgraph "Neo4j"
        Neo4jDSN["DSN: neo4j+s://{host}:{port}"]
        Neo4jBolt["Bolt Protocol"]
    end

    Config -->|"System: sqlite"| SQLiteDSN
    Config -->|"System: postgres"| PgDSN
    Config -->|"System: neo4j"| Neo4jDSN

    SQLiteDSN --> SQLiteFile
    PgDSN --> PgConn
    Neo4jDSN --> Neo4jBolt

Environment Variable Support:

Asset and Entity Lifecycle¶

Assets flow through multiple stages from initial discovery to persistent storage:

graph TB
    subgraph "Stage 1: Discovery"
        Plugin["Plugin Handler"]
        NewAsset["Create OAM Asset<br/>(e.g., oamdns.FQDN)"]
    end

    subgraph "Stage 2: Cache Creation"
        CacheCreate["cache.CreateAsset(asset)"]
        Entity["Returns dbt.Entity<br/>with ID and timestamps"]
    end

    subgraph "Stage 3: Event Dispatch"
        Event["Create et.Event"]
        Dispatcher["Dispatcher.DispatchEvent()"]
    end

    subgraph "Stage 4: Queue Management"
        QueueCheck["Queue.Has(entity)?"]
        QueueAppend["Queue.Append(entity)"]
        QueueElement["SQLite: Insert Element"]
    end

    subgraph "Stage 5: Pipeline Processing"
        PipelineFetch["Dispatcher fills pipeline<br/>from Queue.Next()"]
        HandlerExec["Handler.Callback(event)"]
    end

    subgraph "Stage 6: Persistence"
        CacheFlush["Cache flushes<br/>every 1 minute"]
        GraphDB["Graph Database<br/>Permanent Storage"]
    end

    Plugin --> NewAsset
    NewAsset --> CacheCreate
    CacheCreate --> Entity
    Entity --> Event
    Event --> Dispatcher
    Dispatcher --> QueueCheck
    QueueCheck -->|"false"| QueueAppend
    QueueAppend --> QueueElement
    QueueElement --> PipelineFetch
    PipelineFetch --> HandlerExec
    HandlerExec -->|"New assets"| NewAsset
    CacheCreate --> CacheFlush
    CacheFlush --> GraphDB

Stage-by-Stage Breakdown¶

Stage 1 — Plugin Discovery: A plugin handler discovers new information (e.g., DNS lookup finds an IP) and creates an OAM asset object.

Stage 2 — Cache Creation: Calls session.Cache().CreateAsset(oamAsset). Cache wraps the asset in dbt.Entity with a unique ID and timestamp.

Stage 3 — Event Dispatch: Plugin creates et.Event with the entity and calls dispatcher.DispatchEvent(event).

Stage 4 — Queue Management: Dispatcher checks session.Queue().Has(entity) to prevent duplicates. If not queued, calls session.Queue().Append(entity).

Stage 5 — Pipeline Processing: Dispatcher periodically calls Queue.Next(assetType, 100) to fill pipelines. Handlers execute in priority order (1–9). After processing, marks as processed: Queue.Processed(entity).

Stage 6 — Persistence: Cache flushes to the graph database every 1 minute.

Asset Creation from Configuration Scope¶

Initial assets are created from the configuration scope before enumeration begins:

graph LR
    ConfigScope["config.Scope"]
    Domains["Domains: []string"]
    Addresses["Addresses: []net.IP"]
    CIDRs["CIDRs: []*net.IPNet"]
    ASNs["ASNs: []int"]

    FQDN["oamdns.FQDN"]
    IPAddr["network.IPAddress"]
    Netblock["network.Netblock"]
    AS["network.AutonomousSystem"]

    Assets["[]*et.Asset"]

    ConfigScope --> Domains
    ConfigScope --> Addresses
    ConfigScope --> CIDRs
    ConfigScope --> ASNs

    Domains --> FQDN
    Addresses --> IPAddr
    CIDRs --> Netblock
    ASNs --> AS

    FQDN --> Assets
    IPAddr --> Assets
    Netblock --> Assets
    AS --> Assets

Session-Specific Storage¶

Each session maintains isolated storage in a temporary directory:

graph TB
    Session["Session Object<br/>UUID: {session-id}"]
    OutputDir["Output Directory<br/>config.OutputDirectory()"]
    TmpDir["Temporary Directory<br/>/tmp/session-{UUID}"]

    CacheFile["cache.db<br/>Session Cache"]
    QueueFile["queue.db<br/>Work Queue"]

    GraphFile["assetdb.db<br/>Graph Database<br/>(SQLite only)"]

    Session --> OutputDir
    OutputDir --> TmpDir
    OutputDir --> GraphFile

    TmpDir --> CacheFile
    TmpDir --> QueueFile

    Cleanup["Session Terminated"]
    Cleanup -->|"os.RemoveAll(tmpdir)"| TmpDir
    Cleanup -->|"Kept permanently"| GraphFile

Directory structure:

{OutputDir}/
├── assetdb.db              # Persistent graph database (SQLite)
└── session-{UUID}/         # Temporary session directory
    ├── cache.db            # Session cache (deleted on exit)
    └── queue.db            # Work queue (deleted on exit)

Asset Types and Properties¶

Asset Type Hierarchy¶

graph TD
    Asset["oam.Asset Interface<br/>AssetType() string<br/>Key() string"]

    subgraph "DNS Assets (oamdns)"
        FQDN["FQDN<br/>Name: string"]
    end

    subgraph "Network Assets (oamnet)"
        IPAddress["IPAddress<br/>Address: netip.Addr<br/>Type: string"]
        Netblock["Netblock<br/>CIDR: netip.Prefix<br/>Type: string"]
        AutonomousSystem["AutonomousSystem<br/>Number: int"]
    end

    subgraph "Organization Assets"
        Organization["Organization<br/>Name: string<br/>Industry: string"]
    end

    subgraph "Service Assets"
        Service["Service<br/>Port: int<br/>Protocol: string"]
    end

    subgraph "Contact Assets"
        Contact["Contact<br/>Type: string<br/>Value: string"]
        Location["Location<br/>Address: string"]
    end

    Asset --> FQDN
    Asset --> IPAddress
    Asset --> Netblock
    Asset --> AutonomousSystem
    Asset --> Organization
    Asset --> Service
    Asset --> Contact
    Asset --> Location

Property Attachment Model¶

graph LR
    Entity["dbt.Entity<br/>(Asset Wrapper)<br/>ID, Asset, CreatedAt"]

    subgraph "Entity Properties"
        Prop1["DNSRecordProperty<br/>RRType, TTL, Data"]
        Prop2["SourceProperty<br/>Source, Confidence"]
        Prop3["Custom Properties"]
    end

    Entity -->|"CreateEntityProperty()"| Prop1
    Entity -->|"CreateEntityProperty()"| Prop2
    Entity -->|"CreateEntityProperty()"| Prop3

    Edge["dbt.Edge<br/>(Relationship)<br/>FromEntity, ToEntity"]

    subgraph "Edge Properties"
        EdgeProp1["SourceProperty<br/>Source, Confidence"]
        EdgeProp2["Custom Properties"]
    end

    Edge -->|"CreateEdgeProperty()"| EdgeProp1
    Edge -->|"CreateEdgeProperty()"| EdgeProp2

DNSRecordProperty¶

The DNSRecordProperty attaches DNS record information to FQDN entities.

_, err := session.Cache().CreateEntityProperty(fqdn, &oamdns.DNSRecordProperty{
    PropertyName: "dns_record",
    Header: oamdns.RRHeader{
        RRType: int(dns.TypeTXT),
        Class:  int(record.Header().Class),
        TTL:    int(record.Header().Ttl),
    },
    Data: txtValue,
})

Asset to Entity Conversion¶

sequenceDiagram
    participant Plugin as "DNS Plugin"
    participant Cache as "session.Cache()"
    participant DB as "Graph Database"

    Plugin->>Plugin: Create OAM Asset<br/>e.g., oamdns.FQDN{Name: "example.com"}
    Plugin->>Cache: CreateAsset(asset)
    Cache->>DB: Check if asset exists
    alt Asset exists
        DB-->>Cache: Return existing entity
    else New asset
        DB->>DB: Create new entity<br/>Assign ID, timestamp
        DB-->>Cache: Return new entity
    end
    Cache-->>Plugin: Return dbt.Entity

    Plugin->>Cache: CreateEntityProperty(entity, property)
    Cache->>DB: Attach property to entity
    DB-->>Cache: Success

    Plugin->>Plugin: Dispatch event with entity

DNS Plugin Asset Flow¶

graph TD
    Start["Event with FQDN Entity"]

    Start --> Extract["Extract Asset from Entity<br/>fqdn := entity.Asset.(*oamdns.FQDN)"]
    Extract --> Query["DNS Query<br/>PerformQuery(fqdn.Name, TypeA)"]

    Query --> CreateIP["Create IPAddress Asset<br/>oamnet.IPAddress{Address, Type}"]
    CreateIP --> StoreIP["Store as Entity<br/>session.Cache().CreateAsset(ip)"]

    StoreIP --> CreateEdge["Create DNS Record Edge<br/>oamdns.BasicDNSRelation"]
    CreateEdge --> AddProp["Add SourceProperty to Edge"]

    AddProp --> Dispatch["Dispatch Event with IP Entity"]

Asset Type Summary¶

Relationships and Edges¶

Edge Structure¶

Edges are represented by the dbt.Edge struct with three required components:

• Relation — An object implementing oam.Relation that defines the relationship type
• FromEntity — The source entity of the directed edge
• ToEntity — The destination entity of the directed edge

Edge Creation Pattern¶

graph LR
    Plugin["Plugin Handler"]
    CreateEdge["cache.CreateEdge()"]
    EdgeProperty["cache.CreateEdgeProperty()"]
    SourceProp["SourceProperty"]

    Plugin --> CreateEdge
    CreateEdge --> EdgeProperty
    EdgeProperty --> SourceProp

    SourceProp -.->|"Tracks confidence<br/>and origin"| Edge["dbt.Edge"]
    CreateEdge -.->|"Returns"| Edge

edge, err := e.Session.Cache().CreateEdge(&dbt.Edge{
    Relation: &oamdns.BasicDNSRelation{
        Name: "dns_record",
        Header: oamdns.RRHeader{RRType: 1, Class: 1, TTL: 300},
    },
    FromEntity: fqdn,
    ToEntity:   ipEntity,
})
if err == nil && edge != nil {
    e.Session.Cache().CreateEdgeProperty(edge, &general.SourceProperty{
        Source:     "DNS",
        Confidence: 100,
    })
}

DNS Relations¶

graph LR
    FQDN1["FQDN Entity<br/>example.com"]
    FQDN2["FQDN Entity<br/>www.example.com"]
    IP["IPAddress Entity<br/>192.0.2.1"]
    CNAME["FQDN Entity<br/>cdn.provider.com"]

    FQDN1 -->|"BasicDNSRelation<br/>dns_record<br/>RRType: A(1)"| IP
    FQDN2 -->|"BasicDNSRelation<br/>dns_record<br/>RRType: CNAME(5)"| CNAME

graph LR
    FQDN["FQDN Entity<br/>example.com"]
    MX1["FQDN Entity<br/>mail1.example.com"]
    MX2["FQDN Entity<br/>mail2.example.com"]

    FQDN -->|"PrefDNSRelation<br/>dns_record<br/>RRType: MX(15)<br/>Preference: 10"| MX1
    FQDN -->|"PrefDNSRelation<br/>dns_record<br/>RRType: MX(15)<br/>Preference: 20"| MX2

Identity Relations¶

graph LR
    Org["Organization Entity<br/>Google LLC"]
    LEI["Identifier Entity<br/>LEI:549300GZSUFD98NHH337"]
    Email["Identifier Entity<br/>email:admin@example.com"]
    AviID["Identifier Entity<br/>aviato_company:abc123"]

    Org -->|"SimpleRelation<br/>id"| LEI
    Org -->|"SimpleRelation<br/>id"| Email
    Org -->|"SimpleRelation<br/>id"| AviID

Organizational Relations¶

graph TB
    Parent["Organization<br/>Alphabet Inc."]
    Child1["Organization<br/>Google LLC"]
    Child2["Organization<br/>Waymo LLC"]

    Parent -->|"SimpleRelation<br/>subsidiary"| Child1
    Parent -->|"SimpleRelation<br/>subsidiary"| Child2

Location Relations¶

graph TB
    Org["Organization<br/>ACME Corp"]
    Legal["Location<br/>123 Legal St<br/>New York, NY"]
    HQ["Location<br/>456 HQ Ave<br/>San Francisco, CA"]
    Other["Location<br/>789 Office Rd<br/>Austin, TX"]

    Org -->|"SimpleRelation<br/>legal_address"| Legal
    Org -->|"SimpleRelation<br/>hq_address"| HQ
    Org -->|"SimpleRelation<br/>location"| Other

Network Relations¶

graph LR
    FQDN["FQDN Entity<br/>web.example.com"]
    IP["IPAddress Entity<br/>192.0.2.1"]
    Service1["Service Entity<br/>HTTP on :443"]
    Service2["Service Entity<br/>SSH on :22"]

    FQDN -->|"PortRelation<br/>port: 443<br/>protocol: tcp"| Service1
    IP -->|"PortRelation<br/>port: 22<br/>protocol: tcp"| Service2

graph LR
    IP["IPAddress Entity<br/>192.0.2.1"]
    PTR["FQDN Entity<br/>1.2.0.192.in-addr.arpa"]
    Target["FQDN Entity<br/>mail.example.com"]

    IP -->|"SimpleRelation<br/>ptr_record"| PTR
    PTR -->|"BasicDNSRelation<br/>dns_record<br/>RRType: PTR(12)"| Target

Domain Hierarchy (node relation)¶

graph TB
    Apex["FQDN Entity<br/>example.com"]
    Sub1["FQDN Entity<br/>www.example.com"]
    Sub2["FQDN Entity<br/>mail.example.com"]
    Sub3["FQDN Entity<br/>api.subdomain.example.com"]
    Sub4["FQDN Entity<br/>subdomain.example.com"]

    Apex -->|"SimpleRelation<br/>node"| Sub1
    Apex -->|"SimpleRelation<br/>node"| Sub2
    Apex -->|"SimpleRelation<br/>node"| Sub4
    Sub4 -->|"SimpleRelation<br/>node"| Sub3

Contact Relations¶

graph TB
    DomRec["DomainRecord<br/>example.com"]
    Contact["ContactRecord<br/>Registrant Info"]
    Person["Person<br/>John Doe"]
    Org["Organization<br/>Example Corp"]
    Loc["Location<br/>123 Main St"]
    Email["Identifier<br/>email:admin@example.com"]
    Phone["Phone<br/>+1-555-1234"]

    DomRec -->|"SimpleRelation<br/>registrant_contact"| Contact
    Contact -->|"SimpleRelation<br/>person"| Person
    Contact -->|"SimpleRelation<br/>organization"| Org
    Contact -->|"SimpleRelation<br/>location"| Loc
    Contact -->|"SimpleRelation<br/>id"| Email
    Contact -->|"SimpleRelation<br/>phone"| Phone

Contact relation types include: registrant_contact, admin_contact, technical_contact, billing_contact, abuse_contact, subject_contact, issuer_contact.

Certificate Relations¶

graph TB
    Cert["TLSCertificate<br/>Serial: ABC123"]
    CN["FQDN Entity<br/>*.example.com"]
    SAN1["FQDN Entity<br/>www.example.com"]
    SAN2["FQDN Entity<br/>api.example.com"]
    IP["IPAddress Entity<br/>192.0.2.1"]
    Email["Identifier<br/>email:admin@example.com"]
    URL["URL Entity<br/>http://ocsp.ca.com"]
    Service["Service Entity<br/>HTTPS on :443"]

    Cert -->|"SimpleRelation<br/>common_name"| CN
    Cert -->|"SimpleRelation<br/>san_dns_name"| SAN1
    Cert -->|"SimpleRelation<br/>san_dns_name"| SAN2
    Cert -->|"SimpleRelation<br/>san_ip_address"| IP
    Cert -->|"SimpleRelation<br/>san_email_address"| Email
    Cert -->|"SimpleRelation<br/>ocsp_server"| URL
    Service -->|"SimpleRelation<br/>certificate"| Cert

Financial Relations¶

graph LR
    Investor["Organization<br/>VC Fund"]
    SeedAcct["Account<br/>Seed Round A"]
    Transfer["FundsTransfer<br/>$5M USD"]
    OrgAcct["Account<br/>Startup Checking"]
    Org["Organization<br/>Startup Inc."]

    Investor -->|"SimpleRelation<br/>account"| SeedAcct
    Transfer -->|"SimpleRelation<br/>sender"| SeedAcct
    Transfer -->|"SimpleRelation<br/>recipient"| OrgAcct
    Org -->|"SimpleRelation<br/>account"| OrgAcct

Domain Registration Relations¶

graph TB
    DomRec["DomainRecord<br/>example.com"]
    NS1["FQDN<br/>ns1.nameserver.com"]
    NS2["FQDN<br/>ns2.nameserver.com"]
    WHOIS["FQDN<br/>whois.registry.com"]

    DomRec -->|"SimpleRelation<br/>name_server"| NS1
    DomRec -->|"SimpleRelation<br/>name_server"| NS2
    DomRec -->|"SimpleRelation<br/>whois_server"| WHOIS

Source Attribution¶

Every edge should have a SourceProperty attached to track which plugin discovered the relationship and with what confidence:

type SourceProperty struct {
    Source:     string  // Plugin/source name
    Confidence: int     // 0-100 confidence score
}

graph TD
    Handler["Plugin Handler"]
    CreateAssets["Create Assets"]
    CreateEdge["Create Edge"]
    AttachSource["Attach SourceProperty"]

    Handler --> CreateAssets
    CreateAssets --> CreateEdge
    CreateEdge --> AttachSource

    AttachSource -.->|"Tracks:<br/>• Plugin name<br/>• Confidence<br/>• Timestamp"| EdgeTags["Edge Tags"]