How SkeleTrace works

Overview

SkeleTrace is a working environment for open-source investigation. It brings together three instruments that usually live apart:

• an engine that keeps entities, the relations between them and time-stamped measurements, and produces views of them on request;
• a case notebook that collects public records and search results without letting them blur into conclusions;
• a traffic landscape that reads packet captures, event logs and raw bytes for structure.

All three run from one command-line program, skeletrace. Every command answers in JSON, and that JSON opens in the viewer. Each format is described field by field on the data formats page.

The model

The engine keeps the smallest true picture it can, and builds everything else from it when asked.

Entities

Nodes and edges, each with a stable identifier, a kind, a status, a priority, tags, and the times it was first and last seen. Node kinds include endpoints, relays, exchanges, observers, identities, accounts and topics. Edge kinds include routes, links, inferences, associations, memberships, references and boundary crossings. A node may carry a position.

Measurements

Metric definitions with units, and samples recorded against them as an append-only series. Each metric has a retention policy, so history is kept for as long as it is useful and no longer.

Working set

A small cache of what is active, selected or on screen, held within a fixed budget. Everything else stays in storage until it is needed.

Views

Topology views, sparse geographic views, data cards and snapshots, produced on request. A view is a projection of the stored record at a moment, not a second copy of it.

The engine does not try to hold a model of the whole world in memory. It holds what has been observed, and answers questions about it.

Sources reach the engine through adapters: HTTP JSON endpoints, NDJSON files, polled feeds, manual pushes, and HTTP over Tor. A governance layer records source policies and capabilities, and keeps an audit trail of what was fetched and what failed.

Structure, flow and place

Every investigation can be read in three ways, and SkeleTrace keeps all three on the same footing.

Structure

What is connected to what. Topology views list nodes and links; weak-point analysis finds the nodes and links a group depends on. Commands: materialize-topology, analyze-topology, case graph and case bottlenecks.

Flow

What moves, and whether it moves in a way it should not. The landscape reads packets, events and bytes as they pass. Commands: landscape, landscape-report and verify-packets.

Place

Where things are. Positions, routes and boundaries are kept in WGS 84 coordinates and drawn on a flat map or a globe. Command: materialize-sparse-geo.

The case notebook

The case notebook exists to keep public-source work honest. It records what each source said, separately, and leaves every judgement about identity to a person.

Observations

Each hit from a source becomes one observation: the source, the identifier the source uses for it, a name, a set of fields, the public address of the entry, the time it was collected, its kind and its subject type. Two observations with the same name remain two observations.

Records, leads and imports

Record

An entry in a source’s own directory or register: an NPI entry, a company filing, an LEI record, registration data for a domain.

Lead

A weaker search result: an account with a matching name, a paper, a court opinion, an archived page. Leads are worth reading, and they can never be linked.

Import

Material you bring yourself, from a JSON or CSV file or from an older people-search database. Records from older databases are marked for re-checking, because those tools merged results without keeping their sources.

Every observation also has a subject type: person, organisation or domain. Observations of different types are never linked.

Reviewed links

Two records are joined only when a person links them and gives a reason, for example case link 12 15 --reason "same registration number". The reason is stored with the link, a link can be removed again, and the observations themselves are never altered. Linked records form groups.

Two lists help with review. The review queue shows records that share an exact name and subject type but have not been linked; it is a prompt to look, not a suggestion that they match. Field variations show where the records in a linked group disagree, such as two filings for one company that give different cities.

Source diversity

Each run reports the spread of its sources as Shannon entropy, in bits: H = −∑ p_i log₂ p_i, where p_i is the share of observations that came from source i. Ten observations from ten different sources score log₂ 10, about 3.32 bits; ten from a single source score zero.

The figure describes how the evidence is spread. It is not a measure of confidence, and it is never presented as one.

Storage and exports

A case lives in a single SQLite file, created with owner-only permissions on Unix systems and shared with the Python OSINTropy tool, so either program can open it. Deleting a run removes its observations and links with it. Backups are consistent copies, taken safely while the file is in use.

Cases export as JSON, as CSV, as an HTML notebook and as GraphML. In CSV exports, any cell that a spreadsheet could mistake for a formula is neutralised. The HTML notebook is self-contained and loads nothing from the network.

Sources and connectors

Searches run only when asked. Each connector requests one bounded page of results and keeps only those that match the request exactly; web-search results, which cannot be checked that way, are kept as leads. Requests use HTTPS only, follow no redirects, wait a second between calls to the same service, give up after 12 seconds and refuse responses larger than 2 MB. Keys are read from environment variables and are never written into observations, reports or error messages.

When a source fails, its error is stored with the run and the observations from the other sources are kept; the run is marked partial, or failed if nothing came back. Documentation for every source is listed on the sources page.

Weak points

A chokepoint is the one bridge into a valley town: close it, and part of the map is cut off. SkeleTrace looks for these in any topology view and in any case.

Group

A set of nodes joined to each other by some path. Groups are listed largest first.

Chokepoint

A node whose removal splits its group, reported with the number of pieces it leaves behind.

Bridge

A single link whose loss splits its group. Two parallel links between the same nodes are never bridges.

The analysis is a depth-first search after Tarjan. It runs in time proportional to the number of nodes plus the number of links, without recursion, so very large views are handled without difficulty. In the viewer, chokepoints are ringed and bridges are drawn heavier, in every view.

In a case, the chokepoint is usually the observation that a single reviewed link connects to everything else. It marks the one piece of reasoning on which a whole identity rests, and it is the first thing to check again.

The traffic landscape

The landscape takes its name from Maxwell’s demon, the thought experiment in which a small being sorts fast molecules from slow ones and so creates order where there should be none. In traffic, that kind of sorting appears as structure where the statistics expect noise.

The landscape reads one stream and draws it as a set of lanes that scroll from right to left. For packets, the lanes are traffic families (WEB, DNS, MAIL, MEDIA, CTRL, BULK and OTHER) plus ALL for everything together. For raw bytes, the lanes are classes of byte value (ZERO, CTRL, TEXT, HIGH and FF). Each column is one tick of time, 125 milliseconds by default. Its height shows volume, and its character shows what kind of structure the lane held at that moment.

For each lane the landscape tracks symbol entropy, the dependence between successive symbols measured against a shuffled baseline, the conditional entropy rate and, where it exists, the regularity of arrival times.

A demon marks structure, not intent. A DNS tunnel produces one; so can a well-behaved updater that simply talks too often.

The landscape reads a scripted demonstration, packet events from CSV or JSONL files, any file as raw bytes, and pcap or pcapng captures, which it reads without libpcap. Builds with live capture enabled also read from a network interface, using Npcap on Windows or libpcap on Linux. Live runs draw in the terminal. Offline inputs can also run headless on a virtual clock, which produces a JSON report with statistics for every lane, the alerts raised and the final frame.

Place and geometry

Positions are WGS 84 latitude and longitude in degrees, with an optional altitude. Nodes without a position are kept and listed, and the map views show them in a separate tray, so nothing drops out of sight. Boundaries, such as a jurisdiction or a watch zone, carry a rectangular extent given by its minimum and maximum latitude and longitude.

Sparse geographic views are produced for a requested viewport and delivered as standard GeoJSON. Every link carries a geometry mode, which decides how it is drawn:

The viewer

The viewer opens SkeleTrace output, GeoJSON, GraphML and general graph JSON in four ways: a 2D graph laid out by simulated forces, a 3D graph that can be turned and zoomed, a flat map, and a globe. It opens on the map when most nodes have a position, and on the graph otherwise.

Selecting any node, link, area or track shows it in full: its kind and identifier, its position, its metrics with units and times, every connection, and the raw data exactly as it appears in the file. For case files it also shows each observation’s fields, its source page and the reasons given for its links. Weak points are marked in every view.

The viewer is a single page. Its content policy forbids network requests, so a file opened in it is read on your computer and goes nowhere else. Coastlines are built in, from Natural Earth’s public-domain data. Files saved by Windows PowerShell redirects, which are written as UTF-16, open as they are.

A typical piece of work

1. Collect. Run the searches that suit the subject (case search FIRST LAST, case search-org "LEGAL NAME" or case domain example.org), or bring in your own notes with case import-file.
2. Read. case show RUN lists every observation with its source, along with the review queue and any source errors.
3. Review. Link records only where the evidence supports it, and write the reason down.
4. See the shape. Open the output of case graph RUN or case show RUN in the viewer, or ask for case bottlenecks RUN directly.
5. Check the weak points. Re-examine every chokepoint: each is a link on which an identity depends.
6. Share. case export RUN writes an HTML notebook, CSV, JSON or GraphML.

What SkeleTrace does not do

• It does not decide who is who. It never links records by itself, and identical names alone are never treated as proof.
• It does not score truth. Source diversity describes spread and the landscape describes structure. Neither says whether a claim is correct.
• It does not go beyond published interfaces. Connectors use documented public APIs, at a measured pace.
• It does not hide failure. Connectors are tested against recorded responses. When a service changes or fails, the error is stored with the run and the rest of the run is kept, rather than anything being filled in silently.

Command reference

Engine commands take a profile, the engine’s configuration, as their first argument. Every command prints JSON.

Exit codes: 0 for success, 1 for an error, 2 for a usage mistake, and 3 when a case search was saved but at least one source failed.