Threadle: Population-scale network data management

Threadle is a high-performance software system for large, complex, multilayer and multimode network datasets, designed explicitly for working with the Swedish administrative registers. Unlike other existing general-purpose network libraries, it emphasizes memory-efficient storage and fast querying for heterogeneous multi-mode network data.

Threadle is open source and built in .NET, with a Core module handling data models, analytics, processing, and file I/O for massive, multi-dimensional networks. A CLI console provides direct use or interop from other languages, specifically R. Precompiled binaries are available for Windows, macOS, and Linux, or it can be built from source. An R library - threadleR - is available for R interoperability.

Why Threadle?

Most existing tools, such as igraph , networkx , and NetworKit , assume that edges are relatively homogeneous objects stored in adjacency lists or matrices. Whereas indeed perfectly fine for many datasets, these assumptions break down for real-world population-level data, where:

• relations are heterogeneous (different types of relations, different layers and social domains)
• data often includes both 1-mode and 2-mode relations
• networks include millions of nodes and/or billions of edges
• nodes could have various kinds and types of node attributes that needs efficient internal storage
• derived 2-mode (bipartite) projections take up huge amounts of memory

Purpose-built to handle the complexity found in full-population administrative registers, Threadle handles these aspects efficiently, storing edges and hyperedges directly with their nodes and enabling queries as if 2-mode networks were projected — without extra memory cost for projecting such data.

Key features

Native multilayer support

Each layer memory-optimized for type and directionality.

Distributed edge ownership

Edges and hyperedges are stored with the nodes that own them, improving memory locality and query performance.

Unified 1-mode and 2-mode architecture

Threadle stores:

• 1-mode (unipartite) edges (of layer-specific types)
• 2-mode (bipartite) relations as true hyperedges, not projections

This allows 2-mode data to be queried as if it were projected, without ever materializing the projection.

Built-in node attribute management

Fast addition, lookup and handling of node-owned attributes.

Efficient storage for population-scale administrative data

Designed for massive datasets with millions of nodes and billions of edges.

Clean separation of Core and frontend applications

• Core handles all modeling, processing, management and file handling
• A command-line interface (CLI) console with a full scripting language is provided, either for use as-is or from R using threadleR.
• Threadle.CLIconsole runs in either text-mode to make us humans happy, or in JSON-mode for easy interoperability with threadleR. Additional frontends can thus easily connect to Core, through CLIconsole running in JSON mode, and Core can also be integrated (as a DLL) into other systems needing Threadle capabilities.

Analytics

Threadle started off as a pure network data management system, where all analytical/algorithmic procedures were meant to be implemented in the threadleR R frontend. However, with version 1.3.0, Threadle has obtained a major boost in analytical capabilities, with standard metrics and analytics covering most social-scientific network analytical applications, such as:

• Centrality measures: betweenness, closeness, harmonic, eigenvector, and PageRank, identifying the most important/influential nodes
• Community detection: Louvain, Leiden, label propagation, and Girvan-Newman, for partitioning a network into cohesive subgroups
• Cohesion & clustering: clustering coefficient, transitivity, coreness, and the small-world coefficient, measuring how tightly knit a network is
• Structural holes & brokerage: Burt’s constraint and effective size, for identifying brokers bridging otherwise disconnected parts of a network
• Homophily & assortativity: degree- and attribute-based assortativity, testing whether similar nodes tend to connect
• Distance & reachability: diameter, eccentricity, shortest paths, and random-walker first-passage time, for measuring how close or far nodes are from each other

"Threadle"?

Well, crocheting is fun, and Norrköping has a rich textile history - so... Yeah.