Evaluation & Benchmarks
Deep-dive page: everything that has actually been measured on real LibreSocial nodes — scale, stability, latency, cost — organized oldest → newest, with the current authoritative numbers named.
Supersession note: the authoritative system-level benchmark is Masinde/Khitman/Dlikman/Graffi 2020 (Future Internet) and its dissertation umbrella Masinde 2020; these supersede the numbers reported by the theses they build on (Liu 2013, Peng 2014, Khitman 2020). Those older theses remain the source for method and for details the papers omit. For hardware-current numbers see Imam 2025.
Headline results (2020, real runs — not simulation)
Run on the HHU HPC cluster HILBERT, up to 4,000 launch attempts, measuring up to 2,000 simultaneously logged-in nodes — the authors state this is the first executed P2P-OSN benchmark of this size on a real system (vs. simulated overlays).
| Property | Result |
|---|---|
| Scalability | stable to 2,000 concurrent nodes; limit was cluster memory, not the protocol |
| Availability under churn | 250→1,000 nodes in steps, then churn back to 250: no crashes; overlay rebuilds routes and moves replicas |
| Overlay health | leaf set stays ≈ 25 (configured 24); routing table ≤ 39 entries, adapts dynamically |
| Per-node footprint | RAM allocated ≤ ~900 MB, actually used ≤ ~580 MB |
| Network rates | peak ~1,300 msg/s; typical 10–20 KB/s (peak ~40 KB/s); DDS retrieval bursts to ~900 kB/s |
| Latency (baseline actions) | ≈ 7.5 s (light) / 11 s (medium) per scheduled action — mostly size-independent |
| Latency outliers | file-storage upload 100 s; forum comment 95 s; wall comment 80 s; voting 75 s |
| Hop count (PAST) | mean 1; store/retrieve normally < 1 s |
| Tail latencies | up to 91 s store (2,000 nodes, RF 4), 41.95 s retrieve (2,000 nodes, RF 16) — distributed DDS cost |
The replication-factor trade-off (most reusable result)
Replication factor RF is coupled to the Pastry leaf set (RF ≈ leafset/2 + 1;
rice.freepastry/freepastry.params / LibreSocialConfig.txt). Sweeps at 100/200/1,000/2,000
nodes with RF 4/16/32:
- Storage used grows roughly linearly with RF (~25 MB at RF 4 → ~68 MB mean, ~153–191 MB peaks at RF 32).
- Mean message rates fall with higher RF (e.g. 15.19 → 7.11 msg/s) while RAM rises.
- Effect is non-linear and size-dependent: high RF hurts small networks, low RF hurts large ones. Khitman's runs nominate RF ≈ 16 as the best compromise up to ~2,000 nodes (RF 32 produced inconsistent runs incl. one failed 1,000-node test).
- Practical guidance kept in 2026: pick RF per expected network size via the performance–cost table in §5.4.2 of the 2020 paper; never test k > N (the 2025 S3 sweep had to skip k = 3/5 at N = 2, see revival & build).
Churn cost
Churn is measurably expensive but survivable: in the final churn step mean send traffic rose to ~20 KB/s (from ~900 B/s) and mean messages per node to ~58,000; per-node DDS data grew to ~52 MB instead of shrinking. Older runs localized the visible failure modes:
- 2013: long-term run died after ~25 h (1,500 min) with a queue overflow in FreePastry's
PriorityTransportLayer; friend requests over a master-side RMI name lookup hung nodes under churn. - 2014: with churn, 600 (10 instances) / ~1,900–2,000 (20 instances) lost messages (~0.5–1.7 %);
without churn, zero crashes and zero loss; long-term extended to 5,300 min (3.67 days) before
the same queue overflow (monitoring
historySizeleak fixed on the way). - 2020: randomization bug — with
enableAutomaticLogout=trueall nodes failed ~2.5 h after plan start (algorithm bug, not overlay bug); leaf set/routing table still adapted, proving FreePastry robustness against internal errors.
Modern-hardware baseline (2025)
Same system on one current host: memory median 846 MB/node, 0 exceptions after fixes, lookup 11.5 ms, PAST store 0.048 s, overlay 65.6 KB/s at N = 2 — see revival & build. Useful as regression anchor; not comparable to the 2020 multi-host numbers (no WAN latency, no churn, tiny N).
What has not been measured (open as of 2026-11)
- WAN-latency overlay behavior; inter-datacenter churn.
- Measurements on Java 17 / Gradle build (2026) — regression vs. the 2020 tables is open work (docker-cluster-testing blog).
- DDS structure operation costs isolated per structure type (trees vs. lists) — see DDS internals.
- Security cost model (ECC overhead per operation) — see access control.
Sources (chronological)
- Liu 2013 — first benchmark suite (25/100 instances), long-term 25 h, alpha test.
- Peng 2014 — 10/20 instances, churn loss quantified, 3.67-day run.
- Schulte 2015 · Kaneirya 2015 — framework follow-up studies (see summaries for scope).
- Dlikman 2018 — master thesis feeding the 2020 evaluation.
- Khitman 2020 — HILBERT runs, RF sweeps, 2,000-node scaling.
- Masinde et al. 2020 (Future Internet 12(9):140) — authoritative consolidated benchmark.
- Masinde 2020 (dissertation chapters) — full tables and test-environment description.
- Imam 2025 — modern-hardware regression baseline.