Testing Methodology
Deep-dive page: how LibreSocial has been tested from 2011 to 2026 — the Tester plugin, test-case design, run discipline, and what changed over time.
Supersession note: the description of the current testing bed is Imam 2025 (Reviving LibreSocial) plus the 2026 work items; the method of automated Tester-driven evaluation is documented best by Peng & Balabhadruni 2014 and Khitman 2020; the metrics vocabulary was introduced by Liu 2013 and corrected in 2014 (see below) — the 2014 naming supersedes the 2013 naming.
The Tester plugin (com.p2pframework.tester)
LibreSocial ships an integrated system-test framework, not unit tests:
- Master/slave roles: one node runs the Tester as master (issues the test plan), all other nodes run it as slave (execute received test commands). Every action is a real OSGi/overlay operation on a real node.
- Test cases are small bundles of preconditions + actions + checks. Over time the catalogue grew to 56 test cases with 27 preconditions (counted in 2020, Khitman).
- Test plan: a time-scheduled script, e.g. a line
5x TestCaseCreateCalendarEvent 2mruns the case five times in a two-minute slot (synthesis test), while a random test draws cases at random intervals per node. - Driving the system: OSGi console commands (e.g.
randomTest) configure runs per node.
Maintenance debt (the recurring finding)
The catalogue repeatedly rotted when the framework changed:
| Year | State found | Evidence |
|---|---|---|
| 2014 | only 7 of 26 test cases still worked after the 2013 framework redesign; 19 were deleted/rewritten | Peng 2014 |
| 2020 | 28 of 56 (50 %) broken and repaired | Khitman 2020 |
| 2025 | catalogue dormant during dormancy; revival re-enabled core messaging/wall flows | Imam 2025 |
| 2026 | rebuild of automated system tests is an open goal | blog: test safety net, tester component |
Conclusion drawn in the docs repeatedly: test cases must be independent (no shared global
state in ParameterGenerator) and must be part of every framework refactor.
Metrics model (SkyEye-backed)
Tests are evaluated through the monitoring over the overlay itself (SkyEye, monitoring component):
- Metric tuples: unit/value/aggregation/source/name; sampled globally every 5 s into SQL (2020 test environment).
- Vocabulary introduced 2013:
FRAMEWORK_SENT/RECEIVED_PLUGIN*,..._DDS,STORAGE_DATA_*,MESSAGE_DATA_*,PASTRY_DATA_*, plus per-plugin social metrics;combinedRoundsaveraging at the SkyEye root smooths global curves (Liu 2013). - 2014 corrections (supersede 2013): cache metrics dropped;
STORAGE_DATA_SENT/RECEIVEDrenamed toSTORED/RETRIEVED; plugin data prefixFRAMEWORK_→STORAGEDISPATCHER_STORAGE_; afterwards sent ≡ received globally (Peng 2014). - 2020 additions:
NETWORK_LEAFSET_SIZE,NETWORK_ROUTING_TABLE_SIZE. - Rejected: measuring object sizes inside PAST/Scribe — the instrumentation delayed nodes so much they failed to join. Lesson: measurement must not change participation.
Run discipline (2025/2026 rules, codified from earlier pain)
From the revival runbook (Imam 2025):
- exactly one Tester/monitoring controller per experiment (parallel masters interfere — also flagged as future work in Khitman 2020: a unique master–slave assignment is needed);
- warm-up ≥ 5 min before collecting; fixed random seeds; identical configs; NTP sync;
- never change plugins mid-run (bundle restarts invalidate the run);
- health gates: node count stable, exceptions ≈ 0, sent ≈ received;
- evidence layout
runs/<id>/{configs,metrics,plots,README.txt}keeps every run reproducible.
Scenario families used since 2013
- Synthesis test — fully scheduled test plan; validates every feature works.
- Random test — randomized user behavior (later: algorithm synthesized from a >35,000-user study); with/without churn (logout). Churn is where defects historically appeared (friend-request hangs 2013; message loss 2014; randomization bug 2020).
- Long-term test — stability over days; 2013: ~25 h until FreePastry queue overflow; 2014: 5,300 min (3.67 days) with the same final cause (see bug catalogue).
- Scale/replication sweeps — see evaluation & benchmarks.
- Alpha test with real users — 2013: documented deployment (10 VMs, Xvfb) but failed for social reasons (2 accounts); re-planned as Docker-cluster user studies in 2026 (blog).
Sources
- Liu 2013, master thesis Paderborn — test families, metric vocabulary, long-term and alpha results.
- Peng & Balabhadruni 2014, master thesis Paderborn — test-case repair, corrected metrics, churn results.
- Khitman 2020, thesis HH Düsseldorf — 56 test cases, HILBERT cluster, RF studies.
- Masinde et al. 2020 (Future Internet) — the consolidated benchmark methodology.
- Imam 2025, master thesis TH Bingen — current runbook and discipline rules.