Trapped library study
Methodology
How the sample was drawn, what each package was scored against, and where the design is weak.
Sample frame
Public-sector eLearning packages that were downloadable as files at the time of collection. This is a convenience sample, not a random one, and that is the single most important limitation of the study.
The constraint is structural: you cannot audit a package you cannot obtain. Most public-sector eLearning is streamed rather than distributed, so the population we can reach is systematically different from the population that exists: newer, more deliberately packaged, better maintained.
This biases every figure downward. We treat the results as a floor throughout.
What each package was scored against
| Dimension | Coded as | Rule |
|---|---|---|
| Editable source present | yes / no | Whether the package contains project files that could be re-opened in an authoring tool, rather than only published output |
| Runtime status | live / dead | Whether the package depends on a plug-in or runtime no longer available in any shipping browser |
| Standard | 1.1 / 1.2 / 2004 / AICC / cmi5 / xAPI / none | From the manifest, with the namespace as a tiebreak |
| Broken references | count | Assets referenced by the package and absent from it |
| Hardcoded paths | count | Absolute paths and localhost references that will not survive a move |
| Tracking claim vs. calls | match / mismatch | Whether the API calls present match what the manifest declares |
Harness
Written twice, independently, in two different ways, and the outputs compared rule for rule. Where they disagreed we treated it as a defect in the harness rather than a finding, and fixed it before proceeding.
Known limitations
Convenience sample. Downloadable packages are not representative. Stated above and worth restating.
n=73 is small. Per-publisher figures rest on as few as 5 packages and should not be read as stable estimates of any organization's library.
Automated coding only. No human coder has independently scored a subset, so we cannot report inter-rater reliability. This is the largest outstanding gap.
"No editable source" is a property of the distributed package, not of the organization. A publisher may hold source files internally that were never distributed. We are measuring what can be recovered from the artifact, which is the situation you are in when the source is gone.