Help request to compute the amount of disk space and bandwidth wasted by useless published items?

[ericmorand]

Select Topic Area

Question

Body

Dear maintainers,

I'd like to conduct a survey about the amount of disk space and bandwidth wasted on the npm registry by items that are never used by package consumers - i.e. items that are published in the registry and are never imported by projects and / or products, because they technically can't be imported or because they are not supposed to be.

Typically, the following items are a waste of resources, hanging around, only taking space in the registry and in the client systems:

• Source files, sometimes with inline source maps
• Source maps
• Project documentations (not package documentation)
• Artefacts expected to be deployed to CDNs which are published because CDN tools fetch them from the npm registry
• package.json attributes that are never considered by Node.js resolution algorithm or npm package homepage generator - e.g scripts, devDependencies, files, third-party project attributes like prettier one...

If you need an example of what I mean, svgicons2svgfont is a good one:

https://www.npmjs.com/package/svgicons2svgfont?activeTab=code

The package comes with sources, source maps, tests, and a package.json that holds 190 line when only around 80-90 are actually needed to consume the package. It consumes 425KB on my computer, when only 280KB are reachable and required to make it work. Around 30% of its size is just wasted.

My objective is to compute the total amount of space occupied by this uneeded published information, accross all packages (public and, ideally, private) in order to write a paper about it and, hopefully, raise awareness and initiate a change in both npm and client sides.

Is there a way to achieve such a thing, using a public API, or maybe by having access to some data that npm's maintainer are likely to possess? I don't care about the content - except for package.json files. I'm only interested in file types (path and extension) and their size.

Thanks in advance,

Eric.

[Aqib121201]

There’s no public way to get that data directly. NPM doesn’t expose file by file sizes for every package.What you can do is grab dist.unpackedSize from the registry API and then use something like jsDelivr to list what’s inside each package. From there, you can check how much of it is stuff like .map, test, or src files and figure out roughly what percentage is junk.If you want a global estimate, just sample a few thousand popular packages, calculate the waste ratio, and scale it up. For anything registry wide, you’d need npm’s internal data they don’t publish that.

[ericmorand]

What you can do is grab dist.unpackedSize from the registry API

That would be a good start. My issue is that I'm not even able to find the documentation for the API. The official one only list one endpoint that allows to get a token:

https://api-docs.npmjs.com

[Danindu05]

You can estimate that pretty reliably using the npm registry metadata and a small crawler. The registry.npmjs.org/ endpoint exposes tarball URLs and versions, so you can fetch the dist.unpackedSize field from each manifest — that’s the size after unpacking. Combine that across all versions, and you’ll get a close estimate of wasted disk space.

For a quick proof of concept, try scripting it with Node.js + got or axios to pull package metadata and log unpacked sizes per version. You can filter out .md, .map, and test files before summing if you only want production code.

If you’re analyzing bandwidth too, you might need to approximate based on tarball sizes (dist.size) since download stats aren’t public.

[ericmorand]

If you’re analyzing bandwidth too, you might need to approximate based on tarball sizes (dist.size) since download stats aren’t public.

Actually, they do expose the download stats from their public API that serves HTML representations - aka the www API. The data structure is not documented but it can be inferred.

Anyway, thank you. I may be able to get what I need by mixing all their APIs together (the registry ones, if I happen to find their documentation, and the www ones). It will require quite a few requests to get the job done, but it should work.

[Danindu05]

Thanks — that’s absolutely right. The best way to estimate both disk space and bandwidth waste is by combining data from the npm registry and website APIs, then filtering out unneeded assets.

Disk space estimation
You can use the dist.unpackedSize field from each version’s metadata to estimate how much space each package occupies after extraction. Summing this across all versions gives a close measure of total published size.
To isolate useful vs wasted data, you can unpack a few tarballs (using tar, tar-stream, or similar in Node.js) and exclude files such as .md, .map, and /test directories — leaving only production code.

Bandwidth approximation
The dist.size field represents the compressed tarball size. While npm doesn’t expose raw download stats in the registry API, their public website API (the www one) does include download counts in the HTML. These can be parsed to approximate bandwidth usage:
estimated_bandwidth = dist.size × downloadCount.

Implementation hint
You can automate this with Node.js and axios or got:

Fetch metadata from https://registry.npmjs.org/.

Extract dist.unpackedSize and dist.size for each version.

Query the npm website API for download stats.

Aggregate results and compute waste ratios (e.g., % of non-production files).

That should yield a reliable picture of both wasted disk space and bandwidth, even at scale — without requiring any private data access.