Set up collaborative pinning clusters

[lidel]

ipfs-cluster 0.12.0 shipped collaboratiove clusters, and there is a website with some demo datasets:

Collaborative clusters are public IPFS Clusters that anyone can join to help replicating and re-distributing content on the IPFS network.
– https://collab.ipfscluster.io

TODO

• figure out how to create collaborative cluster from snapshot-hashes.yml
• add Wikipedia mirrors to the list on https://collab.ipfscluster.io
• update README with instructions how to join cluster for specific language

[lidel]

@hsanjuan are you able to provide some thoughts on this?

My hope is to automate things so collaborative cluster so it gets updated every time snapshot-hashes.yml in master of this repo changes (we can change the file format, its not parsed by anything atm), but semi-automatic process could be ok for now too.

[RubenKelevra]

@lidel it's pretty simple. If you have a machine with the data pinned in ipfs, you can add the ipfs-cluster.

You need ipfs-cluster-service and ipfs-cluster-ctl.

The cluster needs to be initiated:

ipfs-cluster-service init --consensus crdt

then the cluster service needs to run continuously as daemon:

ipfs-cluster-service daemon

Note that this machine is by default the only machine able to change the data in the cluster (trusted peer). You can add other trusted peers later.

You can now go ahead and pin a CID to the cluster:

ipfs-cluster-ctl pin add [command options] <CID|Path>

There are command options avaible which defines how often the data should be pinned in the cluster etc.

I think in the end we shard the data in the cluster, meaning that the content is split into multiple pins and this pins will get a minimum and maximum number of replications.

[RubenKelevra]

When we add the current data, we should change the chunker to rabin, which allows for diff updates.

We can also set the --expire-in flag, to unpin old versions of pages and files automatically, like after a year? If someone likes to archive the older versions, than this can be done by pinning them in IPFS.

In the future we could modify the pages a bit and offer a bit of a history, like adding a link to the previous version of the file. When we update to a new version, we just add a static link which will lead to the version before.

When the cluster has automatically unpinned the file they remain available until the last IPFS node has unpinned them and cleared it out of the cache.

The most efficient way to update pins is by adding the new version with IPFS, and then trigger an update of the cluster with ipfs-cluster-ctl pin update. This will push the update to the same nodes of the cluster which had the file before pinned. This reduces the amount of data necessary to safe the new file, since the node has already the old version and can save a diff.

---

We just have to consider that certain files or pages will not be updates for a long time:

• So automatically unpinning them isn't a great solution if we cannot trigger a reset of the counter.
• I don't know if ipfs-cluster-ctl pin update will reset the auto unpin counter as well.

[RubenKelevra]

Cluster setup should better wait a while, until the development of IPFS has advanced a bit:

ipfs/kubo#6841 (comment)

If we get a bunch of people together to be part of the project, it's best to have already all features we like to have at this point, avoiding that people have to upgrade or we to restore the already published informations.

@lidel hope you share the opinion :)

[lordcirth]

go-ipfs 0.5.0 is going to have a lot of changes in general, so waiting at least for that is probably sensible.
There's also the upcoming switch to Badger 2.0, which could potentially be difficult to migrate individual nodes, but doesn't affect the network as a whole.

ipfs/kubo#6776

[RubenKelevra]

@lordcirth can Badger 2.0 be compacted when running an ipfs-cluster? Since the cluster has to unpin older versions after a while for storage considerations. But everyone can pin the old versions and keep them alive in the network.

(that's my current mindset - others might think differently about this!)

[lordcirth]

You mean garbage collection? I expect it should behave pretty much like Badger 1.0 did from the perspective of the user, and have all the same features.

[RubenKelevra]

Garbage collection

Well, no - the compaction of badger itself below IPFS. Since for v1 there was some issues reported, for example this one:

salesforce/sloop#90

[hsanjuan]

@lidel since these are BIG (how big btw?), I'll need to deploy a big machine. Then we can automate adding the pin from the CI on this repo. I'll let you know

[RubenKelevra]

@hsanjuan we could add each page/picture as individual pin with a minimum and maximum amount of replications. This way not each cluster member need to hold the full amount of data and for each new pin the data is stored on the cluster members which got the most free space available.

The en version of 2017 is just shy of 1 TB (don't know the exact number, but it's quite big). If you consider the grow rate and that we want to hold the 13 languages with the most articles (see #63 ) we need considerably more storage as a single machine can easily hold (servers with a large amounts of harddrive slots gets pretty expensive fast).