The InterPlanetary File System (IPFS) can be used to provide files in a more efficient and distributed way than HTTP.
Our filesystem repo already has the go-ipfs client.
You use it with
ipfs daemon --init
And then you can add my Tumbleweed mirror with
zypper ar http://127.0.0.1:8080/ipns/opensuse.zq1.de./tumbleweed/repo/oss/ ipfs-oss
You can also browse the content online at
http://opensuse.zq1.de./tumbleweed/repo/oss/ . During my testing I found that the results are sometimes inappropriately cached on the Cloudflare CDN, so if you used it under this URL without the ipfs client, this might throw signature errors in zypper.
On the server side, the mirror is updated using the syncopensuse script from
https://github.com/bmwiedemann/opensusearchive and consistency of the repo is verified with checkrepo
When a complete repo was synced, dynaname updates a DNS entry to point to the new head:
> host -t txt _dnslink.opensuse.zq1.de.
_dnslink.opensuse.zq1.de is an alias for tumbleweedipfs.d.zq1.de.
tumbleweedipfs.d.zq1.de descriptive text “Last update: 2019-04-03 12:23:43 UTC”
tumbleweedipfs.d.zq1.de descriptive text “dnslink=/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE”
If you got spare bandwidth and 300 GB disk on some public server, you could also host a mirror of today’s version, simply by doing ipfs pin add QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE
This is a permalink: http://127.0.0.1:8080/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE also browsable via any public IPFS gateway. This means, it will always remain on the 20190401 version of Tumbleweed and no changes in content are possible – similar to how a git commit ID always refers to the same data.
So why did I create this IPFS mirror? That is related to my work on reproducible builds for openSUSE. There it regularly happened that published Tumbleweed binaries were built with libraries, compilers and toolchains that were no longer available in current Tumbleweed. This prevented me from verifying that the published binaries were indeed built correctly without manipulation on the OBS build workers.
Now, with this archive of rpms easily available, it was possible to verify many more Tumbleweed packages than before. And most importantly, it remains possible to independently verify even after Tumbleweed moves on to newer versions. This data is going to stay available as long as anyone pins it on a reachable server. I’m going to pin it as long as it remains relevant to me, so probably a bit until after the next full Tumbleweed rebuild – maybe 6 to 12 months.
Thus, it now is even less easy to sneak in binary backdoors during our package build process.
Both comments and pings are currently closed.
Great! I’ll give it a try.
Suggestion: instead of putting “/ipfs/” link in DNS and updating it with script, you can put once an “/ipns/” link, that references an “/ipfs/” link and can be updated.
https://docs.ipfs.io/guides/concepts/ipns/
https://docs.ipfs.io/reference/api/cli/#ipfs-name
I tried the native ipns publishing first, but found that resolving it, always took 60 seconds and was only cached for another minute or so.
You can try it with /ipns/opensuse2.zq1.de./
Sometimes it would not work at all.
I think your repo URI shouldn’t be localhost (127.0.0.1)…
🙂
That’s something to keep in mind even when developing and testing, the localhost interface is completely different than the LAN interface which in some cases can be a drastic difference (eg configuring a website’s address). And, to be even more sure it’s a good idea to publicly deploy before even starting to write the final draft so you’re working off a final, Production configuration.
Actually no. IPFS is a P2P network, quite similar to torrent. The repo URI refers to your local IPFS client.
OK, I see what’s happening. It’s not that IPFS is a P-P network but that you’re implementing a locally running proxy server to transform the network connections, and using the /ipns/ prefix to designate an ipfs name lookup.
OK, I’ll take a look at this soon! Looks interesting! I think I remember when this was first mentioned years ago in an openSUSE Technical Forums post, and there was a short discussion of its benefits and possible drawbacks at the time. Will be interested in seeing how it’s progressed.
FYSA there is IPFS Package Managers Working Group
that aims to improve IPFS experience for use cases like this
https://github.com/ipfs/package-managers
Reach out to them if you have any issues / questions / suggestions / experience reports to share – it will be really appreciated 🙂
Yesterday the IPFS mirror repo was loading correctly, but now it’s not. It just waits forever, as if no none was serving the mirror in the IPFS network.
Ignore this. It works now.
The problem with ipfs is, if your node is not accessible from the internet, it works quite slowly.
There are hickups now and then. One is that the ipfs daemon leaks memory and has to be restarted now and then.
The other seems to be that freshly added files are only announced to the world within the day.
The workaround for that is to run
#!/bin/sh
ipfs swarm connect /ip4/176.9.1.211/tcp/34001/ipfs/QmT7bZwZ3gpBMswUBjtpFhPM42ErFapAPiQHM2ELu3deqs
That sort of makes the DHT-based auto-discovery superfluous, but at least it works.