Step by step guide to deploy a private IPFS cluster on Ubuntu

IPFS stands for InterPlanetary File System which is a protocol and network somewhat similar to BitTorrent. If you want to know what IPFS is, go to their official website for a demo - https://ipfs.io/

In this article, I will be attempting to create a private IPFS network which means that it won't connect to the public IPFS network and the data you put on the nodes of this cluster will be localised to your private network. So let's start.


Step 1 - Install IPFS using the command line
  • Before you start with this, please go to https://dist.ipfs.io/#go-ipfs and copy the link of the latest Go implementation of IPFS. At the time of writing this, the latest version is Version v0.4.13 for Linux 64bit.
  • Now we will install the latest version of IPFS. For this first need to install Go and then we will install IPFS. Installing the distribution won't start the IPFS filesystem so don't worry about it connecting it to the public network. Execute these commands one by one:
    • sudo apt-get update
    • sudo apt-get install golang-go -y
    • wget https://dist.ipfs.io/go-ipfs/v0.4.13/go-ipfs_v0.4.13_linux-amd64.tar.gz
    • tar xvfz go-ipfs_v0.4.13_linux-amd64.tar.gz
    • sudo mv go-ipfs/ipfs /usr/local/bin/ipfs
  • To verify your installation is working, type "ipfs version. You should get an output like this:

    ankit@ankit-K54C:~$ ipfs version
    ipfs version 0.4.13
Step 2 - Initialize the first and second private node
The participants of public network of IPFS usually have just one IPFS node on their machines since that is all they require to participate. However, in our case we will deploy 2 IPFS nodes on the same machine. You can do it on 2 different machines too which are on the same network and are able to ping each other. By default, IPFS is initialized in a hidden directory in your user home directory - ~/.ipfs. Since we will be deploying 2 nodes on the same machine, we will be doing so in two difference directories - ~/.ipfs1 and ~/.ipfs2. You can do it in the default ~/.ipfs and at any other location or if you are using different machines then you don't need to worry about the directories. Let start the deployment:

ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs1 ipfs init
initializing IPFS node at /home/ankit/.ipfs1
generating 2048-bit RSA keypair...done
peer identity: QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83
to get started, enter:

ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme
ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs2 ipfs init
initializing IPFS node at /home/ankit/.ipfs2
generating 2048-bit RSA keypair...done
peer identity: QmToxF88PoBFEkbSLzZ3WLePJn6tvGAM8ZDrJuPrhGhj7P
to get started, enter:

ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme



You should be able to see two directories ~/.ipfs1 and ~/.ipfs2

ankit@ankit-K54C:~$ ls .ipfs*
.ipfs1:
blocks config datastore datastore_spec keystore version

.ipfs2:
blocks config datastore datastore_spec keystore version


If you are getting the same output then you have successfully initialized 2 IPFS nodes. If you want you can initialize a number of nodes if you want.

Creating a Private network

Till now we have just created your IPFS nodes which are one step behind in joining the public IPFS network. However, since we are creating a private IPFS cluster we need to use the still-in-experimental-stage features of IPFS to create a private network.
Create swarm.key file to enable private network feature of IPFS
We will start with adding a key called the swarm.key that tells the IPFS nodes that they will be a part of a private network which all will share this swarm.key file.

There is an application which generates this swarm file but for that, you need to have Go language installed on your system. Install Go before proceeding with the next step.

Once you have Go installed, run the following command to install the swarm.key generation utility:

go get -u github.com/Kubuxu/go-ipfs-swarm-key-gen/ipfs-swarm-key-gen

Now run this utility in one of your node like this:

ipfs-swarm-key-gen > ~/.ipfs1/swarm.key

Copy the file generated to the IPFS directory of each node.

Bootstraping IPFS node
IPFS requires one or more bootstrap nodes which are used by the IPFS daemon to learn about the other nodes that are present in the network. In case of a private network, we need to set up our own bootstrap node since we won't be connecting to the public network node and hence won't have access to the bootstrap nodes of IPFS developers. A bootstrap node is basically the same IPFS node just with the added feature of acting as the bootstrap. There is no separate installation required for this but just a config file entry.

First of all, you need to remove the default entries of bootstrap nodes from all the nodes you have created. Do this by using this command:

ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs1 ipfs bootstrap rm --all
/dnsaddr/bootstrap.libp2p.io/ipfs/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
/dnsaddr/bootstrap.libp2p.io/ipfs/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa
/dnsaddr/bootstrap.libp2p.io/ipfs/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb
/dnsaddr/bootstrap.libp2p.io/ipfs/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt
/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM
/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
/ip6/2400:6180:0:d0::151:6001/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip6/2604:a880:1:20::203:d001/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM
/ip6/2604:a880:800:10::4a:5001/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
ankit@ankit-K54C:~$
ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs2 ipfs bootstrap rm --all
/dnsaddr/bootstrap.libp2p.io/ipfs/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
/dnsaddr/bootstrap.libp2p.io/ipfs/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa
/dnsaddr/bootstrap.libp2p.io/ipfs/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb
/dnsaddr/bootstrap.libp2p.io/ipfs/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt
/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM
/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
/ip6/2400:6180:0:d0::151:6001/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip6/2604:a880:1:20::203:d001/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM
/ip6/2604:a880:800:10::4a:5001/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd



Now add the hash address of your bootnode to each of the nodes including the bootnode.

ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs1 ipfs bootstrap add /ip4/127.0.0.1/tcp/4001/ipfs/QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83

/ip4/127.0.0.1/tcp/4001/ipfs/QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83
ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs2 ipfs bootstrap add /ip4/127.0.0.1/tcp/4001/ipfs/QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83


/ip4/127.0.0.1/tcp/4001/ipfs/QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83



The IP part - 127.0.0.1 will be changed to your machine IP in case you are using different machines. The last part is the node hash key which is generated when you create your node. You can see it above where it shows "peer identity: QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83". Run this for all of your nodes.

Assigning port number for gateway and configuring IP for communication
Inside the .ipfs folder, our as in my case .ipfs1 folder, there is a "config" file. It contains a lot of details including the network details on which our IPFS nodes will work on.
Open this config file and find "Addresses". It will look like this:
 
"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5001",
"Announce": [],
"Gateway": "/ip4/127.0.0.1/tcp/8080",
"NoAnnounce": [],
"Swarm": [
"/ip4/0.0.0.0/tcp/4001",
"/ip6/::/tcp/4001"
]
},


The IP mentioned in the API key is the one on which IPFS will bind on for communication. If you are using different machines then mention the IP address of your computer. In our case, since we are using localhost/127.0.0.1, we will just select different ports for each of the machines and will leave IP as it is. For the first node there is no need to change anything, but for the second, we will increment each of the port like below:

"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5002",
"Announce": [],
"Gateway": "/ip4/127.0.0.1/tcp/8081",
"NoAnnounce": [],
"Swarm": [
"/ip4/0.0.0.0/tcp/4002",
"/ip6/::/tcp/4002"
]
},



Start the node and test!

Now we are done with all the configurations and now it is time to start both the nodes to see if everything went fine. To do this, open two consoles since we need to keep both the nodes running. You can do this by sending the process background too if you prefer that.
We will use an environment variable to make sure that just in case if there is some mistake in our configuration and the private network is not fully configured, the nodes don't connect to the public IPFS network and the daemons just fail.
The environment variable for the same is "LIBP2P_FORCE_PNET" and to start the IPFS nodes you just need to start the daemon using "ipfs daemon" command. In our case the command and the output look like this:

ankit@ankit-K54C:~$ export LIBP2P_FORCE_PNET=1 && IPFS_PATH=~/.ipfs1 ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm is limited to private network of peers with the swarm key
Swarm key fingerprint: 44634569ba70677da9e8950c39a5da9a
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/172.17.0.1/tcp/4001
Swarm listening on /ip4/172.18.0.1/tcp/4001
Swarm listening on /ip4/192.168.0.100/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit/ipfs/QmQVvZEmvjhYgsyEC7NvMn8EWf131EcgTXFFJQYGSz4Y83
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/172.17.0.1/tcp/4001
Swarm announcing /ip4/172.18.0.1/tcp/4001
Swarm announcing /ip4/192.168.0.100/tcp/4001
Swarm announcing /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready


ankit@ankit-K54C:~$ export LIBP2P_FORCE_PNET=1 && IPFS_PATH=~/.ipfs2 ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm is limited to private network of peers with the swarm key
Swarm key fingerprint: 44634569ba70677da9e8950c39a5da9a
Swarm listening on /ip4/127.0.0.1/tcp/4002
Swarm listening on /ip4/172.17.0.1/tcp/4002
Swarm listening on /ip4/172.18.0.1/tcp/4002
Swarm listening on /ip4/192.168.0.100/tcp/4002
Swarm listening on /ip6/::1/tcp/4002
Swarm listening on /p2p-circuit/ipfs/QmToxF88PoBFEkbSLzZ3WLePJn6tvGAM8ZDrJuPrhGhj7P
Swarm announcing /ip4/127.0.0.1/tcp/4002
Swarm announcing /ip4/172.17.0.1/tcp/4002
Swarm announcing /ip4/172.18.0.1/tcp/4002
Swarm announcing /ip4/192.168.0.100/tcp/4002
Swarm announcing /ip6/::1/tcp/4002
API server listening on /ip4/127.0.0.1/tcp/5002
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8081
Daemon is ready


Do note the message log above stating "Swarm is limited to the private network of peers with the swarm key" which means that our private network is working perfectly. Now let's add the file from ipfs1 and try to access it from ipfs2

ankit@ankit-K54C:~$ mkdir ankit
ankit@ankit-K54C:~$ cd ankit/
ankit@ankit-K54C:~/ankit$ echo hello > file1.txt
ankit@ankit-K54C:~/ankit$ IPFS_PATH=~/.ipfs1 ipfs add file1.txt
added QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN file1.txt
ankit@ankit-K54C:~/ankit$ IPFS_PATH=~/.ipfs2 ipfs cat file1.txt
Error: invalid 'ipfs ref' path
ankit@ankit-K54C:~/ankit$ IPFS_PATH=~/.ipfs2 ipfs cat QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
hello


Note that we have added the file to ipfs1 which gave us back a hash. Now when you try to access the same file from ipfs2 with the file name, it won't but work but replace that with the hash and you can see the contents of the file. Now we will try to access the file from the browser.

You can access the file by using the gateway address like this:
http://127.0.0.1:8080/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
OR
http://127.0.0.1:8081/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
You should be able to see the content of the file on browser.

You can similarly add a while directory too.
ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs1 ipfs add ankit/ -r
added QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN ankit/file1.txt
added QmUs71TZY6Vy47vYrgu5baJwfGQNpMQKeHE2eAn3k2r5kv ankit


and when you try to access it you will see the hash of the file inside if any along with the file name:
ankit@ankit-K54C:~$ IPFS_PATH=~/.ipfs2 ipfs ls QmUs71TZY6Vy47vYrgu5baJwfGQNpMQKeHE2eAn3k2r5kv
QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN 14 file1.txt



Popular posts from this blog

Comparison between Airtel Digital and Sun Direct TV DTH providers

Comparison between Tata Sky vs Reliance Big TV DTH service providers

How to configure your router for ION broadband