How decentralized data storage works
THE PARTICIPANTS:
The MasterNodes
The MasterNodes are powerful and specialized nodes in the network that play an important role in file transfer. Their main purpose is to ensure a fast and efficient upload process for users. As soon as a user uploads a file, it is first sent to the MasterNodes. There, the file is compressed to optimize transfer time and storage space requirements. Compression is performed using advanced algorithms to achieve the most efficient compression possible without data loss. This step helps improve transmission speed while reducing resource requirements. After compression, the file is forwarded to the node balancer, which is responsible for further processing and distribution to the nodes in the network. The MasterNodes thus serve as a central hub for the upload process and play an important role in optimizing file transfer. By using dedicated MasterNodes, one can ensure that the upload process is fast and reliable, regardless of the number of users and file size. The MasterNodes are powerful and highly available to ensure a seamless user experience.
In addition, MasterNodes are also responsible for checking files for viruses and applying security measures to ensure the integrity of the transferred data. They play a crucial role in protecting the privacy and security of user data. Overall, the MasterNodes are an essential part of the decentralized network, as they enable fast and efficient file transfer while ensuring the security and integrity of the data. Through their performance and specialization, they contribute significantly to the smooth functioning of the decentral networks. After encryption, each part of the file is assigned a hash code. This code is then used by the indexer to track the location of each part throughout the platform infrastructure. This ensures that the file parts can always be found and correctly assembled if the user wishes to retrieve the file at a later time.
Node Balancer
The Node Balancer has an extensive database of all registered nodes on the network. It regularly indexes and monitors the available nodes to ensure that they meet the requirements and operate reliably. The nodes are evaluated and classified according to criteria such as availability, response time/ping, and speed of data transfer. When the MasterNode reports a file for storage, the Node Balancer contacts the nodes to initiate the upload process. In doing so, it takes into account the current load of the nodes and ensures an even distribution of upload requests in the network. This load distribution ensures that no node is overloaded and that an optimal transmission speed is achieved.
The node balancer also acts as a source of information for other components of the network. It stores the current storage capacities of the nodes to ensure that the available storage space is used efficiently. It also regularly updates the status of the nodes [Mainnet / Testnet] to ensure that only reliable and active nodes participate in the file transfer.
Indexer
The Indexer is a powerful database server in the provider's network that plays a central role in managing and linking data to the corresponding nodes. Its main task is to link hashes of files to their corresponding nodes in order to know which data is located on which nodes and in which storage blocks. The indexer acts as a kind of intermediary between the users and the nodes by managing and monitoring the file distribution information. It stores the mapping of the hashes to the corresponding nodes in its database, allowing efficient access to the stored files.
The indexer regularly checks whether all indexed files are available and accessible on the nodes. If a node fails and a file is no longer available, the indexer automatically takes action. It distributes the file in question to other available nodes to ensure that the data remains accessible. At the same time, the wrong index of the failed node is deleted and the node itself is downgraded to the testnet to check its reachability. Another important function of the indexer is the verification of data integrity. By using checksums, the indexer regularly compares the files stored on the nodes with the original hashes. If a file has been modified or tampered with, the node in question is blacklisted and the node’s stake is locked. This mechanism ensures that the data remains unaltered and that only trusted nodes participate in the network.
Node
The nodes form the most important basis for the decentralized network and provide the storage space for the uploaded files. They serve as data storage and allow users to store and access their files securely and reliably on the network. Each node provides packets of GB of storage to cover the need for different file sizes and types. The nodes are managed by the node operators and meet certain requirements, including a stable Internet connection around the clock and a minimum data transfer rate i.e. 5 MB per second. The nodes actively participate in the network and offer their storage capacity for file transfer. They receive upload requests from the node balancer and securely store the uploaded files. The files are organized into special storage blocks and have to ensure data integrity.
As participants in the network, the nodes are subject to various evaluation criteria, such as availability, response time/ping, and speed of data transfer. These criteria are monitored and evaluated by the node balancer to ensure that the nodes operate reliably and meet the requirements of the network.
The nodes in the network are classified into different measures: Checknet, Testnet, and Mainnet. In Checknet, all nodes are registered, while Testnet and Mainnet must meet stricter criteria to be classified as trusted and high-quality nodes. Nodes are responsible for keeping uploaded files secure and responding to download requests.
Blacklist
The blacklist reviews reported data and perform rechecks to determine if the data has been altered. It aims to detect technical changes and exclude old versions. This helps to ensure that the stored files are unaltered and available in their current version. If a node is found to have intentionally modified the file, the blacklist takes action. The stake of the node in question is shortened (lashed) and the node is permanently blacklisted. These sanctions are used to protect the network from fraudulent activity and to ensure that only trusted nodes participate in the network. The lashed stake is sent to the reward pool and thus increases the total rewards distributed to the node operators. This mechanism creates incentives for node operators to follow the rules and operate their nodes properly.
It is important to note that the blacklist is not based on IP addresses but on hardware IDs. This ensures that nodes with a new IP address cannot simply reconnect to the network. This reduces the possibility that an operator blacklisted for misbehavior will bypass the system by logging in with new identities. The decision to remove nodes from the blacklist rests with the administration of the platform.
Node operators who end up on the blacklist have the option to request a personal review. This allows them to explain their case and, if necessary, remove their node from the blacklist, provided they meet the required conditions and can restore confidence in its integrity. Overall, the blacklist plays an important role in maintaining data integrity and compliance with network rules. Through its actions, it protects the network from fraudulent activities and contributes to the security and trustworthiness of the system.
THE PROCESSES:
Upload
The service starts with the user uploading files via the browser or an app. These files first land on a MasterNode. This MasterNode has the task of compressing the uploaded files and splitting them into ten fragments. Once the compression process is complete, the MasterNode informs the node balancer that the data is ready for transmission. The node balancer then comes into play. As soon as it receives a request to transmit data from a MasterNode, it begins to randomly select free memory locations and allocate them to the data fragments. The data is stored in the mainnet and in the testnet. If the amount of data exceeds the available storage space of the available nodes, the balancer falls back on external Web3 platforms to store the excess data there. Finally, the nodes come into play. Once they receive an allocation from the node balancer, they download the allocated data and store it in the designated locations. Once the download is complete, the node informs the indexer that the upload was successful. In this way, the system ensures that all uploaded data is properly stored and indexed, with extensive redundancy helping to ensure data integrity and continuous availability.
Download
When a user wants to download data, the process starts with a request to the indexer, which is triggered by the call to the website. The indexer searches for the requested data in its index and contacts all associated nodes. In this process, the nodes with the best response time are determined in real-time and the download process is started at the user’s site. During this process, the data packets are merged so that the user ends up with a single zip file.
In the process of revolutionizing the big data industry and ensuring data integrity, some companies have joined the blockchain ecosystem to protect user data from possible abuse by centralized organizations.
Discover more about WeSendit
WeSendit is an established data transfer company based in Zug, also known as the “Swiss Crypto Valley”.
Website: https://wesendit.io
Twitter Linkedin Medium