Put a million rows from flat file to blockchain

Question

I have a flat file with each row containing records such as

AX101 12345 PQR 101
AX102 18390 XYZ 091
AX101 81249 PQR 871

My setup has a few machines connected in a network on Hyperledger with vagrant and docker.

Test 1 

For this test, I am running just one validating peer that should read the flat file with over million such rows and put each row as a new block. The intention is to test the speed for doing so. What could be best way to achieve this?

Approach 1 : The code could be in Golang sitting inside a smart contract
Approach 2 : A separate "reader" in another language that sends the data to the validating peer via APIs (would be slower, I think)

Test 2 

Once (hopefully) when all the data is on blockchain, I need to parse all entries for say AX101, speed is not a concern here but picking up all entries is.

Any pointers would be helpful!

Answer 1

Let's say you are talking about Hyperledger fabric.

You cannot add the data to the database without writing a chaincode. Data in the blockchain is owned by the chaincode that created it, and can only be accessed by the chaincode itself. There is no concept of shared data. You cannot simply write data into the blockchain to update it. So you need to take Approach 2, and send the data to a chaincode via its "add new record" method. And, this method will be created by you. 

You will also have to create a query method for your chaincode. Using this query method you will be able to control the speed of parsing in Test 2 as you store the data. 

Documentation for the database APIs which are available to the chaincode: https://godoc.org/github.com/hyperledger/fabric/core/chaincode/shim

Want to make your career in Blockchain? Enroll in Blockchain Course to acquire the essential skills.