Nifi for database migration

Question

Why would nifi be a good use case for database migration if all it does is sending the same data over and over again?(I have tried to extract data from a database and putting them into a JSON file I was seeing multiple entries of the same tuple.) Wouldn't that be a waste of computing resources?

If I just want to migrate the database once and sometimes update the changed columns only, is nifi still a good tool to use?

Answer 1

When you are doing database migration, what is important is which database you want to migrate to and which database you are migrating from. If, say you want to transfer a bulk of data efficiently, it is not recommended to go for Nifi, but rather the ones like Sqoop. 

Apache NiFi is a tool to build a dataflow pipeline, it is just the right tool for Internet of Things (iOT), Internet of Everything (IoE) or any data in motion using inbuilt connectors (also known as processors in NiFi world) so it can Get/Put data from/to HDFS, Hive, RDBMS, Kafka and many others. It also has a really cool and user friendly interface which can be used to build the dataflow in minutes by dragging and dropping the processors. Nifi is great for just about any simple event processing use case, but for a large scale RDBMS ingest it is not so useful.

Learn more about NiFi dataflow automation and streaming analytics by joining Apache NiFi Training and tutorial.

Answer 2

It all depends on which database you want to migrate from/to which environments. You can use NiFi to do an import as well, using processors such as ExecuteSQL, QueryDatabaseTable, GenerateTableFetch... They all work with JDBC connectors, so depending on if your database supports this, you could opt for this as well. If you want more information regarding the same, refer to the following video tutorial: