Splitting Large Files With Nifi is quite easy when you know the right approach to do that. In this article I will show you step by step on how you can split a large file to smaller files.
See the complete Video or scroll down to continue reading.
Thumb Rule is, You should always use split record processor to split your total records divided by 10. So for example, If you have a file with 1,000,000 records and you want final output with only 10 records per FlowFile. Then you should first split it to 100,000 and then split it to 10,000 and then split it to 1000 and then split it to 100 and then finally to 10 records.
Input FlowFile is having 10,000 records per FlowFile. This is processed by NiFi Split Record Processors and Splitted into from 10,000 records to 1,000 records, then to 100 records and then finally to 10 records per FlowFile
Final Output FlowFile is having only 10 Records Per FlowFile. You see how easy it is to split big files to smaller files with this approach. Also it gives you total control on the number of records you want to see in each file.
Download The Template Here
I hope you liked Splitting Large Files With Nifi article. You can read more about Apache NiFi Here.
- Installing Apache NiFi Cluster on LinuxInstalling Apache NiFi Cluster on Linux is pretty straightforward, to be honest. If you follow the steps mentioned in this article, you can set… Read more: Installing Apache NiFi Cluster on Linux
- Installing NiFi Cluster on Windows ServerIn this article I am going to show you steps for Installing NiFi Cluster on Windows Server. Please follow all the steps as mentioned… Read more: Installing NiFi Cluster on Windows Server
- Apache NiFi Cluster in DetailWhat’s a cluster? To learn about Apache NiFi Cluster in Detail first you should understand about cluster. A cluster is a set of loosely… Read more: Apache NiFi Cluster in Detail
- Splitting Large Files to Smaller Files in NiFiSplitting Large Files With Nifi is quite easy when you know the right approach to do that. In this article I will show you… Read more: Splitting Large Files to Smaller Files in NiFi
- Apache NiFi Best PracticesThis article outlines Apache NiFi best practices for better flow organization, including labeling, color coding, and avoiding root-level storage.