+5 votes
415 views
How to divide large CSV file into several parts

in Guides by (552k points)
reopened | 415 views

1 Answer

+3 votes
Best answer

Split large CSV file from the console in Windows with LargeFileSplitter.
Use the CSV Splitter program with graphical interface for Windows.
Handle large CSV files without dividing them with EmEditor.

Files in CSV format are increasingly present on the network and can be easily handled with office applications such as Excel. The problem that we encounter on many occasions is that the size of the CSV files is too large and it is impossible to handle them with conventional tools. In these situations, our only option is to divide the large CSV file into smaller parts .

image

In this way, we will have several smaller files that we can handle individually with our usual program. The problem of handling large CSV files is accentuated when we use a computer with a very limited amount of RAM. For all these reasons, the solutions we will address below will be focused on achieving our goal without the need for very powerful equipment and will be:

Split large CSV file from the console in Windows with LargeFileSplitter.

This is my preferred option and although it requires the use of the console, it is a simple program to use. The program in question is called LargeFileSplitter and is available for free on GitHub . The source code is also available in C # , so we can even mess with it if we have the necessary knowledge. We will need the .Net Framework 4.5 installed in the system.

Once the executable has been downloaded (from the Releases section of GitHub) we can execute LargeFileSplitter in two ways:

The program needs the path where the CSV file we want to divide is located and the number of parts in which we want to divide the file. An example of a command would be:
LargeFileSplitter.exe "C:\Portables\TechnoWikis tutoriales\archivogrande.csv" 5

It is important to note that the file path contains a blank space, so you have to put it in quotes.

Running this program interactively can be easier and faster , each one choosing the mode they like best.

image

One thing to keep in mind is that the output name of the created CSV files will be preceded by a number, for example if we divide the TechnoWikis.csv file into three parts, the resulting files will be called 1-TechnoWikis.csv , 2-TechnoWikis.csv and 1-TechnoWikis.csv .

Use the CSV Splitter program with graphical interface for Windows.

Secondly, we offer the option of using the free CSV Splitter software, which offers us a very intuitive graphical interface. We will only have to select the file we want to divide, adjust the number of lines that we want each file division to have and press the «Execute» button.

image

In this case, a numerical ending is added to the file names corresponding to the different divisions. For example, if the input file is TechnoWikis.csv , the name of the output files will be TechnoWikis.csv.000 , TechnoWikis.csv.001 , TechnoWikis.csv.002 , etc.

CSV Splitter can also be run in console mode , for this you only have to execute it with a command like the following:
CSVSplitter filename="C:\Portables\TechnoWikis.csv" outputfolder=" C:\Portables" rowcount=10000 firstrowheader=1 repeatheader=1

After several tests, I can say that CSV Splitter is less efficient than our previous solution and with very large files the application has been blocked . For this reason I still recommend LargeFileSplitter as the first option.

Program download: https://www.erdconcepts.com/dbtoolbox/csvsplitter/csvsplitter.zip

Handle large CSV files without dividing them with EmEditor.

Although we are not focusing on how to divide large CSV files in this tutorial, I did not want to overlook the option of handling these files directly, without dividing them.

For this purpose, the program I usually use is EmEditor , which I already warn you that it is paid. I have to admit that EmEditor is impressive, it opens files of almost any size without having large amounts of RAM .

image

If you deal with CSV or text files, of sizes larger than 3GB on a regular basis, it may be a good option to checkout.


by (3.5m points)

Related questions

+4 votes
1 answer
+3 votes
1 answer
asked Aug 9, 2019 in Guides by backtothefuture (552k points) | 947 views
+4 votes
1 answer
+3 votes
1 answer
+3 votes
1 answer
Sponsored articles cost $40 per post. You can contact us via Feedback

Most popular questions within the last 30 days

10,659 questions
10,791 answers
510 comments
3 users