Files in CSV format are increasingly present on the network and can be easily handled with office applications such as Excel. The problem that we encounter on many occasions is that the size of the CSV files is too large and it is impossible to handle them with conventional tools. In these situations, our only option is to
divide the large CSV file into smaller parts
.
In this way, we will have several smaller files that we can handle individually with our usual program. The problem of
handling large CSV files
is accentuated when we use a computer with a very limited amount of RAM. For all these reasons, the solutions we will address below will be focused on achieving our goal without the need for very powerful equipment and will be:
Split large CSV file from the console in Windows with LargeFileSplitter.
This is my preferred option and although it requires the use of the console, it is a simple program to use. The program in question is called
LargeFileSplitter
and is available for free on
GitHub
. The source code is also available in
C #
, so we can even mess with it if we have the necessary knowledge. We will need the .Net Framework 4.5 installed in the system.
Once the executable has been downloaded
(from the Releases section of GitHub)
we can execute LargeFileSplitter in two ways:
The program needs the path where the CSV file we want to divide is located and the number of parts in which we want to divide the file. An example of a command would be:
LargeFileSplitter.exe "C:\Portables\TechnoWikis tutoriales\archivogrande.csv" 5
It is important to note that the file path contains a blank space, so you have to put it in quotes.
Running this program interactively can be easier and faster
, each one choosing the mode they like best.
One thing to keep in mind is that the output name of the created CSV files will be preceded by a number, for example if we divide the
TechnoWikis.csv
file into three parts, the resulting files will be called
1-TechnoWikis.csv
,
2-TechnoWikis.csv
and
1-TechnoWikis.csv
.
Use the CSV Splitter program with graphical interface for Windows.
Secondly, we offer the option of using the free
CSV Splitter
software, which offers us a very intuitive graphical interface. We will only have to select the file we want to divide, adjust the number of lines that we want each file division to have and press the
«Execute»
button.
In this case, a numerical ending is added to the file names corresponding to the different divisions. For example, if the input file is
TechnoWikis.csv
, the name of the output files will be
TechnoWikis.csv.000
,
TechnoWikis.csv.001
,
TechnoWikis.csv.002
, etc.
CSV Splitter can also be run in console mode
, for this you only have to execute it with a command like the following:
CSVSplitter filename="C:\Portables\TechnoWikis.csv" outputfolder=" C:\Portables" rowcount=10000 firstrowheader=1 repeatheader=1
After several tests, I can say that CSV Splitter is less efficient than our previous solution and
with very large files the application has been blocked
. For this reason I still recommend
LargeFileSplitter
as the first option.
Program download:
https://www.erdconcepts.com/dbtoolbox/csvsplitter/csvsplitter.zip
Handle large CSV files without dividing them with EmEditor.
Although we are not focusing on how to divide large CSV files in this tutorial, I did not want to overlook the option of handling these files directly, without dividing them.
For this purpose, the program I usually use is
EmEditor
, which I already warn you that it is paid. I have to admit that EmEditor is impressive, it
opens files of almost any size without having large amounts of RAM
.
If you deal with CSV or text files, of sizes larger than 3GB on a regular basis, it may be a good option to checkout.