For some years now, I’ve been a big fan of RoadKil’s Unstoppable Copier utility which is an easy to use tool to copy or move files from any drive to another drive. We often have to move large amounts of data from old hard drives to new hard drives either because a client is upgrading to a new computer/OS or because a hard drive is failing.
For years, we preferred to use Xcopy from the command prompt over dragging and dropping files in Windows Explorer because in our opinion Xcopy seemed to be faster when copying large numbers of files. However, increasingly Xcopy was becoming unusable because when a path+filename exceeds 256 characters Xcopy dies. It doesn’t skip the file, it doesn’t continue, it just stops cold. This usually occurred in the Temporary Internet files folders which meant we had to do clean up on the old drive prior to executing the copy procedure. In the case of a failing drive, this might be impossible.
After a little bit of research, we discovered Unstoppable Copier which solved a lot of problems for us. First, the program was designed as a data recovery tool (in fact if you visit their website, you will find the download page under Data Recovery). As such, it has the ability to retry the copying of a file a number of times and if it continues to fail to copy a file, the program will log the damaged file and then skip the damaged file and move on. What this does is great because now we can start a large copy process at the end of the day and we know that it will be finished by the time we come in the next morning. No more crashed Xcopy transfers!
But recently, I ran into a situation where I needed to move millions of individual files totaling some 650Gb of data to a large 2TB hard drive to gain some additional space on a server. The problem was that the server and drives HAD to remain in operation because they were used in timed processes that could not be terminated for more than a couple of hours per day. Every tool I had in my arsenal (Acronis, Ghost, CloneZilla, Double-Image, etc.) needed more than a couple of hours to move that volume of data. Since both the old drives and the new drives could be in operation at the same time and because the data was well organized in specific folders that could be moved independently, I made the decision to use Unstoppable Copier to move individual file folders (at the root level) and update any individual applications to point to the new drive locations after the move.
The first folder I attempted to move had 400,000 individual files but only about 5Gb of data. It became clear after the first 50,000 files and several hours of processing time that to continue down this path was going to be a seriously time consuming process. Because Unstoppable Copier copies files one at a time and we were transferring across a network from a NAS RAID device to a local mirrored RAID, as the number of individual files went up the time increased significantly. To completely move 650Gb of data was going to take a couple of weeks of continuous file transfers and this was not possible in this environment.
I want to make it understood that this review is not a criticism of Unstoppable Copier and Unstoppable Copier is not going to be dropped from my toolkit! Unstoppable Copier was designed primarily to recover data NOT to provide highly efficient file copy functions. So, in this case, it was clear that Unstoppable Copier was simply the wrong tool for this particular job, not because of any inherent failure on the part of Unstoppable Copier, but because it was designed to do a slightly different job. It was kind of like using a monkey wrench to drive in a nail. I could accomplish my goal eventually, but not without a lot of work.
Having used FTP transfer software for years and being a FileZilla fan for many of those years, one of the features that I like most about Filezilla is its ability to transfer multiple files at the same time. This means that if you have a large file that will take a while to download and a bunch of smaller files to download as well, Filezilla can continue to download the smaller files at the same time to keep the data flowing efficiently.This multi-threaded approach improves the speed of downloads significantly.
It’s kind of like being at Walmart when there is only one line open for checking out. One person goes through the line at a time, but the line backs up because more than one person at a time joins the line. When the manager opens a second line, the checkout process goes twice as fast. Open five checkout lines and the checkout process goes five times as fast. You get the point.
So I fired up my trusty Google search and found several multi-threaded file copy utilities, but one caught my eye: RichCopy. Apparently, RichCopy was developed internally at Microsoft to improve the speed of file copying in test environments at Microsoft. It was kept in-house for several years and then around 2008-2009, it began to leak out on some file-sharing networks. Finally, Microsoft just released it to the public and it can be downloaded here.
I initially noticed RichCopy because it bore a certain resemblance to Unstoppable Copier and because it looked easy to use. I was surprised after installing it to find it to be as feature rich as you could hope and many of the features are even beyond what my needs are right now, so the program will easily grow with me as my needs change in the future.
The first test of RichCopy was on a folder with about 8Gb of data and over 500,000 individual files. Unstoppable Copier projected a 12-24hours time frame to complete the copy process, but RichCopy finished in under 6hrs. I discovered after the fact that RichCopy was designed to work across networks and in benchmark tests, RichCopy beat fourteen other multi-threaded file copy utilities in network copies. Since that was exactly what I was trying to do, I knew that I now had the exactly the right tool for what I was trying to do.
Another folder copy, I performed was over 50Gb with over 130,000 individual files. I set the options for RichCopy to move and verify each move and started the file copy. With files averaging 3Mb each, RichCopy moved 35,000 files in a little over 1.5hrs across a 100Mbit network. These 35,000 represented about 45% of the total data (over 20Gb).
UPDATE: In a subsequent test using 50 threads, I discovered that RichCopy doesn’t check the files it is trying to move against the internal database of active threads, so what you get is a series of errors indicating that a file is locked by another process. Additionally, with the higher thread count, I saw a about a 15-20% decrease in the bytes per second being transferred, so throughput suffers slightly. However, I also performed a couple of tests with the file verify turned on and what I found was verify slows individual threads down proportionally to the size of the file. The bigger the file, the longer the time to verify, longer it takes to complete the transfer. This result was actually expected by me as I knew that verify would take longer, but it is interesting to actually watch it happen in real-time. My personal recommendation would be to keep your thread count between 15 and 20 to get the best results. I may try another test using only 5 threads to see if that works any better. Will report back.
RichCopy isn’t perfect. You can tell that it was developed for in-house use because some of the help screens haven’t been proof-read and there are typos. And it takes a little bit of practice to find the sweet spot when it comes to the number of file transfers you can run at the same time. Theoretically you can set up 256 threads to run at the same time and while I might test this in the future, 20 is as high as I’ve gone so far and I’m guessing right now for my purposes 15 might be the right number. I haven’t been able to locate any recommendations from anyone on the web about this setting, so if I come up with the perfect number, I’ll update this post accordingly. One negative I’ve noticed is that when you set the transfer mode to “Move”, the files are moved the appropriate folders on the destination drive, but the empty folders are left on the source drive. I don’t know if this has something to do with the multi-thread function or just an oversight on the developer’s part, but it means that I have to do about 5 or 10 minutes worth of clean up after each move. Not a big deal, but you assume that Move means Move and folders would be removed from the source drive after the move is complete.
The bottom line is that if the number of files is huge, the sizes of the files vary greatly and you are copying across a network, RichCopy’s multi-threaded copying is the only way to go. The benchmarks indicate that in a local drive to drive copy process, RichCopy is twice as fast as Unstoppable Copier. In large file copies, Unstoppable Copier is about 2.5 times faster than RichCopy. And, in network copies, RichCopy is about 3 times faster than Unstoppable Copier. My experience so far is matching up well to the benchmarks, so I can positively confirm that in the right situation RichCopy is an excellent solution. As I said before though, Unstoppable Copier will not be leaving my toolkit because while both tools perform basically the same task, one tool can shine over the other in different situations.
Here are my recommendations:
- Unstoppable Copier: Set it and forget it operation; large files; drive to drive; speed not critical
- RichCopy: Network transfers, wild variations in the file sizes with mostly smaller files and few large files; drive to drive; transfer time limited
Give both a try and see if you prefer one tool over the other. I’d be interested in reading your opinions in the comments below.