20

The file is around 170GB. I wouldn't upload it on the FTP, and download it. It is not reliable, sometime, connections drop, and it will have big trouble. So, is there any better way to do so? Anyone suggest? Thanks.

P.S.:two computers are not in same network.

Jens Erat
  • 17,507
  • 14
  • 61
  • 74
Ted Wong
  • 933
  • 8
  • 14
  • 10
    An external USB HD? – mouviciel Mar 13 '13 at 09:22
  • @mouviciel yes, it is. But my environment not allow me to do so, I updated the question, the only way is via network. – Ted Wong Mar 13 '13 at 09:23
  • 9
    http://en.wikipedia.org/wiki/Sneakernet is a network. – ta.speot.is Mar 13 '13 at 09:27
  • 4
    Break it up into smaller pieces, then you only have to worry about one small piece at a time. Pro tip: BitTorrent does exactly this. – ta.speot.is Mar 13 '13 at 09:29
  • 1
    I use SyncApp from BitTorrent to keep my data in sync (~5TB, one file is 300GB). SyncApp has a Windows and Mac application with GUI and a Linux application with a webinterface (perfect for servers). At the moment there is only a closed beta available. You can register for an invite here: http://labs.bittorrent.com/experiments/sync.html - Screenshot: http://i1-news.softpedia-static.com/images/news-700/BitTorrent-Sync-Is-a-P2P-Dropbox.png – fnkr Mar 13 '13 at 10:08
  • 4
    I would use Skype but im not brave enough to put it as an answer :) – Robert Niestroj Mar 13 '13 at 11:31
  • 1
    [FedEx has better bandwidth than the internet](http://what-if.xkcd.com/31/) – zzzzBov Mar 13 '13 at 19:39
  • Do you only care about performance but not security? – Alvin Wong Mar 14 '13 at 00:41
  • 1
    Consider [RFC 1149](http://en.wikipedia.org/wiki/IP_over_Avian_Carriers) – Alvin Wong Mar 14 '13 at 00:57
  • For a 170GB file, there's a high possibility that checksums in IP and TCP will miss some transmission errors, so you should use protocols that actively checks the data. Bittorrent is a good one. – billc.cn Mar 14 '13 at 01:04
  • Yes, thats why I use the SyncApp from BitTorrent. – fnkr Mar 14 '13 at 14:27
  • 4
    Why is this flagged as a duplicate of questions asking how to e-mail 2-30 MB of data? There are a few orders of magnitude between "send 17 MB" and "send 170 GB", and the solutions are completely different. Dropbox/filehosters are not really a feasible suggestion for 170 GB... – Jan Schejbal Mar 15 '13 at 14:51

6 Answers6

27

Wierdly enough bitorrent might work pretty well here, assuming office policies allow it- it breaks up the file for you, checks if its correct and if not redownloads it. You're probably going to want to run your own tracker, but many bitorrent clients do that anyway, and if possible use webseeds to speed things up even more - burnbit makes this easy.

As an alternative I'd also suggest doing the oldschool pirate thing, and splitting up the files and using a parity file, then transferring it by any means you have, FTP or web server. If you use a webserver, downloads can be continued - with something like wget, and the parity archive would allow you to rebuild the file with a few broken pieces.

Naturally consider encrypting the files or file chunks, if the data is of a sensitive nature as well.

A more recent option may be bitorrent sync - It runs on everything but the toaster (Unless your toaster run windows, linux on x86, PPC or ARM), and handles most of the grunt work for you. It uses the underlying bitorrent protocol but is a lot simpler to use.

Journeyman Geek
  • 127,463
  • 52
  • 260
  • 430
  • 1
    additionally if you need to copy to multiple hosts you'll decrease the time significantly. – Colton Mar 13 '13 at 18:02
  • +1 for toaster. U saved the toaster from toasting big chunks – PathToLife May 27 '16 at 08:01
  • BitTorrent Sync was bought up and renamed to Resilio Sync. It's no longer possible to install the free version on WIndows Servers. You need a paid Business license to do that now. – Rudey Jan 15 '22 at 19:49
  • TBH - 2 of the options I've suggested have changed, or are dead - which kinda reflects the problems with software recommendations :/ – Journeyman Geek Jan 16 '22 at 00:46
20

Take a look at robocopy, it supports restarting, and in general is a lot more stable than other options.

Adam Caudill
  • 561
  • 3
  • 11
4

WinZip can produce lots of (relatively) little files (originally designed to archive to floppy disks, but you can choose 1GB now). Then non-resumable FTP (as the Microsoft option is) is OK. Finally, WinZip will retrieve the file at the other end.

Mark Hurd
  • 409
  • 1
  • 6
  • 15
  • 3
    I think 7-zip is much better than WinZip. – Alvin Wong Mar 14 '13 at 00:31
  • @AlvinWong Perhaps; WinZip was first and still pretty good. I just haven't used 7-zip enough. – Mark Hurd Mar 14 '13 at 01:59
  • 1
    something being oldest is not a good reason to use it in the IT world... – wim Mar 14 '13 at 05:36
  • @wim I haven't needed to use anything else. If WinZip is so wrong downvote me, preferably giving reasons why it is actually crap for this purpose. If 7-zip is so right, add it as an answer. – Mark Hurd Mar 14 '13 at 06:20
  • Well, you're not wrong so no downvote.. but Alvin has a point 7-zip capabilities are a superset of winzip, and it's free.. – wim Mar 14 '13 at 06:39
4

Robocopy and BitTorrent have already been suggested and sound like a good idea. Other options that may work better in a restrictive network environment where you cannot e.g. create the SMB connection that seems to be required for RoboCopy:

FTP. I know you don't like it, but with a good server and client, it should work well. Create an FTP server on either source or recipient, make sure that it supports encrypted connections (to prevent transparent proxies etc. from interfering) and files > 4 GB. Then upload/download the file using a good FTP client (making sure to use binary mode). FTP supports connection resuming, so if the connection drops, just resume. A current copy of wget should be fine.

You can do the same with HTTP: Set up a HTTP(S) server supporting large files, and download it with a current copy of wget.

Otherwise, there are rsync binaries for Windows and numerous proprietary rsync-like programs that you could use. Especially if you expect that the file will need to be updated and only small portions of the file will change, you may want to look into that direction.

Remember that at 8 MBit/s (1 MByte/s), it will take you two days to transfer the file. Unless you have really fast connection, sending a physical hard drive with a copy of the file(s) may be faster.

Jan Schejbal
  • 1,112
  • 9
  • 14
  • 1
    Send the hard drive by Fedex. In the end it will cost less and be faster. That is how most larger corporations move backups and such. – Pow-Ian Mar 13 '13 at 19:06
  • 1
    Using the file transfer protocol for file transfer! Who'd a thunk it ??? – wim Mar 14 '13 at 05:37
  • @Pow-Ian: I did not want to advertise a specific company, but FedEx really lives up to their original Federal *Express* name. I handed them a (pretty sizeable) package weighing around 10-12 kg at noon on Saturday in Germany, didn't even pick their fastest service, and got it delivered to Belgium by 10:35 AM on Monday for 29 EUR. If I have anything that needs to get somewhere fast, I will definitely use FedEx again. – Jan Schejbal Mar 15 '13 at 14:48
  • @Jan Schejbal I say call a spade a spade. I know FedEx is the fastest so i don't mind telling others about my experiences. For full disclosure I have seen their facilities and toured their Memphis TN USA air port processing facility. They can literally ship things next day from NY, NY to Japan. – Pow-Ian Mar 15 '13 at 14:51
1

Some things that come to my mind are private P2P Networks (uTorrent, DC++) or a tiny http server with wget

Midhat
  • 474
  • 5
  • 15
0

I would sugest to use some tool (windows) based on rsync http://en.wikipedia.org/wiki/Rsync

Josef Prochazka
  • 101
  • 1