If you know me, you know I'm a huge fan of the program Rsync.  It's fast, it's reliable, it's installed by default on every Mac and Linux machine, and it offers a wealth of options, and it's free!  What's not to like? 

A while ago Dusty McCord and I got together to talk about what the perfect settings for rsync would be. We evaluated the needs of on set data management, and what is most important to a DIT.  We came up with this quick list of what we needed:  file integrity must be preserved, all files hidden or not must be copied, preserve the original time stamp, files must be checksummed to ensure playback, we need to see the progress.

With those options in mind, this is the magic recipe we came up with: 

rsync -rltWDcv --progress

It looks prety confusing doesn't it?  It's pretty simple when you know the parts.  The first parts, rsync, just tells the computer to use the rsync command.  The following arguments -rltWDcv are options for the rsync command (full details on the rsync man page), and --progress indicates that we, the user, would like to see the progress of the file transfer.  It doesn't show a progress bar, but it does tell you how many files its checked and how many are left.

Let's break down the arguments:

r     recursive, copy's the files of the source directory and the sub directories
l     copy symlinks, normally we would not encounter these, but there's always a possiblity
t     copy time, this copies the timestamp of the original file
W    whole file, this copies the whole file instead of just the bytes that are different
D    devices, another item we shouldn't encounter, but just in case...
c     checksum, this is the most secure method of comparing two files, read more here.
v     verbose, displays a log of each file transaction, that way it's easy to spot if something went wrong

There we go.  Using free and open source tools we have crafted a command that is extremely reliable, which on set is extremely valuable.  If this article is too complicated for you, and you need to understand the usage of Rsync, take a look at this primer on how to use Rsync.

Here's a sheet I developed for DIT logging.

P = Playback, C = Copy, V = Verify, S/F = Folder Size over File Size [DOWNLOAD HERE]

Step 1.  Download Air Video - Free, Paid

NOTE: The Free version will let you see only a few videos in the directory, to see all videos you need to buy the premium version.

Step 2.  Connect to HackettWireless with your iPhone

MAKE SURE AIR VIDEO IS DOWNLOADED

Step 3.  Open Air Video, tap the plus symbol.

Step 4. Tap my computer, King Gheedora

Step 5. Navigate to the shared folder.  If a file says "inactive" that means it is currently copying.  Select a file that has a thumbnail and does not say inactive.

Step 6. Select "Play with Live Conversion", this will convert the file on the fly to play on your iPhone.

Typical DIT workflow is copy the same file to two different file locations, typically using a normal Finder copy operation.  This is a flawed method of working for a number of reasons.

Finder Has No File Verification

The thinking behind dragging a folder from the the source to two different destinations is that you won't "Copy the same error twice" assuming an error has occurred.  This is the main drawback of using Finder to copy mission critical files, it does not give you any feedback, there's no log of any kind, and more importantly it lacks file verification using Checksum.  Excerpt from the wiki:

A checksum or hash sum is a fixed-size datum computed from an arbitrary block of digital data for the purpose of detecting accidental errors that may have been introduced during its transmission or storage. The integrity of the data can be checkedat any later time by recomputing the checksum and comparing it with the stored one. If the checksums match, the data were almost certainly not altered (either intentionally or unintentionally).

Copying Twice from CF = Dramatic Speed Loss

This might not always be the case, but currently most DSLR shoots use compact flash cards.  The problem with the current DIT workflow is that it doesn't take into account the limited transfer rate of compact flash readers.   The fastest CF Card as of this writing is 167 Mbyte/s, which is very fast.  But USB 2.0 moves at 480mbit/s, or 60 Mbyte/s and Firewire 800 at 800mbit/s or 100 Mbyte/s, bottlenecking that super fast CF card.

Let's say we're using a USB 2.0 CF reader, only because Firewire CF Readers are getting harder and harder to find.  If I have 8gb of data on my CF card, and I want to copy it to a destination, it will take n number of minutes.  But if I copy that to two locations, it will take n / 2 minutes!  That means it doubles your time and ends up cramping your pipeline of CF cards coming down the tube, leading to a pile up of CF cards, which ultimately increases the risk of mistake.

How Do We Make This Better? 

Film and Video production has only recently found itself at the whim of a computer and it's complex method of storing data.  But there have been industries that have been here for years, and have developed a tool for copying data that uses the aforementioned Checksum to make sure that every byte is exactly the same on the target drive.

Rsync 

The tool is called Rsync, a program heavily used in website development, industrial data management, and any type of backup program worth a damn.   Rysnc was originally written for web developers to synchronize web sites.  That way you could have a redundant server at a Location B mirror the exact website at Location A (this was before the cloud existed).  If the a large website couldn't reliably copy a file from one server to the other, and instead insisted on double copying each file, we would have a very slow and unreliable internet.

Rsync has all of the features that finder does not.  It has a log file, file verification with checksum, it's lighter weight (less system resources) and it's incredibly stable. 

If DIT were to leverage the power of Rsync, gone would be the days of not knowing if a file has successfully copied or not, but what about the bottle neck of transfer speeds?  We simply use Rsync once again.  If you know for sure you're first file copy is 100% the same as your source, you're free to copy from that destination to another destination, and another destination if need be!  Just use Rsync over and over again, moving the data from drive to drive, and paying close attention to the error log.