That Sinking Feeling

When I’m on a photo trip, I have a protocol that I carefully follow regarding image archiving.  I always have my laptop computer with me and at the end of each day I download images from my card(s) to the computer’s internal hard drive, creating a unique folder for each day.  Once the images have been downloaded, I back up all of the newly downloaded files to a small external hard drive.  Once I’ve verified that I have two sets of files saved, I format the card in the camera in preparation for the next day’s shooting.

In the first archiving stage–the process of downloading the images directly from the card to the laptop–I’ve long used a free program from Nikon called Nikon Transfer.  The program is low in frills but it’s functional, easy to use and has always gotten the job done for me.  It allows me to rename files during the download process and makes it easy to create the new folder in which the files will be placed.

This is the process I followed on my recent (I returned two days ago) photo trip to Michigan’s Upper Peninsula.  Upon returning to my home base from a photo trip, the archiving process continues.  I attach the external hard drive to my desktop computer and download the images to the desktop’s internal data drive.  From there, I engage in a thorough (perhaps overly cautious) backup procedure.  The RAW files are ultimately backed up to three additional external hard drives connected to my computer in the Chicago area and two others in the Indianapolis area.  When it’s all said and done, I have five iterative copies of my RAW files stored in two locations 200-plus miles apart.  This may seem a bit anal–particularly for someone who has never had a major loss of data–and perhaps it is, but I’d rather be safe than sorry.  Any loss of my images would be absolutely devastating.

Otter Lake, Hiawatha National Forest, Michigan

After I arrived home a couple of days ago and the extensive backup process that I detailed above was complete, I decided (naturally enough) to take a quick look at an image or two before heading off to bed.  For this purpose, I use another free piece of software from Nikon called View NX2, which is essentially a browsing program.  From there, I can directly open an image in my RAW converter (Capture NX2) and from there I can head into Photoshop.

So, I opened View NX2, navigated to the folder representing my first day’s shooting from the trip, found an image and pulled it into Capture NX2 for a quick white balance tweak.  And that’s where things began to noticeably fall apart.

When I opened the image in Capture, I immediately noted something very strange–the basic options for white balance and exposure adjustment were unavailable.  “That’s odd,” I thought.  I went back to View and pulled another image from the trip into Capture:  same thing; still no white balance or exposure options available.  I then navigated to a folder with images from my spring trip to the Smokies and pulled a RAW file into Capture.  This, I knew, was probably pointless as I’d already successfully edited these shots months earlier, but I wanted to make a comparison.  Sure enough, with the Smokies RAW image open in Capture, all of the normal RAW editing options–including white balance and exposure–were available.  That conclusively ruled out anything that might have been wrong with the View or Capture software.

I went back to one of the UP images in View and took a closer look, and I noticed something telling.  View has an information panel which provides data about the image in question, such as the date, exposure information, and so forth.  Included among that data are the pixel dimensions of the image.  A RAW image from my camera, the D800E, is ordinarily 7360 x 4912 pixels.  The image I was looking at in View was 1632 x 1080–about what I’d expect from a 3-megapixel camera.  The D800E is a 36.3 MP camera.  It was almost as though I had shot tiny JPEG images, but I knew that wasn’t the case.  The file size was displayed at roughly 75 MB–the size I’ve come to expect from D800E RAW files in the 15-odd months I’ve been shooting with the camera.  In fact, I already knew that I hadn’t been shooting tiny JPEGs because I’d sat through the downloads during the trip–the files were their typically huge size.

Near Council Lake, Hiawatha National Forest, Michigan

But the JPEG part of this immediately resonated with me.  JPEGs can’t undergo white balance or exposure adjustment in RAW conversion…because they aren’t RAW files.  That behavior was consistent with what I was seeing with these adjustment options unavailable.  It immediately occurred to me what was going on–I was looking at the embedded JPEG file that is part of every RAW capture.  The question was why.  It was obvious that the RAW data was there:  the size of the files was evidence of that.  So why was I only able to view the embedded JPEG for these files?

I ran a Google search using some appropriate keywords and discovered several on-line discussions dating to the summer of 2012–a few months after the D800 series of cameras was released and around the time I got mine–and found other people who were having a problem identical to what I was experiencing.  Several references were made to problems with RAW files from the camera after they were transferred using the standalone version of Nikon Transfer.

And then it hit me.

Shortly after getting the camera last year I recalled reading something about problems with files from the camera using Nikon Transfer.  The old, standalone version had been abandoned by Nikon prior to the release of the D800.  Instead, Transfer had been embedded inside an updated version of View NX2 and would remain there going forward.  The old, standalone version of Transfer would no longer be updated and should be abandoned by users of newer cameras.  I remembered thinking that I had removed the old version of Transfer from my computer, but I apparently hadn’t gotten around to doing so.

When I insert a card with images into the card reader on my laptop computer, Windows automatically launches a dialogue interface that gives me several options.  Those options include using “Nikon Transfer” to download the images to my computer’s hard drive.  In fact, a careful look at this Windows dialogue interface would have revealed the inconsistency:  two versions of Nikon Transfer were referenced.  The problem is, the only one that showed up directly in the dialogue upon launch was the old version of Nikon Transfer; the newer version doesn’t show up without scrolling down the list.  I had inadvertently downloaded the images–every single image I had taken on this trip–using the old version of Transfer, not the new one.

Otter Lake Sunset, Hiawatha National Forest, Michigan

Why did this matter?  This is just a program to download images, after all–it was just moving the images from the card to the hard drive.  As I continued to read the Internet threads from last year dealing with the problem, I received the answer:  the old, standalone version of Transfer corrupts D800 (and other new camera model) files during the download process.  And the notes in the threads I found were beyond discouraging:  the files were corrupted and unusable.  Nikon had been contacted about this by multiple individuals and said that there was nothing to be done.  All you were left to work with were the tiny embedded JPEGs–totally useless, as far as I was concerned.  There were notes from people bemoaning the permanent practical loss of thousands of images.

As I read these notes, I started to feel physically sick.  I was facing the prospect of having just returned from a 9-day trip that included a host of shooting opportunities that I believed had a chance to result in truly special images and then finding out that it was all effectively gone.  It wasn’t just the time, effort and expense that I was primarily concerned about; in fact, those weren’t really factors at all.

What was making me ill was the loss of the principal payoff for me of engaging in landscape photography in the first place: the creation of visual memories of special moments in time.  I can look at just about any image I’ve made over the years and tell you the story behind the making of the image and the emotions involved in the process.  Viewing these photographs takes me back to the experience of making them and all that entails.  I’m not sure that I’d engage in photography at all if it didn’t have this kind of personal impact on me.  And now I was facing the prospect of losing all that from this trip.  The tiny JPEGs wouldn’t cut it, for a variety of reasons (one being the inability to make large prints which typically is what I use to engender the feelings outlined above and another being that in many cases processing the final image requires combining multiple RAW frames expressly taken for that purpose–that would be impossible).  And these moments couldn’t be recaptured.  I could go back to the UP next fall and every fall for the rest of my life.  I’d create new moments, yes.  But the unique experiences I’d been a part of this time around?  They’d never happen again.  Realizing this and coming to grips with it produced a truly miserable feeling.

Before I completely gave in to the notion that I was going to have to accept this disaster as final, I hung a modicum of hope on two important facts.  The first was discovering that the file corruption, produced by the use of the standalone version of Transfer, was limited to the EXIF information.  This data was required for the file to be read by any RAW converter, but if that could somehow be sorted out, the RAW image file might be salvaged.  The other important piece of information?  The on-line threads I was reading about this problem were well over a year old.  Perhaps something had happened in the interim to address this problem.  So I read on.

Fall Splendor, Little Indian Road, Hiawatha National Forest, Michigan

I eventually found reference, in a note dating to the fall of 2012, that read (and I quote):

“There is a new utility available that will fix these corrupted NEF images.”

“NEF” refers to the proprietary format of Nikon’s RAW images.

A link to the utility was provided and I immediately clicked on it.  On the ensuing Web page I discovered information about this free (!) utility.  By downloading and installing it, folders of images would be “treated” by this command-line program and a new “fixed” version of each file would be created in a separate folder.  Having nothing to lose and (potentially) everything to gain, I immediately downloaded and extracted the program and tried it out on the first day’s folder worth of images.  It went through each image one at a time, and when it had completed the list, it indicated success!  But the proof would be in the pudding.

I immediately called up View NX2 and checked the pixel dimensions of the images now in the “fixed” folder:  7360 x 4912.  Obviously this was a positive sign.  So I exported the image to Capture:  sure enough, all of the normal RAW editing processes were available.    It appeared that this miracle solution was going to do the trick.  I set about fixing each day’s folder of images, checking at least one image per converted folder.  Everything seemed to be fine.

I couldn’t believe my luck!  After this horrific scare, it appeared that everything was as I had originally anticipated.  What a relief!

Is there a moral of the story?  I’m not sure.  I spent some time thinking about what I could or should have done differently and the only thing I came up with is that I should have uninstalled that standalone version of Nikon Transfer last year.  I had meant to do so, but I obviously hadn’t.  (It should go without saying, at this point, that it has now been uninstalled.)

But honestly, I think this matter all lies at the feet of Nikon.  They’ve done next to nothing to warn people about this potentially catastrophic problem, nor have they done anything to reconcile it.  Instead, it’s a private individual (a gentleman by the name of Phil Harvey) who has saved the day for the many individuals who have found themselves in this pickle.  Shame on Nikon for that.

Morning Rainbow, Council Lake, Hiawatha National Forest, Michigan

All’s well that ends well, I guess.  I’ve edited a very small set of images from the trip and have posted a few as accompaniments to this entry, just to visibly demonstrate some of what I feared I had lost.

In any event, I just had to get all of this off my chest.  I’ll have a new post up in a few days–the first installment of my collaboration with Tom Robbins on seeing in the field, and subsequently I’m sure I’ll have a series of posts detailing my experiences in the UP.  At least that’s now a viable option.