Cross-Platform, Multi-NLE, Tape-Less Transcoding Workflow Tips
Monday, September 19, 2011 at 6:00AMIf Workflow is the Answer, then it’s one hell of a Question…!
For a very long time post production workflow has been built upon consistencies - consistent delivery, consistent format, consistent platforms, consistent tools, consistent viewing environments. But then along came a veritable tsunami of digital detritus that washed ashore - in a very short span of time - a greater diversity of tools, formats, platforms and systems that all previous years put together.
And with that diversity came new expectations. Too often unrealistic expectations born of ignorance, but none the less a seismic shift in what was expected of production, the tools and the personal skill-sets that make it all happen.
It is within this digital dilemma that I have often been required to design digital post-production workflows that are flexible enough to satisfy expectations and take advantage of opportunities, whilst being rigid enough to provide some measure of consistency and reliability in an inconsistent and unreliable world. No easy task. Especially when we reflect upon the title of this post, ‘Cross Platform, Multi-NLE, Tape-less, Transcoding Workflow’ and add the suffix of ‘for non-standard variable viewing platforms.’
To put this in practical and specific terms consider a…
- Multi-episode Web-Series
- Shot on a variety of DSLRs and video cameras
- Cut by several hyphenate editors using different software packages
- Delivered in SD and HD; online, DVD, download and projected at festivals and events.
There is actually nothing unusual about this scenario - all over the world advertising agencies, museums and cultural institutions, indie filmmakers, corporate producers and traditional networks stretching out of their broadcast comfort zone are all doing exactly this kind of production everyday.
Yet everything about this scenario is an anathema to the traditional notion of workflow which stemmed from a broadcast mentality and was cemented in to place with consistency and predictability.
What I am presenting below is my recommended workflow tips on some specifications, techniques, ideas, tools and processes that aim to find a balance between digital flexibility and robust consistency whilst imposing managerial order over potential chaos. It’s not a comprehensive workflow plan, it doesn’t include ancillary and parallel processes such as sound-mix, colour grade, VFX work and so on - think of it more as a central spine of key principles and tips for video onto which you can bolt other processes you need. It is a paradigm I use myself and know to work well in both problem-solving and problem-avoiding.
Be aware that the workflow tips below are assuming certain things - assuming you are using low-mid range camera formats, needing to move between editing systems (especially now Final Cut Pro is dead and end-of-life), moving projects between computers and operating systems, working in small teams with overlapping skills and responsibilities and that your prime delivery is online and multi-platform. It will particularly focus on the NLE’s Premiere Pro and Final Cut Pro. It covers 5 elements to keep you organised.
1. FOOTAGE HANDLING AND FILE NAMING
2. TRANSCODING AND FORMAT STANDARDISATION
3. TRANSCODING TO DNXHD
4. TRANSCODING TO PRORES
5. PROJECT NAMING AND SAVING CONVENTIONS
_____
FOOTAGE TRANSFER AND FILE NAMING
(far too often the most neglected part of the process, the point at which the pain begins if not handled well)
1. Create a folder on your hard drive with a unique name based on the date of the shoot, the card number and the camera (DATE_CARD#_CAMERA#). For example, a shoot on the 26th of June 2011 on Camera 1 would be named ‘260611_Cam1_Cd1’
If more than one card of footage was shot on the same date then the folders would be named ‘260611_Cam1_Cd2, 260611_Cam1_Cd3’ etc.
If multiple cameras were used then folder would be named ‘260611_Cam2_Cd1’ and so on.
This 3-part naming of DATE CAMERA# and CARD# will ensure that each project is clearly labeled whilst avoiding any confusion between cards and cameras. It is important that file transfer and folder naming be done immediately after a shoot day.
2. If using a solid-state video camera (such as AVCHD cameras like the Sony VG10 or XDCAMEX cams like the EX3 it is important to maintain the folder structure and metadata as it is on the camera. This specific structure is the equivalent to your Tape Source. Plug the camera in and open the memory card, select all the folders shown and drag to copy them from the camera into the newly created folder (named as per above). These folders - now containing the source material in its exact file structure from the camera - should be back-up archived to external drives as the source-master. They should be treated as you would your source tapes from a tape-based camera.
4. External hard drives used for storage and backup of video projects should be formatted the EXFAT format. EXFAT is a universal cross-platform hard drive format that has no restrictions on file size and can be used on both Mac (Snow Leopard 10.6.8 and above) and PC (Windows Vista and Windows7). Other hard drive formats such as FAT32, NTFS or HFS+ are not suitable in a cross-platform environment as they are operating system specific or have file size restrictions. Such incompatibilities will invariably create time wastage and impede workflow flexibility.
ADDENDUM
To make the whole process of file naming, managment and metadata logging, Adobe Bridge is your best friend. Richard Harrington of Creative Cow has a superb video tutorial on using Bridge to batch rename files in a tapless workflow and using Bridge to manage video files in Premiere Projects.
Check Richard’s Video Tutorial here.
TRANSCODING AND FORMAT STANDARDIZATION
Modern editing systems such as Premiere Pro can work with virtually all file formats and codecs natively on a single resolution-independent timeline. More antiquated systems such as Final Cut Pro cannot work with native formats, nor accommodate mixed resolutions with any efficiency, thus relying on transcoding to a lossless intermediate format simply to be able to edit at all (a la ProRes).
However, whilst a native format workflow in Premiere is effect and simple, there are still many advantages to working in a single unified codec and format based on a transcoding workflow. System performance, colour correction accuracy, faster exporting and consistency of results. Also, more importantly, Transcoding allows for a common video format to be shared between different editing systems and operating systems and take full advantage of project file compatibility such as AAF and XML.
For smaller projects to be edited in Premiere, then transcoding may not be necessary – in which case the files can be imported through the Premiere media-browser and edited as-is directly from the camera. For fast turn-around projects this is ideal. For larger projects however a lossless intermediate transcoding process, unifying all footage, is recommended.
In this regard, there are two main viable lossless intermediate codecs to serve this purpose; Apple ProRes and Avid DNxHD. ProRes is native to the Final Cut system and can only be encoded on a computer with Final Cut Pro installed (though it can be ‘read’ on any system with the free ProRes decoders installed). DNxHD by contrast is a free codec system that is cross-platform and can be installed on any computer, any operating system and encoded from any NLE. Both offer industry grade, 4:2:2 8bit and 10bit, encoding. (ProRes does also have a 4:4:4 variant but this is unnecessary for any work other than high-end visual FX)
(*NOTE: The Cineform digital intermediate codec blows both ProRes and DNxHD out of the water for quality, efficiency and flexibility. But it is not free so Ive left it out of the equation. Suffice to say however that if you spend the extra money you wont regret it and you can dispense with ProRes and DNxHD entirely)
TRANSCODING TO DNxHD
These steps outline using Adobe Media Encoder as a batch-processing and encoding tool for processing and converting of video files separate from the editing system to DNxHD. Adobe Media Encoder is currently the most advanced and comprehensive encoding system of its kind. It’s wholly cross-platform and is bundled as part of the Adobe CS5.5 suite.
1. Create a Project Folder with a unique name for the project (refer project naming protocols section)
2. Launch ADOBE MEDIA ENCODER (AME)
3. In the AME Preferences tick ‘specify output file destination’ and direct it to the Project Folder.
4. Open the source footage folder (named ‘DATE_Cam#_Cd#’) and navigate to the actual video files.
5. Select all files in AME (click the first and then shift+click the last)
6. Change the FORMAT to QUICKTIME
7. Click on the current PRESET to customize the settings.
The following steps relate to creating a DNxHD preset. Once this has been made you will in future be able to select DNxHD as a pre-built template and not have to repeat these steps.
• In the VIDEO tab choose VIDEO CODEC and select ‘AVID DNxHD CODEC’
• Change Resolution WIDTH and HEIGHT to either 1920x1080 or 1280x720
• Change FIELD TYPE to PROGRESSIVE
• Change ASPECT to SQUARE PIXELS
• Click the save icon next to PRESET and name this preset DNxHD
8. Click CODEC SETTINGS and choose the correct RESOLUTIONS setting – 1080p/25 DNxHD 185, 10bit or 720p/25 60, 8bit (*note, despite the preset being saved, this last setting will need to be manually selected every time you use the preset.
9. Press the Start Queue button to begin the Transcoding batch process
TRANSCODING TO PRORES
These steps outline using Adobe Media Encoder as a batch-processing and encoding tool for processing and converting of video files separate from the editing system to ProRes.
1. Create a Project Folder (with a unique name for the project)
2. Launch ADOBE MEDIA ENCODER (AME)
3. In the AME Preferences tick ‘specify output file destination’ and direct it to the Project Folder.
4. Open the source footage folder (named ‘DATE_Cam#_Cd#’) and navigate to the actual video files.
5. Select all files in AME (click the first and then shift+click the last)
6. Change the FORMAT to QUICKTIME
7. Click on the current PRESET to customize the settings.
The following steps relate to creating a PRORES preset. Once this has been made you will in future be able to select PRORES as a pre-built template.
• In the VIDEO tab choose VIDEO CODEC and select PRORES 422
• Change Resolution WIDTH and HEIGHT to either 1920x1080 or 1280x720
• Change FIELD TYPE to PROGRESSIVE
• Change ASPECT to SQUARE PIXELS
• Click the save icon next to PRESET and name this preset PRORES
8. Press the Start Queue button to begin the Transcoding batch process
(* NOTE: Final Cut Pro can also transcode to ProRes through it’s Log+Transfer utility, creating ProRes versions as it imports clips. The file result is the same however using Adobe Media Encoder means that you do not need to create a project or tie up your editing software for batch processing. The ProRes files can then be imported into either Premiere or Final Cut Pro for editing.)
PROJECT NAMING AND SAVING CONVENTIONS
It is crucial to adopt standardized naming conventions both for organization and production efficiency as well as to avoid disaster when project files become corrupted (which they undoubtedly DO and WILL..!)
The following saving and naming procedure will avoid issues of file corruption and workflow confusion.
All editing system project files should be named with the title of the project and the current day date expressed as 6 digits. For Example: ‘AEROPLANES_260611’
(* NOTE: Americans may fail to recognise this date format that most of the rest of the world uses whereby DAY is first followed by MONTH. It’s the most logical way to express date because it works form the smallest increment to the largest - Day Month Year - and is an inversion of the format common in the middle east and parts of Europe where Year is first followed by Month and Day. Unlike the US-centric Month Day Year which has no logic at all… Month first may be fine if you never have to collaborate or send your project to people outside of the US borders, but if you do it will quickly become confusing and problematic - does 11062011 mean the 11th of June or the 6th of November..? If you want to persist with Month first I would suggest you trade numbers for letters in the naming protocol to avoid confusion, thus 11062011 becomes NOV062011)
On each and every day of editing the Editor should open the previous days project file and Immediately save a new project file with the current date (eg ‘AEROPLANES_270611’. This process should be repeated every day of editing so that each separate day of work on an individual project has a discreet dated title.
Since Premiere and Final Cut project files are merely directories that link to the media they take up virtually no hard drive space. By having a separate project file for each days work you are able to return to an earlier part of the process at any time and, more importantly, if a project file becomes corrupted you will always have the previous days file to draw upon. At worst you loose 1 days work rather than ALL your work.
Final Cut Pro project files are particularly notorious for corruptions and errors. With FCP projects it is recommended that you also save each day an XML version of the project file following the same date-naming convention. FCP XML projects contain all the same information as regular project files however they are human-readable, less prone to corruption and can be opened by other editing systems.
Whilst it would be preferable to utilize just a single editing software system there are effective ways to migrate projects between editing systems. Premiere Pro has the ability to open Final Cut Projects saved in the FCP XML format and likewise can also save out Premiere projects in the FCP XML format. This means that timeline, sequences and media (so long as they are transcoded to a lossless codec, eg ProRes or DNxHD because FCP cannot read anything natively like Premiere) can be migrated from FCP to Premiere and from Premiere to FCP, allowing a single project to be shared across users on different NLE systems and even operating systems.
(*NOTE: Premiere can also import and export AAF format projects files for Avid editing systems)
Reader Comments (16)
I've not had much to do with Premier but had a play around with it recently and was very impressed. It now has a chance to leap into the substantial hole left by FCP. I've just completed another longform project on Avid and it will remain my main weapon but I'll be spending more time with Premier and the Adobe suite as I leave my second string (FCP) behind.
Cheers
Mike
"Final Cut Pro project files are particularly notorious for corruptions and errors." I don't really agree with that if you use exactly all tips that you posted here (which I totally agree) and point your scratch disk correctly.
the ONLY and BIG advantage I see to use Premiere is that you can go to AME, After Effects, Encore all the suite easy and fast.
Very nice post.
thanks.
Almost is because i don't save a copy of a project everyday (i can't remember any project file corruption in 5 years) - i've set up autosave to every hour with 10 max project copies.
Also I'm glad someone realizes nonsense with dates, pounds, feets, kelvins and other royal stuff in U.S. :)
And btw, what is that weird workstation on the picture?
Andi
Cheers
Mike
Your point about unique file names for every file is salient. Though I don't believe it's entirely necessary. Each folder is unique and they are in turn stored in unique projects folders. True this means there are multiple files with the same name deep-down on the folder structure but their file paths will be unique. In a large broadcast environment with shared storage this could be a problem for file Search (though unlikely an issue for re-linking as re-linking is path relative. And then your metadata tagging and management provides the other means of sorting and connecting) Still you are right to point out the potential issue.
but for a small institution working discreet projects having unique folder names in unique project bundles should be enough.
And then of course backups and redundancy are crucial. These institutions I've been working with employ external harddrive backups which are the mirrored to either LTO, Server or, increasingly Cloud storage as retrievable archive.
Thanks for your thoughts
Mike
Owen, why not transcode to a smaller file, say SD. That is easy to work with (playable everywhere) but takes less room on the HDD. Then up-res the cut from the source HD files. Just like the olden days of öff-line".