Archive for October, 2013

Preface

This Article is intended for developers experienced with C++ and Low-Level Windows API programming

Introduction

The focus of this article is to discuss an autonomous method of generating dump files w/o the need of any development tool installed.

It starts by giving a high level explanation of what dump files are and what they are used for, then, it present few of the most common development tools used to generate dump files and discuss windows exception model, finally, a way of generating dump files w/o the need of any development tool is presented.

So what a Dump file is?

A dump file is the image of the process at a certain point in time, this process image can include various information such as the call stack & stack variables, loaded module list, and even an image of the raw memory used by the application.
This valuable information can then be used to analyze the process state at the time the dump file was generated.

What is it used for?

In most cases ( but not only ) Dump files are used to identify the root of an exceptional condition causing the process to abnormally terminate ( a 2nd chance exception ), having a dump file generated just before the application has crashed will enable postmortem analysis of the process state when it has crashed, and thus, enables pin-pointing the root of the problem.

Using MS Visual Studio to generate memory dumps

Microsoft Visual Studio enable generation of memory dumps while breaking the execution of a debugged process, this can be done through the Debug->Save Dump As menu item as illustrated in Figure 1 bellow

Figure 1

Two dump file types are support by the IDE, a ‘minidump’ that include stack trace information ( resulting small files ), and a ‘minidump with heap’ including the full memory image ( resulting large files ).

Using ADPlus to generate dump files

Debugging tools for windows is a light weight suite of tools for debugging applications, It is ideal for customer site problem resolution, and for scenarios where it is not possible to install heavy duty development environments such as Microsoft Visual Studio.

ADPlus ( also known as ‘AutoDump+’ ) is a light weight tool used to automatically generate Dump files, that is, upon abnormal process termination a Dump file will automatically be generated enabling postmortem analysis of the process state when it has crashed, it also support automatic dump generation upon deadlocks, Figure 2 bellow present sample ADPlus command line.

ADPlus.exe –crash –pn winword.exe –o d:\Dumps

Figure 2

The above attach ADPlus to winword.exe and generates dump files at ‘d:\dumps’ upon winword.exe crash, click here for the full command line specification.

Analyzing Dump Files

Dump file analysis is the phase where postmortem takes place, Starting with Microsoft Visual Studio 10, it is possible to directly analyze dump files for un-managed applications through the IDE, this is done through the “File->Open->’File…’” menu and then by selecting the dump file to analyze ( ‘*.dmp’, ‘*.mdmp’, ‘*.hdmp’ extensions ).

Once opened, Click the ‘Play’ Icon and the IDE will take you to the point where the application was breaking.

It is important to note that for Dump Analysis to properly work it is essential to keep the symbol files ( .pdb ) associated with the executable for which the dump was created, these should then be used during the analysis process.

Dump file analysis for managed applications is supported by debugging tools for windows and will be covered in a specialized Article.

Process termination due to Exceptional condition

A Process might be abnormally terminated due to an exceptional condition preventing normal process execution, such an exceptional condition is usually due to a programming error ( a SW bug ), A list describing common exceptions can be found here.

The operating system use Structured exceptions to indicate such exceptional behavior, the executing application will get the first chance to deal with the exception, and if not dealt with or if dealt with but not suppressed, the operating system will get the second chance to deal with it, having 1st and 2nd chance exceptions respectively, most of the time when 2nd chance exceptions are generated the operating system will terminate the application ( a crash ), exceptions are eg. debugger breakpoints ( DebugBreak() ) where once intercepted, the OS will open a dialog letting the programmer to choose if he wants to debug the application ( assuming a debugger is installed ) or supress the exception.

Generating Dump files upon abnormal termination

No more than few lines of code are needed to be able to automatically generate dump files when the application is crashing, Figure 3 bellow demonstrate what is needed.

Figure 3

The above code snap uses Structured Exception handling to intercept 2nd chance exceptions, this is done by installing the Unhandled exception handler ‘__TopLevelExceptionHandler’ ( using SetUnhandledExceptionFilter ) that intercept all 2nd chance exceptions.

Once an exception has been intercepted ‘__TopLevelExceptionHandler’ is invoked and does the actual dump file generation.

The un-handled exception handler ( in our case ‘__TopLevelExceptionHandler’ ) is executed on the context of the thread throwing the exception, thread stack is not collected while the handler is executed, this, might limit the exception handler implementation on stack overflow scenarios where there might not be enough space left on the stack to execute the handler functionality, for this, ‘__TopLevelExceptionHandler’ create a separate thread where the actual ~dumping~ process will synchronously execute.

The actual dumping process is executed by the ‘__GenerateDumpFile’ method, in specific by using the MiniDumpWriteDump API.

By default the dump file will be generated at the directory of the executing process, the name of the file include the time, the exception code, and, the name of the process.

The code can easily be integrated in to any C++ application enabling automatic dump file generation, and, reducing the cost of customer site probelm interception.

Final words

I was trying to have the code provided with this article as clear & simple as possible, The generated dump files might take considerable disk space, integrating this code with any commercial product will req implementation of a dump file recycling mechanism.

Introduction

MP4 is a widely used container format for multimedia files, it is an extension of Apple’s QuickTime file format, and is agnostic to the actual codec used for encoding, It can contain multiple streams of video, audio and data ( eg. subtitles ).

MP4 files are broken into two main parts, the payload part where interleaved audio & video are stored and the metadata part where information describing the payload is stored,
that information consists, for example, of the available streams, their payload format/compression type, …

So what are we trying to solve?

MP4 Metadata of specific importance is the file Index, the index is pointing to the offset of the file where the payload ( eg. video ) of a specific time is found, this way, the player knows where the payload for the first video frame is found, and, what data to play at a given time.

The following present a high-level view of the MP4 file structure:

When ~recording~ a video file, the duration of the file and amount of recorded data can ( obviously ) be known only once recording has finished, and thus, the Index is stored at the end of the file.

MP4 files are commonly used on web sites for Video playback, To play the file, a player ( eg. Web Browser ) must read the file from the remote site, files are read sequentially, starting at offset zero.

A player must read the Index before processing any video payload, and thus, must read the file up to it’s end ( where the index reside ) before being able to present the first video frame, for big MP4 files, this limitation might cause playback to start a long time after the play button was actually clicked rendering a poor the user experience.

In this article I will show how to reduce playback latency to a minimum by moving the metadata chunk from the end of the file to it’s start making it available for the player to consume before the first video payload is read, and thus, enabling playback to commence before the file was fully downloaded to the client machine ( also known as progressive download ).

Basic File structure

In accordance with Chapter 1 ( Page 17 ) of the QuickTime file format, The basic structure of MP4 files consists of a data chunk of data called an ATOM, each ATOM has a unique id ( uuid ) and size ( in bytes ).

Specific ATOMs contain data, and others contain a set of other child ATOMs.
ATOMs can have a ‘size’ indicator of either 32bit or 64bit, in this article we assume a 64bit size indicator, the following is the 64bit ATOM structure:

    struct ATOM {
        UINT64	size;
        union {
            UINT uuid;
            CHAR name[4];
        } type;
    };

The following figure present a typical ATOM hierarchy:

There are three types of atoms we will need to deal with in specific

‘mdat’ >This atom is used to hold the raw media data such as compressed Video & Audio samples, the media data is stored according to time in an interleaved fasion as can be seen in the figure to the right:
‘moov’ Holds all metadata related with the media file, “it is essentially a container of other atoms. These atoms, taken together, describe the contents of a movie. At the highest level, movie atoms typically contain track atoms, which in turn contain media atoms. At the lowest level are the leaf atoms, which contain non-atom data, usually in the form of a table or a set of data elements. For example, a track atom contains an edit atom, which in turn contains an edit list atom, a leaf atom which contains data in the form of an edit list table.”
‘stco’ An indirect child of the ‘moov’ atom, available on a per media stream ( ‘trak’ atom ) basis, pointing to the offset of the media payload directly in the ‘mdat’ section, the following simplified diagram present a possible configuration:

Implementation

Moving the Metadata ( ‘moov’ ) ATOM to the beginning of the file require modification of the ‘stco’ offsets so they will be aligned with the new ‘mdat’ position.

The process is finalized by iterating through all of the ‘stco’ ATOMs and updating the offsets after moving the ‘moov’ ATOM to the beginning of the file.

The project consists of A single ‘.cpp’ file implementing the logic described in this article, for simplicity memory mapped files were used for file modification and access

While developing on Windows OS, The code implementation was made as simple as possible so it could easily be ported to any platform.

References

QuickTime file format
Movie Atoms
MP4 Spec