Sustainability of Digital Formats: Planning for Library of Congress Collections

The Desktop Services Store (.DS_Store) file format, developed by Apple Inc., is a proprietary, hidden file created by the Macintosh OS X Finder. DS_Store files are not human-readable and consist of binary data, store information and folder view options, icon positions, and other visual details related to directories.

Notes on terms in this document: Desktop Services Store files are often referred to as ".DS_Store" with a leading period (The leading period classifies the file as invisible according to UnixOS, and thus macOS, conventions.) However, for syntactic reasons in the text, this resource refers to them as "DS_Store" without the leading period. Also "Macintosh OS X" or "Mac OS X" is now known as "macOS").

Each directory accessed with the macOS Finder results in the creation of a corresponding DS_Store file in that directory. The file is permanently generated after viewing the directory with the Finder, and currently, there is no option to disable its creation. The DS_Store file is essential for the Finder to maintain custom attributes of its containing folder, such as icon positions and background image choices.

While its internal structure is proprietary, it has been reverse-engineered. Any attempt to migrate the format, extract files from the binary container, or archive the files could render the code non-functional, as stated by the National Archives and Records Administration (NARA). Overall, DS_Store files are scattered throughout the directory tree, appearing in almost every folder visited with the macOS Finder.

The DS_Store file is prefixed with a dot which hides the file macOS Finder, potentially leaving Mac users unaware of its existence. The file format is proprietary with limited documentation available online regarding its structure and functionalities.

File Structure

The DS_Store file is structured to hold records containing attributes of files within the directory or the directory itself. These records are organized in a B-tree. The B-tree pages, along with metadata, are managed by a "buddy allocator." The allocator introduces a level of indirection, allowing for the relocation of blocks as they grow or shrink.

The file is predominantly big-endian, and an "integer" within it typically refers to a four-byte (32-bit) big-endian number, likely unsigned. The file begins with a 36-byte header that sets the stage for the subsequent content.

The root block section follows the header, storing crucial metadata about the file names in a tree-like structure. This metadata can be categorized into three sections: Offsets, Tables of Content, and Free List. Within the Tables of Content section, there is usually at least one table named DSDB with the value 0x01.

Of particular importance to users are Finder or Spotlight Comments, displayed in the Comments section of the Get Info dialog for a file. These comments may also exist in the com.apple.metadata:kMDItemFinderComment extended attribute (xattr) of the file. However, this is a secondary copy that can fall out of sync with the DS_Store file, and the Finder disregards the xattr.

The structure of DS_Store files is relatively opaque due to the lack of documentation. Knowledge of the file structure is from community reverse engineering. Comments welcome.

Record Structure: Contents

Each DS_Store file contains one or more records with specific formatting which capture diverse metadata about directory components and utilize various data types to represent structured information. Wim Lewis’s article “DS_Store Format” and Mozilla Wiki’s “DS Store File Format” article define the specific components as:

Format of a Record:

• Filename length (4 bytes), an integer.
• Filename (2 * length bytes), in big-endian UTF-16, following HFS+ filesystem Unicode composition rules.
• Structure id or type (4 bytes), a FourCharCode indicating the property of the file this entry describes.
• Data type (4 bytes), indicating the kind of data field that follows.

Structure Type:

• Records information about a directory's contents, including the directory itself, files, subdirectories, icon types, window size and layout, plists (preference/saved settings), and view style (e.g., icon view, column/browser view, list view, and coverflow view). This information is crucial metadata.

Data Types:

• Data types encompass structured data such as integers, boolean types, text strings, or datestamps. These types define the content of the data field following the structure id or type in a record.

Record Structure: B-tree

The records within DS_Store files are organized in a B-tree structure. This B-tree structure includes a small master block with statistics and a pointer to the root node, along with leaf (external) nodes and non-leaf (internal) nodes. The management of B-tree pages and related information is facilitated by a "buddy allocator."

Security

Security concerns can arise when DS_Store files are uploaded to web servers, as they contain crucial folder information that could be exploited to extract details about system configurations. These files hold the names of all files and directories within a folder. Since 2001, security issues and potential vulnerabilities related to DS_Store files have been highlighted. Exploitation could lead to the disclosure of sensitive information, including system configuration and installed applications.

An example illustrating the severity of such vulnerabilities is the HackerOne report on X, formerly known as Twitter, where a DS_Store file revealed private information.

For further insights into security issues related to DS_Store files, the Internetwache.org article "Scanning the Alexa Top 1M for DS_Store Files" provides additional information. It emphasizes the importance of addressing potential risks associated with the presence of DS_Store files on web servers.

Versions

The DS_Store format does not have any official documentation or a specification. There is unofficial documentation created through reverse engineering. It is likely that the version changes with versions of macOS. Comments welcome.

Removal

According to OSX Daily, deleting DS_Store files from Mac folders can result in the loss of specific folder details, such as icons, window placement, and background pictures. While there is generally no significant harm, as these files primarily store folder metadata, it is advisable to retain them unless there is a specific reason for deletion. The macOS Finder relies on DS_Store files, and they are automatically regenerated by the system if deleted. Deleting them might be necessary in certain Windows+Mac shared networking environments.

Some digital preservation systems, such as Archivematica, automatically remove DS_Store files by default, as noted in the Archivematica GitHub codebase.

DS_Store has no official documentation.

Some sources direct users to Mark Mentovai’s reverse engineering work, c. 2007, on the format, hosted on the Mozilla Wiki.

Unknown. Comments welcome.

See: Wikipedia: https://en.wikipedia.org/wiki/.DS_Store

The National Records and Archives Administration (NARA) cites Wikidata for this value, however the Wikidata entry itself does not include the value.Comments welcome.

Two sources, based on reverse engineering, cite these magic numbers:

• “ Parsing the .DS_Store File Format,” by Sebastian Neef.
• “ Mac Dumpster Diving – Identifying Deleted File References in the Trash (.DS_Store) Files – Part 1,” by Jonathon Polig.

Comments welcome.

According to Arno Gourdol’s article “On the Origins of .DS_Store,” in 1999, Apple engineers rewrote the macOSX Finder application. This overhaul aimed to separate the user interface (Finder_FE) from the core functionality (Finder_BE), which handled tasks such as file enumeration, monitoring filesystem changes, and managing metadata, including icon locations and folder settings. .DS_Store emerged as a component integrated into Finder_BE, serving as a repository for private data crucial to the Finder's operations.