Trash or Treasure

Hidden Files Exposed

Matt Willmore Skip to comments 18 Comments (Comments Closed Closed)

Ever wonder what those .DS_Store files are that show up on your system? Matt Willmore is here to explain about it and all the other hidden files your Mac is creating.

When Apple changed the entire strategy of their flagship product, the Mac OS, they also changed the entire Mac community's perception of what an OS is and how it works. Since then, so much has changed. We used to scoff at command-line users; now we have joined their ranks in earnest. We used to look at UNIX, a legacy OS developed in the 1970s by AT&T, and see it as a has-been, a server-only mainframe OS that would never have any practical application in personal computing. Now, we realize the power of UNIX and exactly how well it fits into everything we do on the Mac and the resultant benefits.

UNIX is, for the most part, a strange new world for most Mac users. Luckily, OS X's Aqua takes care of 98% of the command-line tasks that UNIX requires. One must still wonder, though - what exactly makes up OS X? It's immediately evident that OS X (and other UNIX variants) are much more file-based than the classic Mac OS ever was. As a result, questions can arise asking just what these strange foreign files do and why they're on your computer.

The overall goal of this project is to document all of these files - yes, all of them. While it seems insurmountable as a whole, it becomes easier to analyze and document individual types of these files. Hidden files are the first type of file analyzed in this series, with many more on the way. I have done the best I can to document these files and their purpose, even when any existing documentation is hard to come by. My hope is to update and expand this documentation to update and append what is written so far.

Hidden Files

The idea of a hidden file has changed since the days of the classic Mac OS, where a file's "hidden" attribute was set in the resource fork of the file, and wasn't immediately accessible to the user. In OS X (and UNIX in general), it's much easier. All you have to do is preface the name of a file with a period ("."), and the file won't show up in the Finder, as well as Save/Open dialogs, etc. This can be a good way to hide a confidential file, although it's not terribly hard to find it. In UNIX, just typing ls won't reveal the file, but adding the -a flag, as in ls -al, will reveal invisible files as well.

Unfortunately, the Finder won't let you save/rename a file with a period as the first character. It is easy, however, to jump into UNIX and use the mv (move) command to rename the file, like this:

mv SecretFile.pdf .SecretFile.pdf

In most all cases the file should still be perfectly readable by apps, but your mileage may vary with certain apps. If you encounter a problem, send me an e-mail so I can add it to this.

If you would like to view hidden files without having to rename them, however, there's a simple way. By editing the com.apple.Finder.plist (preference list), you can have all invisible files shown. These two steps are all it takes:

  1. Type this into a new Terminal window:
    defaults write com.apple.finder AppleShowAllFiles -bool YES
  2. Restart the Finder. (Logging out/in is an easy way to do this.)
The files should now be visible in the Finder. To reverse it, insert the string above again, but change YES to NO.

.bash_history

The .bash_history file is pretty simple; it records the last commands you typed in bash, the default terminal app in 10.3. It stores a unique file for each user, and is stored in the home directory of the user logged in. When you hit the up arrow in bash, and see your recently used commands, this is the file it's referencing.

.cvspass

The .cvspass file stores successful authentications made to a CVS server, pretty much as typed in Terminal when the initial connection is made. It applies to the individual user and is stored in the user's home directory. Each saved connection has a separate line, and they are separated by carriage returns.

.DS_Store

The .DS_Store file is one of the most common invisible files you'll come across, simply because there are so many of them. The file, in a nutshell, contains information about a directory (and its files) in the Finder. The file is created whenever the directory is viewed in the Finder; thus, directories that have never been viewed in the Finder will not contain this file. The main variables that this file stores include:

The file itself is a proprietary binary format that no one has been able to decipher thus far; it if were open, we could do some really cool tricks and hopefully understand it much more. Until then, though, we can only look at its more noticable roles.

While there are obvious benefits to these files, sometimes they can be a nusiance. For example, browsing an OS X machine (or directories from one) will usually show these files unless you change the preferences of the FTP client. Another example is the use of software like radmind to manage lab computers. Because these software packages list every file on the system, it shows just how many of these files exist on your system. Some software packages (Stuff-It Standard 7.0.3, for example) will contain these files when you download the software. While this is the decision of the developer, most software packages do not contain this file.

While it may be important to save the .DS_Store file for frequent windows like /Applications and /Users/username, other directories don't really need the file kept around. Every .DS_Store file can be safely deleted, but the attributes it retains will be lost upon deletion. Deleting the file can also be beneficial in situations where window settings are not being retained or something else is occurring that might hint towards a corrupted .DS_Store file.

.forward

The role of .forward is that of managing e-mail: it contains rules used by procmail (UNIX mail processing utility), which applies those rules to incoming e-mail messages. E-mail applications in OS X do not by default use procmail, although it can certainly be set up use it. When used correctly, procmail can be an extremely effective tool against spam and sorting mail.

.forward's job is simple: store the rules the procmail uses. By default, OS X's contents of .forward contain /dev/null. This indicates that, until procmail is set up, all mail delivered to procmail will be rerouted to Deep Space Nothing. Unless you have a need to use procmail, there is no need to modify that file.

.hidden

The .hidden file is pretty simple in design and nature; it contains a plaintext list of root level files and directories that should be hidden in the Finder. It is located at the root directory of each OS X volume. It's not the only way to trigger invisibility in a file or directory, and some directories are marked as invisible in multiple ways. Needless to say, this is one of the easiest ways to make something invisible. OS X makes certain directories invisible because they really serve no purpose in the GUI. Here's the list of invisible files and folders in my system (10.3), for example:

automount, bin, cores, Desktop DB, Desktop DF, Desktop Folder, dev, etc, lost+found, mach, mach_kernel, mach.sym, opt, private, sbin, tmp, Trash, usr, var, VM Storage, Volumes

The file isn't actually formatted like this; each file or folder is listed individually, with a carriage return between each. You can easily look at this file in the Terminal with pico (you will need to su/sudo for this):

sudo pico /.hidden

If, for some reason, you wanted to add a file to the .hidden file, simply add the name, in correct alphabetical order, to the current list. Remember that the file or directory you enter must be in the root directory of the boot volume. After that, just restart the Finder; logging out/in is usually the easiest way.

.hotfiles.btree

This file is new to 10.3, as is the feature that uses it. 10.3 introduced a technology called "Hot-File-Adaptive-Clustering" (HFAC) that will do two tasks to improve the system:

  1. Whenever you open a file less than 20MB in size, HFAC checks it for fragmentation. (Files larger than 20MB will need to be defragmented by a 3rd-party utility such as Norton or DiskWarrior. If HFAC finds fragmentation, it repairs it automatically and moves the entire length of the file to a sector on the hard drive big enough for the file. The old storage sectors are then erased and declared empty for use again.
    (NOTE: For those of you that make use of Panther's government-approved "Secure Delete" option when emptying the Trash, this most likely does not apply that. Remember that a normal file deletion merely removes the reference from the file directory; the data is still there. Thus, the same can probably be said for the locations of the once-fragmented file pieces.
  2. HFAC also seems to be moving frequently accessed files to the fastest part of the hard drive (presumably the inner rings) for faster access times. This frequency would be measured over multiple restarts, longer than virtual memory would store frequent data.

This brings us to the purpose of the .hotfiles.btree file. A B-tree file stores the file directory so the system can access a file when requested (read more on the immense coolness of B-trees here. This file most likely stores the file directory of the "hot files" stored on the inner rings of the hard drive for fastest access. The directory is maintained here separately for easy (and fast) access.

.java / .jpi_cache

These two folders, located in the home directory, are used to support Java on OS X. The .java folder is presumably used to store Java applets and other configuration information; mine, however, was completely empty. The .jpi_cache file, on the other hand, contains cached files of anything Java applets you execute. Most of these files will probably have an online source, such as a Java-based game that you would play online. In the .jpi_cache you will find a folder (named "1.0" in my case), and within that directory you find a large number of files: multimedia files, Java classes (.class), and .idx (Internet Data Exchange) configurations.

.lpoptions

This file is used by CUPS (Common UNIX Printing System) to manage some printer attributes. lpoptions is a CUPS utility that is used to manage printers and printer roles (such as which is the default printer). This is different from OS X's printer management, which is also handled by CUPS. In most all circumstances there is no need at all to modify this file; however, if you are experiencing trouble printing, deleting this file may resolve your problem.

.nsmbrc

This file is used wih SMB to manage "favorite" server addresses using a simple, easy to read format. Here's an example entry (thanks to the ADC:

[WINBOX:JLDERA:JLDERA]

addr=192.168.0.7

password=mypass

workgroup=ARTOFTECH

The first line states [netbios_name:username:share]. This is self-explanatory. The second line is the IP address of the Windows share you're connecting to. The third line is your plaintext password. You should have the permissions of this file set to 600 to prevent anyone from root from messing with it.

chmod 600 ‾/username/.nsmbrc

The fourth is the workgroup that the Windows box is located in. Make sure you obey proper syntax when entering this, especially with a mixed-platform network.

Each setting is separated by two carriage returns.

.ssh

The .ssh folder in OS X normally contains one file named known_hosts. This is a user-specific file, and stored in the home directory of the user. The purpose of this file is to store information about known SSH hosts that you have logged into via SSH from your machine. If you have host aliases (eg. you don't have to type the entire address, but just a short name) then there may be multiple entries in your known_hosts files.

The structure of the file is simple. There is one known host per line, with a carriage return between each. The structure of each line is such:

DNS Name, IP Address (if DNS given), checksum algorithm, key fingerprint

.Trash(es)

As the name suggests, the .Trash and .Trashes directories store files that have been placed in the Trash but not deleted. The trick is to understand that there are actually multiple Trashes, and depending where a file was when you put it in the Trash, it could be in one of many .Trash(es) folders.

You will find a .Trashes folder in the root directory of each volume, as well as a .Trash folder in each user's home directory. The .Trashes folder of each volume will contain folders named with the UID of each person who has "contributed" to it. For example, the .Trashes folder of my boot volume has two folders: 0 (user = root) and 501 (user = willmore). If you look in these directories, you will find the files that each user has contributed. For example, if I were logged in as root and moved a file in the root directory of a secondary volume to the Trash, that file would now reside in /volume/.Trashes/0/file. If I deleted a file from within the home directory, however, it would be in /bootvolume/Users/username/.Trash/file. As a third example, if I deleted the file "delete.sh" from the /Library folder while logged in as smithj on my secondary volume, Pomona, it'd end in /Volumes/Pomona/.Trashes/501/delete.sh.

When you log in, the Trash "folder" you see represented by the trashcan in the Dock is "populated" with entries from your home folder's .Trash folder, as well as files in each folder with your UID in each volume's .Trashes directory. For example, when I log in as willmore on my machine, the Trash could contain files from the following directories:

/.Trashes/501/
/Volumes/Pomona/.Trashes/501
/Volumes/überPod/.Trashes/501
/Volumes/FireWire 80GB/.Trashes/501

The UID-named folders are not created until the user with that UID places something in that .Trash(es) folder. Additionally, the UID folder is not deleted when the Trash is emptied; it simply hangs around until it's used again.

.vol

The .vol directory's purpose is to serve the Carbon File Manager (CFM) in mounting the volsf file system. BSD only allows file or directory access by POSIX path; the CFM API also allows file or directory access via its catalog node ID (CNID). What volsf does is provide a bridge between the two, so that the CFM API and BSD's access controls may peacefully coexist.

This is another Apple deep-down-don't-touch things that you will never ever hav to mess with. If you are developing Carbon apps and notice (with fs_usage, for example) that .vol/ is being accessed a lot, this is why. If you look in the .vol/ directory, you'll see other directories, named by numbers. These are decimal equivalents of 32-bit volfs volume identifiers, whose actual volumes are internally mapped by Carbon. The volumes are dynamically created and managed by Carbon as the API is called and volumes are needed. Like I said before, this is deep stuff and should never be modified.

Note: The majority of the information for this hidden directory was obtained from Apple's Developer Connection site. Please visit the link for more information on .vol/ and volfs.

.Xauthority, .Xauth-OSX, .Xresources, .Xdefaults, .Xftcache

The .Xauthority file is used by the UNIX app xauth for saved X11 remote access authorizations, similar to the rlogin/xlogin files. the .Xauthority file stores "magic cookies", which are created strings that X11 (the local X server on your machine) uses to negotiate with the remote X server you're trying to access. If someone were to gain access to your .Xauthority file, others could have that access to remote apps you use. For this reason, the .Xauthority file is stored in the home directory of each user who has ever started an X11 app, whether it be Apple's or another one, like Fink.

The .Xauth-OSX file is an OS X-specific variant of the .Xauth file found on all UNIX systems with X11. As far as I can tell it stores OS X-specific preferences for xauth and/or X11 use on the Mac.

.Xresources is used when the X11 window manager starts up for the first time. Preference values are stored within. .Xdefaults is basically the same thing, except that it can be used at any time during the X11 session. .Xdefaults and .Xresources allow you to customize your X11 windows, such as xterm.

.Xftcache is used by X11 to store a text cache of available fonts for X11. Presumably, it is created when X11 starts up (and no such file exists), or modified (if the file is there) to match the current font list.

Because of the impossibly large number of applications and tools that use hidden files to various types of data, I know that I haven't documented every hidden file that you might come across. Therefore, if you have any hidden files you would like explained and/or documented, please send them to me and I will append this article with that new information.

Matt WillmoreMatt Willmore is a founding partner of MacZealots.com. Matt is also a Resident Assistant at Owen Hall and does Mac support at ECN, and is active in PUMUG. He can be reached at .

Reader Comments (18)

DISCLAIMER: The views expressed below are those of their authors and not necessarily endorsed or supported by MacZealots.com. In all cases, the comments provided here are offered as a courtesy and will be moderated. Any content deemed off-topic or offensive will be removed without notice. Posting a comment here boils down to two things: 1.) Think before you type 2.) Respect the thoughts of others. See our commenting guidelines and/or privacy policy for more information.

1 Darlana remarks:
#1) On December 12, 2003 9:34 AM

Single and willing to mingle?! WHOA YES.

I’m a ravishingly beautiful 34 year old hunk of man-flesh! I swing all three ways… often at the same time ;)

2 nibs remarks:
#2) On December 12, 2003 12:37 PM

if you enable the AppleShowAllFiles flag in the finder preferences, the finder will allow you to create and manipulate dot files and invisible files…etc.

defaults write com.apple.finder AppleShowAllFiles -bool YES

and then restart the finder.

without that, renaming a file with a dot in front would cause it to immediately disappear, which is bad. ;-)

3 Matt remarks:
#3) On December 12, 2003 4:03 PM

Thanks for the hint - I’ll add it into the article!

Apple does have a provision in place to prevent files from disappearing. If you try to insert a “.” at the front of any file, folder, etc., the Finder will shoot up a dialog box:

“You cannot use a name that begins with a dot “.”, because these names are reserved for the system. Please choose another name.”

4 Scott J. Kramer remarks:
#4) On December 13, 2003 1:54 AM

“Finder info” comments are stored in volume headers of filesystems, not .DS_Store files within individual directories.

5 Andy Hughes remarks:
#5) On December 22, 2003 4:43 PM

Matt:
Nice article and great job on the MacZealots web site! I look forward to more.

— andy

6 Matt Willmore remarks:
#6) On March 7, 2004 5:53 PM

In response to Scott Kramer, here’s some information I got from Justin, author of CommentSync:

“The OSX comments are stored in the .DS files, though it is very difficult to directly extract them, I had to use Apple’s own processes to get to them (ie BBEdit won’t work). And the OS9 files are in the HFS metadata - so even MORE difficult to get at. Over all, I have spent quite a long time trying to find the comments directly, but have been very unsuccessful and have simply resorted to using MacOS routines.”

7 Jason Arnold remarks:
#7) On March 29, 2004 4:51 PM

The hidden files on my comp are all being shown. Im not sure how to change it back so they are all hidden again? Do you have any suggestions? Jason Arnold

8 Matt Willmore remarks:
#8) On March 29, 2004 5:54 PM

Jason -

Go into the Terminal and type:

defaults write com.apple.finder AppleShowAllFiles -bool NO

Log out and back in, and that should fix it. Please let me know if it doesn’t. Thanks!

Matt

9 Jeff remarks:
#9) On May 30, 2004 5:40 PM

How do you find hidden files in OS9?

10 Mike remarks:
#10) On June 13, 2004 12:30 AM

Sorry, I’m not to computer saavy and I’m a relatively new fan of the mac. I’m wondering what is the terminal?I’m having trouble finding some video files I misnamed at school.

11 Jeff remarks:
#11) On October 19, 2004 11:53 PM

I successfully used the reveal hidden files command in the terminal (and have in the past), but I still have one problem. Is there any way to search for the hidden files? The apple search box at the top of windows ignores the hidden files. Is there anyway to search for these files (like through the use of third-pary software)?

12 Matt Willmore remarks:
#12) On October 20, 2004 12:33 AM

Jeff -

Instead of using the find box in the Finder window, use the Find window (command-F or File -> Find…). In the criteria, add a second one, select “Visibility” from the popup, and select “visible and invisible items” to search through everything, or “invisible items” to only search through the hidden ones. Hope this helps, and thanks for reading!

13 Nicholas Ink remarks:
#13) On July 23, 2005 10:26 AM

Tiger doesn’t have a /.hidden file, does it?

14 Matt Willmore remarks:
#14) On July 25, 2005 1:41 AM

Nicholas -

That is correct – there is no .hidden file shipped in Tiger. Now the real question is, does the .hidden file still work?

To hide something, the easiest way is still to name it starting with a period; however, there’s obviously something else at work hiding all the system folders by default.

15 Nicholas remarks:
#15) On July 25, 2005 9:10 AM

Actually, I think it does. I run 10.4.2 and it does hide some files. However, the file does not have to be present for most of the hidden files in the system.

16 aman remarks:
#16) On August 31, 2005 7:27 AM

how about .localized? i found it on my desktop.

17 iFrasier remarks:
#17) On September 18, 2005 8:39 AM

‘.localized’ files are for internal OS control only. They contain no data at all, and they are used only as a flag the OS uses to know if the name of a special folder (like ‘Applications’) has to be displayed on Finder depending on the language the operating system is installed.

For instance, talking about the ‘Applications’ folder. If you open a terminal and go to the root (/) you’ll see the ‘Applications’ folder that contains the user common applications on the system. In an english version of MacOSX this folder is displayed as ‘Applications’ on the Finder, BUT in a spanish version of MacOSX, and if there is a ‘.localized’ file in ‘Applications’ folder, this folder will be displayed as ‘Aplicaciones’ (although if you open a terminal and you see the real name of the folder is ‘Applications’).

The same is used for other system folders.

18 karl Hanzel remarks:
#18) On March 16, 2006 2:17 PM

OK, so in 10.4, how are system directories like /usr then hidden from the finder, if not with .hidden? Perhaps it’s a “built-in feature”. Drat.

*——->