XtreemFS - FAQ freqeuently asked questions

Frequently Asked Questions

General

Is XtreemFS open source?

Yes. All components of XtreemFS are licensed under the GPL v2.
Who is behind XtreemFS?

XtreemFS is developed as part of the EU funded research project XtreemOS. For a list of developers see the team.
In which way does XtreemFS differ from network file systems, like NFS or file systems made accessible via Samba/CIFS?

Unlike common network file systems, XtreemFS is a truly distributed file system that runs across multiple servers. It can be easily scaled by adding new machines. By means of striping, it can access files with the aggregated bandwidth of multiple storage servers. Besides, it offers X.509-based authentication and encryption support, as well as features like off-site replication, which renders XtreemFS particularly useful for wide-area networks.
In which way does XtreemFS differ from Lustre?

While Lustre was made for the use in a local cluster, XtreemFS has been especially designed for wide area networks. It can be mounted and accessed from anywhere. Features like replication, RAID and checksums guarantee a high data safety and availability, and make XtreemFS less susceptible to downtimes and crashes of single servers.
I have more questions about XtreemFS. Where can I find the answers?

Please send your questions, comments and suggestions to our public mailing list. It is open to everyone, users and developers alike.

Technical

What is FUSE?

FUSE ("Filesystem in Userspace") is a Linux kernel module that simplifies the development of new file systems. It is part of most recent Linux distributions.
What is a DIR, MRC, OSD?
- DIR - Directory service. Used by the MRC, OSD and client to find other servers, resolve UUID mappings and volume names.
- MRC - Metadata and Replica catalog. Stores the volumes with their directory tree and all metadata for files in the volumes. The MRC also stores the location of the replicas of all files.
- OSD - Objedct Sorage Device. This is a regular server, not a special device. These servers store the file content and implement the replication.
What is BabuDB and why does the MRC use it?

BabuDB is a lightweight database for persistently storing large numbers of key-value pairs. It was developed to offer an efficient and optimized database backend for the MRC. Similar to Google's BigTable, it is based on LSM-trees. BabuDB is a stand-alone library distributed under the BSD license. For more information, visit our BabuDB site at Google Code.
What are the scalability limits of XtreemFS?

Currently, a single volume may contain up to tens of millions of files. So far, no limits are known to the number of volumes and services.
What is FQDN/hostname based replica selection?

By default the client will use a random replica when multiple copies of a file are available. If you use FQDN (hostname) matching for replica selection, the client will use the storage server which has the longest match compared to it`s own hostname. E.g. a client machineY.departmentX.comanyZ.com will prefer to use osd1.departmentX.companyZ.com rather than osd1.departmentY.companyZ.com.
This selection mechanism is particularly useful in static environments where you know the network configuration and geographic distribution.
How do I put a volume in the fstab?

Add the following line to your fstab:

dir.server.com/volume_name /mnt/point xtreemfs defaults,allow_other 0 0

Configuration

How can I configure XtreemFS services?

For each service, there is a config file in /etc/xos/xtreemfs/.
What should I do if my OSD/MRC doesn't get its own IP/hostname right?

Detecting the local hostname and public IP address does not always work. The best way is to set the public hostname or IP address of your MRCs and OSDs in the config file. Set hostname = my.host.name or hostname = 192.168.1.1
What is .autodiscover?

setting dir.host = .autodiscover in osdconfig.properties or mrcconfig.properties allows the OSD and MRC to automatically discover your directory service DIR on the local lan.

Troubleshooting

One or more XtreemFS services failed to start up.

Please look into the log files to get an idea what was going wrong. They are located in /var/log/xtreemfs.
I cannot mount XtreemFS from a remote machine.

Please check your firewall settings on the server side. Are all ports accessible? The default ports are 32636 (MRC), 32638 (DIR), and 32640 (OSD).

Please make sure that you use the DIR service hostname for mounting and not the MRC hostname!
I cannot access a mounted volume.

Please make sure that you have sufficient access rights to the volume root. Superusers and volume owners can change these rights via chmod <mode> <mountpoint>. If you try to access a mount point that belongs to a different user, please make sure that the volume is mounted with xtfs_mount -o allow_other.
An I/O error occurs when trying to create/access/delete a file.

The easiest way to find out what went wrong is to check the client error log file. If you did not specify a path for the logfile, the client will create a file named mount.xtreemfs-PID.log in the working directory (in which you executed the mount.xtreemfs). In this file, you will find log entries with the MRC or OSD error messages, e.g.
2009-12-11T10:01:10.000Z ERR: yield::ipc::Client: connect() to :32636 failed, errno=111, strerror=errno = 111, strerror = Connection refused.
In addition, you can check the error logs of the services. They are located in /var/log/xtreemfs.
I cannot create new files, the client log says "... no suitable OSDs found..."
There are two ways to check the list of suitable OSDs for a volume:
- Open the MRC status page. It can be accessed via http://<MRC-host>:30636 in the default case. For each volume, a list of suitable OSDs is shown there.
- Execute getfattr -n xtreemfs.usable_osds --only-values <mountpoint>.
There may be different reasons for missing suitable OSDs:
- One or more OSDs failed to start up. Please check the log files and status pages of all OSDs to ensure that they are running.
- One or more OSDs failed to register or regularly report activity at the DIR. Please check the DIR status page to ensure that all OSDs are registered and active.
- There are no more OSDs with a sufficient amount of free disk space. Please check the OSD status page to obtain information about free disk space.
XtreemFS doesn't build because FUSE is located in a non-standard path.
Please check whether all OSDs assigned to the file are running and reachable. This can be done as follows:
1. Get the list of all OSDs for the file: getfattr -n xtreemfs.locations --only-values <file>.
2. Check whether the OSDs in (one of) all replicas in the list are running and reachable, e.g. by opening the status pages or via telnet <host> <port>.
The servers don't compile, javac complains about "unmappable character for encoding...".
You can specify the lib and include paths of FUSE as follows:
```
  export LIBPATH="<FUSE-lib-path>"
  export CPPPATH="<FUSE-include-path>"
```

Updating

update from version 1.1.x to 1.2.x

Nothing to do here, it's all automatic :). To be on the safe side, you should always dump your MRC before updating.
update from version 1.0.0 to 1.1.x
Version 1.1.x uses a new protocol version and a different on-disk database format for the directory service (DIR) and metadata server (MRC). You can safely delete the DIR database, all servers will re-register on start up. The DIR database is ususally stored in /var/lib/xtreemfs/dir/.
The MRC database must be dumped before installing the new release!
1. use xtfs_mrcdbtool -mrc oncrpc://mrchost:32636 dump /path/to/dumpfile to save the database to an XML file.
2. check that the file exists on the MRC host and is not empty!
3. shut down the MRC /etc/init.d/xtreemfs-mrc stop (and the DIR and OSD if they are still running)
4. remove the MRC database /var/lib/xtreemfs/mrc/
5. install the 1.1 release of XtreemFS
6. start the MRC /etc/init.d/xtreemfs-mrc start
7. use xtfs_mrcdbtool -mrc oncrpc://mrchost:32636 restore /path/to/dumpfile to restore the database from the XML file.