Home Home
Sign up | Login

Author Archive

Local caching for CIFS network file system – followup

August 5th, 2010 by

Here’s a follow-up to my previous post on Hackweek V: Local caching for CIFS network file system

Since the previous post, I worked on improving the patches that add local caching, fixed a few bugs, addressed review comments from the community and re-posted the patches. I also gave a talk about it at the SUSE Labs Conference 2010 took place at Prague. The slides can be found here: FS-Cache aware CIFS.

This patchset was merged in the upstream Linux kernel yesterday (Yay!) which means this feature would be available starting from kernel version 2.6.35-rc1.

The primary aim of caching data on the client side is to reduce the network calls to the CIFS Server whenever possible, thereby reducing the server load as well the network load. This will indirectly improve the performance and the scalability of the CIFS Server and will improve the number of clients per Server ratio. This feature could be useful in a number of scenarios:

– Render farms in Entertainment industry – used to distribute textures to individual rendering units
– Read only multimedia workloads
– Accelerate distributed web-servers
– Web server cluster nodes serve content from the cache
– /usr distributed by a network file system – to avoid spamming Servers when there is a power outage
– Caching Server with SSDs reexporting netfs data
– where a persistent cache remains across reboots is useful

However, be warned that local caching may not suitable for all workloads and a few workloads could suffer a slight performance hit (for e.g. read-once type workloads). So, you need to careful consider your workload/scenario before you start using local disk caching.

When I reposted this patchset, I got asked whether I have done any benchmarking and could share the performance numbers. Here are the results from a 100Mb/s network:

Environment
————

I’m using my T60p laptop as the CIFS server (running Samba) and one of my test machines as CIFS client, connected over an ethernet of reported speed 1000 Mb/s. ethtool was used to throttle the speed to 100 Mb/s. The TCP bandwidth as seen by a pair of netcats between the client and the server is about 89.555 Mb/s.

Client has a 2.8 GHz Pentium D CPU with 2GB RAM
Server has a 2.33GHz Core2 CPU (T7600) with 2GB RAM

Test
—–
The benchmark involves pulling a 200 MB file over CIFS to the client using cat to /dev/zero under `time’. The wall clock time reported was recorded.

First, the test was run on the server twice and the second result was recorded (noted as Server below i.e. time taken by the Server when file is loaded on the RAM).
Secondly, the client was rebooted and the test was run with caching disabled (noted as None below).
Next, the client was rebooted, the cache contents (if any) were erased with mkfs.ext3 and test was run again with cachefilesd running (noted as COLD)
Next the client was rebooted, tests were run with caching enabled this time with a populated disk cache (noted as HOT).
Finally, the test was run again without unmounting or rebooting to ensure pagecache remains valid (noted as PGCACHE).

The benchmark was repeated twice:

Cache (state) Run #1 Run#2
============= ======= =======
Server 0.104 s 0.107 s
None 26.042 s 26.576 s
COLD 26.703 s 26.787 s
HOT 5.115 s 5.147 s
PGCACHE 0.091 s 0.092 s

As it can be seen when the disk cache is hot, the performance is roughly 5X times than reading over the network. And, it has to be noted that the Scalability improvement due to reduced network traffic cannot be seen as the test involves only a single client and the Server. The read performance with more number of clients would be more interesting as the cache can positively impact the scalability.

Hackweek V: Local caching for CIFS network file system

June 14th, 2010 by

Hackweek

It’s that time of the year when SUSE/Novell developers use their Innovation Time-off to do a project of their interest/wish – called as Hackweek. Last week was Hackweek V. I worked on making the Common Internet File System (CIFS) cache aware, i.e. local caching for CIFS Network File System.

Linux FS-Cache

Caching can result in performance improvements in network filesystems where access to network and media is slow. The cache can indirectly improve performance of the network and the server by reduced network calls. Caching can be also viewed as a preparatory work for making disconnected operation (Offline) work with network filesystems.

The Linux Kernel recently added a generic caching facility (FS-Cache) that any network filesystem like NFS or CIFS or other service can use to cache data locally. FS-Cache supports a variety of cache backends i.e. different types of cache that have different trade-offs (like CacheFiles, CacheFS etc.) FS-Cache mediates between cache backends and the network filesystems. Some of the network filesystems such as NFS and AFS are already integrated with FS-Cache.

Making CIFS FS-Cache capable

To make any network filesystem FS-Cache aware, there are a few things to consider. Let’s consider them step by step (though not in detail):

* First, we need to define the network filesystem and it should be able to register/unregister with the FS-Cache interface.
* The network filesystem has to define the index hierarchy which could be used to locate a file object or discard a certain subset of all the files cached.
* We need to define the objects and the methods associated.
* All the indices in the index hierarchy and the data file need to be registered. This could be done by requesting a cookie for each index or data file. Upon successful registration, a corresponding cookie is returned.
* Functions to store and retrieve pages in the cache.
* Way to identify whether the cache for a file is valid or not.
* Function to release any in-memory representation for the network filesystem page.
* Way to invalidate a data file or index subtree and relinquish cookies.

Implementation

I wanted to get the prototype working within a week. So the way I have implemented it is rudimentary and has lot of room for improvement.

The index hierarchy is not very deep. It has three levels – Server, Share and Inode. The only way that I know of identifying files with CIFS is by ‘UniqueId’ which is supposed to be unique. However, some server do not ensure that the ‘UniqueId’ is always unique (for example when there is more than one filesystem in the exported share). The cache coherency is currently ensured by verifying the ‘LastWriteTime’ and size of the file. This is not a reliable way of detecting changes as some CIFS servers will not update the time until the filehandle is closed.

The rudimentary implementation is ready and the cumulative patch can be found here:

http://www.kernel.org/pub/linux/kernel/people/jays/patches/

[WARNING: The patch is lightly tested and of prototype quality.]

Here are some initial performance numbers with the patch:

Copying one big file of size ~150 MB.

$time cp /mnt/cifs/amuse.zip .
(Cache initialized)

real 1m18.603s
user 0m0.016s
sys 0m8.569s

$time cp /mnt/cifs/amuse.zip /
(Read from Cache)

real 0m28.055s
user 0m0.008s
sys 0m1.140s

Making technology Previews succeed – OSC 09 Unconference session notes

October 9th, 2009 by

I did an Unconference at openSUSE Conference 2009 titled: Roads Less Travelled – Making Technology Previews succeed“.  More number of people than I had expected participated in the unconference session. I wanted to make the discussion notes (rough) available to a wider audience so that we could act on some of those:

Often Technology Previews are not solving their purpose. The objective of this session was to discuss, find out how we could get better feedback on Technology Previews and make them better.

The discussion is focused primarily on these areas:

  • Advertise the feature through proper channels, places
  • Make it easy enough for users to try out and provide feedback
  • Make it less risk-prone
  • What stops users from trying out?
  • Provide better documentation?

Key discussion points, suggestions:

  • Announcement in opensuse.org main page/wiki could grab the attention of community members who could help test Technology Previews.
  • Reduce hassles in providing feedback. For e.g Perhaps facility without authentication/Single-signon?
  • Easy ways/methods to provide feedback/input
  • Bug/Issue reporting made easy, command line tools?
  • Text area to provide feedback (as opposed a authentication based system).
  • Create a dedicated page for preview for e.g. previews.opensuse.org
  • irc channel for previews? (Discussion on all TPs, User testing one TP might get interested in another)
  • Announcement in openSUSE Weekly news could help
  • More Blogs, Articles, Whitepapers etc.. (blog entries should have provision for giving comments)
  • Perhaps, try to get some help from documentation team?
  • Provide instructions, mechanisms to safely try out without breaking things.
  • Additional information about new technology while the community tries to use the old technology For e.g. while a user tries to do nfsv3 mount providing an informational message that NFSv4 is available and can be used
  • Suggest using a VM
  • Caution about what might break and what might not (Make community feel less riskier to try out).

Please feel free to comment on what might work and if you have any more suggestion.

The short presentation I used to introduce the topic can be found here: http://files.opensuse.org/opensuse/en/d/de/Roads_Less_Travelled.pdf