Phoronix has run some tests comparing the openSUSE 11.1 release candidate (RC1), Ubuntu 8.10, Fedora 10 and Mandriva 2009.0 on Intel Atom.
We have looked at the results and they are not good for openSUSE 11.1. I’ve talked with a few engineers and want to present below our first analysis.
While the benchmarks were done on a specific hardware, they might be relevant for other hardware as well.
Note that the numbers I cite below are not benchmark numbers comparable to the one Phoronix measured, they are measured on totally different machines by different engineers and not all are done as real benchmarks. But they show some of the problems.
Disk I/O – Safe or Fast?
We have barriers enabled for the ext3 filesystem. This is needed for filesystem integrity but forces at certain points a flush of disk writeback caches to prevent data corruption.
As Wikipedia states:
“Ext3 does not do checksumming when writing to the journal. If barrier=1 is not enabled as a mount option (in /etc/fstab), and if the hardware is doing out-of-order write caching, one runs the risk of severe filesystem corruption during a crash.”
With openSUSE barrier=1 is the default and even AFAIK openSUSE and SUSE Linux Enterprise are the only distributions enabling barriers by default.
If you want to disable barriers, use “mount -o barrier=0” on ext3 (or change /etc/fstab).
The gzip test for example gives on one of our machines the following results:
kernel-default-2.6.27.8-1.1 / ext3: 1348s
kernel-default-2.6.27.8-1.1 / ext3 barrier=0: 437s
X11 and Graphics – Performance of the Intel Driver
Looking at the graphics results, I see that OpenGL has the same performance but XRender is horribly slow, but Ubuntu sometimes(!) hits the same issue.
We have an upstream bug open about X11 speed (see here), and it’s considered the highest priority bug, still nobody has a clue where it comes from. This needs to be rechecked with the final version of openSUSE 11.1, though, because there are some indications that it got improved. Intel has apparently fixed some of that in a newer driver that is not available yet.
It would also be interesting to know whether Mandriva uses XAA or EXA, we do not use XAA for Intel driver any longer, since Intel as driver author does not want to support it, and it has issues with suspend and resume. The old XAA is currently better optimized than the new EXA.
Performance is Important
Performance is important for us and our engineers are working on performance improvements and try to help with known regressions.
For example, if you look at discussions about the tbench regression (LWN reports about these at “Tracking tbench troubles” and “Tbench troubles II“, you see a couple of Novell engineers involved including Rafael Wysocki and Jiri Kosina – and we have run quite some benchmarks to help track and fix these problems.
Call for Action
I’m interested to see what’s we can do to improve the performance and invite you to discuss on the openSUSE-factory mailing list what to do. Contribution in benchmarking and figuring out bottlenecks is important as well.
Conclusion
So, whom does the problems hit – and how hard? So far, I see two culprits: The ext3 filesystem with barriers enabled and the Intel graphics driver. If you’re not using either, you’re on par with other recent distributions. 🙂
The ext3 filesystem regression can be worked around at the risk of filesystem integrity with disabling barriers. I doubt that I will hit this during normal desktop usage on my system.
The Intel issue has been discussed already at the openSUSE-factory mailing list and it was even suggested to not ship openSUSE 11.1 because of this. It hits some Intel graphics driver users quite card
and I wonder why I don’t see it on my machine. It might be better/worse on specific graphics chip.
If this issue is fixed with a new driver that works in our 11.1 setup, we will consider doing an update via our update repositories.
Both comments and pings are currently closed.
“The Intel issue has been discussed already at the openSUSE-factory mailing list and it was even suggested to not ship openSUSE 11.1 because of this.”
Uf. I hope it’s a bad joke and it will be fixed.
We do not know when it will be fixed and therefore did not delay the release indefinitely for one driver. AFAIK this mainly effects 3D and compiz – so bad but there are ways to turn it off. I also hope it gets fixed.
Thanks for the insight, I somehow had an awkward feeling that opensuse should perform THAT bad in some.
I also have an integrated intel and composite desktop runs kinda slow, but you guys are capable of fixing this, a pushed update will do just fine.
Perhaps on the mount options, it’d help to make them clearer, if when defaults differ from what other distros are doing, that they are explicitly written into fstab(5)?
It seems horrible and pointless to have journal filesystem, without the barriers. How is it better than the writeback cache or ext2? Perhaps reiserfs would hold up in benchmarks better against ext3 & ext4, and yes I know it is effectively single threading I/O thanks to kernel lock, but may be that’s not such a disadvantage on the desktop. I really would not have expected creating 1 file, and having it grow, would present any danger to the file system, once it’s been opened. But I suppose there’s block allocation issues.
The SQLlite benchmark is the one that bothers me the most, is that barrier related?
For CPU intensive benchmarks, pipes (or if must be output files in a tmpfs filesystem), would isolate impact of filesystem issues, for instance computing a checksum of the stdin for verification purposes.
Since I wrote this, I came across comments by Andrew Morton, and a thread discussing barriers and why he refused a patch to make them a default. The sequential layout of journal, in general means the disk does the writes in right order, except when wrapping round (relatively rare); meaning in general running without barriers is a problem only for the unfortunate.
Other filesystems like XFS, may frequently corrupt disk data files on power-failure, so running with a UPS supporting ordered shutdown is advisable.
One performance issue in 11.1 may be due to CPUFREQ demand governor, having an overly long sample size (at least on AMD X2 with Cool N’ Quiet), I was informed this will be reduced in a kernel update.
Another thought. Could i686 compiler optimisations for glibc be part of the problem? The Atom (according to Anandtech http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3276&p=7) has a 2 issue in order core, rather than 3 or 4 issue out of order.
Compiler flags might play into it as well and could explain some noise.
Compiler optimizations are not the issue here. The issue is easily replicated on various setups.The same slowness is observed on P3 / K7 / K8 / P4 / Core’s / Phenoms / Xeons and is present in 32-bit and 64-bit flavors of openSUSE.
I use ext3 on my laptop (because Windows is installed as well and that one can access ext partitions with special tools) and XFS on my desktop PC.
So far I like XFS more. It seems to be way more robust. When I’m forced to turn the computer off without shutting down, ext3 needs to be checked and repaired like forever during the next reboot, while XFS doesn’t seem to care.
ext3’s performance is visibly slower than either xfs or reiserfs and definitely +1 for the robustness of xfs with system crashes an sudden power outages. Opensuse needs to seriously consider making xfs the default file system.
So basically if i have new desktop HP dc7900 with Intel X4500 graphics it will not work…
Ubuntu 8.10 is already not booting from Live CD or alternative CD 🙁
Good job folks, keep up FOSS community!
6205, it works – it’s just not as fast as it should be. Did you test our LiveCDs during the beta phase and report a bug if it failed for you?
Hmm…no, i was at that time without a pc 🙁 Now i’m waiting for GM hoping it will work OK, because it’s better than Ubuntu in many ways, at least 11.0 was.
Andreas: I discovered this article via a link from Roy Schestowitz’s attack on the performance of openSUSE using these benchmarks (I know, I know… what else is new, right?).
Anyhow, even though I am not an openSUSE user myself, I wanted to help you guys out and did a bit of research into your EXA vs XAA question and came up with the following link: http://www.nabble.com/-Cooker–ANN:-Acceleration-method-change-for-intel-driver-td20110862.html
It would appear that Mandriva was defaulting to XAA for Intel cards up until Oct 22nd, which was post release of Mandrake 10. It’s probably safe to assume that Mandriva Linux in the benchmarks was defaulting to XAA.
Dan, thanks for the info. That explains Mandriva’s results.
If you want to do speed vs. safety tradeoffs on filesystems, you might as well do a *thorough* I/O subsystem tuning analysis, rather than just do half-assed benchmarks comparing community distros out of the box. If that stuff matters to you, you’re going to be using hardware RAID with battery-backed caches, you’re going to try all the filesystems under stress conditions, you’re going to be careful about file and filesystem placement, etc. And if you’re interested in high-speed graphics, you’ll use proprietary drivers rather than the open-source ones anyway.
Some feedback for Andreas Jaeger; Firstly your Suse 11 release was solid and fast (I ran many benchmarks and your results were good). Especially the Intel driver + OpenGL + Compositing/Compiz as well KDE4; especially your binary of Firefox 3 (30% faster) blew away both Swiftfox and SwiftWeasel which were supposed to be optimized to my exact core2duo x84_64. BTW could you confirm whether Suse 11.0 Intel driver uses XAA or EXA accel by default. However, Suse 11.1 has turned out to be mixed bag; more bad than good I’m afraid, especially the Intel Graphics performance, inconsistent repositories, missing apps for suse 11.1 repos, revision mismatches; clipper causing corruption in cut and paste during editing openoffice spreadsheets (corrupting production spreadsheets is a really bad thing) when using KDE4.1.3, wierd slowdowns with opensource radeon/radeonhd drivers, in ability even simple OpenGL apps like torcs to even launch (Segmentation Fault). The problems were such that I had to downgrade all our systems backdown to Suse 11.0 (which is fantastic, stable and consistent). With my systems back at Suse 11.0 – ZERO problems and really responsive desktops & servers, back to fresh installing to Suse 11.1 and the problems mentioned, re-manifest (my install media are fine and md5sum just fine). I can’t understand what happened with suse 11.1 after such a beautiful & clean release of Suse 11.0. But these benchmarks that you are commenting on above do not surprise me in light of all the other stuff we experienced here with 11.1.
PS is Suse 11.0 Intel driver using XAA or EXA as a default?
I think it’s XAA, as Intel developed changed to EXA as a default after the 11.0 release.