The gauntlet has been thrown down. I have been slapped in the face with a glove. Reader response to my previous six test comparison has been very strong. More tests were suggested. And I was politely (ahem) informed that my scoring methodology sucked, to put it bluntly.

Well I have expanded my tests, and improved my scoring methodology, and here is what I found.

First off, this entry is going to be light on screenshots. I simply did not have the time to make pretty graphs or take screenies of every step that I took. This is about scoring operating systems and getting unbiased results. Prettiness does not weigh in on my scoring. How much I like a certain feature does not play a role.

What I excluded from testing

Anything hardware related. I took a firm stance to take anything that could be hardware related and chuck it. I COULD compare how much RAM every OS used, or how fast they booted compared to each other, but this is an interface design comparison. YES I strongly agree that how fast something boots is very important to many desktop users, but I have my laptop and a desktop PC to test that with.

A narrow selection of hardware dictates that I cannot cater for every possible scenario out there. I know from personal experience that an OS might have a delay when booting on certain hardware where another might fly past. Windows has a definite delay between finishing boot and loading desktop, Linux has a delay on some systems when loading screen drivers for X to fire up. On my laptop and my desktop Linux does very well. Windows 7 less so, you can read my 7day Going Closed Source series to see what happened with Win7 and my DVD rom.

Printers and USB 3g modems were not included in this test either. Read this entry for a more in depth explanation of why.

Anything that is hardware related is OUT.

Methodology.

Now to the juicy bit. I approached a programmer and maths whizz friend of mine with my testing scoring system. He concurred with you guys that my method “sucks” (his words) and suggested a method. I took his method and adapted it a little. It is a percentages based method. If an OS uses the optimum method for doing something it scores 100% for a test. I just note it as 100. Obviously if none of the Operating systems use what I believe is the optimum way, or shortest way, to get to a task or information none of them will score 100%.

Take the “Get System Usage Information” test, for instance. I believe that Win7 uses the shortest possible way to get to that information. Two steps. Since there is no logical shorter way of getting to that information Win7 scores 100% for that test. If I believed that you could do it in 1 step, Win7 would be scoring 50% for instance. On most tests the impact of an extra step is not that dramatic, of course.

Where something like playing MP3’s are involved, the need to install the codecs deducts a percentage from it’s score. Ease of installing those codecs factor in too. I believe the quickest way to play ONE mp3 should be one step. Double click and the OS should load the correct program and play it. Need to install? One extra step. Need an extra step to install, like if there was no popup dialog to let you install as is the case with Ubuntu, one more step.

Below are the tests, and their results, and some comments along the way.

Now let’s get to it…

The Tests:

Behold:

  • Joining a Wifi Network
    Pretty Straitforward. Select a WIFI network, join it.

    • Win7 – 80
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 80
    • PCLOS – 80
  • Joining a Wired Network
    Plug in a network cable, go on Internet. I assume a DHCP enabled network. I did not bother with testing how many steps it takes to set up a firewall. There is much debate whether Linux actually needs a firewall to be “Internet Safe” (TM) and I decided to sidestep that one all together. Linux Mint and PCLOS comes with excellent Firewall functionality built in. Mint trounces both Win7 and PCLOS when it comes to setting up a firewall. Ubuntu and Kubuntu, because a nice front-end needs to be installed, fare the worst here.
    (Come to think of it, “Internet Safe” is an awesome marketing slogan for Linux – remember kids, you heard it first on sageek.co.za!)

    • Win7 – 50
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Set Proxy
    Again easy. Set up a proxy to access the internet if need be.

    • Win7 – 57.14
    • Ubuntu – 66.67
    • Linux Mint – 57.14
    • Kubuntu – 50
    • PCLOS – 80
  • Get Network Info
    Important to know what your IP and Gateway is. How hard to get that info? Yes I used the better method suggested by some commenters on slashdot.org

    • Win7 – 25
    • Ubuntu – 50
    • Linux Mint – 50
    • Kubuntu – 100
    • PCLOS – 100
  • Change Desktop Background
    How hard is it to change your desktop background.

    • Win7 – 80
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Change Theme
    How hard is it to change your theme?

    • Win7 – 100
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Set Screen Resolution
    Self Explanatory

    • Win7 – 100
    • Ubuntu – 80
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Set Up Network Share
    Here I have to point out that Kubuntu failed. It would not set up a share. After several tries I got it right. KDE failed both PCLOS and Kubuntu here, hence their very bad scores in this department.

    • Win7 – 80
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 33.33
    • PCLOS – 33.33
  • Access Network Share
    How many hoops to jump through to get to your friends, erm, “Poster Girl” collection?

    • Win7 – 66.67
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Get Hardware Info
    How hard is it to find out what hardware is in your computer? I must point out that none of the Operating systems hardware information indicators tell you anything about the harddrives on your computer. I think the optimal would be that harddrive information is included in a central area with other information. PCLOS and Kubuntu give TOO MUCH information. Gnome gives the best information IMO. That said, I tested how easy you got TO the info, since the value of what is actually displayed is very subjective. I should also dock every OS a percentage because Hard Disk Information is not included in this area. Again that is subjective, so I decided to use the ratings as is.

    • Win7 – 100
    • Ubuntu – 66.67
    • Linux Mint – 50
    • Kubuntu – 50
    • PCLOS – 50
  • Get System Usage Info
    Here I have an issue. On neither Kubuntu nor PCLOS could I find where to view system usage information. I Googled, searched for literally more than an hour and asked a long time Mandriva user to help me find it. No joy. If one of my readers knows an easy way to get system usage information a-la task manager in Windows in KDE, please let me know and I will adjust the scores accordingly. Again I should dock each OS a percentage because you cannot get information on harddrive usage from this central system usage tool, but again some might disagree that there is not a need for that. I say it is, but I must remain impartial here. Also of note, Linux can break down usage to folder and file, whereas with Windows you need to install a third party software package to get that kind of info.

    • Win7 – 100
    • Ubuntu – 66.67
    • Linux Mint – 50
    • Kubuntu – 50
    • PCLOS – 2.02 (I just could not find what I needed! Please let me know where to look if you know.)
  • Play Audio CD
    How hard is it to play an audio CD? Ripping was not tested.

    • Win7 – 100
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 66.67
    • PCLOS – 66.67
  • Access USB Drive
    How hard is it to access files on a USB flash drive? Not an image drive with pictures, though. Most flash drives have a mixture of content on them, simply opening up the file manager and letting the user decide from there makes more sense than asking him every time what to do because pictures where detected somewhere.

    • Win7 – 50
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Play Youtube Video (View FLASH Site)
    Does Youtube play out of the box? Is there 64bit flash support for the OS?

    • Win7 – 33.33 (Need to install codec, and no 64bit support for flash)
    • Ubuntu – 50
    • Linux Mint – 100
    • Kubuntu – 50
    • PCLOS – 100
  • Play MP3
    Does MP3’s play out of the box?

    • Win7 – 100
    • Ubuntu – 50 (Will probably never ship with MP3 support out of the box.)
    • Linux Mint – 100
    • Kubuntu – 50 (See above)
    • PCLOS – 100
  • Open PDF
    Do you want to open a PDF? Yes? Okay let’s see if it is supported out of the box then.

    • Win7 – 33.33 (PDF reader not installed, need to get it from the web, no repo available.)
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 100
  • Look for new/specific software and functionality
    How easy is it to get a software package from a trusted source? How easy is it to get software for a specific need?

    • Win7 – 43.86 (Win7 needs a trusted software repository)
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 60
    • PCLOS – 60
  • Check For System Updates
    How easy is it to check if your system is up to date? I did test auto updates since all of the OS’s tested automatically check for updates. Win7 is unique in that if you set it to auto update it will ask you to reboot, and nag you constantly. The longest you can delay the auto reboot is four hours. If you miss the nag screen when booting, or switch on and go and make a cuppa, you might just come back to a rebooting laptop.

    • Win7 – 75
    • Ubuntu – 100
    • Linux Mint – 75
    • Kubuntu – 60
    • PCLOS – 75
  • Supported Architecture
    I was interested in Laptop/Desktop architecture only. Hence 64bit and 32bit only. No ARM, PowerPC or other architectures.

    • Win7 – 100
    • Ubuntu – 100
    • Linux Mint – 100
    • Kubuntu – 100
    • PCLOS – 50 (No 64bit support. A deal breaker for many)
  • Shut Down
    You want to shut down? How many steps does it take?

    • Win7 – 66.67
    • Ubuntu – 66.67
    • Linux Mint – 66.67
    • Kubuntu – 50
    • PCLOS – 100

Totals

If you are smart you would have noticed that out of the twenty tests the absolute maximum score would be 2000 points. After all is tallied, how did they fare? In order from best to worst overall they are:

  • Linux Mint – 1748.81 points or 87 percent.
  • Ubuntu – 1696.67 points or 85 percent.
  • PCLinuxOS – 1597.02 points or 80 percent.
  • Kubuntu – 1480.00 points or 74 percent.
  • Win7 – 1440.00 points or 72 percent.

(Thanks to Picky Statwonk for suggesting the percentage scores in the comments below.)

Over the course of twenty tests Linux Mint trounced Ubuntu. I am not surprised though. The Mint guys try very hard to make the OS more user friendly. Win7 and Kubuntu bring up the rear. I am surprised that Win7 did not do better. I am not enough of a Linux fanboy to think that Microsoft does not spend a lot of time on improving the user interface. Like I noted in my previous comparative test there is a marked improvement in UI since XP in some areas, and a bit of a regression in others.

KDE4 fares much worse than Gnome. The performance of PCLOS versus Kubuntu shows the amount of work that the team has put in to improve the user interface. That KDE fares worse over these tests than Gnome does not surprise me at all. And that Win7 and Kubuntu scored so close to each other does not surprise me either, KDE4 is aimed at being familiar to Windows users. The problem with trying to be familiar is that you carry over mistakes and weaknesses from the one you are trying to emulate.

Once I updated the score for Kubuntu to take into account the availability of the System Information Utility it outperforms Win7 by two percentage points.

Gnome is a better environment in my opinion. I have believed that since switching from KDE. It is not as pretty as KDE4 by a long shot, but it is pretty enough. That it trounces the closest KDE competitor here by more than 100 points proves that.

What Does This Mean for Desktop Readyness?

Not much, I am afraid. You see, Linux falls behind Windows in two large areas. One is hardware support, and the other is software availability, including games. HUGE inroads have been made in these two areas though. It is less and less common to find hardware that is not supported under Linux, and more and more Linux ported software is being made available every day. The fact that Flash on 64bit was made available for Linux before it was made available for Windows speaks volumes of the trend in the market. Also VERY interesting is that Silverlight has a Linux derivative – moonlight.

When Linux’s main competitor, Microsoft, lets one of their partner companies make one of their technologies available to Linux is a major indication of how large a force Linux has become in the market today.

Of note: Skype is available for Linux, Steam is available for Linux, more games are being ported to Linux with the Linux port of Steam being made available for Linux.

I am loathe to comment on the desktop ready debate, and the much meme’d “year of the linux desktop.” I believe the debate is over. Linux as a desktop OS is here to stay. Will it become the No1? I don’t know. I strongly suspect it will match at least MacOS soon, and that that gain will be at the expense of Windows.

Will Windows disappear? No. Microsoft has too much invested in it to allow that. I do think that their current method of protecting their OS with threats of Patent suites is the wrong way to do it. Rather build a better OS, and let the customer decide, than trying to keep a better alternative from them. To quote the Internet Meme: “You’re Doing It WRONG.”

Mark Shuttleworth already called Microsoft’s bluff when he essentially said “Go ahead, sue me.” Microsoft cannot disclose what patents exactly they are threatening Linux companies with, their patent portfolio will be submitted to intense scrutiny, and the boogeyman will be revealed.

Summary

Yes there are tons of other tests that I could have done. Setting up a firewall is one of them, but I would argue again that the architecture of Linux negates the need for a firewall on anything but a purpose designed machine or a production server. I could have weighed my scoring differently, and I am sure many of you will provide better or alternative ways of scoring. Maybe more tests are due for a future entry, maybe that future test roundup will feature a refined or different scoring method.

I could have factored in things like the inclusion or not of an Office Suite (here’s looking at you Windows and PCLinuxOS) or standards support. I could have looked at protection from threats, or any other of a myriad of factors, but then my entry would have taken almost as long to read as it would take to write.

What this exercise has shown though is that Linux does not need to stand back to Windows as far as UI design is concerned. There are areas of improvement for sure. The beauty of Linux, of course, is that you, the user, can take this information, figure out better ways of doing it, and actually have a reasonable chance of suggesting it to your favorite Distro and have your suggestions adopted.

Hope you enjoyd yourself, apologies again for the lack of eye-candy. Cheers!

Related posts:

  1. Comparative Test Problems – Hardware, Windows7 and Linux
  2. Using KDE – Day 6
  3. Using KDE – Day 5
  4. Using KDE4 – Day 2
  5. Using KDE4 – Day 1