IDE vs SCSI Emulation of VMware in Linux

Posted: January 31, 2008 in Linux, VMware

VMware is a marvelous software, especially those who (like me) needs to setup test environment quickly and efficiently. Using it, there is no need for me to search/prepare hardware, there is no need to fix or troubleshoot any potential hardware problem (e.g. disk failure). It really helps my work a lot.

I would consider myself as a new comer/newbie for using VMware. Like anyone of us who started using any new, we tend to select default options, as we don’t really have much experience to really figure out which configurations are the best. Sometime, some default selection is even mark as “recommended” by the system supplier.

This prove to be problematic for me…

The story started when I need to perform an upgrade from version 1.0.3 to 1.0.4.

To start, check out this thread:

http://communities.vmware.com/message/757780

Basically, in version 1.0.3, the default SCSI emulation is done using Buslogic. However, once upgraded to 1.0.4, it generate a lot of errors in scanning SCSI disk. It will still boot eventually, but will take a very long time to finish all the scanning of SCSI disk. We need to change the SCSI emulation from Buslogic to Lsilogic. Though it solve this problem, it has another problem.

My VMware host server has > 10 guest OS. I don’t run all of them concurrently, but there would be situation where I may need to run more than 5 guest OS. This is when problem starts to appear. From time to time, I would get the following messages in the guest OS:

mptscsih: ioc0: task abort: SUCCESS (sc=c411c640)
mptscsih: ioc0: attempting task abort! (sc=c411c780)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 b0 d5 af 00 00 08 00
mptscsih: ioc0: task abort: SUCCESS (sc=c411c780)
mptscsih: ioc0: attempting task abort! (sc=c411c8c0)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 b0 e7 1f 00 00 08 00
mptscsih: ioc0: task abort: SUCCESS (sc=c411c8c0)
mptscsih: ioc0: attempting task abort! (sc=c411ca00)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 b0 ee e7 00 00 08 00
mptscsih: ioc0: task abort: SUCCESS (sc=c411ca00)
mptscsih: ioc0: attempting task abort! (sc=cf8ae280)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 6c e6 57 00 00 08 00
mptscsih: ioc0: task abort: SUCCESS (sc=cf8ae280)
mptscsih: ioc0: attempting task abort! (sc=cf8aeb40)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 21 92 cf 00 00 10 00
mptscsih: ioc0: task abort: SUCCESS (sc=cf8aeb40)
mptscsih: ioc0: attempting task abort! (sc=cf8aea00)
sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 69 c0 d7 00 00 08 00
mptscsih: ioc0: task abort: SUCCESS (sc=cf8aea00)

Worst, in some situation, the entire guest OS may hanged and gave out filesystem errors of disk I/O failure.

I believe these messages are due to high disk I/O operation. So, I did a complete “tar zcvf” of the entire filesystem to test the system. Predictably, these warning messages appeared. Worst, even other guest OS which was idling also showing these messages. If I try to compress a filesystem that contain a large file, say is about few hundred MBytes of file size, the system would became unstable and sometime, generate kernel panic of disk I/O errors.

I spent a lot of my time checking through host and guest OS kernel configurations and trying out newer version of Linux kernel, which I thought would be the root caused of the problem. But after many attempts, the situation remains.

Thankfully, I know someone who is been using VMware long before I started using it. According to him, he always select IDE as the disk emulation, and he never experience such problem. So I decided not to follow the default and “recommended” option for disk emulation, and manually change my installation into IDE disk emulation.

I run a few round of testing (like I did earlier), IDE perform much more stable than SCSI emulation.

I did a few google search, I don’t find anyone facing similar problem as I did. I start wonder how likely for me to face such issue alone. There maybe something else that I am not aware of, but at this moment, I don’t really have the time to continue the investigation. Projects (emphasis plural) deadlines are coming soon, I guess I will revisit this again later, when I have the time.

Advertisements
Comments
  1. Ben D. says:

    I’ve been seeing the same problem, in the same circumstances you describe: heavy disk I/O on the guest OS. We run tar (with compression) and also bzip2 every night on the guest OS (which is Ubuntu Server) while we’re backing up log files and a MySQL installation. Like you, I haven’t found much with Google.

    I’m going to install VMWare Tools on the guest OS to try to keep the clock in sync, but it sounds like I will need to convert to a virtual IDE disk to really solve the problem.

    I’m running Ubuntu 7.10 server edition and a “uname -r” shows me this kernel version: 2.6.22-14. VMWare Server is 1.0.4. What versions are you using?

  2. Ben D. says:

    I should have mentioned in the previous post that when the guest OS has trouble with mptscsih, it slows down and loses a lot of time. Its clock is currently 3 hours behind the host’s, and that’s with VMWare Tools installed on the guest and time synchronization enabled.

  3. lenrek says:

    Hey thanks for your post. I guess, this mean, I am not alone with this problem.

    I tried a few kernels, from 2.6.22.x to 2.6.23.x, and I have 3 VM host servers: 2 using 2.6.23.12 and 1 using 2.6.23.14; 2 use Slackware 12 while another uses Slackware 11. All showing the same symptoms. All 3 machines configure with software RAID-1 mirror (fix disk:2x160GB SATA).

    There is one new development in my investigation. My laptop, which using the same kernel or the newer (2.6.24) kernel, with the same version of VMware 1.0.4, but not using software RAID, I seldom encounter such problem even when there is high disk I/O.

    Are you using software raid with your system?

  4. Ben D. says:

    Interesting… I am also using software RAID-1 on my host system. The host system, like the guest OS, is running Ubuntu 7.10 server edition.

  5. lenrek says:

    Hmm…

    From your post, I think I should try a few more test between non-RAID and soft-RAID systems. I have a few systems left idling, will try to find time over the weekend and try them out.

  6. Ben D. says:

    I’m going to try switching the virtual SCSI controller on the guest OS from LSI to BusLogic — as a temporary fix until I can schedule some downtime to migrate the SCSI disk to an IDE. I know that changing to BusLogic will probably cause me the other problems you first mentioned in your post, but I can live with long boot times temporarily.

  7. lenrek says:

    If you are going to do that, then, it may be simpler to convert the SCSI to IDE, since changing from LSI to Buslogic would require guest OS shutdown as well.

    You may want to check my latest entry:
    https://lenrek.wordpress.com/2008/02/22/linux-software-raid-vmware/

  8. Ben D. says:

    Thanks for the guide! I was looking for something like that. I did end up switching to BusLogic but I didn’t realize how long the guest would take to boot up. After an hour or so (if I remember rightly), I canceled the boot and switched back to LSILogic. Later I followed your instructions to convert to IDE and everything seems to be working much better now.

  9. Terry E says:

    There have been loads of posts on this topic, (e.g. http://communities.vmware.com/thread/76787) and but no decent whitepapers from VMware (e.g. http://www.vmware.com/pdf/WS6_Performance_Tuning_and_Benchmarking.pdf)

    The Virtual SCSI driver mapped through the VM to physical SCSI disks does seem to scale and perform better, which is why this is why ESX uses SCSI disks. However for a Workstation running Server, Workstation or Appliance then the I/O performance delta is marginal and the IDE implementation is simpler and more stable, I go for IDE on my test appliances.

  10. francis.g says:

    I have the same problems. Maybe, switching to IDE will help. I stay tuned to tell you.
    Thx,
    F.

  11. lolz says:

    Switching to IDE is actually not a solution. I switched to scsi prealloc disk because thats the fastest disk access method whatsoever with vmware.

    I started getting this error when I had to run more guests from the same disk pbly due to the large sequential io the oses rot in with waiting.

    When I had the most io sensitive os on a separated disk I had no problem at all just that crashed for now.

    Also on the guest you can experience processes actually die out because they cant write out their logs etc.

    Oh and this is the latest kernel+ latest vmware server so the problem u discussing here is still exist.

  12. lenrek says:

    Oh? I did not try pre-allocate the disk.

    Thanks for the info.

  13. Ondrej Cecak says:

    Hi,

    maybe quite more comfortable way, how to change SCSI disc to IDE (editing of full vmdk wasn’t working well):

    1) Convert disc to flat.
    vmware-vdiskmanager -r scsi.vmdk -t 2 temp.vmdk

    2) Change disc type to IDE.
    In short text file temp.vmdk change “ddb.adapterType” to value “ide”
    ddb.adapterType = “ide”

    3) Convert disc back (in my case to single file, not preallocated).
    vmware-vdiskmanager -r temp.vmdk -t 0 ide.vmdk

    4) Add new existing disc to VM (connected as IDE).


    Ondrej Cecak

  14. Frank M says:

    While searching for comparison’s of IDE V’s SCSI virtual disks I just found a good article on converting between the 2 types:

    http://www.jjclements.co.uk/2010/06/08/vsphere-convert-vm-ide-disk-to-scsi-disk

    Hope this helps 🙂

  15. M.R. says:

    Hi, I experienced the scsi guest problem today.
    I have a dual core proliant server with vmware 2 and about 10 guests. All OSs are debian, except a couple of xp systems. I used to back up the guests with tar and gzip compression. For more than a year I had no problems. Recently I switched some guests to bacula. Still no problem. Two days ago I switched to bacula two more guests, and on them the scsi problem emerged. The two blocked guests sent cpu usage to 100%, and I had to soft-reset them to bring them back.
    VM disks are non-preallocated scsi-emulated, and are located on a softraid-1 1TB array, directly connected to the host.

  16. […] The busiest day of the year was January 4th with 62 views. The most popular post that day was IDE vs SCSI Emulation of VMware in Linux. […]

  17. OleS says:

    Hi,
    had a severe problem with this issue yesterday, found this patch

    VMware ESX 4.1 Patch ESX410-201104401-SG: Updates VMkernel, VMX, CIM

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1035097

    We will have to move from 4.0 to 4.1 first.

    OleS

  18. The end result is that these dysfunctions are removed from the body.
    Although in certain people it’ll certainly not cause pain, often it might
    be a severe hindrance. -Major trauma (serious fall, motor vehicle accident).

  19. Charline says:

    This implies that you have the work and also the psychotherapies
    of two generations of researchers guaranteeing you the success of
    dream therapy. The mental state of your person is judged incidentally a person feels, acts,
    thinks or perceives. We live in the world where poverty prevents a persons development by obliging everyone to forget their moral
    principals as a way to survive.

  20. Ashwagandha or withania somnifera can also be an important herb to take care of cervical osteoarthritis.
    Also, the symptom could function as the inability to boost the big toe on upward direction.
    The herbal home cures for cervical spondylosis will replace the displaced discs.

  21. Wow that was odd. I just wrote an incredibly long comment but after
    I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again.
    Regardless, just wanted to say superb blog!

  22. spondiloza says:

    This pain might be so severe that way of life are also difficult to carry out.
    Such conditions, which result in neck pain, are: osteoporosis or tumors,
    ruptured or degenerated cervical disc, nerve dysfunctions, bone spur, spondylosis or congenital problems.
    Make it a regular habit of consuming leaves and flowers of
    the tree.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s