Virtualization

Expand all | Collapse all

Adding KVM Hypervisor

  • 1.  Adding KVM Hypervisor

    Posted Jun 09, 2017 06:40 AM

    As VIO cannot be used for "proper" OpenStack usage due to there begin no Images that can be used for OpenStack and these images be used for SSH and/or Password Injection or can Instances be resized when using templates from vCenter. This overrides everything OpenStack and we have mistakenly installed and configured VIO and deployed over 180 Instances so I guess we are stuck with it.

    To overcome this we want to install another hypervisor namesly KVM so we can use this hypervisor for the Instances that require resizing and other proper OpenStack options.

    Can another hypervisor we added to an existing VIO deployment? IS it possible to deploy KVM in a host and then add that host to VIO ??

    It is unfortunate that VMware do not provide Images and do not provide guides on using Images so that customers can use OpenStack with all the features of KVM.... I believe with the money that has been outlayed for NSX and all the other costs it would be nice to be able to use VIO in its fullest capacity. Maybe we are missing something but not having Linux Images that can be used for resizing, SSH Key injection is a massive downfall....If someone has a solution we would even be willing to pay handsomely for someone to create Ubuntu, CentOS and Debian images yet from what we have researched I do not think it is possible.

    So if adding a KVM hypervisor to VIO is possible this would solve our issue as we can keep our normal Instances and new Linux Instances we can use KVM so customers can use SSH keys and benefit from image resizing.



  • 2.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 11:16 AM

    First, are you aware that you can use most qcow2 and raw images with VIO?  Simply use the glance-import tool to import them into your cloud.  This allows you take advantage of lots of images from the ecosystem.  Check out the documentation here:

    VMware Integrated OpenStack Documentation

    Alternately, images can be converted manually using qemu-tools if you prefer not to use an import task for some reason.  To do this, have a look here:

    OpenStack Docs: Converting between image formats

    Any image with cloud-init should support keypair injection for SSH access, and you can also use cloud-init to inject passwords.  See for example:

    Changing Passwords with cloud-init · Scott's Weblog · The weblog of an IT pro specializing in virtualization, networking…

    For resize, the standard OpenStack resize API should work:

    OpenStack Docs: Change the size of your server

    Just note that you'll want to detach Cinder volumes before resizing and then reattach them after.



  • 3.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 11:24 AM

    Hello and thank you for the reply. Let me review and try some of your advice.

    In saying this I have imported the qcow2 cloud images for ubuntu and others using both horizon and the cli but they do not boot when adding to VIO. I am an advid supporter of VIO as I am a VMware junkie but not having these images work is affecting the workings of my VIO deployment. Currently I use vSphere templates and link then into when creating and glance image and use temp password but recently have noticed the resizing Instances so not work this way nor does SSH Key injection. It is my understanding that VMware does not support injection but maybe with VIO it does.

    Once I review your reply I will give it a go at 1. importing qcow2 images and boot and I will give you the error and 2. I will build my own using qemu tools which I have already done and received an error when using a Ubuntu Cloud Image ( img ) and using qemu to convert to vmdk even though glance converts it anyway. I will post the screenshot of the error when trying to boot that image. I have tried with LSIlogic and all other adaptors.

    Ill be in touch tomorrow once I try things as it is better knowing that it CAN be dones rather than me hoping it can be done.

    Thank you once again and hopefully I can share positive remarks tomorrow once I review everything.



  • 4.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 12:05 PM

    Just a quick update

    I imported the Ubuntu 16.10.vmdk file from the Ubuntu Cloud Image site and when trying to launch an Instance I receive " No Valid Hosts Were Found"

    It is my understanding that these images are ONLY for KVM and hence cannot be used in VIO with the esxi hypervisor.

    And all the links you provided to have been to before I I have spent the past 12 months trying to find a solution to this. If you want to prove me wrong and create and I am happy to pay anyone for their time. Just PM me and we can talk.



  • 5.  RE: Adding KVM Hypervisor

    Broadcom Employee
    Posted Jun 10, 2017 03:45 PM

    Can you capture output of:

    • glance image show <image id>
    • nova scheduler log from both controllers.  Nova scheduler log should be /var/log/nova/nova-scheduler.log on the controller node.


  • 6.  RE: Adding KVM Hypervisor

    Broadcom Employee
    Posted Jun 10, 2017 03:47 PM

    sorry 1st command is openstack image show <image id>



  • 7.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 04:06 PM

    Hello Again. Thanks. Here is the first... Ill provide the other logs next

    root@loadbalancer01:~# openstack image show 60ea1c30-a198-4b0e-88d7-1ae5bfbd33d3

    +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

    | Field            | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

    +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

    | checksum         | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | container_format | bare                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | created_at       | 2017-06-10T15:58:13Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | disk_format      | vmdk                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | file             | /v2/images/60ea1c30-a198-4b0e-88d7-1ae5bfbd33d3/file                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | id               | 60ea1c30-a198-4b0e-88d7-1ae5bfbd33d3                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | min_disk         | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |

    | min_ram          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |

    | name             | Ubuntu 17.04 Cloud                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

    | owner            | 54b307bd475d4f359953d892e650b3ab                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

    | properties       | direct_url='vi://vcenter.XXXXXXXXXXX.local/Openstack Datacenter/vm/OpenStack/Project (54b307bd475d4f359953d892e650b3ab)/Images/60ea1c30-a198-4b0e-88d7-1ae5bfbd33d3?managed=True', locations='[{u'url': u'vi://vcenter.XXXXXXXXXXX.local/Openstack Datacenter/vm/OpenStack/Project (54b307bd475d4f359953d892e650b3ab)/Images/60ea1c30-a198-4b0e-88d7-1ae5bfbd33d3?managed=True', u'metadata': {}}]', vmware_adaptertype='lsiLogic', vmware_disktype='streamOptimized', vmware_ostype='ubuntu64Guest' |

    | protected        | False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

    | schema           | /v2/schemas/image                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |

    | size             | 330916864                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

    | status           | active                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

    | tags             |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

    | updated_at       | 2017-06-10T16:00:29Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | virtual_size     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

    | visibility       | private                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

    +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

    root@loadbalancer01:~#



  • 8.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 04:12 PM

    Here is the last part of the scheduler log after I try and launch an instance with the Ubuntu 17.04 Cloud vmdk image

    cheduler/host_manager.py:177

    2017-06-10 16:08:49.730 4975 DEBUG oslo_concurrency.lockutils [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Lock "(u'compute01', u'domain-c47.c0c95b3e-0d6e-41c1-bac1-8c6953faed71')" released by "nova.scheduler.host_manager._locked_update" :: held 0.132s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285

    2017-06-10 16:08:49.730 4975 DEBUG nova.filters [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Starting with 1 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70

    2017-06-10 16:08:49.731 4975 INFO nova.scheduler.filters.retry_filter [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Host [u'compute01', u'domain-c47.c0c95b3e-0d6e-41c1-bac1-8c6953faed71'] fails.  Previously tried hosts: [[u'compute01', u'domain-c47.c0c95b3e-0d6e-41c1-bac1-8c6953faed71']]

    2017-06-10 16:08:49.731 4975 INFO nova.filters [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Filter RetryFilter returned 0 hosts

    2017-06-10 16:08:49.732 4975 DEBUG nova.filters [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Filtering removed all hosts for the request with instance ID '4c796ec0-8b1b-4a5f-93c3-bff44c210bfd'. Filter results: [('RetryFilter', None)] get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:129

    2017-06-10 16:08:49.732 4975 INFO nova.filters [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] Filtering removed all hosts for the request with instance ID '4c796ec0-8b1b-4a5f-93c3-bff44c210bfd'. Filter results: ['RetryFilter: (start: 1, end: 0)']

    2017-06-10 16:08:49.733 4975 DEBUG nova.scheduler.filter_scheduler [req-0a35897e-5dc4-43c7-93ec-09804e0d289b 4a45cf83e6e54826b2fddc29bb385a71 54b307bd475d4f359953d892e650b3ab - - -] There are 0 hosts available but 1 instances requested to build. select_destinations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:71

    2017-06-10 16:09:23.587 4975 DEBUG oslo_service.periodic_task [req-7c7c0491-6bf1-4f56-9548-06c11c530362 - - - - -] Running periodic task SchedulerManager._expire_reservations run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215

    2017-06-10 16:09:42.395 4975 DEBUG oslo_service.periodic_task [req-7c7c0491-6bf1-4f56-9548-06c11c530362 - - - - -] Running periodic task SchedulerManager._run_periodic_tasks run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215



  • 9.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 04:15 PM

    All my other images lauch just fine but those images I use as templates from vcenter and just create a link into glance but those images cannot use SSH Keys otr resizing does not work.... to resize i need to boot with gparted and do everything manually and the bad thing about that is it allocated the full storage rather than use thin disk.



  • 10.  RE: Adding KVM Hypervisor

    Posted Jun 12, 2017 09:03 PM

    Ah.  If you're using images that were created from existing templates in vCenter: do you have cloud-init running on them?  Cloud-init is the "agent" which reads metadata that OpenStack provides to instances (such as SSH keys and passwords).  If the templates don't start cloud-init on bootup and try to read metadata either from the metadata service or a config drive, they won't get any of the metadata that OpenStack has provided for them there.  Try installing cloud-init before you import them to VIO.

    Regarding resize: were you using the feature discussed here?

    VMware Integrated OpenStack Documentation

    If so, note that VM's imported from vSphere can't be resized (it's a limitation of the import process).



  • 11.  RE: Adding KVM Hypervisor

    Posted Jun 13, 2017 09:53 AM

    OK here is the import config....Notice the path that the image is saved........not sure if this is correct.....the datastore-h1 is for our glance images and is part of the Management Cluster and NOT the Management Cluster....Not sure if the whole issue is that the Compute Cluster cannot access the Glance storage on the other cluster. Ill post below and contine the process of booting the image.

    +-------------------+----------------------------------------------------------------------------------+

    | Property          | Value                                                                            |

    +-------------------+----------------------------------------------------------------------------------+

    | checksum          | 25da3acab885d89b7045f8844f586c2c                                                 |

    | container_format  | bare                                                                             |

    | created_at        | 2017-06-13T09:48:35Z                                                             |

    | direct_url        | vsphere://vcenter.XXXXXXXXXXXX.local/folder/images/c7781224-0549-4dd6-9344-682e2 |

    |                   | 1c48610?dcPath=Openstack+Datacenter&dsName=datastore-h1                          |

    | disk_format       | vmdk                                                                             |

    | id                | c7781224-0549-4dd6-9344-682e21c48610                                             |

    | locations         | [{"url": "vsphere://vcenter.XXXXXXXXXXX.local/folder/images/c7781224-0549-4dd6- |

    |                   | 9344-682e21c48610?dcPath=Openstack+Datacenter&dsName=datastore-h1", "metadata":  |

    |                   | {}}]                                                                             |

    | min_disk          | 0                                                                                |

    | min_ram           | 0                                                                                |

    | name              | Ubuntu-14.04-Trusty                                                              |

    | owner             | f25adf22c293444bb8e2778df6a12ad9                                                 |

    | protected         | False                                                                            |

    | size              | 262472192                                                                        |

    | status            | active                                                                           |

    | tags              | []                                                                               |

    | updated_at        | 2017-06-13T09:48:54Z                                                             |

    | virtual_size      | None                                                                             |

    | visibility        | private                                                                          |

    | vmare-adaptertype | lsiLogic                                                                         |

    | vmware-disktype   | streamOptimized                                                                  |

    | vmware_ostype     | ubuntu64Guest                                                                    |

    +-------------------+----------------------------------------------------------------------------------+



  • 12.  RE: Adding KVM Hypervisor

    Posted Jun 14, 2017 01:00 AM

    Hello Again..

    I am going to post screen shots of the VIO deployment so as to provide more information about hosts and datastores in case it helps

    FYI the datastore datastore-h1   is on the Management Cluster on esxi-5



  • 13.  RE: Adding KVM Hypervisor

    Posted Jun 10, 2017 04:18 PM

    I was also looking at the vcenter tasks and the error there is interesting. I have attached the screenshot



  • 14.  RE: Adding KVM Hypervisor

    Posted Jun 11, 2017 02:40 PM

    Hello

    I have spoken to Trevor Roberts Jr from VMware and he said the qcow2 images will work as they will be converted upon upload. I have tried all distros and now in every distro I receive the error on boot " Operating System Not Found". I have attached a screenshot and from this it appears to be trying to boot from the network and not the HDD.



  • 15.  RE: Adding KVM Hypervisor

    Posted Jun 12, 2017 08:57 PM

    Perhaps it would be useful if I stepped through a typical workflow or two so we can check if there are differences.  I'll be using a VIO 3.1 lab for the screenshots below.  I'll start by grabbing a generic Ubuntu 14.04 image in qcow2 format from Ubuntu's cloud image download site:

    The first method we'll try is the import task method I mentioned earlier.   When you create an import task, you basically tell Glance "please go get the following image from the specified location and convert it for me".  That means it needs the image to be reachable via HTTP.  To take care of that, I'll just use a Python one-liner to temporarily start a webserver on port 12345 of the machine I downloaded the image to (you could also use a real HTTP server of course):

    That done, we're ready to create the import task.  Notice here that I'm setting the adaptertype and disk type in the process.  An improperly set adapterType is the most common cause of the "Operating system not found" error when booting an instance (if you don't have it set on an image, you can add that property to it via the CLI, API, or Horizon without recreating the image):

    This will take a few minutes to process and Glance downloads the image and converts it for you.  You can monitor progress via Glance or the CLI.  Once it's done, you should see the image available with all the proper attributes set:

    And we can try booting it:

    Since you asked about passwords here, I'll go ahead and use cloud-init to set the password for the Ubuntu user:

    Once I click the Launch button, the image will start spawning:

    Once it's done booting, I can use the Horizon console to verify that Ubuntu started up properly and shows me a login prompt:

    Next, I'll try logging in with the credentials I set via cloud-init:

    Success!

    So, the import method appears to be working well here.  I may want to pre-convert my image to a VMDK instead though.  To see how that works, let's go back to the machine I downloaded the image to in the first place and use qemu-img to convert it (note: you'll need to "sudo apt-get install qemu-utils" if you haven't already).  Notice here that after I run qemu-img, I also run a printf command.  This is due to the fact that in this lab I'm using an older version of qemu tools which had a bug when creating a streamOptimized image.  Off the top of my head, I believe the fix we sent upstream for that got merged in version 2.5 (but don't quote me on it; I'm just working from memory here...if you're curious I'll be happy to check and get back to you):

    I can then upload the image to Glance.  This time I'll just use the "openstack image create" command since we don't need to use an import task (although we could, if we wanted--import tasks don't have to convert between formats; it's just something useful they can do):

    Once the image has been created, I'll boot a server from it.  Notice I used the config drive option here--that's not necessary (we didn't use it when booting the last instance); I was simply in the habit of typing it since I was working on something involving config drives earlier in the day.  Once I had the screenshot and I realized I'd used it, I figured we may as well leave it in there to demonstrate that config drive also works. =)

    And again, we'll switch over to Horizon to verify that Ubuntu started up correctly:

    You also asked about resize, so let's take a look at that too.  First, we verify that the first instance we booted is an m1.small with 1vCPU, 2GB RAM, and 20GB of disk.  Then we'll click on "Resize Instance" in the dropdown menu on the right:

    We'll ask for the instance to be resized to an m1.large with 4 vCPU's, 8GB of RAM, and 80GB of disk:

    The instance should start resizing:

    Eventually we'll be asked to confirm the resize:

    Once we click the confirm button, we're finished and can see that the instance listed as m1.large:

    If we log back in to the console, we can verify the new specs:



  • 16.  RE: Adding KVM Hypervisor

    Posted Jun 13, 2017 08:17 AM

    WOW....Thank you for your detailed answer.

    Let me import the image using your guide.

    I want to move away from vSphere linked clones as they limit many features in OpenStack as you described.

    Give me a few hours and Ill report back with screenshots as I go through the process so you can see what to output os after each task.

    Many thanks again....see you in a few hours.



  • 17.  RE: Adding KVM Hypervisor

    Posted Jun 15, 2017 11:25 PM

    Hello

    Yes I am aware that VIO can use most qcow2 image as VIO converts them when importing.

    I think the whole issue to this problem is that I am not using shared storage and the current datastore that glance is using is local storage on one of the esxi hosts. overnight I purchased a an NFS storage from my server provider and am currently adding that to all hosts and also added glance to that storage but as I currently have images in the old glance datastore I cannot remove the images so I can remove that datastore as I need to remove it so that glance always uses the new NFS datastore and not the old local storage but am yet to find a way to move the old images to the new NFS store.

    More research but Ill get there as I am learning much as I go however there are some things that I need help with because when i ask here it means I have spent weeks finding a solution to no avail.

    I am hopeful that once I add the NFS store to each host so that every host has shared storage when it comes to glance that the issue with booting the cloud images will cease as I think at the moment the instances cannot find the image as it is on local storage on one of the hosts on the management cluster.



  • 18.  RE: Adding KVM Hypervisor

    Posted Jun 13, 2017 10:08 AM

    Well I followed the steps and still get the No Operating System Found.

    Here is the glance import command I used:-

    Booted image and Horizon states instance is running:

    Upon looking at the console it states the same error....

    Maybe it is the glance store that I am using. As I have a lot of unused space in the Management Cluster I assigned the Glance storage there. This is local storage and NOT shared storage. Can it be possible that the Compute Cluster cannot access the image?  Maybe this is not it as I am sure there would be an error in horizon. Would it help if I change the glance storage to the compute cluster ?

    As I am moving to a dedicated OpenStack control panel it is imperitive that I get this working. I have been using vSphere linked clones for 12 months with no issues but now I want to use OpenStack the way it was designed and not rely on linked clones.



  • 19.  RE: Adding KVM Hypervisor

    Posted Jun 13, 2017 04:20 PM

    One thing I have noticed between the image below and your is that yours states that the image size is 2GB and mine is still the 250MB that the original images size is.



  • 20.  RE: Adding KVM Hypervisor

    Posted Jun 14, 2017 03:14 AM

    This virtual size maybe the problem as I noticed in my image upload the virtual_size is zero but in the example provided above the image has an actual size and is not zero like mine below



  • 21.  RE: Adding KVM Hypervisor

    Posted Jun 14, 2017 03:25 AM

    I also noticed the difference between the glance upload that the above tutorial provided is a file link for the location

    My image location is a vcenter URL and not a file like you provided. Just trying to provide as much information to somehow try and get this solved.



  • 22.  RE: Adding KVM Hypervisor

    Posted Jun 24, 2017 01:39 AM

    OK I have some headway.

    Now the virtual image size is the same as the example as before it was displaying a zero size. However STILL the Ubuntu 14.04 Trusty Image states "Operating System Not Found "!!!!



  • 23.  RE: Adding KVM Hypervisor
    Best Answer

    Posted Jun 26, 2017 01:58 AM

    It appears that the only way to get this working was to convert the Openstack Cloud image BEFORE uploading to glance.

    Convert via qemu-img and make sure that when uploading that "IDE" is used as qemu-img converts to IDE.

    I am not sure why VIO did not convert properly but for those with the same issue just convert the OpenStack Image BEFORE uploading to glance.