Writing custom openstack wighter

Two options are available in order to guide the OpenStack cloud infrastructure to schedule  the next Virtual machine on a host with minimum benchmark execution time left :
1.Write a custom Nova scheduling filter.
2.Write a custom Nova scheduling  weigher   .
Due to the fact that the filter logic in this case is complicated a wighter class is more appropriated for this task.

The wighter classes location :
The scheduling implementation in Nova is located in \nova\scheduler directory
The weights folder contains build in Weighers for example the RAMWeigher that returns host score  according to its available ram .

Open stack Weigher classes relationships 

A concrete Weigher implements the weights.BaseHostWeigher interface (its abc python interface ).
The BaseWeigher interface declares the following methods:
The def _weigh_object(self, host_state, weight_properties):
method returns the host un normalized score.
The def weigh_objects(self, weighed_obj_list, weight_properties):
method may used to calculate the weight of an host when information from all hosted is required  .(Not our case ).

We need to create a new concrete Weighter class that's overrides the _weigh_object method .

class BenchMarkResourcesWeigher(weights.BaseHostWeigher):
    def _weight_multiplier(self):
        """the default weight multiplier."""
        return 1
    def _weigh_object(self, host_state, weight_properties):
        """Higher weights win.  """
        return -1 * host_state.BenchMarks_Total_Work_Left

In the next Post we will add the new member : BenchMarks_Total_Work_left  to the host_state class 

benchmark VM host scheduling

Our benchmark system is based on a private cloud that is managed by OpenStack framework.
For each benchmark execution session a new VM is created form an Image that include the benchmark tools .
The VM should be schedule to be host on a machine that the maximum remaining work of other running instances of vm that are executing on this machine is the less from all other machines.
Apsedo code for host selection scoring:

Int MaxWork
Foreach ( Vm nextVM in Host.GetVms())
       If ( MaxWork < nextVM.RemaingingWork )               
                   MaxWork = nextVM.RemaingingWork
Return MaxWork

The goal :
Setting the custom weights in our private cloud.
In the Nova  configuration file.
scheduler_weight_classes=nova.scheduler.weights.all_weighers

Note:The filter can be one of the build in filters

The following tasks should be consider during the solution architecture:
1.How can I get all host vm references ?
2.How can I send a vm remaining time to the nova weight component  ?
3.how can I write a custom weight component  ?
4.how can I consider other instances that executes other jobs then benchmarks in the weight calculation?

References:
http://docs.openstack.org/grizzly/openstack-compute/admin/content/weights.html
https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py
http://docs.openstack.org/trunk/openstack-ops/content/customize.html
http://docs.openstack.org/grizzly/openstack-compute/admin/content/scheduler-filters.html#imagepropertiesfilter
http://www.slideshare.net/guptapeeyush1/presentation1-23249150

Installing DevStack in ubuntu server

Installing devstack is very easy by following the Official documentation .
however there is an issue of firewalls preventing git access in order to solve it in the devstack/stackrc file
change from
GIT_BASE=${GIT_BASE:-git://git.openstack.org}
to
GIT_BASE=${GIT_BASE:-https://git.openstack.org}

install by
./stack.sh

In case of error before reinstalling call
./clean.sh

After Installing if the Result is ok :
Horizon is now available at http://192.168.1.21/
Keystone is serving at http://192.168.1.21:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: a243a75f003a77249a1b
This is your host ip: 192.168.1.21
stack.sh completed in 1917 seconds.
zvika@zvika-openstack:~/devstack$ 

Note that the machine came to me very loaded in work after installation .

To check the version call:
curl http://192.168.1.21:35357/v2.0/
The result:
{"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://192.168.1.21:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/content/", "type": "text/html", "rel": "describedby"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf", "type": "application/pdf", "rel": "describedby"}]}}

References:
http://stackoverflow.com/questions/20390267/installing-openstack-errors
http://api.openstack.org/api-ref-guides/bk-api-ref.pdf
http://devstack.org/guides/single-machine.html
http://docs.openstack.org/developer/keystone/api_curl_examples.html
http://devstack.org/

JCLOUDS blob storage for openstack swift

We use Jclouds blob storage in order to manage the Blobs in our private cload swift blob storage .
The following code demonstrates creating , writing , exploring and reading from swift using jcloud SwiftBlobStore.

package com.gmuav.playwithjcloads;
import java.io.IOException;
import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.domain.StorageMetadata;
import com.google.common.base.Charsets;
import com.google.common.io.ByteSource;
public class PlayWithSwift {
  
   public static void main(String[] args) throws IOException {
 
    
      String identity = " --- ";
      String credential = " --- ";
      String containerName = "myContainer";
      
       ByteSource payload = ByteSource.wrap("This is the Data as text ".getBytes(Charsets.UTF_8));
     
      BlobStoreContext context = ContextBuilder.newBuilder("swift")
                                               .credentials(identity, credential)
                                               .buildView(BlobStoreContext.class);
      try {
         // Create Container
         BlobStore blobStore = context.getBlobStore();
         blobStore.createContainerInLocation(null, containerName);
         String theBlobName = "myFolder/myData";
    
         Blob blob = blobStore.blobBuilder(theBlobName)
            .payload(payload)
            .contentLength(payload.size())
            .build();
       
         blobStore.putBlob(containerName, blob);
           // List Container
         for (StorageMetadata nextBlob : blobStore.list()) {
            System.out.println("NextBlob:" + nextBlob.getName());
         }
         
         Blob theRetreivedBlob =  blobStore.getBlob(identity, theBlobName);
               
         String theRetValue = new String (((ByteSource)theRetreivedBlob.getPayload()).read() , Charsets.UTF_8);
         
          System.out.println("The blob data  " + theRetValue);
         
         
      } finally {
         context.close();       
      }
   }
}

 

References:
jclouds-examples