יום שלישי, 29 באפריל 2014

benchmark VM host scheduling


Capture

Our benchmark system is based on a private cloud that is managed by OpenStack framework.
For each benchmark execution session a new VM is created form an Image that include the benchmark tools .
The VM should be schedule to be host on a machine that the maximum remaining work of other running instances of vm that are executing on this machine is the less from all other machines.
Apsedo code for host selection scoring:

Int MaxWork
Foreach ( Vm nextVM in Host.GetVms())
       If ( MaxWork < nextVM.RemaingingWork )               
                   MaxWork = nextVM.RemaingingWork
Return MaxWork


Capture2


The goal :
Setting the custom weights in our private cloud.
In the Nova  configuration file.
scheduler_weight_classes=nova.scheduler.weights.all_weighers

Note:The filter can be one of the build in filters

The following tasks should be consider during the solution architecture:
1.How can I get all host vm references ?
2.How can I send a vm remaining time to the nova weight component  ?
3.how can I write a custom weight component  ?
4.how can I consider other instances that executes other jobs then benchmarks in the weight calculation?

References:
http://docs.openstack.org/grizzly/openstack-compute/admin/content/weights.html
https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py
http://docs.openstack.org/trunk/openstack-ops/content/customize.html
http://docs.openstack.org/grizzly/openstack-compute/admin/content/scheduler-filters.html#imagepropertiesfilter
http://www.slideshare.net/guptapeeyush1/presentation1-23249150

functools.total_ordering

Example of using the  @functools.total_ordering (based on the sample from http://pymotw.com/2/functools/)

import functools
import inspect
from pprint import pprint
@functools.total_ordering
class MyObject(object):
    def __init__(self, val):
        self.val = val
    def __eq__(self, other):
        print ('  testing __eq__(%s, %s)' % (self.val, other.val))
        return len ( self.val)  == len ( other.val)
    def __gt__(self, other):
        print ('  testing __gt__(%s, %s)' % (self.val, other.val))
        return len (self.val) > len (other.val)
    def TestMethod(self):
        pass
print ('Methods:\n')
pprint(inspect.getmembers(MyObject))
a = MyObject("Long name bla bal ")
b = MyObject("Short mame")
print ('\nComparisons:')
for expr in [ 'a < b','a > b', 'a <= b', 'a == b', 'a >= b' ]:
    print ('\n%-6s:' % expr)
    result = eval(expr)
    print ('  result of %s: %s' % (expr, result))

The Result:
Methods:
[('TestMethod', <function MyObject.TestMethod at 0x03146270>),
('__class__', <class 'type'>),
('__delattr__', <slot wrapper '__delattr__' of 'object' objects>),
('__dict__',
  mappingproxy({'__module__': '__main__', '__le__': <function total_ordering.<locals>.<lambda> at 0x031464B0>, '__dict__': <attribute '__dict__' of 'MyObject' objects>, '__eq__': <function MyObject.__eq__ at 0x031461E0>, '__doc__': None, '__hash__': None, '__lt__': <function total_ordering.<locals>.<lambda> at 0x03146420>, '__gt__': <function MyObject.__gt__ at 0x03146228>, '__init__': <function MyObject.__init__ at 0x03132A08>, 'TestMethod': <function MyObject.TestMethod at 0x03146270>, '__weakref__': <attribute '__weakref__' of 'MyObject' objects>, '__ge__': <function total_ordering.<locals>.<lambda> at 0x03146468>})),
('__dir__', <method '__dir__' of 'object' objects>),
('__doc__', None),
('__eq__', <function MyObject.__eq__ at 0x031461E0>),
('__format__', <method '__format__' of 'object' objects>),
('__ge__', <function total_ordering.<locals>.<lambda> at 0x03146468>),
('__getattribute__', <slot wrapper '__getattribute__' of 'object' objects>),
('__gt__', <function MyObject.__gt__ at 0x03146228>),
('__hash__', None),
('__init__', <function MyObject.__init__ at 0x03132A08>),
('__le__', <function total_ordering.<locals>.<lambda> at 0x031464B0>),
('__lt__', <function total_ordering.<locals>.<lambda> at 0x03146420>),
('__module__', '__main__'),
('__ne__', <slot wrapper '__ne__' of 'object' objects>),
('__new__', <built-in method __new__ of type object at 0x5E774EA8>),
('__reduce__', <method '__reduce__' of 'object' objects>),
('__reduce_ex__', <method '__reduce_ex__' of 'object' objects>),
('__repr__', <slot wrapper '__repr__' of 'object' objects>),
('__setattr__', <slot wrapper '__setattr__' of 'object' objects>),
('__sizeof__', <method '__sizeof__' of 'object' objects>),
('__str__', <slot wrapper '__str__' of 'object' objects>),
('__subclasshook__',
  <built-in method __subclasshook__ of type object at 0x03135E30>),
('__weakref__', <attribute '__weakref__' of 'MyObject' objects>)]


Comparisons:
a < b :


  testing __gt__(Long name bla bal , Short mame)


  result of a < b: False


a > b :


  testing __gt__(Long name bla bal , Short mame)


  result of a > b: True


a <= b:


  testing __gt__(Long name bla bal , Short mame)


  result of a <= b: False


a == b:


  testing __eq__(Long name bla bal , Short mame)


  result of a == b: False


a >= b:


  testing __gt__(Long name bla bal , Short mame)


  result of a >= b: True

Notes
1.The @functools.total_ordering annotation add implementation of the <= and the => compressions sign  based on the gt and eq methods (without the annotation an exception will be raised ) .
2.The pretty print module handle the iterable collection printing (adding the line feed and cr after printing each member of the list)
3.The inspect module is responsible for getting classes meta data .
4.The  __gt__ and the   __eq__ must be override when using  @functools.total_ordering

יום ראשון, 27 באפריל 2014

Simple CQL Query

The following is a simple project that demonstrates Cassandra CQL .

package playwithcassandralocal;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Host;
import com.datastax.driver.core.Metadata;
import com.datastax.driver.core.ResultSet;
import com.datastax.driver.core.Row;
import com.datastax.driver.core.Session;
public class CqlAccessor {
    
    private Cluster cluster;
     private Session session;
     public void connect(String node) {
        cluster = Cluster.builder()
              .addContactPoint(node)
              .build();
        Metadata metadata = cluster.getMetadata();
        System.out.printf("Connected to cluster: %s\n", 
              metadata.getClusterName());
        for ( Host host : metadata.getAllHosts() ) {
           System.out.printf("Datatacenter: %s; Host: %s; Rack: %s\n",
                 host.getDatacenter(), host.getAddress(), host.getRack());
        }
        session = cluster.connect();
     }
     public void createSchema() 
     {
             session.execute("CREATE KEYSPACE GSelfDrivingCarModel WITH replication " + 
                                "= {'class':'SimpleStrategy', 'replication_factor':1};");
             session.execute(
                                "CREATE TABLE GSelfDrivingCarModel.Images (" +
                                      "id uuid PRIMARY KEY," + 
                                      "title text," + 
                                      "Datecreated text," + 
                                      "Location text," +                                    
                                      "Image blob" + 
                                      ");");
     }
     public void loadData() 
     {
             session.execute(
                                "INSERT INTO GSelfDrivingCarModel.Images (id, title, Datecreated, Location) " +
                                "VALUES (" +
                                    "756716f7-2e54-4715-7f00-91dcbea6cf50," +
                                    "'This is my image.'," +
                                    "'25/10/2014'," +
                                    "'34.16234434 32.0245345'" +
                                    ");" );
                        	   
     }
     public void QueryData ()
     {
             ResultSet results = session.execute("SELECT * FROM GSelfDrivingCarModel.Images " +
                          "WHERE id = 756716f7-2e54-4715-7f00-91dcbea6cf50;");
                  System.out.println(String.format("%-30s\t%-20s\t%-20s\n%s", "title", "Datecreated","Location",
                         "-------------------------------+-----------------------+--------------------"));
                  for (Row row : results) {
                      System.out.println(String.format("%-30s\t%-20s\t%-20s", row.getString("title"),
                      row.getString("Datecreated"),  row.getString("Location")));
                  }
                  System.out.println();
     }
     public void close() {
        cluster.close();
     }
     public static void main(String[] args) {
        CqlAccessor theCqlAccessor = new CqlAccessor();
        
        theCqlAccessor.connect("127.0.0.1");
        theCqlAccessor.createSchema();
        theCqlAccessor.loadData() ;
        theCqlAccessor.QueryData ();
        theCqlAccessor.close();
     }
}

The project :
Capture9


When using Maven the following pom the following dependency retrieves the relevant Jars


	<dependency>
  		<groupId>com.datastax.cassandra</groupId>
  		<artifactId>cassandra-driver-core</artifactId>
  		<version>2.0.1</version>
  		<type>bundle</type>
  	</dependency>

And the Result :
Connected to cluster: Test Cluster
Datatacenter: datacenter1; Host: /127.0.0.1; Rack: rack1
title                          Datecreated          Location           
-------------------------------+-----------------------+--------------------
This is my image.              25/10/2014           34.16234434 32.0245345

Note that there is no update operation in Cassandra when doing insert operation with id that is already exists then the old data will be replaced with the new data .
Resources:
Cassandra documentation .

Generating ndk h files using Eclipse

I  found that the best way to generate Android ndk h file is by using Eclipse extended tools options.
By using the following properties dialog run -> external tools -> external tools configuration ,we can declare an external tool that generate the native h files .
Capture5 
-classpath  "${project_classpath};C:\DevTools\Android\adt-bundle-windows-x86-20131030\sdk\platforms\android-19\android.jar" ${java_type_name}

 I found it beater to declare the pathes hard coded .
Capture6
Capture7 
The result:
Capture8
Resources:
The book:Pro Android C++ with the NDK by Onur Cinar



יום שבת, 26 באפריל 2014

Displaying video and annotations using QT

The following code demonstrates displaying video and annotations using QT .
Capture.PNG3
The project is based on QT and libv4l2 library.

The QT project structure in the QT designer :
Capture3
The code :
capturethread.h

class CaptureThread : public QThread
{
public:
    explicit CaptureThread(QWidget *parent = 0);
    ~CaptureThread();
    bool devam;
    videowidget *parent;
    struct v4l2_format              fmt;
    struct v4l2_buffer              buf;
    struct v4l2_requestbuffers      req;
    enum v4l2_buf_type              type;
    fd_set                          fds;
    struct timeval                  tv;
    int                             r, fd;
    unsigned int                    n_buffers;
    char                            *dev_name;
    char                            out_name[256];
    FILE                            *fout;
    struct buffer {
            void   *start;
            size_t length;
    };
    static void xioctl(int fh, int request, void *arg)
    {
            int r;
            do {
                    r = v4l2_ioctl(fh, request, arg);
            } while (r == -1 && ((errno == EINTR) || (errno == EAGAIN)));
            if (r == -1) {
                    fprintf(stderr, "error %d, %s\n", errno, strerror(errno));
                    return;
            }
    };
    void run();
    struct buffer  *buffers;
    void stopUlan();
    void startUlan();
};
#endif // CAPTURETHREAD_H

mainwindow.h


#ifndef MAINWINDOW_H
#define MAINWINDOW_H
#include <QMainWindow>
namespace Ui {
class MainWindow;
}
class MainWindow : public QMainWindow
{
    Q_OBJECT
    
public:
    explicit MainWindow(QWidget *parent = 0);
    ~MainWindow();
    
private slots:
    void on_pushButton_clicked();
private:
    Ui::MainWindow *ui;
    void pushButtonClicked();
};
#endif // MAINWINDOW_H

videowidget.h


#ifndef VIDEOWIDGET_H
#define VIDEOWIDGET_H
#include <QWidget>
#include <QPainter>
class videowidget : public QWidget
{
Q_OBJECT
public:
    explicit videowidget(QWidget *parent = 0);
    QImage img;
protected:
    void paintEvent(QPaintEvent *event);
signals:
public slots:
};
#endif // VIDEOWIDGET_H

capturethread.cpp


#include <QApplication>
#include <QDataStream>
#include <QString>
#include <QDebug>
#include <QBuffer>
#include <typeinfo>
#include <iostream>
#include <QFile>
#include "videowidget.h"
#include "capturethread.h"
CaptureThread::CaptureThread(QWidget *parent) :
    QThread()
{
    //qDebug()<<parent->objectName();
    this->parent=(videowidget*)parent;
    devam=false;
    fd = -1;
}
void CaptureThread::run(){
//do real stuff
fd = -1;
dev_name = "/dev/video0";
    fd = v4l2_open(dev_name, O_RDWR | O_NONBLOCK, 0);
    if (fd < 0) {
           qDebug("Cannot open device");
           //exit(EXIT_FAILURE);
           return;
    }
    static struct v4lconvert_data *v4lconvert_data;
    static struct v4l2_format src_fmt;
    static unsigned char *dst_buf;
    CLEAR(fmt);
    fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    fmt.fmt.pix.width       = 640;
    fmt.fmt.pix.height      = 480;
    fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
    fmt.fmt.pix.field       = V4L2_FIELD_INTERLACED;
    xioctl(fd, VIDIOC_S_FMT, &fmt);
    if (fmt.fmt.pix.pixelformat != V4L2_PIX_FMT_RGB24) {
           printf("Libv4l didn't accept RGB24 format. Can't proceed.\n");
           //exit(EXIT_FAILURE);
           return;
    }
    if ((fmt.fmt.pix.width != 640) || (fmt.fmt.pix.height != 480))
           printf("Warning: driver is sending image at %dx%d\n",
                   fmt.fmt.pix.width, fmt.fmt.pix.height);
    v4lconvert_data = v4lconvert_create(fd);
    if (v4lconvert_data == NULL)
        qDebug("v4lconvert_create");
    if (v4lconvert_try_format(v4lconvert_data, &fmt, &src_fmt) != 0)
        qDebug("v4lconvert_try_format");
    xioctl(fd, VIDIOC_S_FMT, &src_fmt);
    dst_buf = (unsigned char*)malloc(fmt.fmt.pix.sizeimage);
    CLEAR(req);
    req.count = 2;
    req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    req.memory = V4L2_MEMORY_MMAP;
    xioctl(fd, VIDIOC_REQBUFS, &req);
    buffers = (buffer*)calloc(req.count, sizeof(*buffers));
    for (n_buffers = 0; n_buffers < req.count; ++n_buffers) {
           CLEAR(buf);
           buf.type        = V4L2_BUF_TYPE_VIDEO_CAPTURE;
           buf.memory      = V4L2_MEMORY_MMAP;
           buf.index       = n_buffers;
           xioctl(fd, VIDIOC_QUERYBUF, &buf);
           buffers[n_buffers].length = buf.length;
           buffers[n_buffers].start = v4l2_mmap(NULL, buf.length,
                         PROT_READ | PROT_WRITE, MAP_SHARED,
                         fd, buf.m.offset);
           if (MAP_FAILED == buffers[n_buffers].start) {
                   qDebug("mmap");
                   //exit(EXIT_FAILURE);
                   return;
           }
    }
    for (int i = 0; i < n_buffers; ++i) {
           CLEAR(buf);
           buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
           buf.memory = V4L2_MEMORY_MMAP;
           buf.index = i;
           xioctl(fd, VIDIOC_QBUF, &buf);
    }
    type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    xioctl(fd, VIDIOC_STREAMON, &type);
    int di=0;
    char header[]="P6\n640 480 255\n";
    devam = true ;
    while(devam){
        /* bu döngü datanın birikmesini sağlıyor */
        do {
                FD_ZERO(&fds);
                FD_SET(fd, &fds);
                /* Timeout. */
                tv.tv_sec = 2;
                tv.tv_usec = 0;
                r = select(fd + 1, &fds, NULL, NULL, &tv);
        } while ((r == -1 && (errno = EINTR)));
        if (r == -1) {
                qDebug("select");
                //exit(1) ;
                return;
        }
        CLEAR(buf);
        buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        buf.memory = V4L2_MEMORY_MMAP;
        xioctl(fd, VIDIOC_DQBUF, &buf);
        try{
            
        if (v4lconvert_convert(v4lconvert_data,
                                &src_fmt,
                                &fmt,
                                (unsigned char*)buffers[buf.index].start, buf.bytesused,
                                dst_buf, fmt.fmt.pix.sizeimage) < 0) {
                if (errno != EAGAIN)
                        qDebug("v4l_convert");
        }
        unsigned char* asil=(unsigned char*)malloc(fmt.fmt.pix.sizeimage+qstrlen(header));
        memmove(asil, dst_buf, fmt.fmt.pix.sizeimage);
        memmove(asil+qstrlen(header), asil, fmt.fmt.pix.sizeimage);
        memcpy(asil,header,qstrlen(header));
        QImage qq;//=new QImage(dst_buf,640,480,QImage::Format_RGB32);
        if(qq.loadFromData(asil,fmt.fmt.pix.sizeimage+qstrlen(header),"PPM")){
            if(parent->isVisible()){
                QImage q1(qq);
                parent->img=q1;
                parent->update();
              //this->msleep(50);
            }
        //qApp->processEvents();
            if(asil)
                free(asil);
        }
        }catch(...){}
        xioctl(fd, VIDIOC_QBUF, &buf);
        di++;
   }
    try{
    type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    xioctl(fd, VIDIOC_STREAMOFF, &type);
    for (int i = 0; i < n_buffers; ++i)
           v4l2_munmap(buffers[i].start, buffers[i].length);
        v4l2_close(fd);
    }catch(...){}
}
CaptureThread::~CaptureThread()
{
    try{
    type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    xioctl(fd, VIDIOC_STREAMOFF, &type);
    /*for (int i = 0; i < n_buffers; ++i)
           v4l2_munmap(buffers[i].start, buffers[i].length);*/
        v4l2_close(fd);
    }catch(...){}
    fd = -1;
}
void CaptureThread::stopUlan()
{
    devam=false;
    try{
    type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    xioctl(fd, VIDIOC_STREAMOFF, &type);
    /*for (int i = 0; i < n_buffers; ++i)
           v4l2_munmap(buffers[i].start, buffers[i].length);*/
        v4l2_close(fd);
    }catch(...){}
    fd = -1;
}
void CaptureThread::startUlan()
{
    this->start();
}
/*
void CaptureThread::onStopCapture()
{
    type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    xioctl(fd, VIDIOC_STREAMOFF, &type);
    for (int i = 0; i < n_buffers; ++i)
           v4l2_munmap(buffers[i].start, buffers[i].length);
    v4l2_close(fd);
}*/

main.cpp


#include <QtGui/QApplication>
#include "mainwindow.h"
int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    MainWindow w;
    w.show();
    
    return a.exec();
}

mainwindow.cpp


#include "mainwindow.h"
#include "ui_mainwindow.h"
#include "videowidget.h"
#include "capturethread.h"
#include <QtGui>
#include <QtGui/QMessageBox>
MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    ui->setupUi(this);
    ui->pushButton->connect( ui->pushButton, SIGNAL( clicked() ), this, SLOT( pushButtonClicked() )  );
    connect ( ui->pushButton, SIGNAL( clicked() ), this, SLOT( pushButtonClicked() ) );
    QVBoxLayout* layout = new QVBoxLayout();
         layout->addWidget(new videowidget (ui->myFrame));
         ui->myFrame ->setLayout(layout);
         ui->myFrame->show();
}
void MainWindow::pushButtonClicked() // defined in .h under public/private slots:
{
    QMessageBox::information( this, "Information", "Just clicked Ui PushButton" ); // #include <QtGui/QMessageBox>
}
MainWindow::~MainWindow()
{
    delete ui;
}
void MainWindow::on_pushButton_clicked()
{
    QMessageBox::information( this, "Information", "Just clicked Ui PushButton" ); // #include <QtGui/QMessageBox>
}

videowidget.cpp


#include "videowidget.h"
#include "capturethread.h"
videowidget::videowidget(QWidget *parent) :
    QWidget(parent)
{
    setAutoFillBackground(true);
    //setObjectName("video göstergeci");
    CaptureThread * theCaptureThread = new CaptureThread (this);
    theCaptureThread->start ();
}
void videowidget::paintEvent(QPaintEvent *event) {
     try{
    QPainter painter(this);
        if(!img.isNull()){
            painter.drawImage(QPoint(0,0), img);
            painter.setPen(Qt::blue);
            painter.setFont(QFont("Arial", 30));
            painter.drawText(rect(), Qt::AlignCenter, "Qt");
        }
    }catch(...){}
}

The layout file :
Capture4
Note in order to capture the screen in linux ubuntu use the :
gnome-screenshot
command in bash shell‏