--more--

reserve() and capacity() vs resize() and size()

reserve() makes the vector allocate memory internally but that doesn't mean that that memory is being used yet. We can know how much memory a vector has allocated by calling capacity().

resize() allocates memory if it's necesary and it also marks it as used. We can know how much elements are in use by calling size():

vector<int> v;
v.reserve(50);
cout << v.capacity() << endl;    // will print 50
cout << v.size() << endl;        // will print 0

v.push_back(1);
cout << v.capacity() << endl;    // will print 50
cout << v.size() << endl;        // will print 1

v.resize(100);
cout << v.capacity() << endl;    // will print 100
cout << v.size() << endl;        // will print 100

clear() is also related to the number of used elements, not to the allocated memory. So if we call clear we are actually telling that no memory is in use but there's actually no memory deletion: vectors' allocated memory never shrinks[2]:

vector<int> v;
v.reserve(50);
cout << v.capacity() << endl;    // will print 50
cout << v.size() << endl;        // will print 0

v.push_back(1);
cout << v.capacity() << endl;    // will print 50
cout << v.size() << endl;        // will print 1

v.clear();
cout << v.capacity() << endl;    // will print 50
cout << v.size() << endl;        // will print 0

So calling clear is actually fast and reusing a vector is faster than creating a new one cause the memory is already allocated.

push_back()

The push_back method of a vector inserts a new element at the end, but, what happens with memory internally? Well, even if the size of the vector increases by one vectors' capacity grows exponentially[2]:

vector<int> v;
v.push_back(1);
cout << v.capacity() << endl;  // will print 1
cout << v.size() << endl;      // will print 1

v.push_back(1);
cout << v.capacity() << endl;  // will print 2
cout << v.size() << endl;      // will print 2

v.push_back(1);
cout << v.capacity() << endl;  // will print 4 <<<
cout << v.size() << endl;      // will print 3

v.push_back(1);
cout << v.capacity() << endl;  // will print 4
cout << v.size() << endl;      // will print 4

v.push_back(1);
cout << v.capacity() << endl;  // will print 8 <<<
cout << v.size() << endl;      // will print 5

A vector behaves like this to make things faster, and it makes things faster for 2 reasons:

• Allocating memory is slow but it's equally slow no matter how much memory we are allocating[3]
• All the elements in a vector are contiguous in memory. When we push back a new element it has to be after the last one. So if there's no more unallocated memory after the last element, the vector needs to move all the elements that it already contained to a new position. Moving all the elements actually means: allocating new memory, copying all the elements to the new location and deleting the old memory, which is really, really slow

As it can be seen in the previous code, whenever there's no more space left in the vector, and we add a new element, it doubles the size of the allocated memory, that way we can keep adding elements for a while without reallocations.

Now if we print the address in memory of the first element after pushing back new ones, we'll see that probably[4], after exhausting the capacity, the vector moves it's elements to a new location in memory:

vector<int> v;
v.push_back(1);
cout << v.capacity() << endl;  //will print out 1
cout << &v[0] << endl          //will print some address

v.push_back(1);
cout << v.capacity() << endl;  //will print out 2
cout << &v[0] << endl          //will probably print a different address

v.push_back(1);
cout << v.capacity() << endl;  //will print out 4 <<<
cout << &v[0] << endl          //will probably print a different address

v.push_back(1);
cout << v.capacity() << endl;  //will print out 4
cout << &v[0] << endl          //will print the same address

v.push_back(1);
cout << v.capacity() << endl;  //will print out 8 <<<
cout << &v[0] << endl          //will probably print a different address

This is very important for several resons: as i've told before the exponential growth makes thing faster, but even with that, allocating new memory and moving the elements around is slow. So if you know you are going to have at least 50 elements it's better to reserve that memory or more before beginning to add elements:

vector<int> v;
v.reserve(50);
v.push_back(1);
//...

That way there's going to be no memory reallocation till we push back 50 elements.

push_back() is slow

Try to avoid it if you are going to add lots of elements. For the reasons mentioned above this:

vector<int> v;

for(int i=0;i<100;i++){
    v.push_back(1);
}

is way slower than:

vector<int> v;

v.resize(100);
for(int i=0;i<100;i++){
    v[i] = 1;
}

Use the second!

Even this:

vector<int> v;
v.reserve(100);

for(int i=0;i<100;i++){
    v.push_back(1);
}

is slightly slower since push_back has to check if there's still memory available while when using the [] operator there's no check.

If you are adding a few elements at a time or one by one and don't know how many you'll have then push_back is the best because of the exponential growth. For example:

class Shape{
public:
void addVertex(Point p){
    points.push_back(p);
}

private:
vector<Point> points;
}

is correct, trying to optimize it by using resize() like:

void addVertex(Point p){
    points.resize(points.size()+1);
    points[points.size()-1] = p;
}

will usually be slower since it will do more memory allocations in the long term.

push_front() is super slow

A vector only grows from the end, if we add elements in the front it actually moves all the elements one position and adds the new element. So unless you are not concerned with performance don't use push_front. If you need to add elements both in the front and the back very often, then you probably want to use a list or a deque.

If you only need to add elements in the front, add them in the back and go through the vector in the opposite direction.

pointers to elements in vectors: Don't!

As i've said above things in a vector move when it needs more memory to grow, so pointers to elements in a vector can and probably will become invalid:

vector<int> v;
v.push_back(1);
int * p = &v[0];

for(int i=0;i<100;i++){
    v.push_back(1);
}

cout << p << endl;
cout << &v[0] << endl;  // will surely be different than p

We can solve it by using vectors of pointers, storing positions in the vector, using std lists or using a vector as if it was a static array.

using a vector as if it was a static array

In c++ as a general rule, don't use arrays, a vector can do the same and the syntax is more understandable.

If for any reason you are thinking in using an array instead of a vector, you can always use a vector as if it was an array, just don't use push_back:

vector<int> v(100);

for(int i=0;i<100;i++){
    v[i] = 1;
}

What's the advantage of this over a plain c array? well, mainly it's easier to use, for example to copy the contents of a vector to another:

vector<int> v(100);
//...
vector<int> v2 = v;

while with arrays:

int v[100];
//...
int v2[100];
memcpy(v2,v,100*sizeof(int));

Or for example to pass an array as a parameter to a function you need to pass it's size in a different argument:

void doSomething(int * array, int size){
...
}

and at the same time you have to keep that size in some variable making your code more complex. A vector carries it's size in it self, that's called encapsulation and it's the main reasons to use a vector over an array.

vectors and threads

I'm not going to enter into how threads work here but when working with threads and stl vectors keep in mind that:

• If you use a vector as an array, as explained above, you don't need to lock[5], you won't get crashes.
• If you need to change the vector size, then lock always that you access the vector elements both for reading and writing. And, very important, also when you access it's size.
• If you check the size and then access the vector relying on the retrieved size, do it in the same lock. If not, the size can change while you unlock and lock again

Also this post in the OF forum explains lots of details about threads and memory access:

http://forum.openframeworks.cc/index.php?topic=7248.0

an OF example

Here's an OF example that demoes the concepts explained in this article.

https://github.com/arturoc/memoryExamples

[1]: Yes, a c array can also grow by using malloc, realloc and free instead of new and delete.

[2]: Theoretically the statements in this article about memory growing are dependent on the particular implementation but all the implementations i know behave like this.

[4]: This is a simplification but for most cases with vectors it's true

[4]: Sometimes, mostly when the vector is still small, there's enough unused memory after the last element and no reallocation is needed but most of the times when the vector changes it's internal size it needs to move it's elements to a new location.

[5]: There's other reasons for locking apart from having crashes, most of them explained in the forum thread in the link above.

Some days ago this tweet got me intrigued. Some hours later Joel asked for help to @mrdoob since him and @marcinignac did some pointcloud visualization in the browser using mrdoob's Three.js library during OFFF 2011. I had some code for streaming using gstreamer in openFrameworks so it was just a matter of joining the pieces.

@joelgethinlewis and @roxlu were trying to make this work at Art and code 2011 which topic this year was DIY 3D sensing and visualization.

--more--

@Roxlu almost had this running using ffmpeg, the main problem with this approach is that for each client connected to the server there was going to be a stream, and probably the server and the bandwidth weren't going to be enough. The key was to have the server only producing one stream and some online service streaming it to the clients. In the end we didn't manage to make this run because of problems with cross-site security in browsers so probably @roxlu's solution would have worked the same.

There's a great service for free streaming called giss.tv. Yves one of the admins of the site has some python tools that actually use gstreamer. The same pipeline used by that tools with some modifications to get the depth info coming from the kinect worked without problem with ofGstUtils.

Gstreamer has 2 elements which are really useful when you want to process custom data: appsrc, for feeding data into a pipeline and appsink for getting data out of it. appsink is used internally in OF to get data from cameras or video files and upload it to textures in the linux version of the video grabber and the video player.

Using appsrc is really easy to feed the depth data from the kinect into a pipeline and then stream it to giss.tv. Since the video stream has color information (3 channels) and the depth data only 1 the red channel can be used to feed the depth data and the green one to send brightness, something like:

for(int i=0;i<640*480;i++){
    pixels[i*3] = kinect.getDepthPixels()[i];
    pixels[i*3+1] = (kinect.getPixels()[i*3] 
        + kinect.getPixels()[i*3+1] 
        + kinect.getPixels()[i*3+2]) / 3;
    pixels[i*3+2] = 0;
}
// feed pixels into the gstreamer streaming pipeline

The blue channel is unused here so it's even possible to send the raw depth data returned by kinect, which is 16bits, using r & g and b for brightness or do some kind of duplication of the depth data to have more chances of reconstructing it later avoiding compression artifacts.

In HTML5 you can get the pixel data of any element, for example a video and upload it to the graphics card in a texture using webGL. Then using some shaders @mrdoob and @kcmic managed to separate the depth data from the brightness to generate a point cloud.

The main problems we had were with cross-site security in browsers which didn't allowed to use the stream directly from giss.tv so in the last minute we had to do a php script which read the stream and served it as if it was comming from the same server as the web page. This was a really ugly hack and as soon as 1000 petitions came when @golan tweeted about the stream the server couldn't handle it.

The other problem was that theora compression (or any other video compression for that matter) is actually optimized for images not depth data so there was some really noticeable artifacts in the final point cloud.

The source code is here

--col2--

@chrissugrue talking about her lastest work Base8

@golan and @kcmic preparing the stage

Same image as above but the perspective is slightly different from that of the kinect, allowing to see the artifacts introduced by the theora compression in the depth data

Raw depth image, the first data we managed to stream

--more--

Faces works mainly thanks to Jason Saragih's library for face tracking which can be found here

To make it easier to use Kyle McDonald developed some months ago the OF addon ofxFaceTracker. This addon returns a mesh which vertexes match the facial features found in a picture of a face. To understand better how it works take a look at this video which shows the raw output of Jason's library:

The returned vertexes are always the same in the same order for every face, which makes it extremely easy to get the face in a photo and adapt it to match a face found in a video.

In openGL, if you pass a mesh and a texture for that mesh to the graphics card, moving the vertexes makes the texture stretch or srink accordingly. So to adapt the face in the photo to the one in the video you just need to move the vertexes of the mesh in the photo to match the ones in the video, indeed you just need to use the mesh of the video with the texture and texture coordinates of the photo, something like:

void testApp::update(){
    for(int i=0;i<meshVideo.getNumVertices();i++){
        meshVideo.addTexCoord(meshPhoto.getVertex(i));
    }
}

void testApp::draw(){
    video.draw(0,0);
    photo.getTextureReference().bind();
    meshVideo.draw();
    photo.getTextureReference().unbind();
}

Something like this will make the result look very unatural cause skin tones and lighting are usually different in the photo and the video. To correct for that we use some color interpolation. To put it simple we take the color for each vertex in the photo and in the video make an average and use that as the color for the mesh that is going to be drawn. The graphics card does the rest by interpolating the color of each vertex to the rest of the texture. (The actual algorithm is a little bit more complex than this)

for(int i=0;i<meshVideo.getNumVertices();i++){
    ofColor colorPhoto = photo.getPixelsRef().getColor(meshPhoto.getVertex(i));
    ofColor colorVideo = video.getPixelsRef().getColor(meshVideo.getVertex(i));
    ofFloatColor dstColor = (colorPhoto + colorVideo)*0.5
    meshVideo.addColorCoord(dstColor);
}

The source code and osx binaries can be found in my github. Take into account that this cannot be used commercially since Jason's Face Tracking library can only be used for research purposes.