Download Code Samples ZIP 1.07MB
Contents
- Introduction
- Requirements
- Project Structure
- Simple high-level sequence diagram
- Code Walkthrough
- In-Depth Understanding
Introduction
In this document and sample application, I will show you how to use the Intel® RealSense™ camera (R200) and the Enhanced Photography functionality that is part of the Intel® RealSense™ SDK.
Requirements
Hardware requirements:
- 4th generation Intel® Core™ processors based on the Intel® microarchitecture code-named Haswell
- 8 GB free hard disk space
- Intel RealSense camera (R200)
- USB* 3 port for camera connection
Software requirements:
- Microsoft Windows* 8.1 or higher OS 64-bit
- Microsoft Visual Studio* 2010-2015 with the latest service pack
- Microsoft .NET* 4.0 Framework for C# development
- Intel RealSense SDK, which can be downloaded here
Note: This particular sample project was created in Visual Studio 2015 using the latest .NET release.
Project Structure
In this sample application, I have separated the Intel RealSense SDK functionality from the GUI layer code to make it easier for a developer to focus on the R200 Enhanced Photography functionality. To do this, I created the following C# wrapper classes:
- RSStreaming. Wraps the Intel RealSense SDK streaming functionality.
- RSEnhancedPhotography. Wraps the Intel RealSense SDK Enhanced Photography.
Also, for simplicity I created individual WinForms for each type of Enhanced Photography function. While this creates a little more duplicated code, it offers the benefit of keeping things precise in trying to demo each Enhanced Photography functionality.
While there is more to the project structure than what I have mentioned here, I will go into full detail on all the source code files and how they work throughout this document.
I make use of creating my own events to demonstrate how to keep the Intel RealSense SDK functionality loosely coupled from the GUI source code as much as possible. I think this is a cleaner solution than passing an entire form into a class so that a given class can access properties on the form.
Also note that this sample application does not try to enforce proper software engineering techniques. There is little if any runtime checking, no try catch blocks. Providing a simple, clean example project to learn from keeps the code as clean as possible without introducing extra distractions.
While not elegant, the forms in this sample application serve the purpose of demonstrating how to use Enhanced Photography.
Visual Studio Project Structure
The image above shows what the Visual Studio 2015 solution looks like. The various folders contain:
- Forms. The various WinForms that demonstrate a different Enhanced Photography functionality.
- Source. The source code that goes along with the project.
- Source\CustomEventArgs. Contains classes that have been derived from the native EventArgs class.
- MainForm.CS. The main form of the application.
Simple high-level sequence diagram
When you run the sample application, FormMain will display. On this form you can start streaming by clicking the Streaming button 
. When this button is clicked, the application kicks off the streaming by making use of functionality wrapped up in RSStreaming class.
The RSStreaming class is constantly causing updates to FormMain by calling its internal event OnNewStreamingSample. This happens for every frame that comes from the camera.
You can stop streaming by clicking the Stop Streaming button 
. When you do, the streaming simply stops running, and there are no other options available other than to start streaming again. However, if you click the Photo button 
, the image data is saved to disk and streaming stops. Once streaming stops and the photo has been saved to disk, the Photo Enhancement buttons becomes active, enabling you to select from among the various Enhanced Photography dialogs that demonstrate the capabilities included in this sample application.
When one of the Enhanced Photo dialogs is selected, the sample image that was saved to disk is loaded and utilized. I will explain this in more detail later in this document.
Code Walkthrough
The following sections will walk you through the entire application, focusing on the flow of the application and descriptions of the various classes.
High-level overview of the source code files and form
Forms
FormDepthEnhance. Demonstrates how to use two different depth quality settings to display depth data. The user can choose either Low Quality or High Quality.
FormDepthPasteOnPlane. This form demonstrates how to use the paste on plane functionality to import an external image by clicking two points on a flat surface.
FormDepthRefocus. This form shows how to click a point and have the focus point on an image brought to light by blurring the rest of the image. You can click a spot on the image which then becomes the focus point. You can adjust the simulated aperture of the camera lens by moving the slider.
FormDepthResize. Shows the resize functionality that can upsize the depth image to be the same size as the RGB image.
FormMeasure. Demonstrates how to use the Enhanced Photography measure capabilities to obtain distance, precision, and confidence values.
FormMain. This is the main form for the application. It allows the user to start the Intel RealSense camera streaming, stop streaming, and capture a snapshot. Once a snapshot has been taken, the user can perform various Enhanced Photography functions on the image.
Source Code
RSEnhancedPhotography.CS. This is a wrapper class that encapsulates the Intel RealSense camera Enhanced Photography functionality. Its purpose is to remove as much of the Intel RealSense camera functionality from the GUI layer as possible. It uses custom events to publish data back to the client application. It uses the RSEnhancedImageArg class to contain the new image that gets displayed.
RSPaintUtil.CS. This is a static utility class that assists in drawing mouse click points and lines onto the C# PictureBox controls.
RSPoints.CS. This helper class encapsulates PXCMPointI32 point objects and creates functionality to store points and validate point data as well as report point data to be displayed on the GUI.
RSStreaming.CS. This is a wrapper class that encapsulates Intel RealSense camera streaming. It streams data and publishes an event back out to the client. The event uses the RSSampleArg class to store data to be used by the client.
RSUtility.CS. This is a static class that contains source code; that is, as the name implies, utility. None of the functionality really belongs in any particular class.
Source\CustomEventArgs Code
RSEnhancedImageArg. Extends EventArgs by containing a PXCMImage object. This object will contain an image that has been manipulated by the Intel RealSense SDK Enhanced Photography functionality. This image is to be used to display on the individual WinForms PictureBox control.
RSMeasureArg.CS. Extends EventArgs by containing measure data returned from the Intel RealSense SDK Enhanced Photography functionality. This data is used on the WinForm “FormMeasure” to report measurement information back to the user.
RSSampleArg.CS. Extends EventArgs by containing a PXCMCapture.Sample object. This object contains the latest frame captured by the camera and is used for streaming data and displaying it on the WinForm FormMain.
In-Depth Understanding
Now I’m going to describe the underlying supporting classes that support the forms. I think it’s best to learn how the underlying code base works instead of focusing first on the forms. I’ll start with the first class, RSStreaming.
I won’t cover details such as the getter and setter functions, which are self-explanatory. Nor will I cover any function that is clearly obvious for other reasons.
RSStreaming
RSStreaming is a wrapper class around the Intel RealSense SDK streaming capabilities. This class isn’t overly complex. The intent is to show a simple example of how to stream data from the Intel RealSense camera. It has the ability to both stream and take an Enhanced Photo image and send both back to the client via events.
public event EventHandler<RSSampleArg> OnNewStreamingSample;
As mentioned previously in this document, I use Events to send data back to the client apps (Forms). RSStreaming sends data back to the client; in this case, FormMain via the event OnNewStreamingSample. It takes one parameter, RSSampleArg, which will contain the newest sample from the camera.
public bool Initialized
A simple getter flag that indicates whether the class has been initialized.
public bool IsStreaming
A simple getting flag that indicates whether the class is currently streaming data.
public void StartStreaming( )
A public function that a client uses to start the streaming from the camera. Ensures that the class has been properly initialized and if so calls the InitCamera() function to initialize the camera.
One key feature that I would like to mention is that I’m using a feature that does not get a lot of focus. As you have probably seen in a lot of samples for streaming, the sample shows a while loop and the AcquireAccess function with some type of mechanism to cancel streaming via a boolean flag. This sample uses a different approach.
My approach uses the PXCMSenseManager’s StreamFrames function, which causes the SenseManager to kick off its own internal thread and send data back via event handling. This is done by assigning the PXCMSenseManager.Handler object to a function. More on that later in the InitCamera( ) function.
private void InitCamera( )
InitCamera is a private function that initializes the camera and streaming. Starting out we create the PXCMSenseManager and PXCMSession objects. Next, we need the device information (Camera), which is gotten by making use of the RSUtility GetDeviceByType() static function passing in the session and the type of camera we want.
Then I create two PXCMVideoModule.DataDesc objects, one for color streaming and the other for depth streaming. From there I configure each stream. After the streams have been configured, I prompt the PXCMSenseManager to enable the streams.
As mentioned in the function StartStreaming(), I’m using an event-based approach to streaming and gathering data, which is done by creating and initializing a PXCMSenseManager.Handler event handler object and assigning it to the OnNewSample function. Every time the camera captures a new frame, the OnNewSample event handler is called.
Once this has all been accomplished, I initialize the SenseManager, sending it in the handler object and telling it to use this object and its event hander.
private pxcmStatus OnNewSample( int mid, PXCMCapture.Sample sample )
OnNewSample is the event handler for the PXCMSenseManager.Handler object.
Parameters
- Mid. The stream identifier. If multiple streams are requested through the EnableVideoStream[s] function, this is PXCCapture.CUID+0, or PXCCapture.CUID+1....
- PXCMCapture.Sample. The sample image that came from the camera.
When this function is called, I capture the image out of the Sample argument and put it into a new RSSampleArg object, then call the OnNewStreamingSample event for this class. This forces the event to notify the client FormMain that a new image is ready to be displayed.
Release the frame and then return the required pxcmStatus, which is not being used in this case.
public void StopStreaming( )
Stops the streaming by closing the streams and calling Dispose( ).
private void Dispose( )
Frees up resources for garbage collection.
RSEnhancedPhotography
The RSEnhancedPhotography class was created to wrap the Enhanced Photography functionality functionality into one easy-to-use class. It works on an event principle. Once an image has been processed, an event is raised returning the newly created image or measurement data back to the client app/class.
public RSEnhancedPhotography( PXCMPhoto photo )
Constructor initializes several of the global variables that are used in the class. The single input parameter is the original photo that was taken by the main form. It’s fully initialized with image data and used to initialize the local _colorPhoto object.
public void Dispose( )
Releases the memory to be garbage collected.
public void SendOriginalImage( )
Returns the original image back to the calling application by making use of the OnImageProcessed event.
public void MeasurePoints( RSPoints points )
MeasurePoints receives a populated RSPoints object. First I ensure that there are indeed two valid points in this object, the start and end points. Once this has been determined, a MeasureData object is created and sent into the PXCMEnhancedPhoto objects MeasureDistance function.
Next I take the data from the populated measureData object and populate the RSMeasureArg object. Notice the ( mesaureData.distance / 10 ) parameter, which converts to centimeters. Once the arg object has been populated, I send it back to the client via the OnImageMeasured event.
public void RefocusOnPoint( RSPoints point, float aperture = 50 )
With a camera, you set the aperture to get either a large or shallow depth of field. A small aperture creates a large depth of field; a wide-open aperture creates a shallow depth of field, which blurs items in front of or behind your subject.
RefocusOnPoint has the same effect. The function allows you to change your focal point in the image.
Of course, aperture settings don’t work in values of 0–100, but for the purposes of this example they do. If you want, please convert them to proper f-stops and send me the updated code.
RefocusOnPoint uses the PXCMEnhancedPhotos DepthRefocus function to create a new image with a new depth focus using the original color photo—the point where the user clicked on the screen and an aperture setting. Once we have the newly created PXCMPhoto I get the reference image out by calling the QueryReferenceImage() and then supplying the PXCMImage to the RSEnahncedImageArg instance. From there, you just need to pass it back to the client application via the OnImageProcessed event.
public void DepthEnhancement( PXCMEnhancedPhoto.DepthFillQuality quality )
Enhances/Changes the depth quality of an image between two values, either high or low. This is specified in the DepthFillQuality parameter.
First initialize the local PXCMPhoto image by calling the PXCMEnhancedPhoto’s EnhanceDepth function supplying the original PXCMPhoto and the quality specified.
Then to initialize the PXCMImage, I use the enhancedPhoto‘s QueryDepthImage to give the newly created depth image.
Once this has all been done, I create the new RSEnhancedImageArg to be sent back to the client via OnImageProcessed.
public void DepthResize( )
This shows a simplistic way to resize a depth image. In this case it resizes the depth image to be the same size as the color image specified in the original PXCMPhoto that was created in the constructor.
First I need the size information from the color photo. To do this, I query for the original PXCMPhoto object specified in the constructor. I then create the instance of the PhotoUtils object that contains the DepthResize function. After I get the size of the original image, I store the width and height in the required PXCMSizeI32 object.
From there it’s a simple process of telling the PXCMEnhancedPhoto to resize the depth image by calling the DepthResize function, specifying the PXCMPhoto and target size.
Once the resizing is done, it’s the same thing. Create the image by querying for enhancedPhoto, populating the RSEnhancedArg and sending it back to the client via OnImageProcessed.
public void PastOnPlane( RSPoints points )
This function shows how a user can take a PXCMPhoto and paste a new image onto a flat surface. When doing so, the image being pasted adapts to the environment. This means that if the image is pasted onto a wall, the image will look upright and have the same angle. If the image is pasted onto a desktop surface, the image will appear to lay down flat on the desk.
First, we need to ensure that there are two valid points, which the functionality requires.
Next, we load the image we want to paste by using the RSUtility’s LoadBitmap function.
The key object in working on this is the PXCMEnhancedPhoto.Paster class. This is a new class with the R5 release. Paster has a function PasteOnPlane that used to be in the PXCMEnhancedPhoto class, but was moved into this new class.
In this function I’m being cautious by looking to see if the return value from PastOnPlane is not null. This is because there is no guarantee that the PasteOnPlane function was able to successfully perform the operation. For example, if the surface between the two points is not flat, the function will not succeed. I’m simply ensuring that I don’t use a null object.
If we have a successful return value, I get the reference image, store it, pass it into the RSEnhancedImageArg object and post it back to the client application.
RSUtility
RSUtility is a static utility class that contains functionality that does not appropriately fit into any of the other classes.
public static PXCMCapture.DeviceInfo GetDeviceByType( PXCMSession session, PXCMCapture.DeviceModel deviceType )
This function is a helper function that is focused on getting the detailed information about a device—in this case, the R200 camera. This functionality has been seen in multiple RealSense examples.
First, we set up a filter by specifying that we are looking for a sensor as the main group, then for the subgroup we specify a video capture sensor.
Because multiple devices can be on a given system, we must iterate over all the possible devices. For each iteration, I populate the current PXCMSession.ImplDesc data in the currentGroup object. If there is no error we move onto the next step, which is to populate the PXCMCapture object.
Once we have the PXCMCapture object, I iterate over the various devices attached to this object. Check to see whether the current device information is for the camera we are looking for, and if it is, we break out of the loop. If not, we move onto the next device information attached to the PXCMCapture device until all devices have been checked.
Once the device has been found, we return it to the client, which in this case is the RSStreaming object.
public static Bitmap ToRGBBitmap( PXCMCapture.Sample sample )
A polymorphic function simply turns around and calls ToRGBBitmap( PXCMImage image ), passing it the sample argument’s image.
public static Bitmap ToRGBBitmap( PXCMImage image )
A simple wrapper function that uses the PXCMImage objects functionality to get bitmap data using a PXCMImage.ImageData object. Data is pulled out and stored into a .NET bitmap object and returned to the client.
public static PXCMImage LoadBitmap( PXCMSession session )
This function is used to support PasteOnPlanes. It loads a predetermined bitmap into a PXCMImage object and returns it to the client.
First it gets the path to the file and ensures the file exists. If not, it returns null. If the file exists it creates a new .NET bitmap object by loading it from a file.
A PXCMImage.ImageInfo object is used to hold basic information about the bitmap. This is used when creating the PXCMImage object. They are the specifications for the image we will create.
Next we need a PXCMImage.ImageData object, which contains and holds the actual bitmap data. A .NET BitmapData object is created and initialized with data that describes the format and structure of the data we need.
To fully understand what Bitmap.Lockbits is doing, refer to https://msdn.microsoft.com/en-us/library/5ey6h79d(v=vs.110).aspx.
The PXCMImage object releases access to the image data to free the memory used, the bitmap unlocks its bits, and the PXCMImage is returned to the client.
public static int GetDepthAtClickPoint( PXCMPhoto photo, PXCMPointI32 point )
This function receives a PXCMPhoto and a PXCMPointI32.
public static bool SavePhoto( PXCMSession session, PXCMCapture.Sample sample )
Saves the photo to the disk. Uses the session to create a new PXCMPhoto object that has the functionality to save to disk. The PXCMPhoto object uses its import from the preview sample to import the image data into itself. Find out whether the file already exists; if so delete it and save the file.
public static PXCMPhoto OpenPhoto( PXCMSession session )
Straightforward function, ensures that the XDM photo exists on the hard drive, if so, uses the PXCMSession object to create the PXCMPhoto. The photo object then loads the XDM file. The function then returns the PXCMPhoto back to the client.
RSPaintUtil
The RSPaintUtil class is a utility class to encapsulate the drawing of points and lines onto the picture boxes photos. Also, it reduces code duplication between different forms that rely on this functionality.
It ensures that there is a valid start point. If there is, create a new .NET Point object specifying the start point’s x,y values.
Calls the draw circle function to draw a circle around the point that was clicked.
static public void DrawTwoPointsAndLine( RSPoints points, PaintEventArgs e )
Ensures that both points are valid and creates two new .NET Point objects. The function draws circles at those points by calling DrawCircle for each. Then a line is drawn between them via DrawLine.
static private void DrawCircle( Point p, PaintEventArgs e )
This function draws a circle around the x,y coordinates of the .net Point object. This is done by creating a new Pen object. A rectangle is needed to draw any circle. This is what defines the size of the circle to be drawn. I created a utility bounding rectangle function to build this. Once the rectangle has been created, I use the paint event args DrawEllipse function to draw the circle the size of the rectangle.
static private void DrawLine( Point pointA, Point pointB, PaintEventArgs e )
As with the circle, we need to create a .NET Pen object, tell it the mode to use, and draw the line using the event args DrawLine function between the start point and end point.
static public Rectangle BuildBoundingRectangle( Point p )
Builds a .NET rectangle object centered around the x,y values in p. This is done by creating a new Rectangle object. I wanted the bounding rectangle to be 10px by 10px. 10x10 was just an arbitrary value I selected.
RSPoints
RSPoints is a simple wrapper for managing two different possible points. The points represent where a user clicked a given PXCMPhoto being shown in a .NET PictureBox control.
It uses two PXCMPointI32 objects which represent a start point and an end point. In some situations a RSPoints instance will only need the start point, which would be the case for functionality such as RefocusOnPoint. In other cases, two valid points are needed for functionality such as MeasurePoints, which needs both start and end points.
The class operates in two modes: single point mode, meaning we are doing operations that only require one valid point, or multi-point mode, which requires both start and end points to be valid for things like MeasurePoints.
public RSPoints( )
Constructor puts point mode into single, then calls ResetPoints to set all x,y values to 0.
public void AddPoint( int x, int y )
This would be more akin to adding an object to an array or list. However as you can see there is no array or list. Just two points. But, I wanted to give this class a list-type feel from the outside in case I decided to add an array to contain points later. Was this class well thought out? Probably not, but at this point, I’m not too worried about it, since it’s just a supporting class at this time and does what I need it to.
If we are in single point mode, AddPoints will always replace the start point. This is done via the ResetPoints object.
If the mode is single point mode, I don’t necessarily want to add more points, rather just clear out the existing start point and set a new start point.
RSSampleArg
RSSAmpleArg inherits from EventArgs. The intent is that this class be used with RSStreaming. When RSStreaming is streaming, an instance of this class will be created on every new frame and populated with the data from the camera, then sent back to the client via an event class.
public RSSampleArg( PXCMCapture.Sample sample )
The constructor initializes the class’s local PXCMCapture.Sample _sample object with the parameter.
public PXCMCapture.Sample Sample
Simple getter returns the local PXCMCapture.Sample object.
RSEnhancedImageArg
RSEnhancedImageArg inherits from EventArgs. The intent of this class is that it be used with RSEnhancedPhotography. When the RSEnhancedPhotography has completed modifying an image, an instance of RSEnhancedImageArg is created, populated with the newly created image, and sent back to the client by the usage of an Event class that the client subscribes to.
public RSEnhancedImageArg( PXCMImage image )
The constructor initializes the only variable in the class, the PXCMImage object.
public PXCMImage EnhancedImage
A simple getter returns the PXCMImage instance.
RSMeasureArg
RSMeasureArg inherits from EventArgs. This event is used inside the RSEnhancedPhotography classes MeasurePoints function. When the MeasurePoints function has calculated the distance, an instance of this class is used to contain the distance, confidence, and precision data returned. Once this object has been populated, it is used with an Event object to send the data back to the client application.
public RSMeasureArg(float distance, float confidence, float precision)
Parameters
- Float. Measurement distance between two points.
- Float. The level of confidence the SDK has regarding the measurement.
- Float. The precision level the SDK used.
Constructor populates the local data with the input parameters.
public float Distance
Simple getter returns the distance between the two points.
public float Confidence
Simple getter returns the confidence level calculated when the SDK processed the distance between the two points.
public float Precision
Simple getter returns the precision level calculated when the SDK processed the distance between the two points.
FormMain
FormMain is the main form (entry point) for the application. It allows the user to control when to stream, stop streaming, and take a snapshot to be used by the other forms for the various Enhanced Photography functionality.
This form uses the RSStreamingRGB object to stream data from the camera. The data is then rendered to a .NET PictureBox control with the help of an Intel RealSense SDK utility object named D2D1Render.
The form gets its updates from RSStreamingRGB via subscribing to RSStreaming’s OnNewStreamingSample event.
public MainForm( )
This is the forms constructor, so to speak. It initializes the forms global objects, sets up the event handler for the samples as they come in from RSStreamingRGB, and sets the buttons to the proper state.
private void rsRGBStream_NewSample( object sender, RSSampleArg sampleArg )
The event handler for the RSStreamingRGB object. Checks to see if we want to view the color image or RGB image and updates the _render utility object with the proper sample.
It then checks to see whether a new snapshot needs to be taken, and if so, uses the RSUtility to save the photo to disk.
private void btnStream_Click( object sender, EventArgs e )
The event handler for when the Stream button is clicked. Instructs the RSStreamingRGB object to start streaming, and sets the buttons according to the state of the app.
private void btnStopStream_Click( object sender, EventArgs e )
The stop streaming button event handler calls the StopStreaming function.
private void btnTakeDepthPhoto_Click( object sender, EventArgs e )
Event handler for the snapshot button. Sets the flag to take a snapshot to true so that in the streaming event hander, it will know to save the current image data to disk, then instruct the form to stop streaming.
private void EnableEnhancedPhotoButtons( bool enable )
Sets the buttons according to the state of the application via the enable value.
You might look at this function and wonder what’s going on, because you don’t any see multithreading going on. Nothing spun up a thread. So, why am I using multithreading syntax to update the buttons? The reason is that even though the form is not launching any threads, nor is RSStreamingRGB, there IS a separate threading executing.
Inside RSStreaming’s StartStreaming function, you will see a line of code:
_senseManager.StreamFrames( false );
Behind the scene, this line of code (embedded inside the Intel RealSense SDK) is actually spawning a new thread. And because of this, we need to wrap up this functionality in multithreaded syntax.
To start out, I have to check whether the group box surrounding the various enhancement buttons requires an invoke, which is required in dealing with Windows controls in multithreaded applications. If it’s required, then you have to create a new delegate, in this case I’m creating an instance of the EnableEnhancedButtons delegate that was created at the top of the source code file. When creating an instance of a delegate, you must supply the name of the function you want to call, which in this case is the same function EnableEnhancedPhotobuttons. After the delegate has been created, we tell the form to invoke it, sending in the original Boolean value.
When the delegate calls the function, this time the function will not pass the InvokeRequired test and fall into the else statement enabling the group box surrounding the enhancement buttons. The group box will either be enabled or disabled depending on the value in Boolean variable enable.
private void EnableStreamButtons( bool enable )
This works exactly the same as EnableEnhancedPhotoButtons does with the exception of it working with different controls to enable and/or disable.
private void StopStreaming( )
Checks to ensure that the RSStreaming object has been properly initialized, and if so, calls its stop streaming function. This kills the streaming from the camera and closes the thread that was generated.
Then I set the buttons accordingly.
private void btnExit_Click( object sender, EventArgs e )
Click event handler for the exit button. Calls the form Close() function.
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
Handles the forms FormClosing event. Checks to see whether the RSStreaming object was properly initialized, and if so, forces it to stop streaming.
private void btnDepthResize_Click( object sender, EventArgs e )
private void btnDepthEnhance_Click( object sender, EventArgs e )
private void btnRefocus_Click( object sender, EventArgs e )
private void btnPasteOnPlane_Click( object sender, EventArgs e )
private void btnBlending_Click( object sender, EventArgs e )
I grouped all these functions together in one explanation. They all do the exact same thing with the exception that each is initializing a different form. In each I am creating a new session object, which is required by the RSUtility.OpenPhoto function. The OpenPhoto function opens the image that was created and saved to disk. Once the photo has been retrieved, I create a new form object passing it the photo then showing the form. Pretty straightforward stuff.
private void btn_MouseEnter( object sender, EventArgs e )
Event hander when the mouse rolls over either the Start Streaming, Take Photo, or Stop Streaming buttons. I get the button that was rolled over, then look at its “Tag” field. After determining which button was hovered over, I set the toolstrips text value to indicate what each button does.
private void btn_MouseLeave( object sender, EventArgs e )
Sets the toolstrips text back to an empty string.
FormMeasure
FormMeasure demonstrates how to use the Enhanced Photography’s measuring capabilities. It utilizes an RSEnhancedPhotography instance to talk to the Intel RealSense SDK Enhanced Photography functionality.
public FormMeasure( PXCMPhoto photo )
The forms constructor. Accepts a PXCMPhoto object that contains the image to be measured. The constructor initializes any and all global variables included in the class, and registers an OnImageMeasured event handler.
private void InitStatStrip( )
Creates default entries for the status strip. Sets values to empty strings and/or 0,0 for x,y positions.
private void rsEnhanced_OnImageMeasured(object sender, RSMeasureArg e)
The event handler for the OnImageMeasured event on the RSEnhancedPhotography object. Accepts the RSMeasuredArg, which contains the information about the measurement. Updates the status strips text values.
private void pictureBox1_MouseClick( object sender, MouseEventArgs e )
Event handler when a mouse is clicked on the picture box. The event handler starts off by trying to determine how many mouse clicks have been selected. If there is no start point, we know that this is the first time a user has clicked the picturebox/photo. If this is the case then add the click point. If there is a valid start point, then we need to verify the existence of the end point. If it does not exist, add it.
Once we have determined the start and end points, I update the points in the status strip by calling UpdatePointsStatus.
Once we have to valid points; start point, end point, we can then call into the RSEnhancedPhoto’s MeasurePoints(…) function passing in the points object. We don’t need these points anymore so I clear them out by calling the RSPoint object’s ResetPoints() function.
After that, I call the pictureboxe’s invalidate to draw the points on the screen. Keep in mind this function will be called regardless of taking a measurement. This allows us to always see where the picture was clicked on.
private void UpdatePointStatus( )
Updates the status strips two point text values, which shows the x,y on the image where it was clicked.
private void pictureBox1_MouseMove( object sender, MouseEventArgs e )
Mouse move event handler, simply tracks where the mouse is at while moving around the image. Updates the status strips mouse x,y values.
private void pictureBox1_Paint( object sender, PaintEventArgs e )
The picture boxes paint event handler. Responsible for drawing the two points and the line between them onto the image.
Checks to see if we have two valid points and if so, draws both points and the line. If there is only the start point, it only draws the start point marker onto the image.
private void Cleanup( )
Cleans up resources.
private void btnExit_Click( object sender, EventArgs e )
The exit buttons click event handler forces the form to close.
private void FormMeasure_FormClosing( object sender, FormClosingEventArgs e )
The forms closing event handler. Calls cleanup to release resources.
FormDepthEnhance
This form shows the how a depth image can be enhanced and displays the newly enhanced image in the picturebox control. It has the ability to show either low quality or high quality.
public FormDepthEnhance( PXCMPhoto photo )
The forms constructor initializes the variables used by the class sets up the picturebox control with the initial depth image, which defaults to LOW quality.
private void rdo_Click( object sender, System.EventArgs e )
Event handler when one of the two radio buttons is selected. Calls the RSEnhancedPhoto’s DepthEnhancement(…) function, passing in the value indicating high or low quality.
private void _rsEnhanced_OnImageProcessed( object sender, RSEnhancedImageArg e )
The OnImageProcessed event hander when RSEnhancedPhotography sends out the event. Updates the picturebox via the renderer helper object.
private void btnExit_Click( object sender, System.EventArgs e )
Exit button click event handler forces the form to close.
private void Cleanup( )
Nullify the objects to allow for garbage collection.
private void FormDepthScaling_FormClosing( object sender, FormClosingEventArgs e )
Form closing event handler. Forces cleanup of the variables.
FormDepthResize
This form is used to show how a depth image can be resized. In this case, it gets resized to the size of the RGB image from the PXCMPhoto object.
public FormDepthResize( PXCMPhoto photo )
The forms constructor. Just as with the other form constructors, initializes the variables used by the form and sets the OnImageProcessed event handler.
private void btnResize_Click( object sender, System.EventArgs e )
Button resize event handler calls the RSEnahancedPhoto’s DepthResize(…) function.
private void _rsEnhanced_OnImageProcessed( object sender, RSEnhancedImageArg e )
The OnImageProcessed event handler, resizes the picturebox and updates the image displayed via the renderer utility object.
private void btnExit_Click( object sender, System.EventArgs e )
Exit button click event handler. Forces the form to close.
private void Cleanup( )
Nullifies the objects so they can be garbage collected.
private void FormDepthResize_FormClosing( object sender, FormClosingEventArgs e )
Form closing event handler simply ensures that Cleanup is called.
FormDepthRefocus
FormDepthRefocus demonstrates the refocusing capabilities of the Intel RealSense SDK Enhanced Photography functionality. When a user clicks the image, the focal point around that spot is manipulated bringing it into focus while putting everything else out of focus.
Along with clicking a spot on the image to focus on, there is a slider that can be used to simulate changing a camera’s aperture.
public FormDepthRefocus( PXCMPhoto photo )
The forms constructor initializes the variables used by the form. Registers the enhanced photography objects OnImageProcessed event to a function and sets up the picturebox control.
private void pictureBox1_MouseClick( object sender, MouseEventArgs e )
Click event handler for the picturebox. Sets the points objects start point to the cooridnates of the mouse click, calls the RSEnhancedPhotography’s RefocusOnPoint(…). The RSPoints ResPoints() removes the start point values, essentially resetting the object.
private void tbAperture_Scroll( object sender, System.EventArgs e )
The scroll controls scroll event handler. Gets the value of the scroll, sets the forms text value representing the scroll value, and then calls the RSEnhancedPhotography’s RefocusOnPoint this time sending in a second parameter, the aperture size.
private void _rsEnhanced_OnImageProcessed( object sender, RSEnhancedImageArg e )
The event handler for the RSEnhancedObjects OnImageProcessed event. Uses the renderer utility class to update the picturebox control.
private void btnExit_Click( object sender, System.EventArgs e )
Exit button click event handler. Forces the form to close.
private void Cleanup( )
Nullifies the objects so they can be garbage collected.
private void FormDepthResize_FormClosing( object sender, FormClosingEventArgs e )
Form closing event handler simply ensures that Cleanup is called.
FormDepthPasteOnPlane
This form demonstrates how a user can click a flat surface (plane) that is present in the image and force a second image to be rendered onto that surface. The user click two points on a flat surface and if the Intel RealSense SDK functionality can determine the surface, the external image will be pasted onto that flat surface.
public FormDepthPasteOnPlane( PXCMPhoto photo )
The forms constructor initializes the variables used by the form. Registers the enhanced photography objects OnImageProcessed event to a function and sets up the picturebox control.
private void _rsEnhanced_OnImageProcessed( object sender, RSEnhancedImageArg e )
The event handler for the RSEnhancedObjects OnImageProcessed event. Uses the renderer utility class to update the picturebox control.
private void pictureBox1_MouseClick( object sender, MouseEventArgs e )
Event handler when a mouse is clicked on the picture box. The event handler starts by trying to determine how many mouse clicks have been selected. If there is no start point, we know that this is the first time a user has clicked the picturebox/photo. If this is the case, add the click point. After adding the point, I need to invalidate the picturebox control so that the click point can be drawn onto the image.
If there is a valid start point, we need to verify the existence of the end point. If it does not exist, add it.
Once we have the valid points—start point, end point—we can then call into the RSEnhancedPhoto’s MeasurePasteOnPlane(…) function passing in the points object. We don’t need these points anymore so I clear them out by calling the RSPoint object’s ResetPoints() function.
private void btnExit_Click( object sender, System.EventArgs e )
Exit button click event handler. Forces the form to close.
private void Cleanup( )
Nullifies the objects so they can be garbage collected.
private void FormDepthPasteOnPlane_FormClosing( object sender, FormClosingEventArgs e )
Form closing event handler ensures that Cleanup is called.
FormDepthBlending
FormDepthBlending is similar to paste on planes. However the difference is that once an image has been embedded, that image can be manipulated by changing the Yaw, Pitch, Roll, Zoffset, and Scale, which are all indicated by the sliders present on the form.
public FormDepthBlending( PXCMPhoto photo )
The forms constructor initializes the variables used by the form. Registers the enhanced photography objects OnImageProcessed event to a function and sets up the picturebox control.
private void pictureBox1_MouseClick( object sender, MouseEventArgs e )
Event handler for the mouse clicks on the picturebox. Ensures that there is only ever one point in the points object that is the StartPoint. It’s the only point that’s needed by this functionality. I capture the points, and then call the Blend() to do the actual blending of the external image into the PXCMPhoto.
private void Blend( )
Gets the values from each slider, creates the rotation array needed, and then calls the RSEnhancedPhotograpy’s DepthBlend function passing in all the parameters.
private void tbBlend_Scroll( object sender, EventArgs e )
Event handler for any and all of the scrollers. Turns around and calls Blend().
private void _rsEnhanced_OnImageProcessed( object sender, RSEnhancedImageArg e )
The event handler for the RSEnhancedObjects OnImageProcessed event. Simply uses the renderer utility class to update the picturebox control.
private void btnExit_Click( object sender, System.EventArgs e )
Exit button click event handler. Forces the form to close.
private void Cleanup( )
Nullifies the objects so they can be garbage collected.
private void FormDepthBlending_FormClosing( object sender, FormClosingEventArgs e )
Form closing event handler ensures that Cleanup is called.
Conclusion
In this document and sample application, I’ve shown you how to perform some of the enhanced photography functionality that the SDK and camera provide. I hope you’ve enjoyed this article and find it helpful. Thanks for reading.
About the Author
Rick Blacker is a seasoned software engineer who spent many of his years authoring solutions for database driven applications. Rick has recently moved to the Intel RealSense technology team and helps users understand the technology.