How to control Get3DShape() in Microsoft Kinect face tracking SDK?

Question

I'm working on a lip-reading project and I'm using Microsoft Kinect face tracking SDK in order to extract lip features in a generated text file from C# to be processed in MATLAB. The features are 29 angles resulting after calculating the distances between
the 18 feature points related to the lips using Get3DShape() function to get the co-ordinates in each frame and my code looks like that which I added in facetrackingviewer.cs:
public void getangles(FaceTrackFrame fr)
var shape = fr.Get3DShape();
var X_7 = shape[FeaturePoint.MiddleTopDipUpperLip].X;
var Y_7 = shape[FeaturePoint.MiddleTopDipUpperLip].Y;
var Z_7 = shape[FeaturePoint.MiddleTopDipUpperLip].Z;
var X_31 = shape[FeaturePoint.OutsideRightCornerMouth].X;
var Y_31 = shape[FeaturePoint.OutsideRightCornerMouth].Y;
var Z_31 = shape[FeaturePoint.OutsideRightCornerMouth].Z;
var X_33 = shape[FeaturePoint.RightTopDipUpperLip].X;
var Y_33 = shape[FeaturePoint.RightTopDipUpperLip].Y;
var Z_33 = shape[FeaturePoint.RightTopDipUpperLip].Z;
var X_40 = shape[FeaturePoint.MiddleTopLowerLip].X;
var Y_40 = shape[FeaturePoint.MiddleTopLowerLip].Y;
var Z_40 = shape[FeaturePoint.MiddleTopLowerLip].Z;
var X_41 = shape[FeaturePoint.MiddleBottomLowerLip].X;
var Y_41 = shape[FeaturePoint.MiddleBottomLowerLip].Y;
var Z_41 = shape[FeaturePoint.MiddleBottomLowerLip].Z;
var X_64 = shape[FeaturePoint.OutsideLeftCornerMouth].X;
var Y_64 = shape[FeaturePoint.OutsideLeftCornerMouth].Y;
var Z_64 = shape[FeaturePoint.OutsideLeftCornerMouth].Z;
var X_66 = shape[FeaturePoint.LeftTopDipUpperLip].X;
var Y_66 = shape[FeaturePoint.LeftTopDipUpperLip].Y;
var Z_66 = shape[FeaturePoint.LeftTopDipUpperLip].Z;
//some stuff
float[] point1 = new float[3] { X_7, Y_7, Z_7 };
float[] point2 = new float[3] { X_64, Y_64, Z_64 };
float[] point3 = new float[3] { X_41, Y_41, Z_41 };
float[] point4 = new float[3] { X_80, Y_80, Z_80 };
float[] point5 = new float[3] { X_88, Y_88, Z_88 };
float[] point6 = new float[3] { X_31, Y_31, Z_31 };
float[] point7 = new float[3] { X_87, Y_87, Z_87 };
float[] point8 = new float[3] { X_40, Y_40, Z_40 };
float[] point9 = new float[3] { X_89, Y_89, Z_89 };
float[] point10 = new float[3] { X_86, Y_86, Z_86 };
float[] point11 = new float[3] { X_85, Y_85, Z_85 };
float[] point12 = new float[3] { X_82, Y_82, Z_82 };
float[] point13 = new float[3] { X_84, Y_84, Z_84 };
float[] point14 = new float[3] { X_66, Y_66, Z_66 };
float deltax = point2[0] - point1[0];
float deltay = point2[1] - point1[1];
float deltaz = point2[2] - point1[2];
float cetax = point3[0] - point2[0];
float cetay = point3[1] - point2[1];
float cetaz = point3[2] - point2[2];
float notex = point3[0] - point1[0];
float notey = point3[1] - point1[1];
float notez = point3[2] - point1[2];
float deltax1 = point5[0] - point4[0];
float deltay1 = point5[1] - point4[1];
float deltaz1 = point5[2] - point4[2];
float deltax2 = point5[0] - point3[0];
float deltay2 = point5[1] - point3[1];
float deltaz2 = point5[2] - point3[2];
//some stuff
float distance = (float)
Math.Sqrt((deltax * deltax) + (deltay * deltay) + (deltaz * deltaz));
float distance1 = (float)
Math.Sqrt((cetax * cetax) + (cetay * cetay) + (cetaz * cetaz));
float distance2 = (float)
Math.Sqrt((notex * notex) + (notey * notey) + (notez * notez));
float distance3 = (float)
Math.Sqrt((deltax1 * deltax1) + (deltay1 * deltay1) + (deltaz1 * deltaz1));
float distance4 = (float)
Math.Sqrt((deltax2 * deltax2) + (deltay2 * deltay2) + (deltaz2 * deltaz2));
float distance5 = (float)
Math.Sqrt((deltax3 * deltax3) + (deltay3 * deltay3) + (deltaz3 * deltaz3));
float distance6 = (float)
Math.Sqrt((deltax4 * deltax4) + (deltay4 * deltay4) + (deltaz4 * deltaz4));
float distance7 = (float)
Math.Sqrt((deltax5 * deltax5) + (deltay5 * deltay5) + (deltaz5 * deltaz5));
float distance9 = (float)
Math.Sqrt((deltax6 * deltax6) + (deltay6 * deltay6) + (deltaz6 * deltaz6));
float distance8 = (float)
Math.Sqrt((deltax7 * deltax7) + (deltay7 * deltay7) + (deltaz7 * deltaz7));
//some stuff
double feature = CalcAngleB(distance, distance2, distance1);
double feature2 = CalcAngleB(distance2, distance, distance);
double feature3 = CalcAngleB(distance, distance1, distance2);
double feature4 = CalcAngleB(distance3, distance5, distance4);
double feature5 = CalcAngleB(distance5, distance4, distance3);
double feature6 = CalcAngleB(distance4, distance3, distance5);
double feature7 = CalcAngleB(distance6, distance, distance7);
double feature8 = CalcAngleB(distance10, distance8, distance9);
double feature9 = CalcAngleB(distance9, distance10, distance8);
double feature10 = CalcAngleB(distance8, distance9, distance10);
//some stuff
using (System.IO.StreamWriter file = new System.IO.StreamWriter(@"D:\angles.txt", true))
file.WriteLine("{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13},{14},{15},{16},{17},{18},{19},{20},{21},{22},{23},{24},{25},{26},{27},{28}", feature, feature2, feature3, feature4, feature5, feature6, feature7, feature8, feature9, feature10, feature11, feature12, feature13, feature14, feature15, feature16, feature17, feature18, feature19, feature20, feature21, feature22, feature23, feature24, feature25, feature26, feature27, feature28, feature29);
Where CalculateAngleB is a declared function above:
public double CalcAngleB(double a, double b, double c)
return Math.Acos((a * a + c * c - b * b) / (2 * a * c));
and I call getangles(fr) at :
internal void OnFrameReady(KinectSensor kinectSensor, ColorImageFormat colorImageFormat, byte[] colorImage, DepthImageFormat depthImageFormat, short[] depthImage, Skeleton skeletonOfInterest)
this.skeletonTrackingState = skeletonOfInterest.TrackingState;
if (this.skeletonTrackingState != SkeletonTrackingState.Tracked)
// nothing to do with an untracked skeleton.
return;
if (this.faceTracker == null)
try
this.faceTracker = new FaceTracker(kinectSensor);
catch (InvalidOperationException)
// During some shutdown scenarios the FaceTracker
// is unable to be instantiated. Catch that exception
// and don't track a face.
Debug.WriteLine("AllFramesReady - creating a new FaceTracker threw an InvalidOperationException");
this.faceTracker = null;
if (this.faceTracker != null)
FaceTrackFrame frame = this.faceTracker.Track(
colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);
getangles(frame);
this.lastFaceTrackSucceeded = frame.TrackSuccessful;
if (this.lastFaceTrackSucceeded)
if (faceTriangles == null)
// only need to get this once. It doesn't change.
faceTriangles = frame.GetTriangles();
//MessageBox.Show("face has been detected");
this.facePoints = frame.GetProjected3DShape();
//this.FaceRect = new System.Windows.Rect(frame.FaceRect.Left, frame.FaceRect.Top, frame.FaceRect.Width, frame.FaceRect.Height);
Now, whenever the mask appears as if (this.facetracker!=null), the code starts to calculate the angles and generate the text file. My question is how can I add a control (e.g.Timer or button) to decide exactly when to start calculating features or generating
the file because in data collection I need features to be recorded accurately when the speaker starts to pronounce phrases not when the mask appears. I have tried to add a dispatcherTimer as when the timer counts after the mask appears it starts to calculate
the angles like this:
private static void dispatcherTimer_Tick(object sender, EventArgs e)
Microsoft.Kinect.Toolkit.FaceTracking.FaceTrackFrame fr;
fr = facetracker.Track(sensor.ColorStream.Format, colorPixelData,
sensor.DepthStream.Format, depthPixelData,skeleton);
//getangles(fr) and stream writter
and added this to getangles(fr)
System.Windows.Threading.DispatcherTimer dispatcherTimer = new System.Windows.Threading.DispatcherTimer();
dispatcherTimer.Tick += dispatcherTimer_Tick;
dispatcherTimer.Interval = new TimeSpan(0, 0, 10);
dispatcherTimer.Start();
but this gives an error in: var shape = frame.Get3DShape(); as using unassigned variable frame and I tried to declare facetracker in timer's void but this gives me an error too so, if you have an I idea how can I add a timer or a button to control when the
angles to be calculated I would be grateful.
Thanks in advance.

Answer 1

Hi Mesbah15,
I am not sure if I’ve understood the question correctly. You can try using event to communicate with other classes. Define an event, when mask appears you just need call that event handler.
https://msdn.microsoft.com/en-us/library/17sde2xt(v=VS.100).aspx.
Please post more information about your scenario for better understanding if event handler not help.
Regards,
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.