A mouse control application for Kinect v2, including a couple of options for various uses.
I’m so glad that I’ve just become a Microsoft Kinect MVP since this July, so I think I need to do something more.
One request asked most since the original Kinect came out is, “can I use my hand to control the mouse cursor?” There you can find a few applications for doing this with original Kinect on the web.
These days, some people still asked me about this kind of application for Kinect v2. Then I strangely found there’s no such downloadable applications on the web. I gotta release a Kinect v2 version app. And here’s it, with source code!
I initially develop this for Francis Kim(He’s a Magento Developer). Thanks Francis for letting me release this freely!
Options:
- Mouse Sensitivity
- Pause-To-Click Time Required
- Pause Movement Thresold
- Cursor Smoothing
- Grip Gesture
Grip to drag/click - Pause To Click
Hold your hand and don’t move for a little while to click. - No clicks, move cursor only
The time you hold your hand for as a click
How large the circle range you hold your hand inside for a little while would be a click
The large it is, the smoother the cursor would move, more slowly as well.
Github: KinectV2MouseControl
EXE files: [download id=”24″]
Hi,
When I run this the light on the Kinect lights but I don’t seem to be able to control the PC cursor. Help?
Hi, I just tested again with the app and it worked. It should also work for you.
Here’re 2 reasons I guess it may cause the problem:
You can also run Body Basics-WPF the sample application in SDK Browser at the same time. If it’s shows the skeleton view, your body is detected.
Hi can you add distance control cause i must be near of the kinect for the cursor to work well, also add infrared detection to use it in dark room , thank you for your great work ^^
hi, are you planing to make update yes or no ?
Hi ! Thanks for sharing this great piece of software, its the best one I’ve came across for the Kinect V2, the only thing is the mouse seems to move a lot when you try to click on anything using the grab gesture.
Would it be possible to set it so you move the mouse with your right hand and then click with the left ?
Thanks for your time.
Henry P.
Thank you very much!
It’s amazing!
Is there some way to limit range of hand recognition and improve user switching(2 or 3 persons in front of the camera)?
HI,
I have downloaded you code. It working with Drag n Drop of items, Cursor Movement.
But its not working with the OSK (On screen keyboard).
Can you please let me know that why its not working with On Screen Keyboard.
Reply Soon.
hello my name is hasaky.
I want to learn how to coding with kinect v2.
can u suggest some website or material to let me learn kinect v2?
plz
🙂
Hi Hasaky. I’d suggest:
Vangos Pterneas’s blog
Kinect 2 Hands On Labs
From my experiences, I found Vangos Pterneas’s blog and download the source file for body control with Kinect v1, and started from that. Kinect v2 is similar to v1 on most parts. Knowing how to read and do with Body, depth and color data, you can do a lot things. Good luck and enjoy!
hai how to control double click and Drag ?????
I download this and working and i need to double click and drag n drop can pls tell me how to ?
hi!! I want to create a program where the pointer follows the hand. Can I tell if I am using processing3 and what function is following it?
no using kinect v2.
Hi kick, I don’t have experiences with processing on that and I don’t know what functions it has. But I’d share a basic idea, to achieve this, usually you need to find functions to get your body joints positions, which includes your hand positions, and shoulder positions used as what I did for increasing stabilities. And map those positions into expected onscreen pointer positions. (They’re in different dimensions, x-axis of your hand position moving 5 units can possibly be expected to have 2000 pixels movement on screen! Then you just multiply the x value, and maybe add a offset value if needs, and so on.. Based on how you want it to move.)
I have a question, how do you use it? do I need a special program? or just connect the kineckt to the computer and run the exe ? thanks!
Yeah just connect the Kinect(while your PC can already run other regular Kinect programs) and run KinectV2MouseControl.exe 🙂
Sorry! TangoChen ~ I ask the question, I downloaded Kinect v2 MouseControl.exe, but MouseControl.exe no action! ? I use win7 x64 version, kinect already installed driver. Also, install Visual Studioc 2017. Thank you!
Hi Steven, would it work to run other Kinect samples? (Basic demos in SDK browser)
Just to make sure your Kinect installation and setup is fine for any Kinect apps.
TangoChen ~ Right! I downloaded [KinectV2MouseControl-master] and I saw it using [C # Code]. Whether [C # code] to compile, you can do it! ? Thank you (p.s: do you have email?)
Yes it is in C#. You will need to add Microsoft.Kinect.dll into References of the VS project.
My email is tan9ochen@gmail.com
Pingback: New Kinect Interaction – Angel Wu – Week 02 PPJ – Coalescence
Hi,
are you still working on this project or will this be the last version of the app? Just asking because i couldnt find any useful Alternative and Microsoft more or less dumped the kinect and i still have some problems with the usage e.g. Compare the comment of Henry. So you are my last hope for this. In case you will not further work on this i have to drop my use for it aswell or is anybody else here who is still working on this?
Hi buddy, the jittering appears when gripping also causes tracked position moves, which can’t be simply gotten rid of using only the simple body tracking outcome. (May be able to find a workaround somehow. I’m not trying yet. Or may never do.)
As for two-hand controlling(one hand for moving another for clicking) as Henry mentioned. I personally don’t think this is an intuitive, or good-looking way of controlling the cursor. But it does solve the jittering to a certain degree comparing to current function. And if people are okay with this gesture for their scenarios, it’s quite fine.
Am I still working on it or will it be the last version?: I would try to do my stuff mostly when I want. So I rarely determine something is forever done or yet to be continued. But would be a matter of time and arrangement.
I’m going to add the two-hand controlling option. While I’m procrastinating with other things, too. May take around a week or two to have this update..
Hi,
nice to hear, that there is still afford for improvements. Anyway, i did not want to trouble you with this. I agree that a two hand solution is not very intuitive, but for the moment it is probably the easiest way. Maybe some other readers take on off this project any time. It is pretty sad ms dropped the kinect instead of take it as a basis for future development. Nevertheless, i just read ms will introduce a wider range of controlling options for handycaped people with the next os update. We will see, maybe the kinect will have a second awakening.
I agree. I would had been looking forward to a Kinect 3, 4 and more if they didn’t discontinue it. As Kinect is still one of the most practically usable and useful tech devices compared to many gimmicks nowadays. And it still can have a lot of potentials with difference improvements.
HoloLens is partially another form of Kinect but in a more specific scenario. Any enhanced forms of Kinect, or something that may lead the potentials would be nice. 🙂
Hi,
FYI, the KinectV2MouseControl was just updated with two-hand controls.
Link: https://github.com/TangoChen/KinectV2MouseControl
Hi,
just tested the last version and its great. Works very fluently and intuitiv, i have to commit it firts looks a bit strange but because of the prompt recognicen you only have to lift the hand just very short and with a bit of usage you get on it very fast. Thank you very much therefore. It is a great improvement. Now when you support two hand modus would it be difficult to combine two hand with two gestures. Eg. Move and lift for clicking and grip or press for doubleclick. So one hand for moving is the fixed command and then you can combine the command with a second hand alternatively either lift for click or grip/ press for doubleclick?
By using the kinect a bit more intensively there still arise more and more points where ms had to improve its architecture for a future usage. There is e.g one point i have not think about before that could be a problem. I use the kinect on 32 inch touchscreen as a kind of infowall, the problem now is that because of the touchscreen i have no native mouse controll and because of that no constant cursor icon. When i now want to controll the the os with the kinect i first have to plug in a mouse to see where to move. This is e.g. an example for how many scenarious ms has to think of by improving the usage of the kinect. I still hope they will recognice one day what kind of future control option they have on the kinect.
Hi,
As to combining gestures, it’s doable. But currently I’m not planning to have some major updates.
You’re right that people may need a bit of using on lift-clicking before getting on it. There were some intends to make the gesture as “lifting the hand just very short”. At first I was making the lifting to a higher distance over the head, which is what most people would potentially do to “lift their hand”. But in the meanwhile “lift clicking” is the only major gesture besides moving the cursor. It could be frequently used.
So I think there’re two advantages to make the triggering height lower, just around the neck.
1. People can put their hand lower or just put it down before doing a click, it feels less strength consuming, especially when clicking is frequently needed.
2. The approximate triggering height is clearer for users after experiencing. Though the height is not same for everyone. It could be around the height of your neck. Or around the height of your mouth. But then you kind of know where it is. If it’s above your head, you may know or feel less on the height, and may have to think of putting up your hand to the top. It works but also a bigger strength consumption.
These may be very little things though. I like to have some sort of long term convenience. There are also advantages to make it higher above the head. e.g. For short term usages by constant new people. No worry, there’s a HandLiftYForPress property for adjusting the height in KinectCursor.cs for developers.
Hi,
We are using your application for testing purposes. Our application is very straight forward. When hovering a button in windows 10, it has to accept a “click” event. All goes well with one person in the detection area. We are looking for a solution where only one person (eg the first person that is detected) can move the cursor without other people in the detection area influence the working of the app. An application is not fool proof when it is not accurate enough. Now the cursor becomes very unstable and the application crashes (the connection with the Kinect interrupts). We are wondering if you can help us with this? I tried to launch a project on freelancer.com, but all programmers want to use Untiy, but we are not asking for a game, but normal windows click commands.
Hi Wim, I’ve just made an update to make it lock the control with first tracked person. It’s actually not tested with more than one person at the moment but I’d still commit it now. Please give it a try and feel free to leave me some feedback.
Link: https://github.com/TangoChen/KinectV2MouseControl
Hi, I tried to compile your source but i got this error “Error 1 Invalid token ‘=’ in class, struct, or interface member declaration”
please advice how to fix this error.
it points to line 86 column 48…
public double HoverRange { get; set; } = 20;
Hi Sandro, it seems that you are using an old version of Visual Studio(with old C# version). As C# before 6.0 doesn’t support for auto-property initializer, which let you directly initialize the property with an assignment operator(= 20 in this case).
To fix this for you, either try using a newer version of Visual Studio, e.g. VS 2015, 2017.
Or with a little adjustment on the code. You can change line 86 to:
private double _hoverRange = 20;public double HoverRange {
get
{
return _hoverRange;
}
set
{
_hoverRange = value;
}
}
or
// Remove "= 20;" in the end.public double HoverRange { get; set; }
// and insert this at line 119 for KinectCursor constructor:
HoverRange = 20;
Thank you TangoChen. Now I can build and execute without error.
but, how can I use this app? when I try to run, the mouse control window will appear but I can’t control the mouse
Hi Sandro, is the Control Mode on the app set to Disabled? If yes then you need to select other mode options to activate the control.
any option in the control mode doesn’t work. what I know is that, it is not connecting my Kinect because the red light in my kinect is not turning on.
the other kinect programs I downloaded and run is working but the interface is not good.
Hi, is there a Microsoft.Kinect.dll is under the app folder? If not, try copying one from Kinect SDK folder to there. You can also use executables/KinectV2MouseControl_EXE.zip from the source.
Hi TangoChen,
It was really awesome project thanks for sharing it with public. I have few issues. I want to enable the right mouse click does that function is already available in the source or do I need to write it from the scratch?
Thank you again for sharing the source code. Please help me to enable the right mouse button click issue……….
Hi Akila. To have a Right click control. You can add this in MouseControl.cs:
public static void RightClick()
{
mouse_event(MouseEventFlag.RightDown | MouseEventFlag.RightUp, 0, 0, 0, UIntPtr.Zero);
}
And then you can do a right click with MouseControl.RightClick(). For example, you can add it in KinectCursor.cs according to your needs or simply replace MouseControl.LeftClick() with it and you’ll see gestures for left clicks are changed to right clicks.
Hi,
Thanks a lot it worked perfectly
hi thanks for your great effort , but any chance of supporting kinect 360 as its more available than kinect 2 if not please name applications that do that for windows 10 even if paid , also i would donate for your app if its support kinect 360 .
wish all the best .
Hi,
Thanks a lot this amazing program.
Can I use my ankle joint to control the mouse cursor?
I want to change jointPoint.y(hand) to jointPoint.z(ankle) for controlling the mouse cursor with my foot.
Thank you.
To do foot control instead of hand control, you can change line 28 of KinectBodyHelper.cs
CameraSpacePoint handPos = body.Joints[isLeft ? JointType.HandLeft : JointType.HandRight].Position;
to
CameraSpacePoint handPos = body.Joints[isLeft ? JointType.AnkleLeft : JointType.AnkleRight].Position;
And similar to changing y coordinate to z coordinate for hands(of your later question, sorry for answering in a reverse order). You change the coordinate for ankles. This is the overall idea. You may need to adjust some values or methods to make it fit better for ankle movements.
Hi, Thank you for this program.
How to move the mouse pointer with hand using the x, z coordinates not x, y?
Thank you.
Hi Ben, this can be done by changing the y coordinate value it used for cursor position calculation to z coordinate value, like what you mean in words. But please notice it should also be a relative z coordinate value to your body, so that where you stand won’t affect the control. (Unless you do want this kind of absolute position controlling.)
Here’s an untested code modification as an example: try changing the code in line 32 of KinectBodyHelper.cs to:
MVector2 usedHandPos = new MVector2(handPos.X, handPos.Z);MVector2 usedSpineBase = new MVector2(spineBase.X, spineBase.Z);
return usedHandPos - usedSpineBase;
The original code has gestureOffsets[isLeft ? 0 : 1] because the results are a bit off from the desired position for hands(also different for left and right hands). So I add gestureOffsets to move it back to a better result. It could also be off in this case. Then you’ll still need gestureOffsets[isLeft ? 0 : 1]; But values for gestureOffsets should be adjusted to fit it properly. (Can try different values and find out preferred ones.)
Additionally, the original method IsHandLiftForward for doing gesture can be changed to IsHandLiftUpward with its corresponding logics. In case that the cursor isn’t controlled when you move your hand too close to your body.
Hey man
wondering if this can be used with 2 people, if i have 2 mouse cursors onscreen using another program, where each person controls 1 mouse.
Or would this just default to both users fighting over the same original mouse
very specific example i know, for a uni project
Thank you for sharing your software!
do you know if it is more precise than magic cursor app for kinect 360?
Thank you so much
Hi TangoChen,
This is a perfect project thanks for sharing it. I have a question that how can I use right hand to have a Left click and left hand to have a Right click?
Thank you very much if you solve my problem!
Hi TangoChen,
This is a very wonderful project thanks for sharing it. I have a question that how can I use right hand to have a Left Click and left hand to have a Right Click?
Thanks for your help!
Hi, Thank you for this program.
I have a question that how can I use right hand to do Left Click and left hand to do
Right Click.
Thanks a lot!