• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

P3D v2 Hand in Cockpit - Leap Motion

Michael Garbers

Michael Garbers

Messages
30
Country
germany
In order to make my virtual reality cockpit with 3D glasses, track IR and leap motions I am now at the point to display a virtual hand inside the 3d cockpit of an aircraft. Possibly you have some ideas to throw in, how to do it best?
Easiest way would be to design it inside the model. This is ok and no problem to do. But I like to have it in a more generic way. I can think of attaching a simobject to a plane. I am not sure how to do it right now.
Is there something I can use in the aircraft.cfg? Or can I use an effect which uses a simobject? Or can I use simconnect interface? I can think of the mission engine for example. But possible the hand would not stick to the aircraft position during flight.

The hand must move with the plane. An external interface which transfer the data from the leap motion to move the hand inside the cockpit. This will be done by simconnect.

I really would like to hear your ideas. :)

Current status without Leap-Motion:
https://www.facebook.com/video.php?v=1485107651703757
 
Although I am not experienced in programming, I do have some ideas that might work. I think using aircraft.cfg and simconnect are the best way to go. To make the cockpit switches interactive, try to find the script that allows you to click the switches in the virtual cockpit and use it for your program. I think you will learn a lot more from the experts like Arno. I hope this helps. Other than that, I think a virtual reality program is a tremendous idea. I am truly looking forward to its release.
 
Thank you for your response. I like to collect some ideas how to solve the problem "displaying and moving a virtual generic Hand (if everything runs fine also with fingers) inside a virtual cockpit".

The whole setup would be that a "FS Virtual Hand" program retrives input from the leap motion device and translate the position and gestures into actions through simconnect and a "FS Virtual Hand" cockpit specific action-map. These actions will be posted to the sim through simconnect. All of this is already testet and is working. The missing part where I have no final solution yet is, how to display the virtual hand inside the cockpit.

My first idea was to let simconnect add a simobject "Hand" into the scenery and move it with the plane. I just don't know that if I move very fast the hand will work reliable, I would need to check this but I guess I will get some problems.
A better option would be to "attach" the hand to the aircraft somehow. At the moment I have no clue how to do this, I need to inspect the SDK for this. After attaching the hand I need to move it in relation to the virtual cockpit.
Since P3D can attach missles to the plane and move it independantly, this can be also a way to do it.
 
There are some videos on YouTube telling you how to animate a pilot controlling the plane from the exterior view. The pilot's movements are dependent on the input made by the player via the joystick. An ideal approach would be to modify the .mdl of the aircraft cockpit and add a hand to that model. You could assign input movements to the hand in the same manner the virtual yoke and throttle are assigned input movements by the external hardware.
 
This is P3D exclusive, right? PDK offers extra possibilities beyond those of the past FSX SDK.
 
Well - I'm curious how successful you'll be - my fingers are crossed ;-)
I do not want to talk about the showing hand in cockpit but about the LeapMotion itself.
I had very similar idea and so I tested Leap Motion two weeks ago. Unfortunately I had to refuse this idea.

The main issue was the device can perfectly recognize my hand while it is over it, but when you move bit more far from the plane perpendicular to the device, then the recognition becomes really inaccurate and fuzzy. In specification is told the device see your hand in -60 to 60 degrees and roughly 2 feets from the device, what is true.
But when you get "outside" of that perpendicular plane the results are poor.

Maybe my problem was that I wanted to scan my hand in significant part of the cockpit, not just close to one part of particular panel (what would probably work perfect).
I was also considering whether Kinect would not be more suitable in my case. I hope you was able to find more clever idea how to use the leap motion.

Just my two penny for the hand visualization - I'm convinced it should be part of a cockpit as it is definitely possible to calculate the absolute position of the hand (in the WGS-84) and it will be also affordable for HW performance but I think there is a problem of positioning simobject such a way it does not flicker.
If I'm not wrong this problem was the main issue when guys tried to implement AI planes flying in group with user aircraft.
 
Thank you for the response!

@flyboyaces: This is definitely a solution. Only drawback is, that you can use it only with your own plane and it would not be a generic approach.

@Peter: I do it for P3D. If the solution works in FSX as well I am fine, but it does not have to.

@premek: I known what you mean. It depends a bit on positioning the LeapMotion controller. If the whole cockpit is not working I can imagin two or three areas controlled by different LM controller. But first it has to work in general.
The flickering is the problem I expect too, if you try to use WGS/84 positioning. This is why I am talking of "attaching" the hand object to the plane. I need some time to dive into the posibillities of P3D.
 
I think I found something usefull inside simconnect:
SimConnect_AttachSimObjectToSimObject
Here it should be possible to attach one sim object to the user sim object.
The interesting thing to known would be if the attached object is also reacting on user input.
 
Ok, AttachSimObjectToSimObject is working but the problem is, that you cannot move it or you cannot move anything with control axes or send commands to it.
But I have a different solution. I add a sub-view. Here I can control perfectly what I like to see. So in my case the window open as soon as a hand is recognized and closed after removing the hand.
The movement of the hand/finger is moving the view around the cockpit. This is already working and independent of the movement introduced by my head and trackIR.
Of cause, the view is not stable yet, simply because of the zoom factor. So the next step is to define all the spots to click on and move the view from one spot to the next instead of constantly moving the view.
In this way you don't need to hit the spot exactly. An additional question: How to click. Tracking the finger or simply wait for a second if the focus is still at the click-spot? I will see what will work better.

One question into the round: If I open the view the sim displays view informations in red like zoom factor and type of view. Can I remove that? I know there are some switches in the p3d.cfg but the best thing would be to remove it just for my view.

Best,
Micha
 
I have a proof of concept video online. Nothing great but it shows a bit how my solution will work.
First I thought I show a startup but it is more important to see that it is also working while flying.
It still needs some tuning but it works this way:

- If Leap Motion detects the finger it shows the small detailed window.
- For the detection area zones are defined. If the finger is inside a zone the detailed window moves to a defined position inside the P3D.
- If the finger stays inside the zone for a defined time my program sends the related event to the sim by simconnect.
- Turning buttons are working as well, it just fires more clicks.
- If a click spot inside the cockpit has no related event in simconnect I just move the mouse to the center of the detailed window and to a click. I was surprised, this method is really working as well.

There is still some tuning to do and I need to make my program more configurable, but I am on a good way.
Most important is to define good zones for the leap motion controller. It should be big enough and not too much at the border. Currently I split the whole detection area into four major areas. Overhead panel, front panel and left/right panels. If you have a more complicated cockpit I guess more than one controller should be used.

 
Your basic idea is really impressive. And yes, you can use an AI aircraft model to simulate a hand or something similar. The challenging part is to get it under control and to display it precisely. Positioning and axis control is possible . You have to take over the control via a SimConnect command. After taking over control you can deliver Yaw, Pitch, Bank and X,Y,Z position. But there is a limitation in the position resolution. You can see this, if try to follow an AI aircraft in a formation. It jumps. But this could probably be solved if you position your AI object after each frame. Another question would be, how the collision detection reacts if you place the "AI model" directly in the ownship cockpit. If you get this to run, it would be a huge step forward when using Oculus Rift. Good luck.
 
Something else to consider: you don't necessarily need a 3D hand in the cockpit. You could accomplish the same goal with a 2D overlay that shows the position (perhaps you could use the camera to move the mouse pointer) of the user's hand.
 
@TOWSIM: Currently my way is to display a 3-D Cockpit View as shown in the second video. This I can display or hide and move the view-point.
@UnitedExpress4180: Nearly. I need a 3D-Cockpit since there are no 2D-Instruments on a panel to display.

Thank you for input.
 
Hi everybody!
I am also working on a prototype for integrate 3D hands in P3D v2 using the Leap Motion.
What solution did you finally use? Do you add the hands as simObjects? or do you attach them to the aircraft? What SDK do you use?

Thank you for your help
Jean
 
My first solution is working with views and is working quiet nice. I am using simconnect to create a small window to show the positioning. After holding a defined area for a fraction of a second I trigger the Event through simconnect as well.

In this way I can configure the different cockpits and do not need a visible hand inside the cockpit.
 
Ok, thank's for your answer.
My objective is to integrate 3d visible hands inside the cockpit. What do you think would be the best way?

Jean
 
You must log in or register to reply here.
Back
Top