Scott Guthrie has blogged about a nice little how-to twitter silverlight application that Celso Gomes and Peter Blois created. There is a full 10 minute video and source code for this neat application. The blog post shows you detailed steps on how to create the application too!
Twitter away!
Virtual Earth is the 3D interface to Microsoft's Live Maps service. Normally this control is loaded via the web browser and allows interaction with a keyboard, mouse, and Xbox 360 controller. In this article, we will take the Virtual Earth 3D control out of the web browser, use it in a WinForms application, and control it with a Nintendo Wii Remote (Wiimote) and a pair of Vuzix VR920 glasses, while also providing a stereoscopic 3D image to the glasses, creating the illusion of a fully three dimensional environment. Note that use of the Virtual Earth 3D control in this way is undocumented and unsupported at the moment. Because of this, some of the descriptions in this article are educated guesses and may not be 100% accurate…
Originally, this project started as a simple Wiimote interface to Virtual Earth 3D as shown in the video below. Since I wrote that application, I learned of the VR920 glasses and the Wii Fit Balance Board was released, so I’ve decided to create a more immersive experience using all of these controls which was demonstrated at PDC2008.
Before we get started, you will need to install the Virtual Earth 3D control. If you haven't done this already, browse to http://maps.live.com/ and click on the 3D link to install the control and supporting software.
Additionally, if you haven't already, please review my Managed Library for Nintendo's Wiimote article on this site. We will be using the library in this article, but I will not repeat the basic information that is located in the original article. You will also need to have the Vuzix VR920 glasses installed and setup according to its own user manual. That will also not be covered here.
The Virtual Earth 3D (VE3D) control is intended to be used through a well documented JavaScript interface from a web page, however we would not be able to access the Wiimote or VR920 glasses from JavaScript. Therefore, we will be using the VE3D control through its native, but wholly undocumented interface.
Start by creating a new Windows Forms application named WiiEarthVR in C# or VB. As with all controls and 3rd party libraries, a reference needs to be set to the Virtual Earth 3D libraries. Add references to the following items:
If they do not show up in the .NET references, they can be found by selecting the Browse tab and navigating to C:\Program Files\Virtual Earth 3D\ or C:\Program Files (x86)\Virtual Earth 3D\ . With the references in place, the project file can now be opened and the references will be seen in the References folder in the Solution Explorer as usual.
Creating an instance of the control can be done in code just like any other control. Used in the constructor or load event of the form, the following code will create a VE3D control and add it to the form as fully docked:
C#
private GlobeControl _globeControl;
public MainForm()
{
InitializeComponent();
_globeControl = new GlobeControl();
SuspendLayout();
_globeControl.Location = new System.Drawing.Point(0, 0);
_globeControl.Name = "_globeControl";
_globeControl.Size = ClientSize;
_globeControl.Anchor = AnchorStyles.Left | AnchorStyles.Right | AnchorStyles.Top | AnchorStyles.Bottom;
_globeControl.TabIndex = 0;
_globeControl.SendToBack(); // we want the button to be on top
pnlGlobe.Controls.Add(_globeControl);
ResumeLayout(false);
}
VB
Private _globeControl As GlobeControl
Public Sub New()
InitializeComponent()
_globeControl = New GlobeControl()
SuspendLayout()
_globeControl.Location = New System.Drawing.Point(0, 0)
_globeControl.Name = "_globeControl"
_globeControl.Size = ClientSize
_globeControl.Anchor = AnchorStyles.Left Or AnchorStyles.Right Or AnchorStyles.Top Or AnchorStyles.Bottom
_globeControl.TabIndex = 0
_globeControl.SendToBack() ' we want the button to be on top
pnlGlobe.Controls.Add(_globeControl)
ResumeLayout(False)
End Sub
This sets up the VE3D control in its default state. If you were to run an application with only this code, you would see nothing but the earth. The navigation controls and other extras would be missing.
We can start adding items to the VE3D control by listening for the FirstFrameRendered event of the GlobeControl and then setting the appropriate properties. Setting these properties prior to this point can lead to some unexpected results.
In the FirstFrameRendered event handler, if you wish to add the default navigation controls to the screen, the PlugInLoader object is used. The PlugInLoader is created by using the CreateLoader static method, passing in an instance of the GlobeControl's Host object. Then, the NavigationPlugIn can be loaded and activated as shown:
// load all the spiffy UI navigation goodies
PlugInLoader loader = PlugInLoader.CreateLoader(this.globeControl.Host);
loader.LoadPlugIn(typeof(NavigationPlugIn));
loader.ActivatePlugIn(typeof(NavigationPlugIn).GUID, null);
' load all the spiffy UI navigation goodies
Dim loader As PlugInLoader = PlugInLoader.CreateLoader(Me.globeControl.Host)
loader.LoadPlugIn(GetType(NavigationPlugIn))
loader.ActivatePlugIn(GetType(NavigationPlugIn).GUID, Nothing)
The last thing to be added for basic functionality is the data. As it stands, the only data that will appear on the globe is the image of the continents. Zooming in only produces a blurry representation of that base image.
Data layers are created from specially formatted data sources provided by maps.live.com known as content manifests. These are XML files which tell the VE3D control how to load the data required for any view. Content layers can be added by adding them to the DataSources object of the GlobeControl. We can add any of the following layers (note that there may be other content manifests provided by maps.live.com, but these are the only 5 that I am aware of):
URL
DataSourceUsage Type
Description
For the best display, add the ElevationMap, Model and Aerial TextureMap layers as shown:
// set various data sources, here for elevation data, terrain data, and model data.
_globeControl.Host.DataSources.Add(new DataSourceLayerData("Elevation", "Elevation", @"http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap));
_globeControl.Host.DataSources.Add(new DataSourceLayerData("Texture", "Texture", @"http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap));
_globeControl.Host.DataSources.Add(new DataSourceLayerData("Models", "Models", @"http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model));
' set various data sources, here for elevation data, terrain data, and model data.
_globeControl.Host.DataSources.Add(New DataSourceLayerData("Elevation", "Elevation", "http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap))
_globeControl.Host.DataSources.Add(New DataSourceLayerData("Texture", "Texture", "http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap))
_globeControl.Host.DataSources.Add(New DataSourceLayerData("Models", "Models", "http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model))
By passing the URL of the content manifest, a name for the layer, and what the manifest represents, a new DataSource is created, which is in turn used to create a DataSourceLayerData object which is then given to the VE3D control to consume. This should be done in the FirstFrameRendered event handler as well.
We also need to setup VE3D to turn off any UI elements, turn on the atmosphere effects, and ensure we have a full unobstructed view. Again, in the FirstFrameRendered event handler, we can use the following code to achieve this:
// turn on the nice atmosphere
_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering;
// default to all off
_globeControl.Host.WorldEngine.Display3DCursor = false;
_globeControl.Host.WorldEngine.SetWindowsCursor(null);
_globeControl.Host.WorldEngine.ShowNavigationControl = false;
_globeControl.Host.WorldEngine.ShowCursorLocationInformation = false;
_globeControl.Host.WorldEngine.ShowScale = false;
_globeControl.Host.WorldEngine.ShowUI = false;
_globeControl.Host.WorldEngine.Environment.SunPosition = null;
_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = true;
_globeControl.Host.WorldEngine.BaseCopyrightText = " "; // workaround for a display issue
' turn on the nice atmosphere
_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering
' default to all off
_globeControl.Host.WorldEngine.Display3DCursor = False
_globeControl.Host.WorldEngine.SetWindowsCursor(Nothing)
_globeControl.Host.WorldEngine.ShowNavigationControl = False
_globeControl.Host.WorldEngine.ShowCursorLocationInformation = False
_globeControl.Host.WorldEngine.ShowScale = False
_globeControl.Host.WorldEngine.ShowUI = False
_globeControl.Host.WorldEngine.Environment.SunPosition = Nothing
_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = True
_globeControl.Host.WorldEngine.BaseCopyrightText = " " ' workaround for a display issue
If you were to run the application at this point, you would see a fully functioning Virtual Earth 3D control with proper data and navigation.
The user will control VE3D with the Wiimote by holding the nunchuk in the left hand, which will move the user forward/back/left/right using the joystick. The C and Z buttons on the front of the nunchuk will be used to raise and lower the altitude of the camera. The Wiimote, held in the right hand, will be used to toggle various things on or off and interact with menus. The user will also stand on the Balance Board which will use their center of gravity to turn them in the environment.
VE3D bindings allow you to change or create new control schemes for VE3D by placing an XML file in a specific directory as follows:
In this directory you will find a Bindings.xml file. This XML schema defines the default keyboard, mouse, Gamepad and other input device properties. Open the file to see the schema used to define events and parameters.
By default, VE3D will load any file named Bindings*.xml from this directory. For the Wiimote control scheme, create a new file named BindingsWiiEarthVR.xml in this directory. Set the contents of the file to the following:
<?xml version="1.0" encoding="utf-8" ?>
<Bindings>
<BindingSet Name="WiiEarthVRBindings" AutoUse="True" Cursor="Drag">
<!-- Nunchuk joystick -->
<Bind Event="Wiimote.NunchukX" Factor="0.5"><Action Name="Strafe"/></Bind>
<Bind Event="Wiimote.NunchukY" Factor="1"><Action Name="Move"/></Bind>
<!-- Nunchuk buttons -->
<Bind Event="Wiimote.NunchukC" Factor="0.20"><Action Name="Ascend"/></Bind>
<Bind Event="Wiimote.NunchukZ" Factor="-0.20"><Action Name="Ascend"/></Bind>
<!-- Balance Board -->
<Bind Event="Wiimote.BalanceBoardX" Factor="-0.0009"><Action Name="Turn"/></Bind>
<Bind Event="Wiimote.BalanceBoardY" Factor="0.0009"><Action Name="Ascend"/></Bind>
<!-- Wiimote buttons -->
<Bind Event="Wiimote.Home"><Action Name="ResetOnCenter"/></Bind>
<Bind Event="Wiimote.A" Factor="1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Left" Factor="-1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Up" Factor="-1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Down" Factor="1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<!-- FPS-style keyboard controls in case we don't have a nunchuk -->
<Bind Event="Key.W" Factor="22"><Action Name="Move" /></Bind>
<Bind Event="Key.S" Factor="-22"><Action Name="Move" /></Bind>
<Bind Event="Key.D" Factor="22"><Action Name="Strafe" /></Bind>
<Bind Event="Key.A" Factor="-22"><Action Name="Strafe" /></Bind>
<Bind Event="Key.Space" Factor="20"><Action Name="Ascend" /></Bind>
<Bind Event="Key.C" Factor="-20"><Action Name="Ascend" /></Bind>
<!-- Other keys -->
<Bind Event="Key.F1"><Action Name="VR920SetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Key.F2"><Action Name="BalanceBoardSetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Key.F3"><Action Name="ToggleVR920Stereo, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Minus" Factor="-0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Plus" Factor="0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.One"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Wiimote.Two"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Key.B"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Key.V"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
<Bind Event="Key.F"><Action Name="FullScreen, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
</BindingSet>
</Bindings>
The <BindingSet> tags wrap groups of control bindings. It requires a Name and optionally a Cursor. If the binding set is to be used automatically, as it would be in most cases, set the AutoUse parameter to True. Inside of that are <Bind> tags. The tag requires the Event parameter and optionally the Factor parameter. The Event parameter will be used to match the binding to its handler which will be written later. The syntax is <Handler Name>.<Event Name>. The Factor parameter is optional and can be used to scale the data value up or down to increase or decrease sensitivity of the input method. The Action tag inside the Bind tag is used to map the specific binding to a particular method. Once the handler is written, these will make more sense.
The bindings above create the control scheme described above: NunchukX/Y describe what happens when the analog joystick is moved, NunchukC/Z describe what happens with the C/Z buttons are pressed, and so on.
The bindings also allow for several variations. Bindings are defined for both the IR position (IRX, IRY) and accelerometer values (AX, AY). If an IR sensor bar is not available, the accelerometer values of the Wiimote can be used instead. Additionally, keyboard bindings are created in the style of a first person shooter using WASD. These can be used if a Nunchuk is not available.
Note that some bindings append two Events together with a + sign. This allows for button combinations. In this case, for the accelerometer and/or IR sensor, we only want to register the action if a button is pressed down. So, those events which require the button to be held down contain Wiimote.B+ and the event it is combined with.
For those events which require a custom action that will be written separately and not part of the VE3D control, the Action parameter must contain the action name, followed by a comma, and then the full assembly name:
<Bind Event="Wiimote.One"> <Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>
An EventSource is needed which will grab data from the Wiimote and pass it along to VE3D as defined by the bindings file above. Create a new class named WiimoteEventSource which derives from Microsoft.MapPoint.Binding.EventSource as follows:
Next, add an enumeration named WiimoteEvent (the name isn't important) which contains all of the Name items from the bindings XML file above. It should look like this:
// all events handled by this event source from XML filepublic enum WiimoteEvent{ IRX, // IR X position IRY, // IR Y position NunchukX, // Nunchuk joystick X position NunchukY, // Nunchuk joystick Y position NunchukC, // Nunchuk C button NunchukZ, // Nunchuk Z button AX, // Wiimote accelerometer X AY, // Wiimote accelerometer Y Up, // Dpad up Down, // Dpad down Left, // Dpad left Right, // Dpad right A, // A button B, // B button Minus, // Minus button Home, // Wiimote Home button Plus, // Plus button One, // Wiimote One button Two, // Wiimote Two button BalanceBoardX, // Balance Board COG X BalanceBoardY // Balance Board COG Y}
' all events handled by this event source from XML filePublic Enum WiimoteEvent IRX ' IR X position IRY ' IR Y position NunchukX ' Nunchuk joystick X position NunchukY ' Nunchuk joystick Y position NunchukC ' Nunchuk C button NunchukZ ' Nunchuk Z button AX ' Wiimote accelerometer X AY ' Wiimote accelerometer Y Up ' Dpad up Down ' Dpad down Left ' Dpad left Right ' Dpad right A ' A button B ' B button Minus ' Minus button Home ' Wiimote Home button Plus ' Plus button One ' Wiimote One button Two ' Wiimote Two button BalanceBoardX ' Balance Board COG X BalanceBoardY ' Balance Board COG YEnd Enum
Next, several methods from the EventSource object need to be overridden: GetEventData, IsModifier, CanModify, TryGetEventId, TryGetEventName, Name. The methods do the following:
Method/Property
The code for these methods is presented below:
// return out value of the passed enum
public override bool TryGetEventId(string eventName, out int eventId)
eventId = (int)Enum.Parse(typeof(WiimoteEvent), eventName);
return true;
// return out the string name of the passed in enum value
public override bool TryGetEventName(int eventId, out string eventName)
eventName = ((WiimoteEvent)eventId).ToString();
// unknown
public override EventData GetEventData(int eventId, EventActivateState state)
throw new NotImplementedException();
// can the event be used as a modifier?
public override bool IsModifier(int eventId)
// yes to all for now
// can the supplied event be used as a modifier?
public override bool CanModify(int eventId, EventKey other)
// only if it's from us
return (other.Source == this);
// this must match the Source name in the bindings XML file
public override string Name
get { return "Wiimote"; }
' return out value of the passed enum
Public Overrides Function TryGetEventId(ByVal eventName As String, <System.Runtime.InteropServices.Out()> ByRef eventId As Integer) As Boolean
eventId = CInt(Fix(System.Enum.Parse(GetType(WiimoteEvent), eventName)))