PDF – Using TiltBrush in Unity with DICOM Datasets

KTVR-TiltBrush_Unity.pdfDownload

How to import DICOM data into TiltBrush for immersive editing and art-making with the TiltBrush VR controllers within Unity

Scripts required: KTVR TiltBrush Extension, GenericLoader

GitHub link: https://github.com/Know-Thyself-as-a-Virtual-Reality/KTVR-DICOM-Tools

VIMEO link: https://vimeo.com/735183720

Note:

Step by step (initial setup): 

  1. If you don’t already have SteamVR installed, use the link to visit the Steam website or search for Steam. Make an account if you don’t already have one. 
  2. Download and install Steam and follow their instructions to set it up. Then open the program and ensure Store is selected at the top left of the screen. 
  3. Search for SteamVR and install the extension by the same name (fig. 1).
  4. If you don’t already have the Oculus Integration package installed, download it from the Oculus website (full instructions for this step can be found here).  
  5. Acquire the project file from GitHub:
  6. If using Git Bash: Copy the code from GitHub (fig. 2). Then open Git Bash and enter the text “git clone” then hit shift and paste the GitHub link (fig. 3). You can then skip step 7.
  7. If not using Git Bash: Skip step 6. Download the ZIP file from GitHub (fig. 4) and unzip it. Then open Unity Hub and click Open to access the project file (it’ll be in your computer’s downloads folder unless you saved it elsewhere).

Step by step (link TiltBrush and Unity): 

  1. With TiltBrush and Unity open, ensure the KTVR Tiltbrush Extension from the project file is in the Unity Hierarchy.
  2. To import your DICOM data, find the cube within the DICOM loader in the KTVR Tiltbrush Extension in the hierarchy (fig. 5) (this is within the TiltBrush Main Canvas, meaning you’ll be able to edit (resize, reposition, etc) the DICOM dataset in TiltBrush). 
  3. With the cube selected, go to the project folders at the bottom of the Unity window and navigate to Assets, then KTVR-tiltbrush-Extension, Resources, 3D, then Chest (fig. 6). Right click the bottom T04 file in the Chest folder and select “show in explorer” (or “show in finder” on Mac). 
  4. The Chest folder should open on your computer. This is where the .mhd and .raw files for your DICOM data should be placed for importing (meta files will be created for you).
  5. In Unity, with the cube still highlighted in the hierarchy, go to the inspector window and under GenericLoader update the Image Dir (image directory) field to match the location of the dataset you’d like to import (fig. 7).
  6. Hit the Play button and TiltBrush should start up in Unity with your dataset automatically in the scene. 

Step by step (working in TiltBrush): 

  1.  Using your VR headset and controllers, the left hand holds the palette and the right hand operates the main controls and cursor. 
  2.  Use the right-hand cursor to select New Sketch from the palette (fig. 8).
  3.  Moving left/right on the right-hand joystick changes brush size (fig. 9).
  4.  Use the bottom grips to allow moving/resizing objects.
  5. Exporting from TiltBrush will export everything you’ve drawn in the scene but not the DICOM dataset you loaded in. Don’t worry though, it can be re-imported alongside the drawings afterwards using the KTVR DICOM tools.
  6. Save works in progress within TiltBrush using the Sketchbook function in the left-hand menu. They should be listed next to New Sketch when reopening Tiltbrush so you can continue working. 

Fig. 1
Install SteamVR (center)

Fig. 2
Copy the GitHub link if using Git Bash

Fig. 3
Type ‘git clone’ and paste link if using Git Bash

Fig. 4
If not using Git Bash, download and open the ZIP folder

Fig. 5
Location of Cube within DICOM loader

Fig. 6
Navigate to Assets, then KTVR-tiltbrush-Extension, Resources, 3D, then Chest

Fig. 7
Under Generic Loader, change the Image Dir field to the location of the files to be imported

Fig. 8
Open a new sketch (left), or if you have saved sketches they’d be found on the right side

Fig. 9
Use right-hand joystick to make the brush larger or smaller