Unity Integration
There are a couple of different ways of incorporating FAtiMA Toolkit in your Unity project.
FAtiMA Toolkit works as an independent C# library which means you need to import it to your project.
However we have made available a Unity project: FAtiMA Toolkit Unity Demo that already has all of the files needed and examples on how to use them. We recommend new users to start by downloading this project and using it as a basis to work and play with.
However if you prefer can easily add FAtiMA to your existing Unity project by downloading its .dll files or generating them on your own.
There are 2 different ways of obtaining/use them:
- Download the files from our NAS repository
- Compile the Fatima Toolkit solution (FAtiMA-Toolkit.sln) on your IDE of preference and then, using Windows PowerShell, use the following command:
.\ExportDlls.bat debug ‘path of desired directory’
It is important to note that to use an external library in Unity the .dll files need to be in a sub-folder called “Plugins” in the “Asset ” folder of the Unity Project. If there is no “Plugins” folder, please create one.
These are the aforementioned FAtiMA .dll files:
A simple way to test if the integration is properly working is to create a new C# script in the Unity Project and try to import a FAtiMA-specific library, such as Integrated Authoring Tool Asset, as seen in the figure below:
Now that you have the dll files you can either use the FAtiMA Toolkit Unity Demo to learn how to incorporate the toolkit in your project using its scripts as an example or you can take a look at the FAtiMA as a C# library post to learn how to create FAtiMA powered scripts by your own.
In case you go for the first option here is a guide to better help you understand the Unity example project.
The FAtiMA Toolkit Unity Demo
The FAtiMA Toolkit Unity Demo repository project is frequently updated with new scenarios and use cases. It serves as an example of how to use FAtiMA specific functions to handle character’s dialogues, emotions and decision making.
We encourage developers to use this project to get familiar with FAtiMA, use it as an example on how to get an interaction game of the scenarios created using the authoring tool.
The main Unity Scene we use to test our examples is the “SingleCharacterScene”. It’s purpose is to test the scenarios created for dyadic conversations as in only one on one interactions are supported. Multi Character interactions are recreated in the “Office” Unity scene.
Adding a new Scenario
In this project all the scenarios used must be in the StreamingAssets folder. Inside you will find a .txt file describing the path of the scenario and the name of the folder that contains its text-to-speech files. As you can see in the image below:
The first line points to the .iat file of the scenario and the line afterwards let’s the code know where are the voice files used in it. Inside the “Scenarios” folder you can find the FAtiMA specific files for each scenario, its “.iat” file, its world model (“.wmo”) and its characters (“.rpc” files) and their components (“.edm”, “.ea”, etc…).
If you have a scenario you created yourself, add it to this project by copying the files to a folder and editing the .txt file with the path of the .iat file, just like the others.
The Manager and Agent Controller scripts
In the Unity Scene if you inspect the “Manager” game object of the scene you will find that it has a script attached to it, the “Single Character Demo” script. This script is responsible for controlling and implementing this scene.
As you can see, the script needs a lot of objects associated to it to function properly. Let’s look at the most relevant ones:
- “Character Anchor” is a “Transform” from where the Character will “appear/spawn” in.
- “Dialog Controller” serves to indicate where the Agent’s dialogue will appear in
- “Bodies” indicates what is the body of each type of body. For instance if the Role Play Character says its body is Male what will the Engine spawn, which 3D-Model is suited for the “Male” body tag.
- “Dialogue Button Archetype” is a Prefab and determines the button the Player will press to chose a dialogue option.
- Dialog Button Zone defines where the dialog options for the Player will appear.
Please note that this script has been created by us for this particular example and users should change it towards its scenario. Feel free to edit the scripts and remove or move things around as much as you’d like, this is a simple example only.
The SingleCharacterDemo script receives a lot of UI Objects that are needed to display the Dialogues from both the agent and the Player. Some of the UI objects should be on the Scene, such as the Menu Button Holder, the Dialog Button Zone and the Score Zone and others are Unity Prefabs, such as the Score Text Prefab and the Dialogue Button Archetype.
Once the scene starts, the script runs several initialization methods and then locates the .txt file and finds all of its referenced .iat files. For each of the .iat files it creates a button with the name of their scenario. Once the player selects one of the scenarios the manager loads the file, its World Model and all of its Role Play Characters. Once again it creates a button for each rpc and the player selects which one does he want to talk to, once this process is finished it enters its main loop (update method).
Each active Role Player Character, except the Player, has an “Agent Controller” script associated with it. Essentially the Manager script handles the Player’s decisions and the Agent Controller handles its Role Play Character’s decisions.
Each agent’s EDM component is essential to the scenario. The main loops of the game are based around asking their Role Play Characters “what they want to do”.
In the examples we created, the Emotion Decision Making component of most of the Role Play Characters had the following condition:
Has(Floor) = SELF
This condition says that unless the agent believes he has the floor he will not decide to talk. This allows us to make sure that each agent does not decide to talk when another is talking. When the agent finishes talking it “passes the floor” to the agent whom he just talked to (the target of his Speak action).
Currently this is what typically happens in the first main loop of the Single Character Script:
- Manager script asks Player RPC to decide
- Player RPC decides to talk, aka, perform action: Speak(Start, *, *, *)
- Manager script retrieves all dialogue options that correspond to that action
- Manager script creates a button for each dialogue option
- Human Player selects an one of the available dialogue option
- Manager script informs the agent controller of the Player’s decisions
- Manager script asks the World Model what are the consequences of the action selected
- Manager script informs all affected of what the consequences are.
As we mentioned beofre the Agent Controller script handles the agent whom the Player will be interacting with. Once it is initialized by the Manager it loads its Role Play Character, and instantiates its body according to the “BodyName” field. Afterwards in starts its main loop. Similarly to the Manager’s main loop it is constantly asking its RPC what does it want to do.
Note: The project’s “Office” Unity scene thas both of these scripts adapted to take into account multiple RPCs and as a consequence multiple Agent Controllers in the same scene.
RPC bodies and 3D Models
FAtiMA Toolkit is one of many products developed through the RAGE project. In order to get our characters to display emotion and to appear as if they were talking we use another RAGE originated asset: the BML Realizer Asset. The Models used in the Unity Demo were created using Daz3D Software and were then imported to Unity using the guide in the Virtual Human Controller website.
FAtiMA Toolkit’s Unity integration however does not require bodies, it is just the way this particular example is set up.
While this page is in construction we highly recommend users to check the Unity Demo Repository to understand how to use these “Fatima specific Unity Scripts”.
While using the already written scripts is advised, in some cases and to better understand how FAtiMA works we will now show how users can use FAtiMA assets in their C# code.
We have also created a starter kit that is using a compiled version of the toolkit in a Unity game, to use it, download the following zip and read the instructions.
WebGL-Build
WebGL is a graphics library that works in mysterious ways, specially in Unity.
To run in WebGL, all code needs to be JavaScript. Unity uses use the emscripten compiler toolchain to cross-compile the Unity runtime code (written in C and C++) into asm.js JavaScript. This means that the Unity WebGL Build converts all your c# code into javascript and only then does it compile the code and buld the game.
Unfortunately the FAtiMA dll files can be lost within this process, as such, in order to successfully build a Unity game using FAtiMA to WebGL the project itself needs some help. If you are using FAtiMA Unity Demo as a basis for your work you don’t need to do this:
- A link.xml file needs to be in the Assets folder, you can download this file here
- In the Plugins folder also within the Assets folder there needs to be a folder named “WebGL”, this folder can be downloaded here