[NC2] AoT iOS Jit Error on Startup

[Matt Newcombe]

So I get a JiT error for AoT on startup on iOS, it looks like it’s a deserialisation issue. In general I don’t know if it’s possible to avoid non AoT compilation for Node Canvas as it makes it somewhat limiting for use with app development, stack trace is:

https://gist.github.com/newcommat/e4c1561b142bd6504a49

[Gavalakis]

Hello Matt,

There are a realy whole lot of stuff in the core that use generics now and the reason of some very nice new features.
With that said, I will take a look and see what can be done. I researched a bit and seems to exist some solution that Im going to try. The only problem is that it will be a bit difficult testing it without actualy being able to compile an iOS project or force an AOT only compilation somehow. Will look at possible workarounds.

Thanks for the report

Join us on Discord: https://discord.gg/97q2Rjh

[Matt Newcombe]

No worries.

With multiplatform development being such a strong part of the Unity ecosystem I would say that if you could solve compatibility of AoT platforms it would be a huge win, as my project is a mobile project it unfortunately removes the reliance I’m able to put on NodeCanvas.

Maybe a second hand mac mini / iphone 4 won’t break the bank for testing, but understand this may be outside of the scope of what you envisage NC2 to be.

[tedw4rd]

I’ve seen the same issue on our project as well (which isn’t iOS). I believe (but haven’t tested) that you can reproduce this issue on Windows by making a Windows build and using Mono’s AoT post-processor: http://www.mono-project.com/docs/advanced/aot/

[Gavalakis]

Hello,

Basicaly, IOS has a bunch of limitations of what great things c# support, copared to the other platforms.
Ted, what platform did you see the errors other than iOS and WebGL? Also thanks a lot for that link, will take a look at it as soon as possible 🙂

I will really look into supporting iOS, even though some features will not be fully supported.
This is something I will do very soon.

Sorry for that,
Gavalakis Vaggelis

Join us on Discord: https://discord.gg/97q2Rjh

[tedw4rd]

You’re welcome! We’re building for PS4.

JiT compilation is probably not going to be supported on any device, since it allows for potential security and performance issues. NodeCanvas is such a great utility, and NC 2 is such a huge improvement in terms of performance, that it’d be a shame for it not to work on consoles and mobiles.

I’m working on a workaround (hack) for these JiT issues. If I find a good solution, I’ll post it here!

[tedw4rd]

I’ve made some progress, but things aren’t working completely.

One should be able to fix the error we’re seeing by forcing the compiler to create appropriate entries in the function table. Doing that is fairly easy; all you need is to explicitly call each function somewhere in your code with each type you’ll ever be calling. Ours kind of looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

void SpoofNodeCanvasTypes<T>()     {         NodeCanvas.Framework.Variable<T> v = new NodeCanvas.Framework.Variable<T>();         NodeCanvas.Tasks.Conditions.CheckVariable<T> c = new NodeCanvas.Tasks.Conditions.CheckVariable<T>();         NodeCanvas.Tasks.Actions.SetVariable<T> s = new NodeCanvas.Tasks.Actions.SetVariable<T>();         NodeCanvas.Framework.Blackboard b = new NodeCanvas.Framework.Blackboard();         b.GetVariable<T>("");         b.GetValue<T>("");     }       void RegisterTypesForAOT()     {         SpoofNodeCanvasTypes<Vector3>();         SpoofNodeCanvasTypes<float>();         SpoofNodeCanvasTypes<bool>();         SpoofNodeCanvasTypes<int>();         SpoofNodeCanvasTypes<string>();         SpoofNodeCanvasTypes<Rect>();         SpoofNodeCanvasTypes<Keyframe>();         SpoofNodeCanvasTypes<AnimationCurve>();

… and so on for every value-type in your graphs (i.e. Enums, structs, and anything else that’s passed by value).

You don’t even need to call RegisterTypesForAOT. Just having it in your compiled code will force the compiler to create the entries you need.

After adding this to our code, though, we’ve been getting a different error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.ExecutionEngineException: Attempting to JIT compile method ‘(wrapper runtime-invoke) <Module>:runtime_invoke_void_Rect*_single (UnityEngine.Rect*,intptr,intptr,intptr)’ while running with –aot-only.

at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod*,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0
— End of inner exception stack trace —
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0
at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <filename unknown>:0
at ParadoxNotion.Serialization.FullSerializer.Internal.fsMetaProperty.Write (System.Object context, System.Object value) [0x00000] in <filename unknown>:0

Something similar happens to Keyframes as well. I’ll update again if we fix that.

[Gavalakis]

Hello,

Sorry for the late reply. I got sick past few days :/

Yeap. This is also the valid fix I am reading up while researcing this case.
There is not a problem with the generic methods, rather with the open generic instances created in runtime.
At least from what I understood, the following code would suffice to work for the variables and parameters.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

using NodeCanvas.Framework; using UnityEngine;   public static class AOTDummy{   #pragma warning disable 0168   public static void DummyVariables(){   Variable<string> s; Variable<float> f; Variable<int> i; Variable<bool> b; Variable<Vector2> v2; Variable<Vector3> v3; Variable<Rect> r; Variable<Color> c; Variable<Object> obj; }   public static void DummyParameters(){   BBParameter<string> s; BBParameter<float> f; BBParameter<int> i; BBParameter<bool> b; BBParameter<Vector2> v2; BBParameter<Vector3> v3; BBParameter<Rect> r; BBParameter<Color> c; BBParameter<Object> obj; }   #pragma warning restore 0168   }

It would be awesome if someone can check until a mac arrives home 🙂

Thanks

Join us on Discord: https://discord.gg/97q2Rjh

[tedw4rd]

That solves some of the issues we’re seeing, but not all. We’re still seeing the error I mentioned in my most recent post.

We’ve made some progress by modifying the Read and Write functions of fsMetaProperty to intercept Rects, Keyframes, and AnimationCurves and explicitly getting and setting the appropriate properties. This seems to break some functionality, and I’m not sure why. I’m also not sure why Read is getting called at runtime, since that seems to be a method related only to serialization.

Matt, I’d be interested for our purposes to see if these errors are PS4-specific. Can you try Gavalakis’s solution and see if you’re also getting the error I have above?

Our version of the fsMetaProperty methods follows. I will update if we figure out what else is breaking.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

public object Read(object context) {             if (context is UnityEngine.Rect)             {                 UnityEngine.Rect r = (UnityEngine.Rect)context;                 switch (_memberInfo.Name)                 {                     case "xMin":                         return r.xMin;                     case "yMin":                         return r.yMin;                     case "xMax":                         return r.xMax;                     case "yMax":                         return r.yMax;                     default:                         return null;                 }             }             else if (context is UnityEngine.Keyframe)             {                 UnityEngine.Keyframe a = (UnityEngine.Keyframe)context;                 switch (_memberInfo.Name)                 {                     case "inTangent":                         return a.inTangent;                     case "outTangent":                         return a.outTangent;                     case "tangentMode":                         return a.tangentMode;                     case "time":                         return a.time;                     case "value":                         return a.value;                     default:                         return null;                 }             }             else if (context is UnityEngine.AnimationCurve)             {                 UnityEngine.AnimationCurve a = (UnityEngine.AnimationCurve)context;                 switch (_memberInfo.Name)                 {                     case "keys":                         return a.keys;                     case "length":                         return a.length;                     case "postWrapMode":                         return a.postWrapMode;                     case "preWrapMode":                         return a.preWrapMode;                     default:                         return null;                 }             }             else if (_memberInfo is PropertyInfo) {                 object ret;                 try                 {                     ret = ((PropertyInfo)_memberInfo).GetValue(context, new object[] { });                 }                 catch (Exception e)                 {                     UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                     UnityEngine.Debug.LogError("Error reading property.  Is this a value-type?");                     UnityEngine.Debug.LogError(context);                     UnityEngine.Debug.LogError(_memberInfo.Name);                     UnityEngine.Debug.LogError(e);                     UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                     throw e;                 }                 return ret;             }               else {                 return ((FieldInfo)_memberInfo).GetValue(context);             }         }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

public void Write(object context, object value) {             var field = _memberInfo as FieldInfo;             var property = _memberInfo as PropertyInfo;               if (field != null) {                 field.SetValue(context, value);             }             else if (context is UnityEngine.Rect)             {                 UnityEngine.Rect r = (UnityEngine.Rect)context;                 switch (_memberInfo.Name)                 {                     case "xMin":                         r.xMin = (float)value;                         break;                     case "yMin":                         r.yMin = (float)value;                         break;                     case "xMax":                         r.xMax = (float)value;                         break;                     case "yMax":                         r.yMax = (float)value;                         break;                 }             }             else if (context is UnityEngine.Keyframe)             {                 UnityEngine.Keyframe a = (UnityEngine.Keyframe)context;                 switch (_memberInfo.Name)                 {                     case "inTangent":                         a.inTangent = (float)value;                         break;                     case "outTangent":                         a.outTangent = (float)value;                         break;                     case "tangentMode":                         a.tangentMode = (int)value;                         break;                     case "time":                         a.time = (float)value;                         break;                     case "value":                         a.value = (float)value;                         break;                 }             }             else if (property != null) {                 var setMethod = property.GetSetMethod(/*nonPublic:*/ true);                 if (setMethod != null) {                     try                     {                         setMethod.Invoke(context, new object[] { value });                     }                     catch (Exception e)                     {                         UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                         UnityEngine.Debug.LogError("Error writing property.  Is this a a value-type?");                         UnityEngine.Debug.LogError(context);                         UnityEngine.Debug.LogError(setMethod.Name);                         UnityEngine.Debug.LogError(_memberInfo.Name);                         UnityEngine.Debug.LogError(value);                         UnityEngine.Debug.LogError(e);                         UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                         throw e;                     }                 }             }         }

[tedw4rd]

Ok! We’ve got it fully working on PS4 now. I think…

After making the modifications to fsMetaProperty I listed above, we noticed that AnimationCurves were not deserializing properly, and somehow showed up with completely blank Keyframes. Since Keyframes are structs (and are therefore passed by value), fsMetaProperty.Write wasn’t modifying the Keyframes that would eventually be passed back to the AnimationCurves. The same thing was probably happening to Rects, but we didn’t notice because we don’t use Rects in our Graphs (except for layout, of course).

Our solution was to force pass-by-reference the context object into fsMetaProperty.Write using the ref keyword and then reassign the modified struct to that reference at the end. So, our final, working version of fsMetaProperty.Write looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73

public void Write(ref object context, object value) {             var field = _memberInfo as FieldInfo;             var property = _memberInfo as PropertyInfo;               if (field != null) {                 field.SetValue(context, value);             } // TODO: Find general value-type setter             else if (context is UnityEngine.Rect)             {                 UnityEngine.Rect r = (UnityEngine.Rect)context;                 switch (_memberInfo.Name)                 {                     case "xMin":                         r.xMin = (float)value;                         break;                     case "yMin":                         r.yMin = (float)value;                         break;                     case "xMax":                         r.xMax = (float)value;                         break;                     case "yMax":                         r.yMax = (float)value;                         break;                 }                 context = r;             }             else if (context is UnityEngine.Keyframe)             {                 UnityEngine.Keyframe a = (UnityEngine.Keyframe)context;                 switch (_memberInfo.Name)                 {                     case "inTangent":                         a.inTangent = (float)value;                         break;                     case "outTangent":                         a.outTangent = (float)value;                         break;                     case "tangentMode":                         a.tangentMode = (int)value;                         break;                     case "time":                         a.time = (float)value;                         break;                     case "value":                         a.value = (float)value;                         break;                 }                 context = a;             }             else if (property != null) {                 var setMethod = property.GetSetMethod(/*nonPublic:*/ true);                 if (setMethod != null) {                     try                     {                         setMethod.Invoke(context, new object[] { value });                     }                     catch (Exception e)                     {                         UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                         UnityEngine.Debug.LogError("Error writing property.  Is this a a value-type?");                         UnityEngine.Debug.LogError(context);                         UnityEngine.Debug.LogError(setMethod.Name);                         UnityEngine.Debug.LogError(_memberInfo.Name);                         UnityEngine.Debug.LogError(value);                         UnityEngine.Debug.LogError(e);                         UnityEngine.Debug.LogError("|||||||||||||||||||||||||||||||||||||||");                         throw e;                     }                 }             }         }

Obviously, this is not the perfect solution. If we ever introduce new structs to NodeCanvas, we’ll likely have to update this function again. I also fear that we will have to update fsMetaProperty.Read if we ever introduce anything containing a struct to NodeCanvas. However, this works, and seeing as we have a deadline rapidly approaching, we’re going to roll with this fix for the time being.

Finally, we ran into a strange hard crash on our dev kits containing the error message Ran out of trampolines of type 2. Turns out trampolines are artifacts within Mono that are used in dealing with generics. Since NC2 by-and-large relies on generics more, you may run into this error. To fix it, go to your Player Settings, and under Other Settings, update AOT Compilation Options with nimt-trampolines=512. Trampolines ostensibly incur a memory cost, but I honestly haven’t researched it enough to know how many trampolines is too much, resource-wise.

So, complete (janky) solution:

• Spoof the function calls for each of the value-types (i.e. primitives, enums, and structs) NodeCanvas needs to handle, using either Gavalakis or my templated method calls
• For each of your structs that need to be deserialized, update fsMetaProperty.Write to include an explicit setter for each of the properties on that struct
• For each of the objects that need to be serialized which contain a struct, update fsMetaProperty.Read to include an explicit getter for each of the fields on that object
• If your game crashes with an error referencing “trampolines”, update your AOT Compilation Options to increase the number of trampolines available.

Hope that helps!

[Gavalakis]

Hey,

Wow, you realy dag into the stuff and made some progress on that. I really thank you a lot for providing all this information!!
Yeah NC2 is so much into generics and basicaly the issue of all this incombatibility.

Thanks a lot!
Gavalakis Vaggelis

Join us on Discord: https://discord.gg/97q2Rjh

[Matt Newcombe]

Nice work ted, I’ll take a look at taking this change over for our project and let you know how it goes.

[jacobdufault]

Hey everyone,

I’m the developer of Full Serializer (the JSON library in usage here) and I’ve implemented a solution for this issue. I’ve been submitting some issues with Unity regarding il2cpp but it looks like part of reflection is broken (and won’t be fixed). As a result, I’ve just finished up implementing some AOT compilation capability in Full Serializer that can eliminate most of the usage of reflection.

As it turns out, that capability is extremely useful here too – especially after taking a look at those modifications to fsMetaProperty.Read/Write (ouch!).

First, you’ll have to get an updated version of Full Serializer (I assume NC can ship an updated version at some point), but here is the repository if you want to manually update it.

After that, you can run your game in the editor and do all of the standard serialization you expect. Add the following method to your project (and invoke it) while you’re in play mode inside the editor (not a deployed player). Refresh assets (ctrl-R) after you exit play mode and you’ll see a bunch of converters appear top-level inside of the Assets folders. These will be used automatically by Full Serializer and will remove the usage of reflection.

1 2 3 4 5 6 7 8 9 10 11

using FullSerializer; using FullSerializer.Internal; // for CSharpName   public void AddAotCompilations() {     foreach (var aot in fsAotCompilationManager.AvailableAotCompilations) {         var path = "Assets/AotConverter_" + aot.Key.CSharpName(true, true) + ".cs";         System.IO.File.WriteAllText(path, aot.Value);     } }

NOTE: If you change one of your models, make sure to regenerated the AOT compiled converters.

Jacob Dufault

[Gavalakis]

Thanks a lot Jacob! Great work as always.
The new NC version will include latest Full Serializer.

Cheers and best regards 🙂

Join us on Discord: https://discord.gg/97q2Rjh

[Bosaku]

Hello,

I have done the following :
Update Nodecanvas to 2.1 ( from 1.6 )
Running in iOS has JIT compilation issues, as seen above.
Upon jacobdufault’s suggestion, I have downloaded the source for Full Serializer
Added the “Source” folder to the assets folder (within a subfolder for cleanliness sake)
In the Editor, OnAwake, I am calling the AddAoTCompliations method shown in the above post.
Command R to refresh.
Built, and observed the same problem.

So, I went back and printed out a line after every loop in the foreach statement, to see if there exist any ‘Available AOTCompliations”. There currently are not any.

So I have not found a solution yet.

Any ideas why that may not have worked? Thank you ahead of time.

[Author]

Posts