This uses a newly exposed UnityEditorInternal layermask api to handle the layer masking logic. Note that I haven’t fully tested this new approach for bugs and because it’s a unity internal function, it would be hard to fix if there are bugs in Unity’s implementation.
Hope this helps anyone that noticed this issue with layer masks!