Boxed nullable underlying type can be cast to enum but boxed enum type can't be cast to nullable type

• Boxed nullable underlying type can be cast to enum but boxed enum type can't be cast to nullable type.

And similarly,

• Boxed nullable enum can be cast to underlying type but boxed underlying type can't be cast to nullable enum.

Ok, I know "boxed nullable type" is not the best way to describe it, but it's for the sake of the question. I'm aware it's the underlying value type that's getting boxed.

I will show it with examples. Assume I have an enum with int as the underlying type.

enum Sex { Male, Female }

Case I:

int? i = 1;
object o = i;
Sex e = (Sex)o; //success

//but

Sex e = Sex.Male;
object o = e;
int? i = (int?)o; //invalid cast

Case II:

Sex? e = Sex.Male;
object o = e;
int i = (int)o; //success

//but

int i = 1;
object o = i;
Sex? e = (Sex?)o; //invalid cast

In a nutshell,

(enum)int? -> succeeds
(int?)enum -> the reverse fails

(int)enum? -> succeeds
(enum?)int -> the reverse fails

Or in even simpler terms,

cast to non-nullable -> succeeds
cast to nullable -> fails

Now I do know that once you box a value type, it can be cast back only to the original type. But since by C# rules, a boxed int can be cast to enum and a boxed enum can be cast to int, and a boxed int to int? and a boxed int? to int, I was looking for a consistent understanding of other scenarios as well, ie the ones listed above. But I dont get the logic. For one, I feel if they all failed or if they all succeeded, it made more sense to developers. Two, even the successful casts look a little odd. I mean since a value type can be implicitly cast to its nullable equivalent (and not the other way around), a cast to nullable should anyway succeed, but with the current implementation a nullable type is being successfully cast to non-nullable which can even fail if the former had a null value. Had this whole thing been other way around, it would have been easier comprehending. An example like:

Sex? e = null;
object o = e;
int i = (int)o; //succeeds, but sure to explode on cast

//but

int i = 1;
object o = i;
Sex? e = (Sex?)o; //invalid cast, even though its always a safe cast

Questions:

1. So what C# rule is letting this happen?

2. Is there a simple way I can remember this?