gh-131798: split recursion check to _CHECK_RECURSION_LIMIT and combine checks

gh-131798: split recursion check to _CHECK_RECURSION_LIMIT and combine checks #5841