The casting for Matrix was stellar. Even the mid-tier sequels had some incredible performances on their own merit. Possible Smith recognized he wasn’t a good fit and passed on it. Also possible he picked up on the underlying themes too well and got freaked out (there’s plenty of speculation that Smith has been in the closet going back decades).