I can’t believe it’s been a day and no one has said Kenshi yet.

Kenshi is an open world rpg where you control one or more characters in an enormous wasteland. It’s super complicated with a ton of systems and that game really hates you. When you start your first save, you’d be lucky if you only get captured and enslaved instead of killed. Your character is a regular person in the world, not the powerful main character (the real main character is Beep) and you’ll constantly feel it.

I would also like to know what games are like that, so I can avoid them.

Nethack, a very deep and actively obtous game. If you want to figure it out yourself, its complexity is nearly on a level with dwarf fortress.

Outer Wilds does not fit 100 % your description, as it doesnt hate the player. But it is a game all about exploration and figuring stuff out yourself. And if you do things wrong, you die.

Stationeers. It’s a game about building and maintaining a station on another planet, with complex and realistic gas/liquid mechanics, properly recreating the refrigeration cycle and everything. You have to grow food and create oxygen and water, and manage the temperature of your station. It has one of the steepest learning cliffs ever, but there are guides for everything online.

(It’s being made by the same company making Kitten Space Agency.)

Honestly Elden Ring kinda fits that description

Edit: And the other souls games, but they’re not open world in the same way. All of them are about learning how the game works, without quest markers or hand holding, and can feel quite punishing until you do.

Mon Bazou

Not really unforgiving, but I feel like Voices of the Void could have other gameplay aspects you’re looking for. If you like ambient horror, it’s still free to download from itch.

Noita may be close. It nails the no directions and unforgiving part.

I would argue that Tunic and probably most soulslikes would fit your description.

Morrowind kinda does this. Unlike the other Elder Scrolls games that came later, you don’t have any quest markers, or anything really to help you find where you’re looking for. Most of the quests give directions like, “follow the road south till you get to the fork in the road with a big tree. Go on the eastern branch till you find a cave entrance. If you get attacked by cliff racers you’re either on the right path or you went too far. Good luck.”

Raft is kind of like this. You’re basically told nothing and you’re out at sea on a raft that you can then build with stuff you find.

It was not for me, but almost entirely for the reasons you want a game like this, from what I’m reading.

“Don’t Starve”? Checks all your boxes and has a ton of dlcs. I’ve yet to find the will to play it again.

I’m looking for other complex open world games that throw you into the deep end without any explanation, and are completely unforgiving when you make mistakes.

UnReal World? It’s never really grabbed me when I’ve tried it, personally, but it might be up your alley.

Kenshi has a pretty difficult start, though it gets easier later in the game. I don’t think that it’s as complex.

Cataclysm: Dark Days Ahead is quite complex and has a very steep learning curve, with a lot of hidden stats and the like and limited documentation.

I’m looking for other complex open world games that throw you into the deep end without any explanation, and are completely unforgiving when you make mistakes.
Games that even actively hate the player, but give you a deep sense of satisfaction when you finally figure it out.

So no tutorial, no quest markers, no mini-map, no quick-saving, etc.

Boy have I got a game for you. I’m playing Darkwood right now and this should tick most of your boxes. It’s not a huge open world, more like several open zones as you progress through the story, but otherwise should do what you want. It’s also not just deeply unsettling but also genuinely terrifying despite being a top-down game.

You might dig The Forest.

I died a LOT in the fucking forest.