Warning that I'm lacking in overtly "monster" books but some of these might hit your other points, here's a jumbled list of ideas:

Piranesi by Susanna Clarke

Maybe you've already read? But by the author of JS&MN. Very different to that (and honestly I preferred Strange & Norrell) but kind of has an ~experimental vibe where you don't know wtf is going on.

Our Wives Under The Sea by Julia Armfield

Maybe a monster book?! I haven't read but it's been on my TBR forever and a book!friend I would trust with my life gave it 5 stars. Modern/queer/horror/a submarine expedition???

Hell Followed With Us by Andrew Joseph White

Again, haven't read yet but someone who met me all of five minutes prior recommended it to me without blinking? Trans author, trans characters, sounds like monsters. Maybe YA?

Frankenstein by Mary Shelley

Just in case you haven't read before xD The "monster" is not the monster but very much a metaphor, gothic horror, a classic but feels way more modern than it is.

The Lathe of Heaven by Ursula K. Le Guin

This is almost one of her lighter?? books I've read so far and set in "our world". A man can dream things into reality. I can't recommend Le Guin more highly, her prose is everything to me.

The Raven Boys by Maggie Stiefvater

Have you read this? Beanie and I were very into this whole series back in the day xD Modern, YA, magic and sentient forests and ghosts and quests and seers. A man can ALSO dream things into reality?

Ninth House by Leigh Bardugo

Has some dark and mildly depressing themes in it (TW for some graphic violence/sexual assault etc) BUT if/when you are in the mood for something ~gritty, it's magic societies/murder mysteries at modern Yale and so much fun. Plus a ghost I would die for.

Rivers of London by Ben Aaronovitch

Again, I haven't read yet but has been on my list forever. London urban fantasy. Written by a man so idk yet but I think he wrote a few Doctor Who novelisations.

Gideon The Ninth by Tamsyn Muir

Another one I haven't read but it has been recommended to me. The short logline is "necromancers in space" but it sounds quite experimental. I think one of the books in the series is in second person? IDK.

This so you got into TMA list of book recs might be useful too xD

A wild TV show rec:
The Terror (AMC): has horror. And monsters. And naval expeditions. And is my favourite. (Steer clear of the book at any cost, it's not good. xD)

Honourable mention to All The White Spaces by Ally Wilkes and Dark Matter by Michelle Paver for being polar ghost stories... I preferred the former to the latter but I personally find both have pacing issues/didn't quite make enough of the horror. Still solid tho!