i've seen lots of people confused about how private the fediverse is, especially when it comes to things like direct messages. i can understand why people are confused about it; many Big Tech companies don't openly explain the privacy on their platforms, and a lot of people seem to assume there's much more than there actually is.
to put things simply: nothing you say on the fediverse is private, not even your DMs. for public posts, this makes sense; but for followers-only and unlisted posts, people are a bit surprised by this sometimes. there is nothing in the ActivityPub protocol that prevents someone who isn't supposed to from reading your messages; an unscrupulous server operator could easily look at everything their server has access to, not just what you tell them they're supposed to.
of course, it's worth noting that most other social media is the same way. the people at Twitter, Instagram, and Discord can read your DMs, and the people at Snapchat can see your pictures as much as they want. the only exception are specifically end-to-end encrypted applications, like Whatsapp and Signal.
while this obviously could be seen as a downside, i personally think it's okay that things are like this on the fediverse. encryption is a hassle to deal with sometimes, especially for those who aren't tech-savvy, and i've yet to find an open source application that handles encrypted communications without making things so user-hostile that it's practically impossible.
there is also a legal component; having people post stuff you can't see on your server is a bit of a problem if people start doing stuff that could get you into trouble for hosting it. since most volunteer server admins don't have the money to pay for a lawyer if they get told to shut down by the cops, it's hard to put up a fight. ideally this wouldn't have to be a problem, but we live in a world where it is.
personally, i think that if you want to communicate privately, you should just use a dedicated instant messaging app. you don't need one app to do everything for you, and not keeping all your eggs in one basket is generally a good idea anyways. i personally use Signal for most of my messaging needs currently, although i hope one day i can replace that with a fully open source alternative.