trista

Yes, it is a real word. Trista is in PIV, it is in Zamenhof-era verse, and it has been in continuous literary use since the 1890s. It is not a Romance speaker's mistake for malĝoja, even though that is what it looks like to somebody who has just learned the mal- prefix.

What separates them is register, not meaning. Malĝoja is the built word, transparent to every learner on the planet: ĝoja joyful, mal- reverses it, done. It is what you say in conversation, in an email, in a lesson. Trista is borrowed, opaque to anyone without a Romance language, and it belongs to poetry, song lyrics and deliberately heightened prose. Reaching for it in ordinary speech sounds the way "doleful" sounds in English: not wrong, but dressed for a different occasion.

So the working rule is simple. Say malĝoja unless you are writing something that would also tolerate a metaphor. Read trista and understand it, because the literature is full of it.

This is a small instance of a large argument inside Esperanto, between the people who think everything expressible with affixes should be expressed with affixes, and the people who think a language needs synonyms with different textures before it can carry poetry. The second camp won, quietly, by writing better poems. Trista, tristo (sorrow) and tristeco are their souvenirs.