nukso

Esperanto has no suffix -ks-, so there is nothing to take apart here: nuks- is one root, straight from the Latin nux. It is worth saying out loud only because learners at this stage have started dismantling everything, and a word with an awkward consonant cluster in the middle invites a guess.

The generous word is ŝelo, which covers the shell, the peel, the rind and the bark all at once, so a nut's shell is simply its ŝelo and the kernel inside is the kerno. That page ends on the phrase nuksoŝele, in a nutshell, which is a compound worth keeping: it says the same thing English does and it is built entirely out of parts you already own.

Specific nuts get compounds or their own roots - avelo for a hazelnut, juglando for a walnut - but nukso alone is perfectly good for all of them in ordinary speech, and nuksarbo is the tree they grow on. One warning about the metaphor: English calls a difficult problem a hard nut to crack, and while malmola nukso will be understood, the natural Esperanto for a stubborn problem is simply malfacila problemo. Borrow the shell, not the idiom.