I used foam/foil soundproofing material available through West Marine or Defender or similar marine stores. The material is 1.5 or 2 inches thick. My installation was around the engine on an I/O, so totally enclosed.
The noise reduction was significant and worthwhile, but not as quiet as I hoped. My boat is also made of plywood, so the whole thing is a sound board. The best place to ride is in the bow where the dash and windshield block the sound.
The foam in soundproofing is open cell. It absorbs sound, but also water. I don’t know how effective it is when wet. The foam is also flame retardant to a modest degree.
You might first fashion a shell around your motor well to see if that helps. When the boat is at speed remove the shell to hear the difference in sound levels. Then proceed to add foam.
Good luck. I admire your work.