What's with the baffle? is it a camera trick, looks like stacked MDF?
when I make these kinds of enclosures, I put each layer, as I assemble, on the table router and flush it up. Bonus points for inside and outside. That way, it's smooth, and ready for filler-primer to edge seal and prep for paint.
I think it would make it look "more finished" next time.
Another tip I like to use is the roundover bit to knock down the edges all around. small radius is OK. Reason being, I find the edges for painted surfaces wear quick. But roundovers are more durable and forgiving than a hard edge. Another consideration if you ever fix up the front baffle.
Another idea, if you can't flatted the front, is to fake it. You can cut out a facia out of 1/4" MDF and plaster it on the baffle. SHould be OK. Make the sub cutout larger than the OD and it will "countersink" the subwoofer, though noone will really see that. Then you fill the edges to 100% smoothness with bodyfiller, then re-paint. That would flatten up that baffle without spending 5 hours with the belt sander trying to get it flat.