The noise floor with the engine off I believe is native to the dsp. I can get the master volume up to about 60/66 before i hear any noise with the engine off. With the engine running the alternator whine becomes audible about 38/66 on the master volume.
If I open the circuit between the HU and the dsp the alternator whine goes away but the base system noise remains. the system hiss near max volume is something I can live with as I can get my spl up to where I want it around 52-55/66.
I am starting to think I may have 2 bad HU's because I can disconnect the rcas from the HU and leave them connected to the dsp and the whine stops.
I am using twisted pair type rcas which after doing some google fu last night may be the wrong type for my needs. I need to confirm what type of input the dayton dsp 408 uses. I suspect it is single ended and not differential.
I am going to go digging through my boxes of odds and ends to see if I can dig up some shielded rcas to try as the ones I tested with yesterday were also a twisted pair design.
I am also going to bug my brother to see if I can borrow his HU for a test. His is only a few months old and I know it does not have any noise issues in his car.
Best case at this point is looking like a couple pairs of new rcas of a shielded type might solve the issue and worst case is I may need a new HU.
I will report back as things develop as well as checking periodically to respond to other peoples posts.
thank you all for the help and support. having other minds to bounce ideas off of is helping me keep my sanity as I work through this issue.