Right. I need help with this.
Quick life story - music teacher mother (classical), piano when young, several years of clarinet, sight reading, exams etc. Gave up. Couldn't cope without playing an instrument so went to guitar at 13, eschewing the theory and all that. Lots of metal. Now my tastes are extremely broad, but mainly it's metal and free -> avant jazz.
What I really struggle with is getting ideas from my head and into some kind of structure. I want to be able to come up with a plan, a storyboard of sorts, that I can expand upon, give to other musicians etc.. What I envisage is some kind of avante metal fusion. Structure verging on chaos, with groove. I love what Zorn is doing with Electric Masada, I love what Uri Caine gets up to with his interpretations of Mahler.
Where do I start?
Heh. This is a pretty advanced (and open) question, Dan.
"I want to understand the universe. Where do I start?"
I'd say, obviously, the key to improve in jazz is typically a) falling into the rhythm, and b) having a concise melodic idea. With metal, you're not going to be as constrained with using syncopated lines, but those really do help with a groove. Having a bass line grooving over some tonal element is probably the main hallmark, and maybe where one would "start." I guess it wouldn't even have to be strictly diatonic, but you'd need to have a pretty clear idea where to go together (with the bass and/or other harmonic instruments) from the outset.
Hmm. So -
A) decide on key/harmonic backbone. (Major? Minor? Modal?) Juxtaposing different tonic elements can be interesting, but you have to be smart, and careful.
B) develop a theme. Some sort of melodic expression. Even a riff can carry this. Or perhaps the theme is more rhythmic in nature, a la Meshuggah. Either is fine, but you want some sort of overall idea. Even if you just sprinkle it in here and there like sugar on a donut.
C) for me, a good bass groove is next. It's almost a key element for anything improvisational. Like with jazz, the structure of the bass groove is probably not extremely important, but rhythm and key structure are.
4) know where the bass is going is going, though! If you throw too many non-scalar notes in there, you can introduce perhaps unpleasant dissonance, or make the elements you're using too chromatic. Same thing applies with time and rhythm. Now, if you want something like that, that's fine, but realize you'll lose any real sort of melodic ideas if you push that envelope too far (see 20th century classical music for an example.) Is atonality what you want? Then go for it. But I get the idea you just want to flirt with it. So, in my opinion, the clever use of passing tones, to hint at some odd modality, or perhaps even chromaticism, would be in order. And you can even use all 12 tones over the course of a phrase or run, as long as you do it smart.
My best ideas on it. I get what you want to do, because I play with that sort of thing myself occasionally. And sometimes I envision some crazy, technical metal-acoustic-soft jazz improve monster band.