I think that approach might walk you into the domain of philosophical languages. In order to subdivide the space you need to establish boundaries, which then must become explicit vowels. Any variability is then merely allophony that can be mapped to real phonemes, and the whole point of it is lost. The phonetic realization might be within any arbitrary, non-overlapping subdivision of the possible vowel space, but those still can (and must) be labelled for communication to be meaningful. Like with any oligosynthetic language, you'll find you run out of short words sooner than you'd like.

The original formula of using vowels as a kind of coordinate system for continuous traits (which I think was intended as a conceptual broadening of the bouba/kiki effect) is fundamentally a lot more interesting. You're right that it wouldn't be very possible/desirable to transcribe communication with exacting fidelity in an alphabet like the IPA, but you could use floating-point numbers for academic purposes, and a non-discrete scheme for a native script, such as the angle and length of a single line, or the intersection point of two lines—ideas that sometimes manifest in featural abugidas.

As to what to use these marking systems for, you're probably right that environment is key. Maybe there's a special set of adjectival morphemes, CVC, where the consonants indicate the qualities on each axis, and the vowel indicates the value. For example, "g" = size and "t" = taperedness, so "gɯt" indicates that something is big but not tapered, but "tɯg" (a synonym of "gat") means small and sharp. Insert rules about correct ordering (and the meaning of null consonants) per the chef's taste.