Intuitionalistically, I don't see a problem with it; there are lots of examples of systems where affixes can (appear to) have inverse effects on words just because you provide a different root. "Inflammable" is a great English example of this—a naïve English speaker will assume it means "not flammable."

Given that, I think this is actually a really natural problem for a language to have, and it would probably be solved by memorizing individual words from context. Over time, I bet the boundary between the two categories would break down and some nouns would adopt augmentative reduplication and some adjectives would adopt diminutive reduplication. This could readily result in many hilarious shibboleths.

If you haven't done so already it'd probably be a good idea to include all reduplicated forms in your dictionary, especially so that nouns acting as adjectives are readily apprehended in their correct form.