It's the latter. As for which form to use with a definite time adverbial, I think that without any other temporal constructs in the same clause, either goes, simply because non-fixed reference forms are — hence the name — unmarked for time. So if I say:
orothis-wag̃o-LOC buwayesterday jabsuhole urube-∅-AOR-g̃-3s
it can't really mean anything other than "there was a hole here yesterday". The same thing goes for the ~past~, of course. I mean, essentially, the feature of the fixed reference forms is that they are marked for tense. The tense just happens to be unspecified. So they can either double up on other explicit time marking or act as a sort of discursive temporal switch reference marker, and they're only obligatory in the latter case. Probably there is also a thing where, if either would work, fixed reference forms are preferred for perfectives and non-fixed forms for imperfectives. It seems like it would easily acquire that particular implicature.

The important question I don't have a good answer for yet is what that means for time marking across a paragraph, or between speaker turns. I figure that fixed-reference forms are used whenever it needs to be established what time is being talked about. For instance, if you need to establish that you're *not* switching back to the default. So if you say the above sentence, then whoever you are talking to might reply:
urukureal je?Q pasuthing gaNEG toresee-ne-PST-wa.-1s
"Really? I didn't see anything."

The function of the fixed form there is to anchor what he is saying in your time context. The aorist would mean "Really? I don't see anything.", because the present is the default time context. And so on.