A few thoughts here:
It doesn't seem to be the case that if a belief about the universe is, if
non-scientific, untrue. E.g., suppose someone thought that the Bible
teaches that the universe had a beginning, despite (suppose this is last
century or early this century) the steady-state theory being in vogue. Even
though there belief was non-scientific, indeed, perhaps even unscientific,
it was true, right? (Was it justified or warranted? I'm not sure.)
Also, many general beliefs about the material universe -- that it exists (as
opposed to being an illusion, as in some Eastern religions, or the
challenges of skeptics), that it has a past, that we can trust our sensory
experiences and memories of it, that the future will suitably resemble the
past, that we can generally (though obviously not always!) trust what others
tell us about it -- these beliefs are scientific in the sense of being
required to do science, but they are not (unless one accepts circular
arguments) justified by science. But presumably that doesn't mean that
they're unjustified. (I would call these beliefs basic, foundational
beliefs [at least in part; evidence can support them, but our trust in them
goes way beyond what any sort of non-circular evidential argument offers],
and I [and presumably you, and nearly everyone] would deem them very
properly basic.)
But I don't think you meant to deny any of this -- none of these things are
what you mean by "investigation of the natural universe", right? And I
think very much in general, particularly when one is discussing some
detailed hypothesis about physical reality, you're quite right.
John