This phrase is used four times in Scripture. Habbakuk 2:4, Romans 1:17, Galatians 3:11, and Hebrews 10:38.
Habbakuk, of course, coined the phrase, so to speak. More accurately, these are the words he used to convey what the Spirit was saying through him. A close look at the context of Habbakuk shows us the backdrop of the phrase. Judah is in a state of rebellion against God. Habakkuk cries out asking God why He lets this continue. God answers Habakkuk's prayer by telling him that Judah's sin will not go unpunished. He is raising up the Babylonians, who will come into Judah and carry them away captive.
But for Habakkuk this raises more questions. Surely God would not allow a nation more wicked than Judah to be the executers of His judgment on them. God responds to this prayer of Habakkuk by saying that the Babylonians are coming to execute judgment on Judah, but that they in turn will also be judged by another nation that God will raise up. As for Judah, they are to hear the proclamation of judgment, repent, and if they will not repent, then to wait for that judgment.
And then we come to the verse that we started with.
"Behold, his soul which is lifted up is not upright in him: but the just shall live by his faith."
These words are spoken in a context of warning and assurance. Warning, because when Judah thought that they were strong, they were found to be weak. Assurance, because when those who trusted God for their strength were found to be strong. Not by their own merits, but by faith in God who is their Strength and Sustenance.
May these be words of warning to us, that we may by faith cast out all prideful thoughts of human abilities and trust in God who makes us just and alive. For it is by faith that the just will live.