Could someone explain proper usage of infinitive phrase as modifier please .

I concieved three very simple examples .

1. To build his muscles, he goes to gym daily.
2. He goes to gym daily to build his muscles.
3. He ,to build his muscles, goes to gym daily.
4. He goes to gym daily, to build his muscles.

I can feel that first two (1,2) are correct and the role played of the infinite phrase (underlined) is of adverb.

But the question is that an infinitive phrase can play a role of modifier at the end of the sentence or in the middle as shown above ?

Need authentic comments please.