I would have liked to have gone swimming yesterday.
A. to have gone swimming
B. to go swimming
C. to had gone swimming
D. to go to swim
E. to of gone swimming
The form "to have VERBed" means that the action took place prior to some other action or time, as in "He is believed to have left," meaning "People believe that he left." So A would mean "Yesterday, I would have liked it if I *had already gone swimming earlier*." People often talk this way because it feels right to mark everything as "past" rather than potentially still coming, but from a strictly logical perspective, it doesn't make sense.
B is correct because the bare infinitive just describes the action without any time attached to it, so it can mean that the swimming and the liking take place at the same time, and/or that at the time the person was thinking that she'd like it, the swimming was hypothetically in the future.
C just isn't English. There is no verb form "to VERBed" -- combining 'to' and past tense marking on the verb.
D isn't idiomatic. We don't "go to swim," we "go swimming," just as we "go shopping," "go skiiing," "go running," etc. The standard pattern "go VERBing" means "to participate in an activity".
E is absolutely not English. Some speakers write "of" in place of "have" because they sound the same in fast speech, but it's considered illiterate usage.
