First one that comes to mind is Candy O by The Cars. Concludes and ruins a perfectly good album side which I actually like more than that of the first album (which is honestly saying a lot).
The Smiths' "Meat is Murder" and Sly Stone's "There's a Riot Goin' On" are both dreary. [EDIT: Nipped at the wire. Good thing I mentioned a second one!]
Beach Boys - Pet Sounds Not bad per se, but I consider it to be weaker compared to all of the other songs.
Gotta say I disagree with "Invisible Touch" & "Candy-O". In fact, IMO, there IS no worst song on Candy-O. Anyway, the two which come to my mind are "Mental Notes" by Split Enz & "Ship of Memories" by Focus. The Split Enz track is cute enough but it's also pointless. At 30 seconds, it seems like a P.S. to the album instead of an actual piece of music. I think the Focus track similarly feels like a P.S. but with the additional liability of sounding like a child wrote and played it even though it's clearly intended to be serious.
Ok, this one's controversial, but... Steely Dan - "The Royal Scam" I like every other song on the album better than that one.