I dunno, there're a whole slew of games made in the pre-NES days that weren't even beatable. They also had no point or purpose. Where you're supposed to go? I dunno. What you're supposed to do? Good question. I'd say those were probably the worst.
As per today's standards though, I'd say Superman 64 is a good candidate, but I always liked being able to pick up a tanker truck, fly way up high and throw it towards the Earth. Apparently since the game has no real physics engine though, the truck maintains a completely singular speed as it sorta speeds towards the Earth. Clipping was the real problem with Superman 64 in my opinion though. I enjoyed the mind-boggling stupidity of this game though.
Worst game of all time though?
Pepper's Adventures in Time, Definitely.
Kidding, only kidding. This game was actually quite fun when I was a kid. Actually, this was one of the first games I played (and it was one of the games that got me interested in video games), and I enjoyed it quite a bit.
I'd say the Final Fantasy series after 6 has become pretty crappy though. SquareSoft's learned the art of milking the cash cow, much like Sony. Who needs quality when you have name recognition?