Yep, have to go with Rainbow Road in the N64 version. I had no problem with the steering, and the jump-over-the-wall trick was a bit fun if you did it properly, but the track was just way too long. Like, horrendously long. Usually just turned the game off when I reached the end of that cup.
Now, if you look at that same track in Mario Kart 8, it flips to one of my favourites. Going around it once is amazingly hectic. I've had some of my best online races on that Rainbow Road.