I uploaded the same HD video file (1280×720 progressive) to YouTube and Vimeo. Watch these on full screen, with resolution set to 720p on YouTube and ‘HD Mode’ selected on Vimeo.
This is an unusual video in that it consists only of moving black lines on a pure white background, so compression artifacts are quite noticeable. The raw data (1800 png image files) is over 200MB but artefacts were barely visible in the ~100MB H.264-compressed mp4 file I uploaded to both Vimeo and YouTube. That noted, it seems clear that YouTube has the advantage in terms of quality. Since I’ve read otherwise in informal reports on the web, I’m not sure whether or not the quality might improve with Vimeo Plus, a paid upgrade currently available for the discounted price of US$60/year. Overall, Vimeo offers a calmer, more pleasant user experience than YouTube. The user interface is nicely designed and the online help files are easy to navigate and genuinely helpful. The content and the community are generally more edifying: no denying it there’s a lot of trash on YouTube. The advertising on YouTube is also more obtrusive and distracting. As for pure spec: time to upload and process a video is much quicker with YouTube. Upload is slower with Vimeo and they also make non-paying users wait for at least 30 minutes before the video goes online. Moreover there are weekly limits on total data and only one HD video may be uploaded per week. Third party advertising is less intrusive but lately Vimeo is pushing the paid-subscription fairly hard.
Addendum: Vimeo offers a slightly better experience when browsing from iOS devices in that it’s easier to directly open the 720p viewer. But compression artefacts are still more noticeable, with this file, than with YouTube. Note also that YouTube allows upload of higher resolution videos such as ‘Full HD’ (1080p).