For what it’s worth, Federer won his first four Grand Slams (2003–2004) without Nadal or Djokovic.
In 2005, he won two of four Grand Slams, losing in the Australian Open semis to former #1 Marat Safin, and in the French Open semis to the soon to be #2 player in the world, Rafael Nadal.
In 2006, he won three out of four Grand Slams, with #2 Nadal beating him in the finals of the French Open.
In the 2007 Rogers Cup in Montreal, Djokovic made his first big mark in tennis when he beat the top three players in the world in the same tournament (Federer, Nadal, Roddick). Federer won two slams, and lost again in the finals of the French Open to Nadal.
In 2008, he won the US Open, but lost in the finals of the French Open and Wimbledon to Nadal, with Djokovic winning the Australian Open.
In 2009, Federer won two slams, and lost in the finals of the Australian and US Open.
That’s still eleven Slam wins and six Slam finals in five years during which Nadal was his chief rival (with Rafa winning six Slams during that period of time).
In 2012, he won Wimbledon, beating Djokovic and eventually regained the #1 ranking, at the age of 31.
If anything, Federer started showing the slightest drop off in his level due to age and the difficulty in playing five set matches when he lost to Soderling, Tsonga, Berdych in the quarters of three Slams between 2010 and 2011, and then blew match points twice in the fifth set against Djokovic in the 2010 and 2011 US Open semifinals. (Federer had won two of the first three sets in both matches against Djokovic, so the matches would have been over in normal tournaments.)
