On solo-game (one to one):
Overall, I find it difficult more difficult to play/master, because it is intense and requires a constant concentration from both the DM and the player. The player is constantly under the spot light and has to come with idea and decide the next action. And if he cannot, the MJ has to "save" the game by guiding the player to the next scene. When there is more players, there is more "brain power" to drive the game and as DM, you only have to deal with the consequence of the player's choice instead of having to find ways to guide them (ideally without making it look too much "rail road").
It is probably because most of the game I master are focussed on roleplaying and investigation that I have this feedback. If it is a dugeoncrawler, it good be quite fun as the player has to really optimise and think strategy when confronting several monsters at the same time.
Also, the player has to remember everything, as nobody else will be there to remind him the critical bit of information he needs to understand/complete whatever task/situation is at hands.
There is another issue with solo game: they have to be tailor-made to the character to be sure that he has the skill/competencies/spells to get through it.
Two to three people: that's good numbers to have a mixture of abilities, different opinions and fast-pace action, without the downside of the solo-game.
Four people: it is my usual group size, you have a good mixture of abilities, some being redundant, making sure that if there is a critical test/roll to make, on will likely succeed, ensuring smooth transition to the next scene. It is also very interesting in term of RP since you can have stand off of 2 vs 2, and players have to sort out themselves how to weight one way or the other.
You might start to see some drifting as two players might engage in a non-game-related discussion, but that's easy to catch and bring back on track.
Five people: occasionnaly, I run game with 5 players, and I don't enjoy it has much. Drifting becomes more common, having to repeat and describe several times the same element increases, overally slowing down the pace of the game.
Above: Never tried, nor want to.