Honestly, the reason I use youtube is because I refuse to actually spend any money on doing this.
As for the portability thing, I do understand where you're coming from...but I had kind of assumed people know how to download youtube videos as mp3s? Would it make things better for the audience if I had a mediafire folder of the audio recordings or something?
I agree with Wilzuma, but I think it goes further then that. I can do all that, I just can't be bothered to do it. There is other material out there for me that is prepacked and I have more options then I can fill my day with. And I don't think I'm the exception.
If you want listeners outside the group of people that can listen to youtube at work or at home then you need to make the podcast easily available though the classic mediums.
If you are just doing it for fun and youtube is just the gravy, then it's not a problem. Just keep enjoying it.
Cheers mate.