We're homeschooling an only child (with no plans for more children). We find more value in one-on-one interaction at this point - Sam is almost a year older than your son, so they are similar in age. We do go to our homeschool park days, but he probably has more fun at the playdays with only one or two other children. I'm not sure if that will change with age.
I'd say that it'd be good to become familiar with your local homeschooling community even if you don't do a whole lot with them right now. Ours has informal field trips and other activities that we sometimes take advantage of, and I think as Sam gets older we'll be more involved.
Also, a lot of kids like being out and about in the world, interacting with people of all ages - which they don't usually do if they are in school all day. I found that homeschooled children are more at ease with adults, elderly people, etc., because they aren't separated from them the bulk of the time.
I just thought of an article that might be good for you to read:
article

Tara