Fabulous. This is very close to how I think about writing, and also quite close to how I came about writing my own book. It didn't start with a big idea, but a multitude of smaller ones meshed around the most important aspect...character.
With this knowledge in mind it would be interesting to read this book and see how well things fit together. My guess is that the story fits inside a whole perfectly. Here's why: you may not have a big picture right away, but once you have that character that won't leave your head you can trust her or him to find the story.
"It came to pass that I realised, about half way through, that this is how I write best. Letting the writing carry me along and bubble up more ideas – that the act of writing begets ideas, and more writing."
Now we're speaking my language. No plan writing! How we roll.