I love the idea. I was doing something similar, where I was making some music inspired by a piece of writing a friend had written, she would then write with my music as her inspiration and back and forth, following indirect interpretations of our own work.
I would imagine the soundtrack to her story and descriptions and she would imagine a story to my soundtrack.