We are currently using Oracle's Coherence product as our primary data store behind our website. Coherence provides a scalable, fast data grid that allows for developers to easily access data and allows our data to be versioned so that structures can morph over time.
To date we have deployed our grid following the same pattern we used for our relational databases creating a separate grid for each integration environment. As integration environments grow we need to create more and more grids. Our ultimate goal for environments is to have a virtual stack that development teams can spin up and down as needed, given our current deployment architecture this would greatly multiply the number of grids we need.
A potential solution is to treat our data services more like a shared service that can be used by anyone. In this model there would be one production data grid that provides tested, approved data services to development, qa, and production environments.
The upside is that data consistency is easy to maintain and deployment and management would be easier. The data services would roll through their own deployment model and have internal integration environments that the services team could use for testing prior to releasing a new version of their service. Service upgrades would be independent of all other code (in many cases they already are) and be instantly available to all consumers. The instant availability would also cause the services team to focus on backwards compatibility.
There are also many downsides, most notably non-production applications could negatively impact our production website. Such impacts could be mitigated by ensuring that only production released data services clients are used by applications (our services all have client access libraries), however, that may not be enough to prevent a rouge process from impact end user performance and thus, revenue.
Perhaps there is a hybrid approach based on SLAs, such a hybrid approach could be the use of an internal facing production grid and an external facing production grid. Regardless of the eventual topology our move towards virtualization has an enormous impact on "shared" resources and I believe that as technologists we need to ensure we are taking in the big picture to ensure that small decisions seemingly disconnected from one another do not lead us towards a future in which trap ourselves in a corner. Every day we make small decisions that without regard to the bigger picture lead us towards an architecture that is not thought through and can have long term negative impacts. As a technology manager/leader it is my job to ensure that the big picture questions are at least asked.