That we "outsource" our brains to the environment is the main idea behind the distributed cognition paradigm of research.

For instance, one good primer is Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research, from which is the following excerpt:

In several environments we found subjects using space to simplify choice by creating arrangements that served as heuristic cues. For instance, we saw them covering things, such as garbage disposal units or hot handles, thereby hiding certain affordances or signaling a warning and so constraining what would be seen as feasible. At other times they would highlight affordances by putting items needing immediate attention near to them, or creating piles that had to be dealt with. We saw them lay down items for assembly in a way that was unambiguously encoding the order in which they were to be put together or handed off. That is, they were using space to encode ordering information and so were off-loading memory. These are just a few of the techniques we saw them use to make their dedecision problems combinatorially less complex.

We also found subjects reorganizing their workspace to facilitate perception: to make it possible to notice properties or categories that were not noticed before, to make it easier to find relevant items, to make it easier for the visual system to track items. One subject explained how his father taught him to place the various pieces of his dismantled bicycle, many of which were small, on a sheet of newspaper. This made the small pieces easier to locate and less likely to be kicked about. In videos of cooking we found chefs distinguishing otherwise identical spoons by placing them beside key ingredients or on the lids of their respective saucepans, thereby using their positions to differentiate or mark them. We found jigsaw puzzlers grouping similar pieces together, thereby exploiting the capacity of the visual system to note finer differences between pieces when surrounded by similar pieces than when surrounded by different pieces.

Finally, we found a host of ways that embodied agents enlist the world to perform computation for them. Familiar examples of such off-loading show up in analog computations. When the tallest spaghetti noodle is singled out from its neighbors by striking the bundle on a table, a sort computation is performed by using the material and spatial properties of the world. But more prosaically we have found in laboratory studies of the computer game Tetris that players physically manipulate forms to save themselves computational effort [Kirsh 2001; Kirsh and Maglio 1995]. They modify the environment to cue recall, to speed up identification, and to generate mental images faster than they could if unaided. In short, they make changes to the world to save themselves costly and potentially error-prone computations.

Interpreting the idea a bit more radically, from what I wrote elsewhere:

Information processing doesn't only happen inside brains and computers. The paradigm of distributed cognition studies human societies as information-processing systems, with individuals being parts of the larger system. For instance, the operation of an airliner cockpit's crew has been studied from this perpective [1]. For a flight to proceed without trouble, the different crew members need to be aware of information relating to their areas of responsibility at any given moment. If the crew is experienced and well trained, they'll constantly stay up to date by e.g. simply listening to other crew members converse with flight control. As flight control informs the captain of a new flight altitude, the rest of the pilots begin to adjust the altitude even while the captain is still finishing up the communication. The cockpit functions as a unified system, and relevant information is propagated to wherever needed. Several crew members hearing the same information also allows for error correction. If the message is unclear and the captain can't make out flight control's words, he can ask the others for clarification. The co-pilot answers the captain's query: even though one part of the system has failed to absorb the information received from outside the system, the same information has been stored in another part, which may then attempt to re-send it where needed.

Several other fields have been studied in the same manner, ranging from a child's language learning [2] to creativity [3]. A child doesn't learn language by itself and in a vacuum, but via interaction with adults and older children. Creativity, on the other hand, requires common, shared "idea resources" which individuals may use to come up with their own inventions and then give them back for others to refine further. Another theory of innovation considers inventions to be responses to problems encountered by the community. Things such as bad laws or ineffective ways of doing things show up in community, and are considered problems by its members. This leads the community - the system - into a need state, mobilizing its members to seek solutions until they're found.

One central idea is that social communities are cognitive architectures the same way that individual minds are [4]. The argument is as follows. Cognitive processes involve trajectories of information (transmission and transformation), so the patterns of these information trajectories, if stable, reflect some underlying cognitive architecture. Since social organization - plus the structure added by the context of activity - largely determines the way information flows through a group, social organization may itself be viewed as a form of cognitive architecture.

[1] Hutchins, E. & Klausen, T. (1995) Distributed Cognition in an Airline Cockpit.

[2] Spurrett, D. & Cowley, S.J. (2004) How to do things without words: infants, utterance-activity and distributed cognition. Language Sciences, 6, 443-466.

[3] Miettinen, R. (2006) The Sources of Novelty: A Cultural and Systemic View of Distributed Creativity. Creativity and Innovation Management. Vol. 15, no. 2.

[4] Hollan, J. & Hutchins, E. & Kirsh, D. (2000) Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research. ACM Transactions on Computer-Human Interaction. Vol 7, no. 2.

So, no, probably nothing to be worried about. Just normal human use of the environment.