ICMC98 Paper version E

This paper examines notions of interaction in order to synthesize an approach to the use of computers in the arts which respects the fact that, oftentimes, creative work is facilitated by task environments in which "surprises" can happen. Typically, interface design concerns the rendering of an interaction such that it requires minimal cognitive engagement with the task in question, relying heavily on historically and culturally determined patterns of behavior and cognition. Ill-structured problems (like music composition), however, benefit when the interface presents concepts and interactions in ways that are novel. Computers can be understood as tools for the projection of such an interface when they are conceived as generators of semiotic rather than symbolic ordering frameworks.

Computer music is concerned with the design and composition of acoustical and musical representations made with a computer. In this context, representations articulate domains of interaction into which one might enter while designing musical processes and structures. A representation is projected within a collection of objects and operating principles that define an "interface." Typically, an interface encapsulates a notion of representation that is highly denotative: it is based primarily on some cultural norm on the basis of which a task environment is conceptualized. In such a case, the interface effectively conceals the potential underlying complexity of a system by situating the interaction according to familiar and therefore cognitively redundant patterns of action and observation.

Within the context of facilitating task domains (e.g. driving a car, using a word processor, etc.), such a notion of interaction makes sense: we want to be able to leverage our history of experiences (both cultural and personal) in order to reduce the cognitive strain that might otherwise be involved. In problem-posing task domains, however, like music composition, such a notion of interaction can have an inhibitory effect. One way to encourage creative problem-solving is to problematise the very representations and interactions in the context of which tasks and goals are historically conceived [Truax 1986]. By this means, a human is freed, at least to some extent, from historical patterns of thought and action which often block fresh insight—a necessary precursor to creative thought and action [Smith 1995].

Toward this end, computers can be understood as tools for problematising interaction. Understood in such a manner, the computer becomes a semiotic, rather than a symbolic processor, in that it assists in the postulation of possible, though as-yet non-existent real-world scenarios, rather than referencing already existing ones.

An interaction is an exchange of energy between two differentiated existents, be they imagined or physically realized. An "interface" defines the mechanisms—physical, conceptual, and cultural—by which a set of interactions appropriate to a particular cognitive domain is engendered. An interface can be as simple as a door knob or as complex as an airplane cockpit, or it can delineate a conceptual framework such as a text or a score.

Interface design frequently concerns the rendering of an interaction such that it requires minimal cognitive engagement with the task in question, relying heavily on historically and culturally determined patterns of behavior and cognition [Norman 1986]. A well-designed interface engenders a historical performance in which the particular objects which populate an interaction become virtually invisible. In such a context, 'things' become pieces of equipment, whose manner of appearance is closely correlated with the objective toward which the interaction is directed [Heidegger 1962]. Heiddeger refers to the cognitive orientation which such an interaction instills as "circumspective being."

Circumspective being is that way of being which is elicited by familiar tasks and task environments; it is a way of being in which subject and object are submerged in the apparent immediacy of a task and the actions required for its execution. In the carpenter's experience, for instance, both hammer and hammerer disappear in the immediacy of the task of hammering. Similarly, when writing a document using a word processor, if one is skilled in the use of that word processing software, the functional aspects of the software disappear in the interaction. According to such an understanding of interaction, the human is directed toward the appropriation of an expected performance; an arrangement by which her/his subjectivity is neutralized under the imperative of the signified task. Only when something "breaks down" can the objects which constitute the circumspective unfolding of the interaction begin to appear as things .

One way to retard the rate at which human subjectivity is consumed by the signifying imperative of an interaction, is to engineer such breakdown. No longer a vehicle for replicating historical methodologies, the interface instead orients a hypothesized domain of interaction, thus engaging the generation of an unexpected performance. Such an interaction projects a notion of the 'subject' that is emergent rather than transcendent. As emergent, the subject arises in the moment at which something unfamiliar, or foreign, appears, and, in its labor over the comprehension and synthesis of that something, projects itself toward it [Adorno 1993]. In a sense, the moment of the appearance of an object represents the very commencement of the subject—its beginning, that is, as an activated and activat ing agent, as opposed to a static, a priori , existent.

To create such an environment is to constitute the appearance of things as foreign objects—to cause the appearance of the object of interaction as Other . By situating the object as Other—as something which confronts the subject with its very Otherness—the subject itself comes into crisis.

As a tool for the construction of interfaces, the computer enables the design of mutable constructs by which such a breakdown might be engineered. However, oftentimes (particularly in the increasingly dominant commercial software industry) the search for the proper representation is motivated by performative criteria: what will best assist a human in comprehending and synthesizing her/his experience vis a culturally appropriated understanding of the task domain. The underlying metaphysical assumption is that there exists a "real" world which the computer system references in the domain of interactions that it projects [Anderson 1997]. As a consequence of such design criteria, computers become 'virtual' machines which project representations based on historical imperatives. In this regard, they effect symbolic modes of representation.

A symbol is a special kind of sign, one that is "based on conventional relations between signifier and signified." As Julia Kristeva notes, "in the case of the symbol the signified object is represented by the signifying unit through a restrictive function-relation" [Kristeva 1986]. This function-relation defines an epistemological framework that is both immutable and non-porous; it is, effectively, a black box . It orients an ontology in which interaction is circumscribed by a history of use, and thus prone to that circumspective mode of being in which—as was the case with the carpenter and hammer—both subject and object disappear into the apparent immediacy of the interaction.

Within the context of problem-posing and problem-solving task domains—such as music composition—one frequently wishes to free oneself from the historical constructs according to which those "pre-existing" things have been defined. Here, strongly referential interfaces can actually block creative activity[Smith 1995]. In order to facilitate this kind of working process, one wants an environment in which that which is as-yet unimagined might be formulated and realizable through the projection of an unfamiliar pattern of interactions. In such an environment, historically determined—and therefore circumspective—modes of interaction are thrown into crisis. Interaction thus becomes a context for hypothesizing a domain of actions and descriptions, rather than the simulation of already existing ones.

As a consequence of this shift in emphasis, representations move from being immutable referencing agents—through which particular kinds of objects and artifacts are referenced and understood—to becoming orienting agents wherein the actions by which such objects and artifacts are produced are themselves hypothesized and thrown into question. Such an understanding renders the computer as semiotic rather than symbolic. While the symbolic is concerned with representations of objects through a fixed "function-relation", the semiotic is concerned with the "play of signifiers" [Derrida 1981]. One becomes empowered to construct the very processes through which a world is constituted. As an agent in this empowerment, the computer is no longer bound to its otherwise denotative imperative. Rather it, becomes an epistemological tool for the construction of representations and of domains of interaction within which those representations are engendered.

Such a notion of interaction invigorated so-called experimental music composition of the '50s and '60s. The musical score, for instance, became a context for the problematisation of performance practice: musical notation articulated a set of interactions not accounted for by historical practice. Without the cues which an already well-known notation engenders, the performer could not fall back upon habitual and acculturated patterns of relationship; s/he was forced to re-invent those relations from scratch. Moreover, compositional procedure itself was problematised through the concretization of compositional process (e.g. serialism, chance procedures, stochastic processes).

The electronic music studio introduced not merely a new set of sound producing equipment (such had already been accomplished with instruments like the Theremin), but the very restructuring of compositional activity itself. The relationship between the particularity of a technology and the means by which musical structures might be conceived and realized were understood as mutually determinative [Di Scipio 1997]. With the computer, this mutual determination is deepened: composition, computation, and interaction become deeply intertwined.

In this paper, I propose that computation is, in great part, a context for representation and interaction. As the stipulation of a system which encapsulates a set of decisions or invites a set of descriptions, a computation engenders a domain of interaction. Consider, for instance, two different approaches to modeling the computation of a plucked string sound. The first example is generated within CSound:

In CSound parameter definitions are structured around the properties of amplitude, frequency, initial attack buffer size, and index to a table from which initial attack values are produced. In Modalys they are structured around various types of modal data. Both interfaces orient a particular mode of thought and action. Modalys, however, structures a more problematic interaction than CSound. Frequency, for instance, is only indirectly defined, through the specification of string length, tension, and density. With such an interface, a composer may at first stumble around more than that given with CSound. However, in the process, s/he is likely to discover a considerable variety of principles according to which sounds might be composed. Through the discovery of sound structuring principles, the composer participates in the descriptive framework according to which sound might be generated.

In both of these examples, the referencing agency is fixed to an indexical order, in this case, a "plucked string" sound. An index conditions the activity of a human actor according to an appropriated signifying order, thus delineating a domain of actions and descriptions according to a pre-existing domain of objects and things.

An artifact, by contrast, begins as a design context in which the "subject" (i.e. the "goal") of activity is the product of that activity itself. It thus circumscribes a domain of actions and descriptions according to an emergent order. Human performance is contingent on the emergence of a particular order and is not, as such, bound to an already fixed presentation [Simon 1969].

Such an interface manifests directive comportment in that it includes the particular involvement of a human actor. Many computer music systems project such an interpretation of the involvement of the composer. The following set of instructions, taken from Herbert Brün's SAWDUST [Blum 1979], reflects one such approach:

Here six "elements" (e1 through e6) each define sequence of integer samples all of which have the same amplitude. For instance, e1 consists of 100 samples all with an amplitude of 200 while e2 is defined as a sequence of 70 samples all having an amplitude of -18000. A waveform is created when 2 or more elements are "linked". More complex waveforms result from operations such as MINGLE. Such operations (others include MERGE, VARY, etc.) can be applied and combined such that there is no differentiation, from the point of view of compositional procedure, between operations which determine timbral features and those which determine aspects of musical pattern and form. Moreover, no external sound model is referenced; sound structure is predicated upon the very principles by which sound is generated within the computer: patterns of amplitude values that are supplied to the DAC.

Many other music software systems take this approach. I. Choi's Chua's Oscillator is a physically based model whose dynamic behavior is defined by a set of three ordinary differential equations plus a non-linear function [Choi 1997]. With K. Corey's Ivory Tower , waveforms are generated through the composition of interactions of self-modifying code bins [Corey 1997]. In Foo, each sound is understood within the context of the larger musical context in which it is deployed [Eckel 1994]. M. Hamman's ResNet is an experiment in signal processing wherein signals are generated through networks of dynamically configurable networks of delays and feedback multipliers [Hamman 1994].

In each such system—along with the many others—sound/music structure and computation are closely intertwined within the context of a particular idiosyncratic framework. Computation and musical process are mutually determinative, and the notion of a computer system is tightly coupled with a particular compositional project. Interaction becomes a form of mediation—a process in which an otherwise unperturbed cognitive frame is broken in order to introduce the possibility of an elective subjectivity. Such a notion of interaction presupposes that for every object, there is a particular "I" with respect to which it arises, and that "the real subject matter is not exhausted in its purpose, but in working the matter out" [Hegel 1967].

[16] Smith, S. M., 1995. "Fixation, Incubation, and Insight in Memory and Creative Thinking," in The Creative Cognition Approach , ed. S. M. Smith, T. B. Ward, and R. A. Finke. Cambridge, MA: The MIT Press. pp. 135-156.