Biologists have been making considerable progress in identifying members of a special class of genes - those that shape an animal's behavior toward others of its species. These social behavior genes promise to yield deep insights into how brains are constructed for certain complex tasks.

Last month researchers reported on the role of such genes in the sexual behavior of both voles and fruit flies. One gene was long known to promote faithful pair bonding and good parental behavior in the male prairie vole. Researchers discovered how the gene is naturally modulated in a population of voles so as to produce a spectrum of behaviors from monogamy to polygamy, each of which may be advantageous in different ecological circumstances.

The second gene, much studied by fruit fly biologists, is known to be involved in the male's elaborate suite of courtship behaviors. New research has established that a special feature of the gene, one that works differently in males and females, is all that is needed to induce the male's complex behavior.

Social behavior genes present a particular puzzle since they involve neural circuits in the brain, often set off by some environmental cue to which the animal responds. Catherine Dulac of Harvard has found that the male mouse depends on pheromones, or air-borne hormones, to decide how to behave toward other mice. It detects the pheromones with the vomeronasal organ, an extra scent-detecting tissue in the nose.

The male mouse's rule for dealing with strangers is simple - if it's male, attack it; if female, mate with it. But male mice that are genetically engineered to block the scent-detecting vomeronasal cells try to mate rather than attack invading males.

The mice have other means - sound and sight - of recognizing male and female. But curiously, nature has placed the sex discrimination required for mating behavior under a separate neural circuit aroused through the vomeronasal organ.

"It was very surprising for us," Dr. Dulac said.

The gene that was eliminated from the mice is a low-level member of a presumably complex network that governs the inputs and outputs necessary for mating behavior. The most striking behavioral gene discovered so far is a very high level gene in the Drosophila fruit fly.

The gene is called fruitless because when it is disrupted in males they lose interest in females and instead form mating chains with other males. The male's usual courtship behavior is pretty fancy for a little fly. He approaches the female, taps her with his forelegs, sings a song by vibrating his wing, licks her and curls his abdomen for mating. If she is impressed she slows down and accepts his proposal. If not, she buzzes her wings at him, a gesture that needs no translation.

All these behaviors, researchers discovered several years ago, are controlled by the fruitless gene - fru for short - which is switched on in a specific set of neurons in the fly's brain. The gene is arranged in a series of blocks. Different combinations of blocks are chosen to make different protein products. The selection of blocks is controlled by a promoter, a region of DNA that lies near but outside the fru gene itself.

So far four of these fru gene promoters have been found. Three work the same way in both male and female flies. But a fourth selects different blocks to be transcribed, making different proteins in males and in females. This difference, it seemed, was somehow the key to the whole suite of male courtship behaviors.

Last month Barry J. Dickson of the Austrian Academy of Sciences provided an elegant proof of this idea by genetically engineering male flies to make the female version of the fruitless protein, and female flies to generate the male version. The male flies barely courted at all. But the female flies with the male form of fruitless aggressively pursued other females, performing all steps of male courtship except the last.

How does the male form of the fruitless protein govern such a complex behavior? Dr. Dickson and his colleagues have found that the protein is produced in 21 clusters of neurons in the fly's brain. The neurons, probably connected in a circuit, presumably direct each step of courtship in a coordinated sequence.

Fruitless serves as a master switch of behavior, just as other known genes serve as master switches for building an eye or other organs. Are behaviors and organs constructed in much the same way, each with a master switch gene that controls a network of lower level genes?

Dr. Dickson writes that other such behavior switch genes may well exist but could have evaded detection because disrupting them - the geneticist's usual way of making genes reveal themselves - is lethal for the fly. (Complete loss of the fruitless gene is also lethal, and the gene was discovered through a lucky chance.)

Though researchers like to focus on specific genes, they are learning that in behavior, an organism's genome is closely linked to its environment, and that there can be elaborate feedback between the two.

Honeybees spend their first two to three weeks of adult life as nurses and then switch to jobs outside the hive as foragers for the remaining three weeks. If all foragers are removed from a hive, the nurse bees will sense the foragers' absence through a pheromone and assume their own foraging roles earlier. As the colony ages however, there are too few nurses, so some bees stay as nurses far longer than usual.

Gene Robinson, a bee biologist at the University of Illinois, has found that a characteristic set of genes is switched on in the brains of nursing bees and another set in foraging bees. This is an effect of the bees' occupation, not of their age, since both the premature foragers and the elderly nurses have brain gene expression patterns matched to their jobs.

Evidently the division of labor among bees in a hive is socially regulated through mechanisms that somehow activate different sets of genes in the bees' brains.

A remarkable instance of genome-environment interaction has been discovered in the maternal behavior of rats. Pups that receive lots of licking and grooming from their mothers during the first week of life are less fearful in adulthood and more phlegmatic in response to stress than are pups that get less personal care.

Last year, Michael J. Meaney and colleagues at McGill University in Montreal reported that a gene in the brain of the well-groomed pups is chemically modified during the grooming period and remains so throughout life. The modification makes the gene produce more of a product that damps down the brain's stress response.

The system would allow the laid-back rats to transmit their behavior to their pups through the same good-grooming procedure, just as the stressed-out rat mothers transmit their fearfulness to their offspring.

"Among mammals," Dr. Meaney and colleagues wrote in a report of their findings last year, "natural selection may have shaped offspring to respond to subtle variations in parental behavior as a forecast of the environmental conditions they will ultimately face once they become independent of the parent."

A full understanding of these behavior genes would include being able to trace every cellular change, whether in a hormone or pheromone or signaling molecule, that led to activation of the gene and then all the effects that followed. Dr. Robinson has proposed the name "sociogenomics" for the idea of understanding social life in terms of the genes and signaling molecules that mediate them.

The genes discovered so far mostly seem to act in different ways and it is hard to state any general rules about how behavior is governed.

"It's early days and we don't have enough information to develop theories," Dr. Robinson said.

A question of some interest is how far the genetic shaping of behavior exists in people. Larry J. Young of Emory University, who studies the social behavior of voles, said that, in people, activities like the suckling of babies, maternal behavior and sexual drives are likely to be shaped by genes, but that sexual drives are also modulated by experience.

"The genes provide us the background of our general drives, and variations in these genes may explain various personality traits in humans, but ultimately our behavior is very much influenced by environmental factors," he said.

Researchers can rigorously explore how behavioral genes operate in lower animals by performing tests that are impossible or unethical in people. "The problem with humans is that it is extremely difficult to prove anything," Dr. Dulac said. "Humans are just not a very good experimental system."