Literature, or more precisely 'literary fiction' has no real set of rules. There is, though there is the vague notion that it should be about people coping with something, and not the event itself. Literary fiction defies classification. If you can explain to someone what kind of story you're reading (or writing) in a sentence, then you, my friend are likely in some genre or another. Literary fiction defies cliché. The literary author can take something cliché, like the amnesiac story arch, and manage to make it his own.
Literary fiction exists to break the rules, by sacrificing technical perfection to achieve a voice that is the author's own. You could compare two literary authors and tell the difference, almost instantly from one to the other. On the reverse, comparing two 'bestsellers', or, one, and a piece of dime-store smut, and that difference in voice most assuredly goes the way of the dinosaur.
Literary fiction also tends not to offer closure, in the traditional sense, but rather a pragmatic, functional ending where the players continue their lives, better or worse, for the experiences they have encountered in the story.
The literary author is recognized, sometimes more for his stylized destruction, than anything. When looking at Raymond Carver, for example, it is impossible not to notice how blunt and cold his descriptions are, or how his characters relate to one another. But at the same time, that is what makes a Carver story so distinct to Carver. Almost everything that does not need to be said is cut away, and what is left is barebones story. When this is compared to someone like Tolkien, the difference is immediate. Tolkien spends pages and pages describing a field, or a sword, or a lake, or a mole on someone's ass.
Some literary authors take this a step further and eliminate whole sections of the language. Adverbs, and pronouns are common enemies of this writer, for example. An adverb is something that the literary author destroys to create voice. People can do things stupidly, we all would agree, but the literary author cares more about what that looks like, or smells like, or sounds like. What is stupidly really saying about this action, or this person? The personal pronoun “I” is often eliminated from first-person writing because it can act as a speed bump, every time a reader hits it, he is removed from the story and the disbelief the author has created for him. Cliché expressions, such as 'wringing your hands' may also be cut away in the name of explaining them physically.
You might also see these writers use a lot of comparison. Everything is something else. The moonlight is the color of ice cubes, maybe. This puts the reader more into the story, and also makes their picture unique to them. The writer might go further though, and incorporate a body part. A foot, or penis or something of that sort, so that the color of ice cubes effects a specific skin tone. This might also further differ the reader's picture.
Physical descriptions of the characters are also kept to a minimum by some literary authors. All of this is to keep the reader picturing. The literary author has no need to describe a person's whole wardrobe, or every curl of their hair, or the exact shade of green their eyes might be.
So, that about covers it. Wouldn't you say?