From: eLinux.org
X11 also known as X.org, XFree or XServer is the most used graphics on Linux, at least on desktop.
X11 works in a client-server architecture with communication going via fast UNIX Sockets if local or TCP/IP if remote, it's totally transparent to users.
Protocol is simple and optimized, often used via libraries like Xlib or Xcb (asynchronous). Most common commands like mouse movement and expose events are small so they don't impact too much. Since communication happens using file descriptors (either local unix sockets or tcp) one can easily integrate it in event loops (or main loops) with easy polling with poll(2) or select(2).
It is extensible and can make use of hardware acceleration. With extensions like XRender one can optimize 2d rendering paths. With Xvideo it's possible to use extra planes and also pass through raw YUV data. Others like Composite can allow a composite manager to do actual drawings, possible adding nice effects like semi-transparent windows or shadows. One can use OpenGL (and OpenGL-ES) with X11.
One extension that worth special note is XShm, or the shared memory extension, that is can be used to avoid sending images or other heavy data over the wire. When using XShm images, one just need to send the image identifier and other image parameters, not the image pixels. This, however, have the impact that image creation is more expensive since Linux Kernel needs to zero memory to avoid data disclosure, but it's negligible.
Recent efforts are being made to optimize X11 more and its Linux integration even better, like Kernel Mode Setting, which will avoid flickers during system boot and also reduce X11 server complexity.
Although one can write directly to wire or use low level libraries like Xlib or Xcb, usually one write X11 applications using various helpers and toolkits like GTK, Qt, Evas/EFL, FLTK and more. These libraries are the base of most graphical user interfaces available for Linux, including big projects as Firefox and Thunderbird, Pidgin Instant Messenger, GNOME Desktop, KDE Desktop, Enlightenment Desktop and more. These applications and libraries are primarily developed with focus on X11, so they're always up to date and require no porting at all and receive much more testing. This ready/availability of most famous applications is the strongest point of using X11, even on embedded systems as showed by Maemo, OpenMoko, OLPC and more.
X11 was for a long time neglected from embedded systems tagged as slow and big. While not the very truth, it do have some true facts.
There are benchmarks that show that software-on-software comparison X11 does not add much overhead compared to DirectFB and raw framebuffer: http://profusion.mobi/node/11