James Gleick's book on information theory is very enjoyable to read as it provides intuitive explanations for this very tricky and information-dense field. Gleick goes through the history of information expounding on different personalities starting with Charles Babbage and Ada Byron, the Countess of Lovelace, progressing through Alan Turing and moving onto more present day actors, such as Claude Shannon at Bell Labs and Charles Bennett at IBM. Most importantly, beyond the personalities, the book gives a good intuitive explanation of information theory. In particular, it explains how information can be measured by -log p where p denotes the rarity (i.e. "surprisal") of an event (getting a particular "symbol," such that a symbol from a larger alphabet would carry more information than one from a smaller one). It also has a nice explanation of the Kolmogorov complexity of a message (as the shortest possible computer program that can describe the message) and a good accounting of a variety of logical head-twisters such as Russell's recursion and Gödel's paradox. Finally, the book delves into name spaces and the importance of very simple printing interventions such as alphabetization in coming up with information organizing principles. I would highly recommend this book to anyone who desires to learn about information theory. It is both entertaining and quite educational.
Another interesting tidbit: Charles Bennett at I.B.M had some interesting connections to biology. He developed a subject called "logical depth." It seems that logical depth quantifies information but does so in a way that purely random information or purely "uniform" sequences are down weighted and things that get most strongly up weighted are those "in the middle," with a non-trivial but hidden pattern in them. I thought this was actually a kind of good match to what people are interested
in Genome Annotation. Bennett also goes into how chain letters can mutate and evolve like biological sequences.
http://www.amazon.com/Information-History-Theory-Flood/dp/1400096235
The Information: A History, A Theory, A Flood
James Gleick
Did a review (http://www.amazon.com/gp/cdp/member-reviews/A255VTLSSZIVT9/ref=cm_cr_dp_auth_rev?ie=UTF8&sort_by=MostRecentReview)
My tag: infoth0mg
Related links:
http://aminotes.tumblr.com/post/4872331472/what-defines-a-meme-james-gleick-our-world-is-a
http://en.wikipedia.org/wiki/Minimum_description_length
http://en.wikipedia.org/wiki/Jacquard_loom
