{"id":14,"date":"2006-04-27T16:38:19","date_gmt":"2006-04-28T00:38:19","guid":{"rendered":"http:\/\/www.tranzoa.net\/~alex\/blog\/?p=14"},"modified":"2006-04-27T16:38:25","modified_gmt":"2006-04-28T00:38:25","slug":"expected-characters","status":"publish","type":"post","link":"https:\/\/www.tranzoa.net\/~alex\/blog\/?p=14","title":{"rendered":"Expected Characters"},"content":{"rendered":"

After reading all the Buffett yearly letters<\/a> it sure seemed like a good idea to experiment with programs to help read repetitive stuff – stuff containing lots of boiler plate, for instance.<\/p>\n

So …<\/p>\n

There are lots of ways to address the issue. I did something with assembling a large tree that could represent a Markov Model of the text. That took a lot of memory and a lot of CPU. And, things get very interesting when it comes time to prune the tree. More work to be done with that.<\/p>\n

Meanwhile, there’s a really quick, easy way to play with this sort of thing:<\/p>\n

Starting with, say, 10 documents, ordered in some way, “read” the first 9. Build a memory of sub-strings of the 9 documents. Then display the 10th with each character rendered in a font size that represents how well it’s expected to be at that position in the document. In particular, render “surprising” characters big and “expected” characters small.<\/p>\n

Well, without going in to details of the current code in random_byte_strings_in_file.py, here is an example paragraph from the Buffett 1999 letter processed with data from the ’77 through ’98 letters:<\/p>\n

\"Paragraph<\/p>\n

Ugly.<\/p>\n

But, nice try.<\/p>\n

Hmm. Well, let’s note how the script works:<\/p>\n

It stores a big dictionary\/hash keyed by unique strings. The hash’s values are the number of times the string has been found in a document.<\/p>\n

For each document the script reads, it picks lots of random characters in the document.<\/p>\n

For each random character, it remembers strings in the document that include the character. It does this by, first, storing the character as a string. Then it tries to extend the string on both ends, continuing to store strings until either a new string is stored or some limitation is reached.<\/p>\n

To process the last document, the script uses the string:value hash table to assign a numeric value to each character of the document. I’ve experimented with several ways to do this. They all lead to words that look like kidnappers created ’em.<\/p>\n

There are, of course, a gob of ways to make the output more visually appealing, if not usable.<\/p>\n

But, what the heck. Another interesting thing I’ve not done is to convert the output font sizes to audio samples and listen to the thing.<\/p>\n

One wonders, for instance, whether the ear can distinguish between various texts. But, then, recognizing the differences between writings of, for instance, one person and another, is an old story – and there sure are better ways to do so than this rather round-about scheme.<\/p>\n","protected":false},"excerpt":{"rendered":"

After reading all the Buffett yearly letters it sure seemed like a good idea to experiment with programs to help read repetitive stuff – stuff containing lots of boiler plate, for instance. So … There are lots of ways to … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-14","post","type-post","status-publish","format-standard","hentry","category-programing"],"_links":{"self":[{"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=\/wp\/v2\/posts\/14","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14"}],"version-history":[{"count":0,"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=\/wp\/v2\/posts\/14\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tranzoa.net\/~alex\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}