It was time to scan the B2 todo.doc idea file (hashes of which are published to alt.security.keydist for lack of a better, public place to dump ’em).<\/p>\n
One of the odd items in B2 was a note of curiosity about what words would change in the future. Specifically, what words would be shortened because they are used a lot? And what words would drop out of use because they are too short for their own good? I speculated that words that are too short are pompous, fuddy-duddy words, scheduled to go out of use, and words that are too long are hip-happening words, scheduled to be replaced by shortened forms of the word (“something” becomes “sum’em”, “about” becomes “bout”, “OK” becomes “K”).<\/p>\n
The thing is, words that are in common use are short, e.g. “I the you me”. And rare words are usually big words. That makes sense. Huffman type compression is a natural phenomenon.<\/p>\n
<\/dir><\/p>\n
There are lots of word list out there. I turned to Wiktionary’s<\/a> TV script word frequency list.<\/p>\n And, from a while back, I just happened to have a copy of all the audio word recordings from Merriam Webster<\/a>.<\/p>\n Now, the durations of these recordings are not a very good indication of the duration of the words, but it’s a start. (I considered using a phoneme count from Wiktionary’s pronunciation guides)<\/i><\/p>\n If you sort the words by word-count and give them each an index corresponding to where they are in the list, and do the same for durations, you should be able to figure out which words have very different indices\/rankings in the two sorted lists.<\/p>\n The sorted, absolute-value results should order the words in “stability”. That is, the words at the top of the list should either be too-short words, or too-long words.<\/p>\n Here they are, grouped:<\/p>\n Too-short, fuddy-duddy words:<\/p>\n Too-long, hip-happening words:<\/p>\n Well, it was a thought, anyway.<\/b><\/p>\n Some data and code:<\/p>\n Here is a selection of the results in “stability” order from least to most stable:<\/p>\n Here is the code:<\/p>\n It was time to scan the B2 todo.doc idea file (hashes of which are published to alt.security.keydist for lack of a better, public place to dump ’em). One of the odd items in B2 was a note of curiosity about … Continue reading \r\n\r\n aught 6 15795 4074 37 0.99775\r\n kiddy 6 15795 4331 93 0.99434\r\n chirp 6 15795 4516 170 0.98965\r\n pomp 6 15795 4599 220 0.98661\r\n clunk 6 15795 4844 390 0.97626\r\n teat 6 15795 4925 473 0.97121\r\n hera 6 15795 4965 524 0.96810\r\n peat 7 15269 4161 56 0.96329\r\n pic 7 15269 4163 57 0.96323\r\n debit 6 15795 5040 607 0.96305\r\n wort 6 15795 5101 678 0.95873\r\n cud 6 15795 5158 750 0.95435\r\n deft 7 15269 4574 205 0.95422\r\n yolk 7 15269 4687 267 0.95045\r\n berth 7 15269 4778 338 0.94612\r\n putter 7 15269 4804 352 0.94527\r\n aright 6 15795 5282 925 0.94369\r\n airy 7 15269 4868 414 0.94150\r\n capper 6 15795 5345 1013 0.93834\r\n heller 7 15269 4934 488 0.93699\r\n amuck 7 15269 4973 542 0.93371\r\n heady 7 15269 4989 552 0.93310\r\n lite 7 15269 4990 554 0.93298\r\n punt 8 14787 4359 107 0.92967\r\n alum 7 15269 5051 619 0.92902\r\n bauble 8 14787 4426 132 0.92815\r\n pellet 7 15269 5070 637 0.92792\r\n breadth 8 14787 4554 188 0.92474\r\n beagle 8 14787 4554 188 0.92474\r\n erie 8 14787 4702 278 0.91926\r\n buoy 7 15269 5176 785 0.91891\r\n bap 7 15269 5189 802 0.91788\r\n simp 7 15269 5203 824 0.91654\r\n wisp 6 15795 5519 1378 0.91612\r\n whist 6 15795 5537 1402 0.91466\r\n ardent 8 14787 4821 372 0.91354\r\n beech 8 14787 4842 385 0.91275\r\n heath 8 14787 4852 399 0.91189\r\n dewy 8 14787 4859 404 0.91159\r\n\r\n<\/pre>\n
\r\n\r\n relationship 3880 543 10302 15572 -0.91352\r\n responsibility 1157 1262 11876 16298 -0.91219\r\n apologize 1932 885 10730 15895 -0.91153\r\n themselves 1117 1287 11861 16296 -0.91048\r\n affair 865 1527 22434 16425 -0.90314\r\n investigation 951 1433 11692 16260 -0.89905\r\n sharon 1750 957 10540 15751 -0.89820\r\n outside 4260 514 10052 15284 -0.89782\r\n situation 3359 595 10107 15354 -0.89695\r\n opportunity 1601 1011 10526 15742 -0.89423\r\n experience 1767 952 10388 15638 -0.89164\r\n mia 1105 1295 10838 15953 -0.88910\r\n information 3063 635 10028 15251 -0.88815\r\n realize 4114 527 9922 15110 -0.88641\r\n explanation 847 1545 11186 16119 -0.88337\r\n surprise 3439 585 9908 15091 -0.88158\r\n otherwise 1499 1051 10273 15537 -0.87922\r\n suppose 2952 652 9838 15009 -0.87234\r\n security 2237 817 9960 15164 -0.87133\r\n necessary 1215 1219 10288 15561 -0.87005\r\n conversation 2266 808 9919 15107 -0.86843\r\n besides 3355 596 9738 14868 -0.86731\r\n absolutely 4704 482 9648 14724 -0.86576\r\n ridiculous 1943 881 9941 15138 -0.86570\r\n downstairs 1071 1327 10333 15596 -0.86534\r\n girlfriend 2325 789 9846 15014 -0.86397\r\n eventually 985 1400 10380 15634 -0.86303\r\n ourselves 1525 1041 9980 15200 -0.85934\r\n sometime 871 1518 10440 15680 -0.85836\r\n someplace 1165 1253 10132 15384 -0.85712\r\n grandfather 1006 1377 10191 15444 -0.85292\r\n sacrifice 472 2244 11921 16308 -0.85063\r\n recognize 909 1470 10225 15486 -0.84959\r\n psychiatrist 463 2272 11731 16269 -0.84648\r\n necessarily 478 2222 11402 16186 -0.84459\r\n champagne 1085 1316 9998 15220 -0.84315\r\n understand 16724 191 9299 14020 -0.84133\r\n meantime 701 1741 10275 15540 -0.83572\r\n imagination 459 2284 11062 16060 -0.83300\r\n\r\n<\/pre>\n
\n
\n\r\n\r\n; Thu Oct 30 16:38:35 2008\r\n; counts=15795 durations=16428 unique_counts=2110 unique_durations=5812\r\n\r\n; Word count cnti dur duri offness\r\n\r\n aught 6 15795 4074 37 0.99775\r\n kiddy 6 15795 4331 93 0.99434\r\n chirp 6 15795 4516 170 0.98965\r\n pomp 6 15795 4599 220 0.98661\r\n clunk 6 15795 4844 390 0.97626\r\n teat 6 15795 4925 473 0.97121\r\n hera 6 15795 4965 524 0.96810\r\n peat 7 15269 4161 56 0.96329\r\n pic 7 15269 4163 57 0.96323\r\n debit 6 15795 5040 607 0.96305\r\n wort 6 15795 5101 678 0.95873\r\n cud 6 15795 5158 750 0.95435\r\n deft 7 15269 4574 205 0.95422\r\n yolk 7 15269 4687 267 0.95045\r\n berth 7 15269 4778 338 0.94612\r\n putter 7 15269 4804 352 0.94527\r\n aright 6 15795 5282 925 0.94369\r\n sometimes 5596 431 10772 15920 -0.94179\r\n airy 7 15269 4868 414 0.94150\r\n capper 6 15795 5345 1013 0.93834\r\n heller 7 15269 4934 488 0.93699\r\n amuck 7 15269 4973 542 0.93371\r\n heady 7 15269 4989 552 0.93310\r\n lite 7 15269 4990 554 0.93298\r\n punt 8 14787 4359 107 0.92967\r\n alum 7 15269 5051 619 0.92902\r\n bauble 8 14787 4426 132 0.92815\r\n pellet 7 15269 5070 637 0.92792\r\n breadth 8 14787 4554 188 0.92474\r\n beagle 8 14787 4554 188 0.92474\r\n erie 8 14787 4702 278 0.91926\r\n buoy 7 15269 5176 785 0.91891\r\n bap 7 15269 5189 802 0.91788\r\n simp 7 15269 5203 824 0.91654\r\n wisp 6 15795 5519 1378 0.91612\r\n whist 6 15795 5537 1402 0.91466\r\n ardent 8 14787 4821 372 0.91354\r\n relationship 3880 543 10302 15572 -0.91352\r\n beech 8 14787 4842 385 0.91275\r\n responsibility 1157 1262 11876 16298 -0.91219\r\n heath 8 14787 4852 399 0.91189\r\n dewy 8 14787 4859 404 0.91159\r\n apologize 1932 885 10730 15895 -0.91153\r\n themselves 1117 1287 11861 16296 -0.91048\r\n catty 8 14787 4892 433 0.90982\r\n contra 6 15795 5575 1485 0.90961\r\n droop 6 15795 5579 1502 0.90857\r\n gluck 6 15795 5582 1512 0.90796\r\n yammer 8 14787 4921 468 0.90769\r\n affair 865 1527 22434 16425 -0.90314\r\n cherub 6 15795 5630 1603 0.90242\r\n inca 8 14787 5005 567 0.90167\r\n ogle 7 15269 5385 1082 0.90084\r\n millet 7 15269 5385 1082 0.90084\r\n bey 6 15795 5642 1636 0.90041\r\n creak 8 14787 5025 588 0.90039\r\n bunt 8 14787 5032 593 0.90009\r\n amah 6 15795 5644 1645 0.89987\r\n whet 9 14316 4394 119 0.89912\r\n investigation 951 1433 11692 16260 -0.89905\r\n wrought 8 14787 5050 617 0.89862\r\n sharon 1750 957 10540 15751 -0.89820\r\n dietrich 6 15795 5655 1677 0.89792\r\n outside 4260 514 10052 15284 -0.89782\r\n dour 7 15269 5412 1140 0.89730\r\n situation 3359 595 10107 15354 -0.89695\r\n velour 6 15795 5660 1694 0.89688\r\n hatter 6 15795 5665 1711 0.89585\r\n conk 8 14787 5109 687 0.89436\r\n opportunity 1601 1011 10526 15742 -0.89423\r\n cooker 9 14316 4579 210 0.89358\r\n ilk 9 14316 4620 231 0.89230\r\n sot 7 15269 5455 1227 0.89201\r\n experience 1767 952 10388 15638 -0.89164\r\n batty 9 14316 4682 264 0.89029\r\n mia 1105 1295 10838 15953 -0.88910\r\n thomson 6 15795 5720 1826 0.88885\r\n baroque 6 15795 5724 1831 0.88854\r\n eth 6 15795 5725 1833 0.88842\r\n information 3063 635 10028 15251 -0.88815\r\n tusk 7 15269 5487 1294 0.88793\r\n anima 7 15269 5496 1312 0.88683\r\n realize 4114 527 9922 15110 -0.88641\r\n vigor 8 14787 5199 820 0.88627\r\n brusque 7 15269 5513 1349 0.88458\r\n corker 8 14787 5234 867 0.88341\r\n explanation 847 1545 11186 16119 -0.88337\r\n woolly 8 14787 5237 872 0.88310\r\n demur 6 15795 5759 1925 0.88282\r\n coolant 6 15795 5760 1928 0.88264\r\n peeve 6 15795 5765 1939 0.88197\r\n.\r\n.\r\n. Somewhere in the middle of the list...\r\n.\r\n.\r\n cockpit 42 8131 6575 4544 0.23818\r\n dummy 166 4147 4858 401 0.23814\r\n hut 169 4103 4809 356 0.23810\r\n home 22901 156 6450 4073 -0.23805\r\n shorthand 28 9689 9285 13988 -0.23805\r\n twirl 34 8949 6793 5397 0.23805\r\n free 5433 440 6531 4368 -0.23803\r\n aspire 25 10179 7094 6677 0.23800\r\n sheila 352 2688 7102 6704 -0.23790\r\n component 40 8299 8687 12538 -0.23779\r\n pathetic 1115 1289 6749 5247 -0.23779\r\n tune 344 2716 7109 6731 -0.23777\r\n sped 21 10897 7282 7428 0.23775\r\n excel 16 12021 7561 8598 0.23769\r\n chemotherapy 23 10525 9729 14851 -0.23766\r\n conduct 234 3426 7293 7465 -0.23750\r\n tribal 47 7696 6456 4103 0.23749\r\n hitch 121 4863 5421 1157 0.23745\r\n cult 170 4091 4808 355 0.23740\r\n envision 13 12907 7797 9526 0.23729\r\n glib 33 9036 6819 5500 0.23729\r\n libel 18 11532 7441 8097 0.23723\r\n willed 39 8399 6649 4839 0.23719\r\n lecturing 65 6604 8146 10765 -0.23718\r\n grapefruit 65 6604 8146 10765 -0.23718\r\n severance 36 8727 8846 12973 -0.23717\r\n virtual 57 7018 6261 3403 0.23717\r\n from 59972 74 6420 3973 -0.23716\r\n fungus 58 6968 8257 11143 -0.23714\r\n illinois 153 4326 7511 8395 -0.23713\r\n decoration 27 9842 9351 14131 -0.23707\r\n exploitation 17 11781 11263 16147 -0.23703\r\n champ 144 4460 5151 745 0.23702\r\n compartment 55 7129 8307 11307 -0.23693\r\n iceberg 60 6862 8220 11028 -0.23685\r\n.\r\n.\r\n. The most \"stable\" words...\r\n.\r\n loose 1069 1331 5516 1367 0.00106\r\n scottie 56 7070 7258 7336 0.00106\r\n arrival 149 4385 6584 4578 -0.00105\r\n radioactivity 6 15795 13593 16411 0.00103\r\n sung 65 6604 7147 6885 -0.00099\r\n pilgrimage 9 14316 9766 14906 -0.00099\r\n eyeball 35 8825 7702 9163 0.00095\r\n heal 466 2260 5904 2335 0.00095\r\n scrabble 39 8399 7596 8751 -0.00094\r\n provoke 93 5547 6886 5784 -0.00089\r\n iron 314 2875 6128 2976 0.00087\r\n extortionist 7 15269 10730 15895 -0.00086\r\n rubbish 54 7186 7299 7488 -0.00085\r\n cavalier 28 9689 7948 10091 -0.00083\r\n get 126849 37 3948 25 0.00082\r\n nick 2699 704 5132 719 0.00080\r\n integration 15 12307 8785 12813 -0.00078\r\n pedestal 65 6604 7140 6856 0.00077\r\n ringing 281 3078 6201 3213 -0.00071\r\n yo 1347 1138 5429 1172 0.00071\r\n platter 116 4967 6726 5155 0.00067\r\n stifler 14 12603 8898 13119 -0.00066\r\n cat 1742 960 5330 988 0.00064\r\n relate 172 4061 6487 4214 0.00059\r\n machismo 8 14787 10116 15370 0.00058\r\n altitude 52 7303 7325 7605 -0.00057\r\n fetish 43 8040 7501 8353 0.00056\r\n ton 304 2941 6157 3068 -0.00056\r\n twentieth 41 8212 7542 8532 0.00055\r\n telltale 10 13925 9533 14492 -0.00054\r\n montage 7 15269 10700 15873 0.00048\r\n toll 113 5030 6742 5224 0.00046\r\n fabricate 13 12907 9016 13418 0.00038\r\n misrepresentation 6 15795 15362 16422 0.00037\r\n buster 249 3295 6271 3433 -0.00036\r\n primal 49 7517 7375 7824 -0.00035\r\n alvin 65 6604 7142 6863 0.00034\r\n tumor 225 3506 6328 3641 0.00034\r\n book 5027 468 4937 492 -0.00032\r\n voyage 106 5215 6799 5429 -0.00030\r\n hug 697 1751 5720 1826 -0.00029\r\n demon 1703 977 5349 1020 -0.00023\r\n marietta 21 10897 8313 11330 0.00023\r\n empty 1261 1183 5457 1234 -0.00022\r\n bootleg 19 11306 8435 11756 0.00019\r\n jordan 266 3167 6229 3297 -0.00019\r\n theses 9 14316 9761 14892 -0.00014\r\n letter 1839 925 5311 960 0.00013\r\n cheer 550 2046 5834 2130 -0.00012\r\n maiden 82 5896 6965 6134 -0.00010\r\n toaster 76 6107 7016 6352 -0.00002\r\n castle 408 2451 5977 2549 0.00001\r\n\r\n<\/pre>\n
\r\n\r\nclass a_word(object) :\r\n def __init__(me, word, cnt, dur) :\r\n me.word = word # the word\r\n me.cnt = cnt # the word's use count\r\n me.dur = dur # the word's shortest .wav file byte length\r\n me.cnti = 0 # the normalized ranking of the count (low rank are frequent words)\r\n me.duri = 0 # the normalized ranking of the duration (low ranks are short words)\r\n me.off = 0.0 # how far off the two rankings are\r\n pass # a_word\r\n\r\n#\r\n#\r\nif __name__ == '__main__' :\r\n import os\r\n import re\r\n import sys\r\n import time\r\n\r\n import TZCommandLineAtFile\r\n import tzlib\r\n\r\n\r\n sys.argv.pop(0)\r\n\r\n TZCommandLineAtFile.expand_at_sign_command_line_files(sys.argv)\r\n\r\n wc_fn = sys.argv.pop(0)\r\n\r\n wcs = tzlib.read_whole_text_file(wc_fn) # lines of: \"word count (wav_size (...))\" - we use the shortest .wav size\r\n wa = re.split(r\"\\n\", wcs)\r\n wa = [ wc for wc in [ re.split(r\"\\s+\", ln) for ln in wa ] if (len(wc) >= 3) and (wc[0][0] != ';') ]\r\n words = []\r\n for wc in wa :\r\n wc[1] = int(wc[1])\r\n words.append(a_word(wc[0], wc[1], min([ int(ln) for ln in wc[2:]])))\r\n\r\n\r\n words.sort(lambda a, b : cmp(b.cnt, a.cnt))\r\n i = 0\r\n j = 0\r\n icnt = 0\r\n ucnt = 0\r\n for w in words :\r\n if icnt != w.cnt :\r\n icnt = w.cnt\r\n i = j\r\n ucnt += 1\r\n w.cnti = i\r\n j += 1\r\n icnt = float(i)\r\n\r\n\r\n words.sort(lambda a, b : cmp(a.dur, b.dur))\r\n i = 0\r\n j = 0\r\n idur = 0\r\n udur = 0\r\n for w in words :\r\n if idur != w.dur :\r\n idur = w.dur\r\n i = j\r\n udur += 1\r\n w.duri = i\r\n j += 1\r\n idur = float(i)\r\n\r\n\r\n for w in words :\r\n w.off = (w.cnti \/ icnt) - (w.duri \/ idur)\r\n\r\n words.sort(lambda a, b : cmp(abs(b.off), abs(a.off)))\r\n\r\n print \"; \" + time.asctime()\r\n print \"; counts=%i durations=%i unique_counts=%i unique_durations=%i\" % ( int(icnt), int(idur), int(ucnt), int(udur) )\r\n print\r\n print \"; %-30s count cnti dur duri offness\" % \"Word\"\r\n print\r\n for w in words :\r\n print \" %-30s %8u %5u %6i %5u %8.5f\" % ( w.word, w.cnt, w.cnti, w.dur, w.duri, w.off )\r\n\r\n print\r\n print \";\"\r\n print \"; eof\"\r\n\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"