Maximum entropy! Is this "sufficiently" random?

Question

note: Yeah, it's a long post. I tend to do that when tired. The first part just gives some context. Skip down to the code if you're not interested.
Threads keep popping up about pacman-key "not working" when it's waiting for entropy from /dev/random to generate keys. The advice in those threads is to move the mouse around (if you have Xorg), run some cpu intensive apps, and wait.
I started to wonder if the advice about running apps was actually good. I really don't know how the kernel collects entropy but the little that I've read only mentions mouse, keyboard and disk IO. I did some really basic testing with different commands (CPU intensive, disk IO intensive) to see if they increased the available entropy (/proc/sys/kernel/random/entropy_avail). They didn't. At least not enough to have an immediate impact from what I could tell.
So, what else can we do? The first thing that came to mind was using the microphone as an entropy source. I found audio-entropyd (old and new) but there was no package for it. Instead of creating one I decided to test some some ideas. I spent some time reading up on how to actually inject bits into the entropy pool before I ended up installing rng-tools as recommended in the wiki.
At first I tried piping the microphone output from arecord into rngtest but that failed the FIPS test (the first few times it failed because my microphone was plugged into the headphones jack, but it still failed after that ).
I read something about hashing inputs while searching for ways to increase entropy, so I wrote the following script:
#!/usr/bin/env python3
from time import time
from hashlib import sha512
from sys import stdin, stdout, stderr, argv
h = sha512()
if argv[1:]:
RATIO = int(argv[1])
else:
RATIO = h.block_size
buf = stdin.buffer.read(h.block_size)
n = 1
while buf:
h.update(buf)
buf = stdin.buffer.read(h.block_size)
n += 1
if n >= RATIO:
stdout.buffer.write(h.digest())
stderr.write('\r%f' % time())
h = sha512()
n = 0
It takes an input stream and hashes it with sha512 at a configurable ratio. Let's call it "hashpipe" for now. Piping the microphone output through it passes rng's FIPS test. Great.
So I tried this:
mkfifo foo
# change hw as needed
arecord -c 1 -f cd -t raw -D hw:1,0 - | hashpipe > foo
# in another terminal
sudo rngd -f -r foo -W 4096 -t 5
Lo and behold, maximum entropy! The entire pool is refreshed as soon as it's depleted. 4096-bit gpg keys take no time at all.
Now, accepting that my crypto-fu is weak, I ask you... does passing rng-tool's FIPS test mean that the data is "sufficiently" random for secure cryptography? If not, what about if the input/output hash ratio is increased?
I'm thinking that hashing the input stream is either masking its regularity or condensing its entropy.  I'm expecting the former as the authors of audio-entropyd do much more to extract the entropy of the audio input, and that project has been around long enough to receive scrutiny. I haven't actually looked at what they do though. I've just seen some messages around the internet. I didn't see any hashing in randomsound, but there is a debias function. Of course, I have no idea if either of those generate "good" entropy to begin with.

Answer 1

keenerd wrote:It means each bit is completely independent of all the others...
Hmmm.  I am refering to "entropy" in the sense of Shannon's information theory, in which case it is just a concrete measurement on a probability distribution.  Apologies if I'm not understanding -- I'm new here.  Anyway, using my definitions, you can have a funny distribution on strings in which each bit is independent of the others, but it still has very low entropy.  Say if the bits are usually 0, but their likelyhood of being 0 has nothing to do with any of their neighbors.  In this sense, independent =/=> lots of entropy.  Anyway, differences in language aside, I think we actually agree on the meaning:  "perfect" <==> the value is sampled uniformly and independently at random from {0,1}.  Right?
For example, all pseudo RNGs have 0% entropy because they can be perfectly predicted.
Provided you know the seed.  In general, entropy <= size of seed.  But again, this is using my definition...
So in this context, "percent entropy" is the ratio of kolmogorov size divided by input size.  Sources of perfect entropy don't exist, so improving the quality (debiasing) of entropy sources is a huge deal.
Intuitively I like the definition.  Although Kolmogorov complexity is not so easy to compute and also not so easy to estimate.  Well, I guess it is easy to come up with upper bounds, but we are interested more in lower bounds, which I think Shannon is more suited towards.
Personally I avoid the term "high entropy", because it can either mean high quality or high rate depending on context.
I think I'm coming from a different background.  Perhaps I mistook my world for the universe.  Sorry if that's the case.
When I mentioned extracting high quality bits, I did not mean in a literal sense of pointing at a 0 or 1 and saying "You, you there are completely random."  Rather in the vague sense of a bit as a component of information, discarding all the components that can be predicted or (more specifically) do not add to the Kolmogorov complexity of the random sequence.
I guess the bit you mentioned about the LSB sent me down that path.
The leftover hash lemma just proves that debiasing is possible.  It does not let you gain any extra randomness.  Randomness extractors are just debiasing by a different name.
Regarding the extra randomness, that's exactly what I said.  Regarding the lemma and extractors, I wouldn't say that it just proves it is possible: it gives you a generic and concrete way to do it.  Bonus: it comes with formal statements of what it accomplishes, and proofs that it works!  But to be honest, I am unfamiliar with the literature on debiasing.  I would be interested in any references if you have them.
But provable does not mean practical.
Unfortunate, but true.  But if the intersection is NOT the empty set...  win!  I looked (briefly) at some of the other libraries out there, and didn't find any that claimed an information-theoretic kind of proof.  Meh.  In reality, most of this is probably overkill, and the ad-hoc method is probably fine.  But people have been burned before, and although we certainly aren't doing anything that dumb nowadays, I still would prefer to be on the safe side if possible.
If you want to learn about collecting entropy, then experiment with it.
I did once upon a time.  Maybe I will bring that back to life and share the results (if they don't suck).  But I thought Xyne's idea had some promise...
Last edited by wes (2012-04-02 01:32:12)