add log abstraction to store records in disk

[hongchaodeng]

@xiangli-cmu
This adds log and message abstraction.
I haven't thought about how to batch encoding yet. It had better be considered in snapshotting.

[yunxing]

There is no need to have rev number persisted on disk.

[yunxing]

No needs to have bufio here since OS already buffers IO for you. Double buffer is evil in database.

[hongchaodeng]

That's interesting. We need to investigate more regarding buffering behavior.

It currently uses bufio to write a single request: (length | record).

[yunxing]

Buffer should only be done in kernel space.

In kernel space, OS's page cache will buffer any incoming writes until you fsync on the file or until the flusher thread kicks in. There is no point buffering them in user space unless you want to speed up the reads for buffered data.

[hongchaodeng]

This isn't about buffering. It's just a way to construct a message to be written into disk.

[yunxing]

You can still construct the message with io.writer.

[yunxing]

How about we don't buffer stuff until we know there is a performance issue without buffering? : ) The code will be much simpler.

[hongchaodeng]

Nope. You can keep a downstream branch without buffer. I don't understand this part in OS so I will keep it in upstream.

[yunxing]

This is not upstream, this is just a pull request towards upstream. I'm just pointing out things that doesn't make sense in the pull request.

[hongchaodeng]

Just a follow up on this. According to KAFKA doc, "As a result of these factors using the filesystem and relying on pagecache is superior to maintaining an in-memory cache or other structure". It sounds OS page-cache reliance is a better option. And thanks for the discussion here.

[yunxing]

Thanks for doing the due diligence before adding a feature :)

To clarify, most of stuff we were discussing here is buffering instead of caching. Caching is a super set of buffering in the sense that you can read from a cache but cannot read from a bufio writer.