chunksOf :: Int -> ByteString m r -> Stream (Of ByteString) m r

[ocharles]

What do you think about adding a function:

chunksOf :: Int -> Data.ByteString.Streaming.ByteString m r -> Stream (Of Data.ByteString.ByteString) m r

This would yield consecutive chunks of n bytes as a strict bytestring. If the ByteString doesn't have n bytes, it ceases to yield and returns.

[ivan-m]

My preference would be that if the ByteString has 0 < len < n bytes it yields it rather than silently discard it.

[ivan-m]

If we at least modify this to (using Int64 to match the behaviour of splitAt which would probably be used in the implementation):

chunksOf :: Int64 -> Stream.ByteString m r -> Stream (Of Strict.ByteString) m (Maybe Strict.ByteString, r)

(where Just in the return value indicates a non-empty < n chunk) then we can use >>= \ (rem, r) -> mapM_ yield rem >> return r to append that final chunk to the stream.

[fosskers]

Hi there, is this still desired?

[vdukhovni]

Don't know how useful it is to rechunk to fixed sizes, but assuming it is:

{-# LANGUAGE BangPatterns, LambdaCase #-}
import qualified Data.ByteString as Strict
import qualified Streaming.Prelude as S
import qualified Streaming.ByteString as Q
import Data.Int (Int64)
import Streaming (Of(..), Stream)
import Streaming.ByteString (ByteStream)

chunksOf :: Monad m => Int64 -> ByteStream m r -> Stream (Of Strict.ByteString) m r
{-# INLINE chunksOf #-}
chunksOf n = S.unfoldr step
  where
    step str = do
        (c :> rest) <- Q.toStrict $ Q.splitAt n str
        if Strict.null c
        then Left <$> Q.effects rest
        else return $ Right (c, rest)

Demo:

λ> show $ (chunksOf 5 $ Q.chunk $ fromString $ ['a'..'m'] :: Stream (Of ByteString) Identity ())
"Effect (Identity (Step (\"abcde\" :> Effect (Identity (Step (\"fghij\" :> Effect (Identity (Step (\"klm\" :> Effect (Identity (Return ())))))))))))"

[fosskers]

I could see somebody wanting to do this for some networking application with specific requirements. Feel like opening up a PR?

[vdukhovni]

I could see somebody wanting to do this for some networking application with specific requirements. Feel like opening up a PR?

I think I have enough things going on that yet another PR may be too much... But if you like the above example, feel free to add it, the only non-obvious thing is whether chunksOf is the right name. I haven't spent any time thinking about whether it is the natural name for this function. If it is, then just cut/paste and you're done. Unless there's a better implementation.

The above carefully avoided using any Internals from either Streaming or Streaming.ByteString, it is likely possible to construct a more direct implementation that performs the required right fold without resorting to S.unfoldr, and could then perhaps not need to go through the Either r (a, s) unfoldr wrapper. Perhaps with enough inlining GHC optimises those away, but it should be possible to build the Stream directly, if there's a compelling performance advantage.

For now, I'd just like to see #42 merged, and perhaps make more progress on the README (ultimately for inclusion directly in the modules).

[fosskers]

We can always come back to this, there doesn't seem to be pressing demand for it.

[vdukhovni]

I guess the direct implementation would be something like:

chunksOf :: Monad m => Int64 -> ByteStream m r -> Stream (Of Strict.ByteString) m r
{-# INLINE chunksOf #-}
chunksOf n = go
  where
    go str = Effect $ do
        (c :> rest) <- Q.toStrict $ Q.splitAt n str
        if Strict.null c
        then Return <$> Q.effects rest
        else return $ Step (c :> go rest)

I am still not up to speed on when to INLINE and when to use SPEC in loops, ... so that too could be tweaked as required.