Skip to content

Support for streaming mode#144

Open
swjz wants to merge 648 commits intoscanner-research:masterfrom
swjz:streaming
Open

Support for streaming mode#144
swjz wants to merge 648 commits intoscanner-research:masterfrom
swjz:streaming

Conversation

@swjz
Copy link
Contributor

@swjz swjz commented Feb 9, 2018

Fixes #104.

Usage:

python python/scannerpy/python_master.py  # start running python master
python tests/test_realtime.py  # start test client

API:

input = db.ops.MemoryInput()
output = db.ops.MemoryOutput(columns=[test_out])
# ...
input.push(frame.tobytes())
out_frame = output.pull()
# ...
input.close()

Known issue:
The client does not completely shut down after finishes. Might need to kill -9 the process manually for now. I'll try to fix it in a few days.

willcrichton and others added 30 commits November 24, 2017 22:03
@willcrichton
Attempt to update travis DOCKER_PASS
141e8b2
@willcrichton
Attempt to fix login/push on branch updates
7ca0c60
@fpoms
Fix docker rmi not removing tagged imaged
a3135d4
@fpoms
Fixes scanner-research#89
cde8dea
@fpoms
Fixes scanner-research#88
92888e8
@fpoms
Do not publish build if branch is not master
b5dc23a
@fpoms
Use exit instead of return
f6b3f60
@fpoms
Shutdown server when process receives SIGINT, fixes scanner-research#78
3cacff4
@qianl15 @fpoms
Support batched input for python kernel
fb851e5 
add can_batch feature, restore my_kernel.py, and add a test_python_batch_kernel test to py_test.py

restore my_kernel.py
@qianl15 @fpoms
add s3 support
6d8f69d 
add s3 support

Pass region and endpoint to s3 config

Restore thirdparty/CMakeLists.txt and 00_basic.py

Restore thirdparty/CMakeLists.txt and 00_basic.py
@fpoms
Support incomplete jobs/tables, fixes scanner-research#72
33eeea4
@fpoms
WIP new README
c232b69
@fpoms
WIP new README
273ebf7
@fpoms
Update README, change face pipeline to new NN app style
0b48cb9
@fpoms
Update face detect doc
8382bb3
@fpoms
Use absolute python imports (for py3), add basic sphinx docs
b3f3143
@fpoms
Fix tests not passing due to unicode strings
480a347
@fpoms
Add OpenPose Op, closes scanner-research#97
fed2370
@fpoms
Add facenet CPU support
6c8fd7a
@fpoms
Fix db new_table for new commit table
f96ca7f
@fpoms
Fix issue with stenciled kernels
4278dd6
@fpoms
Reorganize examples directory
3667047
@fpoms
Update README to match new example structure
c5783d3
@fpoms
Build facenet kernel always
e47fcf6
@fpoms
Fix saved io packets being overwritten
af7c2be
@fpoms
Update apps based on new dir structure
049b673
@fpoms
Build docker from new base images
c5901c0
@fpoms
Use most recent docker on travis
c0addd5
@fpoms
Assume Caffe uses all CPUs
a9c8188
@fpoms
Support different devices for input/output cols, py kernel config
b16f613
fpoms and others added 28 commits January 14, 2018 22:04
@fpoms
Roll back keyframe slicing for sparsity and roll back roll back
ea04247 
large_distance and exactly_keyframes were causing hangs on cinema
videos
@fpoms
Use stop cluster command to shutdown master
cd524fa
@willcrichton
Fix io packet multiple issue for pipelines
8d10bcf
@willcrichton
Optimize sparse loading via Python API
be49822
@willcrichton
Add last row hacky optimization
880305f
@fpoms
Update hwang to fix gpu decode issue
4039d5c
@willcrichton
Autoscheduler for CPU
08bd255
@fpoms
Fix bug causing decoder to output empty frames
165ffb2
@fpoms
Fix switched arguments in range sampler
946f827
@fpoms
Remove unused functionality
9fc956a
@fpoms
Remove user_function from cmake
386d4bc
@swjz
First (buggy) version of streaming support
197f26c
@swjz
Fixed some compile-time errors
e3eec7d
@swjz
Added dummy table to support streaming mode
b9f935b
@swjz
Fix bugs in implementing dummy table
397ea8a
@swjz
Use alternative way to implement dummy table
0d3846d
@swjz
Finished switching between C++ master and python master
37777b7
@swjz
Override a lot of stuff and add checkpoints for debugging
06f5e19
@swjz
Fix many bugs and remove many special cases for streaming mode
321cfc7
@swjz
Updated test case for streaming mode
6f16e2f
@swjz
A seemingly working version of streaming mode
9437999
@swjz
Removed special treatment of last_in_io_packet for streaming mode
99c65c5
@swjz
Removed special treatment of last_in_task for streaming mode
62febc3
@swjz
Shutdown worker when finished
6357f5b
@swjz
Revert unnecessary changes and clean code
d087573
@swjz
Use try_pop() instead of pop() to prevent deadlock
6d4c82f
@swjz
Fix '\0' truncating and remove unnecessary locks
2544e47
@swjz
Update test case for streaming mode
f72349f
@willcrichton willcrichton force-pushed the master branch 2 times, most recently from 6822cde to 5d4d222 Compare April 19, 2019 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support for streaming mode

4 participants

@swjz @willcrichton @fpoms @qianl15