Reverse Pull Request#19
Open
iamrobinhood12345 wants to merge 105 commits into
Open
Conversation
layout and result templates. routes.
…o development merging from harvester#
Harvester
…into development Pulling Marc's changes into local development branch.:wq
…added template for computing results.
…d temp test_harv.py.
…tializedb.py to drop all on init and input Keyword models, adjusted views/default.py to return results from postgres pysearch database.
…w in views/default.py.
Integration thursday
add about me template
Dev branch
Final testing
…search description.
nhuntwalker
reviewed
Feb 3, 2017
| """ This function returns a Pyramid WSGI application. | ||
| """ | ||
| settings["sqlachemy.url"] = os.environ["DATABASE_URL"] | ||
| # settings["sqlachemy.url"] = os.environ["DATABASE_URL"] |
| session.rollback() | ||
| raise | ||
| finally: | ||
| session.close() |
There was a problem hiding this comment.
This monolith of a function could probably be broken down into some smaller, more testable functions.
| #HTTPCACHE_EXPIRATION_SECS = 0 | ||
| #HTTPCACHE_DIR = 'httpcache' | ||
| #HTTPCACHE_IGNORE_HTTP_CODES = [] | ||
| #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' |
There was a problem hiding this comment.
You could probably remove all of the commented lines since they're not being used anyway.
| del word_count[key] | ||
|
|
||
| to_add = [] | ||
| session = self.Session() |
There was a problem hiding this comment.
This is a lot of logic for one method. You could've broken up this monolith of a method into several smaller, more testable functions.
| {'keyword': 'soccer', 'weight': 5, 'url': 'url2', 'count': 25}, | ||
| {'keyword': 'football', 'weight': 10, 'url': 'url3', 'count': 5}, | ||
| {'keyword': 'soccer', 'weight': 5, 'url': 'url3', 'count': 5} | ||
| ] |
There was a problem hiding this comment.
Why is this here? What's it used for?
| def computing_results_view(request): | ||
| """Initiates crawler and routes to results.""" | ||
| url = request.params["url"] | ||
| print('computing results view ', url) |
| # entries = query.all() | ||
| unique_urls = [] | ||
| for val in request.dbsession.query(Match.page_url).distinct(): | ||
|
|
|
|
||
| unique_urls.append(val[0]) | ||
|
|
||
| print(unique_urls) |
| unique_keywords = [] | ||
| for val in request.dbsession.query(Match.keyword).distinct(): | ||
| unique_keywords.append(val[0]) | ||
| print(unique_keywords) |
| results.append({'keyword': kw, 'weight': url_q.keyword_weight, 'url': url, 'count': url_q.count}) | ||
|
|
||
| except DBAPIError: | ||
| return Response(db_err_msg, content_type='text/plain', status=500) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.