[FEAT] contest api 구현#105
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive packages, services, and routes for crawling, transforming, and rendering card news for contests, festivals, and policies, including dedicated API clients for TourAPI and Policy News. Key feedback from the review highlights several critical improvements: strengthening date validation to prevent invalid calendar dates, fixing a bug in the error handler where custom error messages were not being passed to the response, adding robust exception handling for XML parsing and JSON decoding of external API responses, and ensuring the contest card news font fallback chain includes a Korean-supporting font like Pretendard to prevent text rendering issues.
| def validate_yyyymmdd(value: str, *, label: str) -> str: | ||
| text = (value or "").strip() | ||
| if len(text) != 8 or not text.isdigit(): | ||
| raise ValueError(f"{label}는 YYYYMMDD 8자리여야 합니다 (받음: {value!r})") | ||
| return text |
There was a problem hiding this comment.
현재 validate_yyyymmdd 함수는 단순히 문자열의 길이(8자리)와 숫자 여부만 검증하고 있습니다. 이로 인해 "20260231"과 같이 존재하지 않는 잘못된 날짜가 입력되어도 검증을 통과하게 됩니다. datetime.strptime을 사용하여 실제 유효한 날짜인지 교차 검증하도록 개선하는 것이 안전합니다. (이 파일에 datetime 임포트가 필요합니다.)
def validate_yyyymmdd(value: str, *, label: str) -> str:
from datetime import datetime
text = (value or "").strip()
if len(text) != 8 or not text.isdigit():
raise ValueError(f"{label}는 YYYYMMDD 8자리여야 합니다 (받음: {value!r})")
try:
datetime.strptime(text, "%Y%m%d")
except ValueError as exc:
raise ValueError(f"{label}는 올바른 날짜 형식이어야 합니다 (받음: {value!r})") from exc
return text| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | ||
| response = failure_response(error_code=code, result=result, status_code=http_status) |
There was a problem hiding this comment.
_error_response 함수의 매개변수로 message를 전달받고 있으나, 내부적으로 failure_response를 호출할 때 message 인자를 전달하지 않고 누락하고 있습니다. 이로 인해 unhandled_exception_handler 등에서 로컬/개발 환경일 때 예외 상세 정보를 담아 전달하려는 message가 클라이언트 응답에 반영되지 않고 유실됩니다. failure_response 호출 시 message 매개변수를 함께 전달하도록 수정해야 합니다.
| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | |
| response = failure_response(error_code=code, result=result, status_code=http_status) | |
| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | |
| response = failure_response(error_code=code, result=result, status_code=http_status, message=message) |
| pin_content = clean_text(raw_contents) | ||
|
|
There was a problem hiding this comment.
ET.fromstring(xml_text) 호출 시, API가 점검 중이거나 인증 오류 등으로 인해 XML이 아닌 HTML 에러 페이지나 빈 응답을 반환할 경우 xml.etree.ElementTree.ParseError가 발생하여 서버가 예기치 않게 종료될 수 있습니다. 이를 방지하기 위해 try-except 블록으로 감싸 예외를 안전하게 처리하고, 디버깅을 위해 응답 본문의 일부를 포함한 명확한 에러 메시지를 제공하는 것이 좋습니다.
| pin_content = clean_text(raw_contents) | |
| try: | |
| root = ET.fromstring(xml_text) | |
| except ET.ParseError as exc: | |
| raise ValueError(f"정책뉴스 XML 파싱 실패: {exc}. 응답 내용 일부: {xml_text[:200]}") from exc |
| url = f"{self._base_url}/{endpoint}?{query}" | ||
| response = await self._http.get(url) | ||
| response.raise_for_status() | ||
| payload = response.json() |
There was a problem hiding this comment.
TourAPI(한국관광공사 OpenAPI)는 트래픽 초과나 서비스 키 오류가 발생했을 때, 요청 포맷(_type=json)과 관계없이 XML 에러 메시지(<OpenAPI_ServiceResponse>...)를 반환하는 경우가 빈번합니다. 이 경우 response.json() 호출 시 json.JSONDecodeError가 발생하여 원인을 파악하기 어려운 500 에러로 이어질 수 있습니다. 따라서 json.JSONDecodeError를 예외 처리하여 실제 API 응답 내용을 포함한 명확한 에러 메시지를 제공하도록 개선하는 것을 권장합니다.
| payload = response.json() | |
| try: | |
| payload = response.json() | |
| except (json.JSONDecodeError, ValueError) as exc: | |
| raise RuntimeError(f"VisitKorea API 응답 JSON 디코딩 실패 (API 에러 가능성): {response.text[:200]}") from exc |
There was a problem hiding this comment.
Pull request overview
Implements end-to-end ingestion and transformation pipelines for Linkareer contests (crawl → JSONL → Gemini copy → cardnews PNG → DB handoff), and additionally adds similar search/transform/handoff flows for TourAPI festivals and PolicyNews OpenAPI policies, including new FastAPI routes, clients, schemas, and cardnews rendering templates.
Changes:
- Added Contest pipeline: Playwright crawler + JSONL normalization + Gemini-based 3-slide cardnews generation + DB handoff outputs.
- Added Festival/Policy pipelines: OpenAPI clients, search/transform/handoff services & routes, plus cron/CLI scripts and handoff docs.
- Added Policy/Contest cardnews rendering: Pillow template renderer, palette/term utilities, and optional Gemini image-model generation.
Reviewed changes
Copilot reviewed 78 out of 87 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/preview_policy_cardnews.py | Local preview script for policy cardnews rendering. |
| scripts/preview_contest_cardnews.py | Local preview script for contest 3-slide deck rendering. |
| rag/scripts/transform_policy_content.py | CLI transformer: policy documents JSONL → Gemini easy-read → DB handoff JSONL. |
| rag/scripts/transform_festival_content.py | CLI transformer: festival documents JSONL → Gemini Instagram tone → DB handoff JSONL. |
| rag/scripts/run_festival_pipeline.py | Cron-oriented festival batch pipeline (fetch → transform → report). |
| rag/scripts/run_contest_cardnews.py | CLI runner to generate contest cardnews + DB handoff JSONL. |
| rag/scripts/preprocess_module.py | Adds unicode whitespace normalization to text cleaning utilities. |
| rag/scripts/normalize_contest_jsonl.py | Normalizes existing contest JSONL without re-crawling. |
| rag/scripts/fetch_policy_news.py | CLI fetcher for PolicyNews OpenAPI → policy_documents.jsonl. |
| docs/policy_pin_handoff.md | Policy pipeline handoff documentation and Swagger flow. |
| docs/festival_pin_handoff.md | Festival pipeline handoff documentation and Cron guidance. |
| docs/contest_pin_handoff.md | Contest pipeline handoff documentation and constraints (no crawled images). |
| app/utils/visitkorea_facilities.py | Extracts pet-friendly / stay-available hints from TourAPI payloads. |
| app/utils/policy_news_parse.py | Parses PolicyNews XML, cleans content, and extracts/filters image URLs. |
| app/utils/festival_date_filter.py | Date-range overlap helper for festival/contest filtering. |
| app/services/VectorStoreService.py | Updates domain table-name resolution (standalone-table support). |
| app/services/vector_domains.py | Refactors vector domain config builder (currently complaint-only). |
| app/services/prompts/policy_pin.py | Prompt template for policy “easy read” pin content generation. |
| app/services/prompts/policy_cardnews.py | Prompt templates for policy cardnews slide JSON + image prompt. |
| app/services/prompts/festival_pin.py | Prompt template for festival Instagram-style content generation. |
| app/services/prompts/contest_cardnews.py | Prompt templates for contest cardnews slides + caption generation. |
| app/services/PolicyPinService.py | Policy search/transform/handoff service wiring and image enrichment. |
| app/services/policy_pin_transform.py | Policy JSONL → Gemini text + cardnews generation → handoff JSONL. |
| app/services/policy_cardnews.py | Orchestrates policy cardnews slide generation and rendering (image model or Pillow). |
| app/services/internal/ai/PolicyCardnewsImageService.py | Gemini/Imagen image-generation wrapper with retries/fallbacks. |
| app/services/FestivalPinService.py | Festival search/transform/handoff service with optional date filtering at handoff. |
| app/services/festival_pin_transform.py | Festival JSONL → Gemini Instagram text → DB handoff JSONL. |
| app/services/ContestPinService.py | Contest crawl/documents listing/cardnews/handoff service (incl. Windows Playwright handling). |
| app/services/contest_pin_transform.py | Contest JSONL → Gemini slides(+caption) → cardnews PNG → handoff JSONL. |
| app/services/contest_cardnews.py | Contest cardnews generation orchestration (Gemini → template render). |
| app/services/init.py | Exposes build_vector_domain_configs from services package. |
| app/schemas/PolicyPinDTO.py | Adds policy DTOs for source, transform result, and handoff result payloads. |
| app/schemas/FestivalPinDTO.py | Adds festival DTOs for source/transform/handoff payloads. |
| app/schemas/ContestPinDTO.py | Adds contest DTOs for documents/crawl/transform/handoff payloads. |
| app/routes/PolicyPinRoute.py | Adds policy-pins search/transform/handoff endpoints. |
| app/routes/FestivalPinRoute.py | Adds festival-pins search/transform/handoff endpoints (handoff date filter supported). |
| app/routes/ContestPinRoute.py | Adds contest-pins crawl/documents/cardnews/handoff endpoints. |
| app/routes/init.py | Registers new routers (festival/contest/policy) with env gating. |
| app/policy_cardnews/visual.py | Image paste helpers for rounded-fit/cover hero images. |
| app/policy_cardnews/terms.py | Terminology simplification and auto term-guide enrichment for slides. |
| app/policy_cardnews/template/metrics.py | Policy template layout metrics constants. |
| app/policy_cardnews/template/draw.py | Shared drawing primitives for policy template rendering. |
| app/policy_cardnews/template/init.py | Exports policy template dispatch/render helpers. |
| app/policy_cardnews/slides.py | Parses and normalizes policy slide JSON into supported layouts. |
| app/policy_cardnews/render.py | Renders policy cardnews slides to PNG with hero image/mascot selection. |
| app/policy_cardnews/mascot.py | Loads mascots via manifest and renders speech bubbles/mascot blocks. |
| app/policy_cardnews/images.py | Downloads and decodes background/cover images with headers+referer. |
| app/policy_cardnews/copy.py | Copy cleanup, filler removal, slide compaction, and speech derivation logic. |
| app/policy_cardnews/constants.py | Policy cardnews canvas/frame/spacing constants. |
| app/policy_cardnews/init.py | Policy cardnews package exports. |
| app/main.py | Adds Windows event-loop policy tweak and refactors vector domain configs init. |
| app/core/handlers.py | Enhances unhandled exception logging and dev/local message detail propagation. |
| app/core/deps.py | Adds FastAPI dependencies for contest/festival/policy services. |
| app/core/config.py | Adds TourAPI/PolicyNews/cardnews settings and (currently) duplicates some RAG fields. |
| app/contest_cardnews/template/palette.py | Defines contest palettes and applies a consistent deck palette. |
| app/contest_cardnews/template/dispatch.py | Contest layout dispatch and normalization logic. |
| app/contest_cardnews/template/chrome.py | Browser-chrome drawing for contest card layout. |
| app/contest_cardnews/template/init.py | Exports contest template renderer helpers. |
| app/contest_cardnews/slides.py | Parses contest slide JSON, normalizes layouts, and compacts to a 3-slide deck. |
| app/contest_cardnews/render.py | Renders contest slides to PNG using contest templates + mascots. |
| app/contest_cardnews/copy.py | Contest slide cleanup, compaction, and speech derivation. |
| app/contest_cardnews/constants.py | Contest cardnews constants (canvas, spacing, chrome). |
| app/contest_cardnews/init.py | Contest cardnews package exports. |
| app/clients/VisitKoreaClient.py | Async HTTP client for TourAPI endpoints with pacing. |
| app/clients/PolicyNewsClient.py | Async HTTP client for PolicyNews OpenAPI (XML) with pacing. |
| app/assets/policy_cardnews_templates/template_three_col.json | JSON template spec for policy three-column layout. |
| app/assets/policy_cardnews_templates/template_numbered.json | JSON template spec for policy numbered layout. |
| app/assets/policy_cardnews_templates/template_grid.json | JSON template spec for policy grid layout. |
| app/assets/policy_cardnews_templates/template_cta.json | JSON template spec for policy CTA layout. |
| app/assets/policy_cardnews_templates/template_cover.json | JSON template spec for policy cover layout. |
| app/assets/policy_cardnews_palettes.json | Custom palette definitions for policy templates. |
| app/assets/mascots/mascots.json | Mascot manifest listing allowed mascot PNGs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| rag_retrieve_top_k: int = Field(default=10, ge=1, le=100, alias="RAG_RETRIEVE_TOP_K") | ||
| rag_rerank_top_k: int = Field(default=5, ge=1, le=100, alias="RAG_RERANK_TOP_K") | ||
| rag_enable_rerank: bool = Field(default=False, alias="RAG_ENABLE_RERANK") | ||
| rag_vector_query_mode: str = Field(default="hybrid", alias="RAG_VECTOR_QUERY_MODE") |
| _PET_INTRO_KEYWORDS = ("반려동물", "반려견", "애견", "pet") | ||
| _STAY_KEYWORDS = ("숙박", "숙소", " lodging", "stay") | ||
|
|
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive pipeline for crawling, processing, and rendering card news for both public policy news and contest activities, utilizing specialized Pillow-based templates and mascot integrations. The code review highlights several critical areas for improvement: potential API authentication failures due to double-encoding of service keys, a bug in error handlers ignoring custom messages, and significant performance bottlenecks from sequential image generation and redundant HTTP client instantiations. Additionally, the reviewer noted hardcoded font directories that bypass configuration settings, timezone discrepancies that could lead to data omission, and reliability issues in the crawler due to a lack of incremental saving.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| query_params = { | ||
| "serviceKey": self._service_key, | ||
| **{k: str(v) for k, v in params.items() if v is not None}, | ||
| } | ||
| query = urlencode(query_params) | ||
| url = f"{self._base_url}/{endpoint}?{query}" |
There was a problem hiding this comment.
공공데이터포털(data.go.kr)의 서비스 키(serviceKey)는 이미 URL 인코딩되어 제공되는 경우가 많습니다. 이를 urllib.parse.urlencode에 그대로 전달하면 % 문자가 %25로 이중 인코딩되어 SERVICE_KEY_IS_NOT_REGISTERED_ERROR 인증 에러가 발생할 수 있습니다. 따라서 serviceKey는 인코딩하지 않고 URL에 직접 결합하고, 나머지 파라미터만 urlencode하는 것이 안전합니다.
other_params = {k: str(v) for k, v in params.items() if v is not None}
query = urlencode(other_params)
url = f"{self._base_url}/{endpoint}?serviceKey={self._service_key}&{query}"| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | ||
| response = failure_response(error_code=code, result=result, status_code=http_status) |
There was a problem hiding this comment.
현재 _error_response 함수는 message 매개변수를 정의하고 있지만, 내부에서 failure_response를 호출할 때 이를 전달하지 않고 완전히 무시하고 있습니다. 이로 인해 local 이나 dev 환경에서 예외의 상세 내용을 담은 커스텀 에러 메시지를 생성하더라도 클라이언트에게는 기본 메시지만 전달되는 버그가 있습니다. failure_response 호출 시 message를 함께 전달하도록 수정해야 합니다.
| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | |
| response = failure_response(error_code=code, result=result, status_code=http_status) | |
| def _error_response(http_status: int, code: ErrorCode, result: Any = None, message: str | None = None) -> JSONResponse: | |
| response = failure_response(error_code=code, result=result, status_code=http_status, message=message) |
| ) -> list[str]: | ||
| saved_paths: list[str] = [] | ||
| total = len(slides) | ||
| for index, slide in enumerate(slides, start=1): | ||
| prompt = build_policy_cardnews_image_prompt( | ||
| pin_title=pin_title, | ||
| minister=minister, | ||
| slide=slide, | ||
| slide_index=index, | ||
| slide_total=total, | ||
| ) | ||
| image_bytes = await image_service.generate_slide_image_bytes(prompt=prompt) | ||
| slide_no = int(slide.get("slide") or index) | ||
| saved_paths.append( | ||
| save_slide_image_bytes( | ||
| contentid=content_id, | ||
| slide_no=slide_no, | ||
| image_bytes=image_bytes, | ||
| output_dir=output_dir, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
현재 _generate_with_image_model 함수는 여러 슬라이드의 이미지를 생성할 때 for 루프를 돌며 순차적으로(await) API 요청을 보내고 있습니다. 이미지 생성 API(Imagen 등)는 요청당 수 초 이상 소요되므로, 순차 처리 시 전체 카드뉴스 생성 시간이 매우 길어집니다. asyncio.gather를 사용하여 모든 슬라이드 이미지를 병렬로 동시에 생성하면 성능을 획기적으로 개선할 수 있습니다.
total = len(slides)
async def _generate_and_save(index: int, slide: dict[str, Any]) -> str:
prompt = build_policy_cardnews_image_prompt(
pin_title=pin_title,
minister=minister,
slide=slide,
slide_index=index,
slide_total=total,
)
image_bytes = await image_service.generate_slide_image_bytes(prompt=prompt)
slide_no = int(slide.get("slide") or index)
return save_slide_image_bytes(
contentid=content_id,
slide_no=slide_no,
image_bytes=image_bytes,
output_dir=output_dir,
)
import asyncio
tasks = [_generate_and_save(index, slide) for index, slide in enumerate(slides, start=1)]
return await asyncio.gather(*tasks)| _REPO_ROOT = Path(__file__).resolve().parents[3] | ||
| _FONT_DIR = _REPO_ROOT / "app" / "assets" / "fonts" |
There was a problem hiding this comment.
app/core/config.py에 policy_cardnews_font_dir 설정 필드가 추가되었음에도 불구하고, 실제 폰트를 로드하는 dispatch.py에서는 _FONT_DIR을 app/assets/fonts로 하드코딩하여 사용하고 있습니다. 이로 인해 환경 변수나 설정을 통해 폰트 디렉토리를 변경하더라도 무시되는 문제가 발생합니다. settings.policy_cardnews_font_dir 값을 동적으로 참조하도록 수정해야 합니다.
| _REPO_ROOT = Path(__file__).resolve().parents[3] | |
| _FONT_DIR = _REPO_ROOT / "app" / "assets" / "fonts" | |
| _REPO_ROOT = Path(__file__).resolve().parents[3] | |
| def _get_font_dir() -> Path: | |
| from app.core.config import settings | |
| if settings.policy_cardnews_font_dir: | |
| return Path(settings.policy_cardnews_font_dir) | |
| return _REPO_ROOT / "app" / "assets" / "fonts" | |
| _FONT_DIR = _get_font_dir() |
| _REPO_ROOT = Path(__file__).resolve().parents[3] | ||
| _FONT_DIR = _REPO_ROOT / "app" / "assets" / "fonts" |
There was a problem hiding this comment.
app/core/config.py에 policy_cardnews_font_dir 설정 필드가 추가되었음에도 불구하고, 실제 폰트를 로드하는 base.py에서는 _FONT_DIR을 app/assets/fonts로 하드코딩하여 사용하고 있습니다. 이로 인해 환경 변수나 설정을 통해 폰트 디렉토리를 변경하더라도 무시되는 문제가 발생합니다. settings.policy_cardnews_font_dir 값을 동적으로 참조하도록 수정해야 합니다.
_REPO_ROOT = Path(__file__).resolve().parents[3]
def _get_font_dir() -> Path:
from app.core.config import settings
if settings.policy_cardnews_font_dir:
return Path(settings.policy_cardnews_font_dir)
return _REPO_ROOT / "app" / "assets" / "fonts"
_FONT_DIR = _get_font_dir()| def default_date_range() -> tuple[str, str]: | ||
| today = date.today() | ||
| lookback = max(1, settings.policy_sync_lookback_days) | ||
| start = today - timedelta(days=lookback - 1) | ||
| return start.strftime("%Y%m%d"), today.strftime("%Y%m%d") |
There was a problem hiding this comment.
date.today()는 서버의 로컬 시스템 시간대(기본적으로 UTC인 경우가 많음)를 사용합니다. 한국 정책 브리핑 뉴스는 한국 표준시(KST, UTC+9) 기준으로 운영되므로, 클라우드 환경(UTC)에서 실행 시 날짜 계산 차이로 인해 당일 오전 시간대에 수집 누락이 발생할 수 있습니다. 명시적으로 KST 시간대를 적용하여 오늘 날짜를 구하도록 개선하는 것이 안전합니다.
| def default_date_range() -> tuple[str, str]: | |
| today = date.today() | |
| lookback = max(1, settings.policy_sync_lookback_days) | |
| start = today - timedelta(days=lookback - 1) | |
| return start.strftime("%Y%m%d"), today.strftime("%Y%m%d") | |
| @staticmethod | |
| def default_date_range() -> tuple[str, str]: | |
| from datetime import datetime, timezone, timedelta | |
| kst = timezone(timedelta(hours=9)) | |
| today = datetime.now(kst).date() | |
| lookback = max(1, settings.policy_sync_lookback_days) | |
| start = today - timedelta(days=lookback - 1) | |
| return start.strftime("%Y%m%d"), today.strftime("%Y%m%d") |
| # korea.kr은 WAF로 인해 Referer가 없으면 연결이 막히는 케이스가 있습니다. | ||
| # (특히 actuallyView.do류 URL) | ||
| headers["Referer"] = url | ||
|
|
||
| last_exc: Exception | None = None | ||
| for attempt in range(1, 4): | ||
| try: | ||
| async with httpx.AsyncClient( |
There was a problem hiding this comment.
재시도(for attempt in range(1, 4)) 루프 내부에서 매번 httpx.AsyncClient를 생성하고 닫는 것은 매우 비효율적입니다. 매 시도마다 커넥션 풀이 새로 생성되고 해제되므로 오버헤드가 발생하고 소켓 고갈 위험이 있습니다. AsyncClient를 루프 바깥에서 한 번만 생성하여 재사용하도록 수정하는 것이 성능과 자원 관리 측면에서 훨씬 효율적입니다.
async with httpx.AsyncClient(
timeout=timeout,
follow_redirects=True,
headers=headers,
) as client:
for attempt in range(1, 4):
try:
response = await client.get(url)| async def download_cardnews_image( | ||
| url: str, | ||
| *, | ||
| timeout: float = 20.0, | ||
| referer: str = "", | ||
| ) -> Image.Image | None: | ||
| if not url.startswith(("http://", "https://")): | ||
| return None | ||
| headers = { | ||
| "User-Agent": ( | ||
| "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " | ||
| "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" | ||
| ), | ||
| "Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8", | ||
| } | ||
| if referer: | ||
| headers["Referer"] = referer | ||
| try: | ||
| async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=headers) as client: | ||
| response = await client.get(url) | ||
| response.raise_for_status() | ||
| with Image.open(BytesIO(response.content)) as img: | ||
| return img.convert("RGBA") | ||
| except (httpx.HTTPError, UnidentifiedImageError, OSError): | ||
| logger.warning("카드뉴스 배경 이미지 다운로드 실패: %s", url) | ||
| return None | ||
|
|
||
|
|
||
| async def download_cardnews_images( | ||
| urls: list[str], | ||
| *, | ||
| timeout: float = 20.0, | ||
| referer: str = "", | ||
| ) -> list[Image.Image]: | ||
| images: list[Image.Image] = [] | ||
| seen: set[str] = set() | ||
| for url in urls: | ||
| url = (url or "").strip() | ||
| if not url or url in seen: | ||
| continue | ||
| seen.add(url) | ||
| img = await download_cardnews_image(url, timeout=timeout, referer=referer) | ||
| if img is not None: | ||
| images.append(img) | ||
| return images |
There was a problem hiding this comment.
download_cardnews_images 함수는 여러 이미지 URL을 순회하며 download_cardnews_image를 호출하고 있습니다. 하지만 download_cardnews_image 내부에서 매번 새로운 httpx.AsyncClient를 생성하고 닫기 때문에, 이미지 개수만큼 불필요한 클라이언트 생성/소멸 오버헤드가 발생합니다. download_cardnews_images에서 단일 AsyncClient 인스턴스를 생성한 뒤 이를 download_cardnews_image에 인자로 전달하여 재사용하도록 개선하는 것이 좋습니다.
async def download_cardnews_image(
url: str,
*,
timeout: float = 20.0,
referer: str = "",
client: httpx.AsyncClient | None = None,
) -> Image.Image | None:
if not url.startswith(("http://", "https://")):
return None
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
),
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
}
if referer:
headers["Referer"] = referer
try:
if client is not None:
response = await client.get(url)
with Image.open(BytesIO(response.content)) as img:
return img.convert("RGBA")
else:
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=headers) as local_client:
response = await local_client.get(url)
response.raise_for_status()
with Image.open(BytesIO(response.content)) as img:
return img.convert("RGBA")
except (httpx.HTTPError, UnidentifiedImageError, OSError):
logger.warning("카드뉴스 배경 이미지 다운로드 실패: %s", url)
return None
async def download_cardnews_images(
urls: list[str],
*,
timeout: float = 20.0,
referer: str = "",
) -> list[Image.Image]:
images: list[Image.Image] = []
seen: set[str] = set()
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
),
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
}
if referer:
headers["Referer"] = referer
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=headers) as client:
for url in urls:
url = (url or "").strip()
if not url or url in seen:
continue
seen.add(url)
img = await download_cardnews_image(url, timeout=timeout, referer=referer, client=client)
if img is not None:
images.append(img)
return images| doc.get("event_end_time"), | ||
| ) | ||
| continue | ||
|
|
||
| new_docs.append(doc) | ||
| existing_by_id[contentid] = doc | ||
|
|
||
| if args.delay > 0: | ||
| await asyncio.sleep(args.delay) | ||
|
|
||
| await browser.close() | ||
|
|
||
| all_rows = [ |
There was a problem hiding this comment.
현재 크롤러 스크립트는 모든 상세 페이지 수집이 완전히 끝난 후에만 결과를 파일(write_jsonl)에 저장하고 있습니다. 만약 수십 페이지를 크롤링하는 도중 네트워크 오류, 세션 만료, 혹은 사용자의 강제 종료(Ctrl+C) 등으로 스크립트가 중단되면 기존에 성공적으로 수집했던 모든 데이터가 유실되는 문제가 있습니다. 성공적으로 수집할 때마다(혹은 주기적으로) 중간 저장을 수행하여 데이터 유실을 방지하는 것이 안전합니다.
new_docs.append(doc)
existing_by_id[contentid] = doc
# 중간 저장으로 크롤링 중단 시 데이터 유실 방지
all_rows = [
normalize_contest_row(existing_by_id[k])
for k in sorted(existing_by_id.keys())
]
write_jsonl(output_path, all_rows)
if args.delay > 0:
await asyncio.sleep(args.delay)
await browser.close()
Contest API와 Policy Admin 파이프라인을 동시에 유지하도록 DI·라우터·폰트 경로 충돌을 해결한다. Co-authored-by: Cursor <cursoragent@cursor.com>
RAG 설정 필드 이중 정의를 제거하고 prod 동작값(rerank off, top_k=10)을 단일 정의로 고정한다. Gemini 팩토리·민원 서비스 조립·pin_content 유틸·카드뉴스 경로 해석을 공통 모듈로 추출해 deps/main/transform 간 중복 코드를 줄인다.
Linkareer 공모전을 정책 핀과 같은 3단계(크롤→transform→import)로 자동 적재한다. contest_api_id 기준 중복과 종료일(KST) 경과 건은 LLM·DB 모두 스킵하고, attachments/media-cdn 이미지 분류·S3 카드뉴스 업로드·매일 KST 12시 sync를 포함한다. - ContestAdminRoute: /sync, /status, crawl·transform·import 배치 API - ContestEventIngestService: pin/event_pin/community/pin_image/cardnews INSERT - EventPin.contest_api_id 및 EventPinRepo 조회·중복 제거 - contest_pin_transform: handoff 병합, S3 카드뉴스, pending/만료 스킵 - ContestPinSchedulerService 및 CONTEST_* 설정·deps·main 등록 - fetch_linkareer_contests: start_page, pin_images, KST 만료 필터 - tests/test_contest_admin_batch.py
Feat/104 contest crawler db
🔗 Related Issue
✨ 작업 개요
Linkareer 공모전 데이터를 자동 수집하고, AI 기반 CONTEST 핀 및 CARDNEWS 생성까지 이어지는 공모전 파이프라인을 구현하였습니다.
체크리스트
📷 이미지 첨부 (선택)
🧐 집중 리뷰 요청