Seonglae ChoCollector is an asynchronous task queue system for calling external cloud service (Azure, GCP) APIs.
api key for collector
replace endpoint to double queue with double credential
Time-consuming issue
- Request Creation: When the system needs to call an external API, it stores the request in the collector table
- Worker Processing: The agent-collector written in Python fetches pending tasks every 5 seconds and processes them
- Result Return: The worker calls the API and stores the result back in the database
- Client Waiting: The client that created the request polls while waiting for the result
Columns
- status: pending → completed/failed → processed
- connectionType: azure, gcp, etc.
- requestUrl: API URL to call
- response: API response result
Python Worker (agent-collector/main.py):
- Azure/GCP authentication handling
- API call execution
- Result reporting
Client (packages/core/src/collector/collector-client.ts):
- Checks result every 5 seconds after creating request
- Waits until completion
Project Scan (Connection Scan)
Project Scan is the process of scanning resources in connected cloud accounts to collect artifacts (assets).
Rescan
- Rescan: Manual rescan (packages/functions/src/events/rescanConnection/)
- Auto Scan: Automatically executed when connection is created packages/functions/src/events/connectionConnected.ts