본문 바로가기
프로그래밍언어/Python

파이썬 웹 크롤링(Python web crawling) 만들기 크롤링 데이터 RestAPI 제공할수 있는 conda기반 FastAPI 패키지 설치 및 실행 2부

by by 앵과장 2022. 10. 14.
반응형

파이썬 크롤링 이후 데이터를 전달할때

Rest API 인터페이스를 만들어서 다양한 플랫폼에서도 사용 가능하게 하기위해서 

파이썬 프레임워크가 필요할것 같아서 간단하게 사용할수 있는 프레임워크중 

이름부터 빠른 FastAPI 설치 해보도록 하겠습니다.

 

필요한 파이썬 FastAPI 설정을 해보도록 하겠습니다.

 

fastapi 설치
renzo@renzoui-MacBookPro jobkorea % conda install -c conda-forge fastapi
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/renzo/opt/anaconda3/envs/jobkorea

  added / updated specs:
    - fastapi


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    anyio-3.6.1                |     pyhd8ed1ab_1          83 KB  conda-forge
    ca-certificates-2022.9.24  |       h4653dfc_0         150 KB  conda-forge
    certifi-2022.9.24          |     pyhd8ed1ab_0         155 KB  conda-forge
    fastapi-0.85.0             |     pyhd8ed1ab_0          45 KB  conda-forge
    openssl-1.1.1q             |       ha287fd2_0         1.8 MB  conda-forge
    pydantic-1.9.0             |   py39h1a28f6b_1         1.8 MB
    sniffio-1.3.0              |     pyhd8ed1ab_0          14 KB  conda-forge
    starlette-0.20.4           |     pyhd8ed1ab_0          48 KB  conda-forge
    typing-extensions-4.4.0    |       hd8ed1ab_0           8 KB  conda-forge
    typing_extensions-4.4.0    |     pyha770c72_0          29 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         4.1 MB

The following NEW packages will be INSTALLED:

  anyio              conda-forge/noarch::anyio-3.6.1-pyhd8ed1ab_1
  fastapi            conda-forge/noarch::fastapi-0.85.0-pyhd8ed1ab_0
  pydantic           pkgs/main/osx-arm64::pydantic-1.9.0-py39h1a28f6b_1
  sniffio            conda-forge/noarch::sniffio-1.3.0-pyhd8ed1ab_0
  starlette          conda-forge/noarch::starlette-0.20.4-pyhd8ed1ab_0
  typing-extensions  conda-forge/noarch::typing-extensions-4.4.0-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-4.4.0-pyha770c72_0

The following packages will be UPDATED:

  ca-certificates    pkgs/main::ca-certificates-2022.07.19~ --> conda-forge::ca-certificates-2022.9.24-h4653dfc_0

The following packages will be SUPERSEDED by a higher-priority channel:

  certifi            pkgs/main/osx-arm64::certifi-2022.9.2~ --> conda-forge/noarch::certifi-2022.9.24-pyhd8ed1ab_0
  openssl              pkgs/main::openssl-1.1.1q-h1a28f6b_0 --> conda-forge::openssl-1.1.1q-ha287fd2_0


Proceed ([y]/n)?  y


Downloading and Extracting Packages
ca-certificates-2022 | 150 KB    | ###################################################################################################################################################### | 100%
typing-extensions-4. | 8 KB      | ###################################################################################################################################################### | 100%
certifi-2022.9.24    | 155 KB    | ###################################################################################################################################################### | 100%
fastapi-0.85.0       | 45 KB     | ###################################################################################################################################################### | 100%
openssl-1.1.1q       | 1.8 MB    | ###################################################################################################################################################### | 100%
sniffio-1.3.0        | 14 KB     | ###################################################################################################################################################### | 100%
typing_extensions-4. | 29 KB     | ###################################################################################################################################################### | 100%
starlette-0.20.4     | 48 KB     | ###################################################################################################################################################### | 100%
pydantic-1.9.0       | 1.8 MB    | ###################################################################################################################################################### | 100%
anyio-3.6.1          | 83 KB     | ###################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

 

uvicorn 설치
renzo@renzoui-MacBookPro jobkorea-web % conda install uvicorn
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.13.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/renzo/opt/anaconda3/envs/jobkorea

  added / updated specs:
    - uvicorn


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    asgiref-3.5.2              |   py39hca03da5_0          39 KB
    h11-0.12.0                 |     pyhd3eb1b0_0          47 KB
    uvicorn-0.16.0             |   py39hca03da5_0          82 KB
    ------------------------------------------------------------
                                           Total:         168 KB

The following NEW packages will be INSTALLED:

  asgiref            pkgs/main/osx-arm64::asgiref-3.5.2-py39hca03da5_0
  click              pkgs/main/osx-arm64::click-8.0.4-py39hca03da5_0
  h11                pkgs/main/noarch::h11-0.12.0-pyhd3eb1b0_0
  uvicorn            pkgs/main/osx-arm64::uvicorn-0.16.0-py39hca03da5_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
uvicorn-0.16.0       | 82 KB     | ################################################################################# | 100%
asgiref-3.5.2        | 39 KB     | ################################################################################# | 100%
h11-0.12.0           | 47 KB     | ################################################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

 

FastAPI 를 사용하기 위해서는 fastapi, uvicorn 2개의 패키지를 설치 해야합니다.

uvicorn 패키지를 통해서 Phthon 서버가 실행되기에 필수적으로 진행하셔야합니다.

 

main.py 생성

# main.py

from fastapi import FastAPI # FastAPI 모듈 가져오기

app = FastAPI() # 객체 생성

@app.get("/") # Route Path
def test_index():

    # Json 타입으로 값 반환
    return {
	    "Python": "Framework",
	}

@app.get("/something")
def something():
    return {
        "Something": "Page",
    }

코드는 java만 해본저도 어렵지 않게 이해가됩니다.

fastAPI를 import

Route 할수 있는 Rest API GET방식 2건 입니다.

 

Springboot  기반에 Rest API만들려면 이것보다는 손도 많이가고 코드의 양도 많아지는데 Python은 정말 

Fast 하네요!!

 

Python 실행 명령어 
uvicorn main:app --reload --port 8000
  • uvicorn : 파이썬 서버 실행을 위해 필요한 기본 명령어
  • main : 실행할 초기 파이썬 파일 이름. 위의 소스코드를 test.py 파일에 작성하였을 경우, test:app으로 변경하여 실행
  • app : FastAPI() 모듈을 할당한 객체명을 뜻한다. practice = FastAPI() 일 경우, main.practice로 변경하여 실행
  • reload : 소스코드가 변경되었을 때, 서버를 자동으로 재시작해주는 옵션이다. 개발단계일 때 유용하게 사용가능
  • port : 해당 포트번호를 통해 서버를 실행시켜주는 옵션. 포트 옵션을 생략시 자동으로 8000번 포트가 할당
renzo@renzoui-MacBookPro jobkorea % uvicorn main:app --reload --port 8000
INFO:     Will watch for changes in these directories: ['/Users/renzo/opt/anaconda3/envs/jobkorea']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [97004] using statreload
INFO:     Started server process [97007]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:65428 - "GET / HTTP/1.1" 200 OK
INFO:     127.0.0.1:65428 - "GET /favicon.ico HTTP/1.1" 404 Not Found

정상적으로 기동이 되면 오류 없이 INFO 에 정상적으로 startup된걸 확인할수 있습니다.

로그정보도 Springboot보다 심플하네요 

그래 옷이던 사람이던 개발언어든 심플한게 좋네요 !!

 

chrome 브라우저에서 아래 내용을 호출하면 코드레벨에 정의한 Response 가 정상호출되면 RestAPI 준비는 완료입니다.
http://localhost:8000/something 

{"Something":"Page"}

url 호출하면 아래처럼 Log 확인가능합니다.

INFO:     Will watch for changes in these directories: ['/Users/renzo/opt/anaconda3/envs/jobkorea']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [97354] using statreload
INFO:     Started server process [97361]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:65466 - "GET /something HTTP/1.1" 200 OK
API Doc

와 Springboot 는 Springdoc, Swagger config  설정하고 Java에 별도 Annotation 해주고 그래야되는데 
그냥 되네 역시  Python은 정말 짱인거같아요