Gfrktyl

Preface: This is my first work script, normally I'm just mucking around in console or writing snippets. This script has a lot of firsts for me (configs, logging instead of print, try/except). Typically when I just write snippets I don't care if things are breaking. Given this is set up as a scheduled task for work, I tried to make it a bit less... um... brittle.

Problem

The web app i'm logging into stores log files on a system admin profile as attachments. I needed to write a script to automatically log in and download these attachments as it only has a small retention period before they disappear.

General Plan of Action

1. GET webapp/ page to retrieve cookies


2. GET webapp/login to retrieve form input ids used for posting credentials

GET login response

form    -23nuyjabt3o6jyz20bze4lt19wjxqihqpzcja0kiy0osistaqb,-2jnri62v92sbs0qooyazpdfrwv2561kl49fulslch1thh43c0s,2qvew89xgsjkg0u0uhhmuxjxqp8kb71t6k827iaofkmepu7c47

3. POST username to webapp/uuid to retrieve UUID

uuid params

userName    testuser

uuid response

uuid    c8637a56-1495-4388-888b-0c35aff86974

4. Hash password using UUID and random string (SHA3-KACCAK)


5. POST credentials (hashed pw, username) mapped to the random form IDS from above.

POST /login params using the form IDS from previous GET /login

-23nuyjabt3o6jyz20bze4lt19wjxqihqpzcja0kiy0osistaqb 

-2jnri62v92sbs0qooyazpdfrwv2561kl49fulslch1thh43c0s a3a3574d05e0de1fa30d898e8ac425e44be2f19b7f8119a1ae27eeb4e3d0e445679554ba12fd44f120784465bc18cc512eaababec00a4ec9c03f11b2d64208ae

 2qvew89xgsjkg0u0uhhmuxjxqp8kb71t6k827iaofkmepu7c47 testuser

6. Get the current users profile page id by finding "View My Profile" link and getting value of kUserID parameter.




7. GET worker profile page and find all attachments



8. Download attachments to disk if they don't already exist.

log_downloader.py

Logs into web app, navigates to own worker profile and downloads attachment files that match the whitelist.

import configparser

import logging

import requests

import sha3

from logging.handlers import RotatingFileHandler

from pathlib import Path

from urllib.parse import urlparse, parse_qsl

from bs4 import BeautifulSoup

from getpass import getpass



logger = logging.getLogger(__name__)

handler = logging.StreamHandler()

fh = RotatingFileHandler('log.txt', maxBytes=1024, backupCount=5)

formatter = logging.Formatter(

        '%(asctime)s %(name)-12s %(levelname)-8s %(message)s')

handler.setFormatter(formatter)

fh.setFormatter(formatter)

logger.addHandler(handler)

logger.addHandler(fh)

logger.setLevel(logging.DEBUG)





config = configparser.ConfigParser()

config.read('config.ini')

URL = config['Penelope']['url']



sess = requests.session()





def get_uuid(username: str):

    """

    Returns a UUID for the provided username.

    :param username: Penelope User Name

    :return: str

    """

    try:

        r = sess.get(f'{URL}acm_loginControl')

        r = sess.post(f'{URL}acm_loginControl/uuid', {'userName': username})

        _uuid = r.json()['uuid']

        logger.debug(f'UUID Retrieved: {_uuid}')

        return _uuid

    except Exception as e:

        print(e)





def digest_password(uuid: str, password: str):

    """

    Digests a password using **SHA3 (Keccak)** to match Crypto.JS lib used in Penelope login

    :param uuid: Generated UUID for username.

    :param password: pw to be digested

    :return: str

    """

    h = sha3.keccak_512()



    h.update(uuid.encode())

    h.update('algo != null && algo !== 0'.encode())  # Random string in Penelope JS script

    h.update(password.encode())



    hexdigest = h.hexdigest()

    logger.debug(f'Password Hashed: {hexdigest}')

    return hexdigest





def login(username: str, digest: str):

    if 'authentype' not in sess.cookies or 'JSESSIONID' not in sess.cookies:

        try:

            # Gets token from login page

            r = sess.get(f'{URL}acm_loginControl')

            # Gets form IDs generated by server based off timestamp

            r = sess.get(f'{URL}acm_loginControl/login')



            # Set Credentials to form IDS

            form_ids = r.json()['form'].split(',')

            creds = {form_ids[0]: '', form_ids[2]: username, form_ids[1]: digest}



            # Post Credentials

            logger.debug('Trying to Log In')

            r = sess.post(f'{URL}acm_loginControl/login', creds)

            logger.debug(r.json())

            if r.json()['state'] == 'ok':

                logger.debug('Logged in Successfully')

                r = sess.get(f'{URL}acm_loginControl/create')

                r.raise_for_status()

            else:

                raise ValueError(r.json()['errorCode'])

            return r

        except ValueError as e:

            logger.error(e)



        except Exception as e:

            logger.error(e)



    else:

        logger.debug('Not Logging in')

        return f'Session already exists: {sess.cookies}'





def navigate_to_workerprofile(user_id: int):

    return sess.get(f'{URL}acm_userProfileControl?actionType=view&kUserID={user_id}')





def get_user_id(login_response: requests.models.Response):

    """Get Worker Profile kUserID (wruser table)"""

    soup = BeautifulSoup(login_response.content, 'html.parser')

    home_page_link = soup.find(id='frm_content_id')['src']

    r = sess.get(f'{URL}{home_page_link}')

    soup = BeautifulSoup(r.content, 'html.parser')

    href = next(link.get('href') for link in soup.find_all('a') if link.text == 'View My Profile')

    qs = urlparse(href)

    params = dict(parse_qsl(qs.query))

    logger.debug(f'kUserID Retrieved: {params["kUserID"]}')

    return params['kUserID']



def find_log_attachments():

    logger.debug('Searching for stdout log files')

    soup = BeautifulSoup(r.content, 'html.parser')

    attachment_table = soup.find(id='attachListTable_worker')

    logs = [link.text for link in attachment_table.find_all('a') if link.text.startswith('stdout')]

    logger.debug(logs)

    return logs





def download_attachment(file_name):

    download_url = f'{URL}acm_attachmentControl?actionType=download' 

                   f'&attachCat=worker&attachCatID={kuserid}&attachName={file_name}'

    logger.debug(f'Downloading {file_name}')

    r = sess.get(download_url)

    if 'JSESSIONID' not in sess.cookies:

        raise ConnectionError

    return r





def save_log_files():

    log_attachments.sort()

    for log_name in log_attachments:

        log_file_path = Path(f'E:/Penelope Logs/RAQ.Athena-au.com/{log_name}')



        if not log_file_path.exists():

            try:

                log_download_response = download_attachment(log_name)

                with open(log_file_path, 'wb') as log_file:

                    log_file.write(log_download_response.content)

                    logger.debug(f'Log File {log_file_path} saved to disk')

            except ConnectionError as e:

                logger.error('Connection no longer valid. No session token.')

        else:

            logger.debug(f'{log_file_path} already exists')



if __name__ == '__main__':

    if not config.has_option('Penelope', 'user'):

        user = input('Enter Username to login with')

        config['Penelope']['user'] = user

    if not config.has_option('Penelope', 'password'):

        password = getpass('Enter Password')

        user = config['Penelope']['user']

        uuid = get_uuid(user)

        digest = digest_password(uuid, password)

        config['Penelope']['password'] = digest

        with open('config.ini', 'w') as config_file:

            config.write(config_file)



    user = config['Penelope']['user']

    digest = config['Penelope']['password']



    r = login(user, digest)

    kuserid = get_user_id(r)

    r = navigate_to_workerprofile(kuserid)

    log_attachments = find_log_attachments()

    save_log_files()

Output log file - PW already hashed / 1 new log file to DL

2019-03-01 08:00:00,969 urllib3.connectionpool DEBUG    Starting new HTTPS connection (1): mywebapp:443

2019-03-01 08:00:01,099 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl HTTP/1.1" 200 None

2019-03-01 08:00:01,140 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl/login HTTP/1.1" 200 None

2019-03-01 08:00:01,141 root         DEBUG    Trying to Log In

2019-03-01 08:00:01,183 urllib3.connectionpool DEBUG    https://mywebapp:443 "POST /acm_loginControl/login HTTP/1.1" 200 None

2019-03-01 08:00:01,222 root         DEBUG    {'next': 'create', 'state': 'ok'}

2019-03-01 08:00:01,223 root         DEBUG    Logged in Successfully

2019-03-01 08:00:01,331 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_loginControl/create HTTP/1.1" 302 83

2019-03-01 08:00:01,378 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_topFrameControl HTTP/1.1" 200 None

2019-03-01 08:00:01,491 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET //acm_homepageControl?actionType=top_frame&curval=1551391201355 HTTP/1.1" 200 None

2019-03-01 08:00:01,823 root         DEBUG    kUserID Retrieved: 1077

2019-03-01 08:00:01,877 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_userProfileControl?actionType=view&kUserID=1077 HTTP/1.1" 200 None

2019-03-01 08:00:01,941 root         DEBUG    Searching for stdout log files

2019-03-01 08:00:01,960 root         DEBUG    ['stdout_2019-02-26.0.log.gz', 'stdout_2019-02-25.2.log.gz', 'stdout_2019-02-25.1.log.gz', 'stdout_2019-02-25.0.log.gz', 'stdout_2019-02-24.0.log.gz', 'stdout_2019-02-23.0.log.gz', 'stdout_2019-02-22.1.log.gz', 'stdout_2019-02-22.0.log.gz', 'stdout_2019-02-28.1.log.gz', 'stdout_2019-02-28.0.log.gz', 'stdout_2019-02-27.1.log.gz', 'stdout_2019-02-27.0.log.gz', 'stdout_2019-02-26.2.log.gz', 'stdout_2019-02-26.1.log.gz']

2019-03-01 08:00:01,960 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-22.0.log.gz already exists

2019-03-01 08:00:01,961 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-22.1.log.gz already exists

2019-03-01 08:00:01,961 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-23.0.log.gz already exists

2019-03-01 08:00:01,961 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-24.0.log.gz already exists

2019-03-01 08:00:01,962 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-25.0.log.gz already exists

2019-03-01 08:00:01,962 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-25.1.log.gz already exists

2019-03-01 08:00:01,962 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-25.2.log.gz already exists

2019-03-01 08:00:01,962 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-26.0.log.gz already exists

2019-03-01 08:00:01,963 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-26.1.log.gz already exists

2019-03-01 08:00:01,963 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-26.2.log.gz already exists

2019-03-01 08:00:01,963 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-27.0.log.gz already exists

2019-03-01 08:00:01,964 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-27.1.log.gz already exists

2019-03-01 08:00:01,964 root         DEBUG    E:Penelope Logsmywebappstdout_2019-02-28.0.log.gz already exists

2019-03-01 08:00:01,964 root         DEBUG    Downloading stdout_2019-02-28.1.log.gz

2019-03-01 08:00:01,989 urllib3.connectionpool DEBUG    https://mywebapp:443 "GET /acm_attachmentControl?actionType=download&attachCat=worker&attachCatID=1077&attachName=stdout_2019-02-28.1.log.gz HTTP/1.1" 200 8964328

2019-03-01 08:00:02,509 root         DEBUG    Log File E:Penelope Logsmywebappstdout_2019-02-28.1.log.gz saved to disk