Gfrktyl

I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and can extract the table of information I need.

The HTML for the table looks like this (yes, it has the commented out lines)

The I have below works, but it feels inefficient. I feel like I have a lot of loops and these feel unnecessary. I'm not sure how to be more efficient though.

The only change between this and my real code is that I've moved the HTML to pastebin to cut down on the length of this post. That pastebin is here and the HTML is not written or editable by me. It's generated on the cable modem. Since you aren't able to see the results of my cable modem by running the code I use to extract it from the modem, I hope this is good enough.

from bs4 import BeautifulSoup

import requests



results_url = "https://pastebin.com/raw/bLZLFzy6"

content = requests.get(results_url).text



measurement = "modem"

hostname = "home-modem"





def strip_uom(data):

    """Strip Unit of Measurement



    Some of our fields come with unit of measure. Rip that out and keep only 

    the value of the field.

    """

    uom = ["dB", "Hz", "dBmV", "Ksym/sec"]

    dataset = 

    for d in data:

        if d.split(" ")[-1] in uom:

            dataset.append(d.split(" ")[0])

        else:

            dataset.append(d)

    return dataset





soup = BeautifulSoup(content, 'html.parser')

dstable = soup.find('table', {'id': 'dsTable'})



# Pull the headers from the downstream table 

# We want to make these tag friendly, so make them lowercase and remove spaces

dstable_tags = [td.get_text().lower().replace(" ", "_") for td in dstable.tr.find_all('td')]



# Pull out the rest of the data for each row in the table; associate it with the correct tag; Strip UoM 

downstream_data = 

for row in dstable.find_all('tr')[1:]:

    column_values = [col.get_text() for col in row.find_all('td')]

    downstream_data.append(dict(zip(dstable_tags, strip_uom(column_values))))



# Print line protocol lines for telegraf's inputs.exec plugin to handle

for data in downstream_data:

    line_protocol_line = f"{measurement},hostname={hostname}"

    fields = 

    for key, value in data.items():

    # Check if our value is a number. If it's not, surround it in quotes. 

    # Don't actually use the float() value, as some numbers are returned as 

    # valid integers

        try:

            _ = float(value)

            fields.append(f'{key}={value}')

        except ValueError:

            fields.append(f'{key}="{value}"')

    fieldset = ",".join(fields)

    line_protocol_line = line_protocol_line + f" channel={data['channel']} {fieldset}"

    print(line_protocol_line)

Finally, the output this script generates is this:

modem,hostname=netgear-cm400 channel=1 channel=1,lock_status="Locked",modulation="QAM 256",channel_id=121,frequency=585000000,power=5.6,snr=37.0,correctables=19443,uncorrectables=11263

modem,hostname=netgear-cm400 channel=2 channel=2,lock_status="Locked",modulation="QAM 256",channel_id=6,frequency=591000000,power=5.7,snr=37.0,correctables=19531,uncorrectables=9512

modem,hostname=netgear-cm400 channel=3 channel=3,lock_status="Locked",modulation="QAM 256",channel_id=7,frequency=597000000,power=5.7,snr=36.8,correctables=17457,uncorrectables=9736

modem,hostname=netgear-cm400 channel=4 channel=4,lock_status="Locked",modulation="QAM 256",channel_id=8,frequency=603000000,power=5.9,snr=37.0,correctables=12750,uncorrectables=11156

modem,hostname=netgear-cm400 channel=5 channel=5,lock_status="Locked",modulation="QAM 256",channel_id=122,frequency=609000000,power=2.6,snr=36.3,correctables=1855538,uncorrectables=18388

modem,hostname=netgear-cm400 channel=6 channel=6,lock_status="Locked",modulation="QAM 256",channel_id=10,frequency=615000000,power=2.6,snr=37.0,correctables=846194,uncorrectables=14615

modem,hostname=netgear-cm400 channel=7 channel=7,lock_status="Locked",modulation="QAM 256",channel_id=11,frequency=621000000,power=2.6,snr=37.6,correctables=281431,uncorrectables=13998

modem,hostname=netgear-cm400 channel=8 channel=8,lock_status="Locked",modulation="QAM 256",channel_id=12,frequency=627000000,power=2.4,snr=36.1,correctables=78059,uncorrectables=13695

I have loops and list comprehensions to get the tags, associate data with tags for each row, and to generate the line protocol lines. I have two additional comprehensions in those and a loop to see if I need to strip out an included unit of measure. Can I eliminate some of these to be more efficient?

I am running all of this code on Python 3.6.7.