Automated Class timetable optimize crawler?

Overall Plan

Get my class information to automatically optimize and select my uni class timetable

Overall Algorithm

  1. Logon to the website using its Enterprise Sign On Engine login
  2. Find my current semester and its related subjects (pre setup)
  3. Navigate to the right page and get the data from each related subject (lecture, practical and workshop times)
  4. Strip the data of useless information
  5. Rank the classes which are closer to each other higher, the ones on random days lower
  6. Solve a best time table solution
  7. Output me a detailed list of the BEST CASE information
  8. Output me a detailed list of the possible class information (some might be full for example)
  9. Get the program to select the best classes automatically
  10. Keep checking to see if we can achieve 7.

6 in detail Get all the classes, using the lectures as a focus point, would be highest ranked (only one per subject), and try to arrange the classes around that.

Questions

Can anyone supply me with links to something that might be similar to this hopefully written in python? In regards to 6.: what data structure would you recommend to store this information in? A linked list where each object of uniclass? Should i write all information to a text file?

I am thinking uniclass to be setup like the following attributes:

  • Subject
  • Rank
  • Time
  • Type
  • Teacher

I am hardly experienced in Python and thought this would be a good learning project to try to accomplish. Thanks for any help and links provided to help get me started, open to edits to tag appropriately or what ever is necessary (not sure what this falls under other than programming and python?)

EDIT: can't really get the proper formatting i want for this SO post ><


Asked by: Madaline942 | Posted: 01-10-2021






Answer 1

Depending on how far you plan on taking #6, and how big the dataset is, it may be non-trivial; it certainly smacks of NP-hard global optimisation to me...

Still, if you're talking about tens (rather than hundreds) of nodes, a fairly dumb algorithm should give good enough performance.

So, you have two constraints:

  1. A total ordering on the classes by score; this is flexible.
  2. Class clashes; this is not flexible.

What I mean by flexible is that you can go to more spaced out classes (with lower scores), but you cannot be in two classes at once. Interestingly, there's likely to be a positive correlation between score and clashes; higher scoring classes are more likely to clash.

My first pass at an algorithm:

selected_classes = []
classes = sorted(classes, key=lambda c: c.score)
for clas in classes:
    if not clas.clashes_with(selected_classes):
        selected_classes.append(clas)

Working out clashes might be awkward if classes are of uneven lengths, start at strange times and so on. Mapping start and end times into a simplified representation of "blocks" of time (every 15 minutes / 30 minutes or whatever you need) would make it easier to look for overlaps between the start and end of different classes.

Answered by: Brad528 | Posted: 02-11-2021



Answer 2

BeautifulSoup was mentioned here a few times, e.g get-list-of-xml-attribute-values-in-python.

Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:

  1. Beautiful Soup won't choke if you give it bad markup. It yields a parse tree that makes approximately as much sense as your original document. This is usually good enough to collect the data you need and run away.
  2. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. You don't have to create a custom parser for each application.
  3. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't autodetect one. Then you just have to specify the original encoding.

Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. You can tell it "Find all the links", or "Find all the links of class externalLink", or "Find all the links whose urls match "foo.com", or "Find the table heading that's got bold text, then give me that text."

Valuable data that was once locked up in poorly-designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup.

Answered by: Elian964 | Posted: 02-11-2021



Answer 3

There are waaay too many questions here.

Please break this down into subject areas and ask specific questions on each subject. Please focus on one of these with specific questions. Please define your terms: "best" doesn't mean anything without some specific measurement to optimize.

Here's what I think I see in your list of topics.

  1. Scraping HTML

    1 Logon to the website using its Enterprise Sign On Engine login

    2 Find my current semester and its related subjects (pre setup)

    3 Navigate to the right page and get the data from each related subject (lecture, practical and workshop times)

    4 Strip the data of useless information

  2. Some algorithm to "rank" based on "closer to each other" looking for a "best time". Since these terms are undefined, it's nearly impossible to provide any help on this.

    5 Rank the classes which are closer to each other higher, the ones on random days lower

    6 Solve a best time table solution

  3. Output something.

    7 Output me a detailed list of the BEST CASE information

    8 Output me a detailed list of the possible class information (some might be full for example)

  4. Optimize something, looking for "best". Another undefinable term.

    9 Get the program to select the best classes automatically

    10 Keep checking to see if we can achieve 7.

BTW, Python has "lists". Whether or not they're "linked" doesn't really enter into it.

Answered by: Kellan651 | Posted: 02-11-2021



Similar questions

django - Flickr API automated login using Python library flickrapi

I have a web application that I want to sync with Flickr. I don't want the users to have to log into Flickr so I plan to use a single login. I believe I'll need to do something like this: import flickrapi flickr = flickrapi.FlickrAPI(myKey, mySecret) (token, frob) = flickr.get_token_part_one(perms='write', my_auth_callback) flickr.get_token_part_two((token, frob,)) flickr.what_have_you(...


python - How do I retrieve an automated report and save it to a database?

I've got a web server that will take scripts in Python, PHP or Perl. I don't know much about any of those languages, but of the three, Python seems the least scary. It has a MySql database set up, and I know enough SQL to manage it and write queries for it. I also have a program that I want to add automated error reporting to. Something goes wrong, it sends a bug report to my server. What I don't know ho...


database - python django automated data addition

I have a script which reads data from a csv file. I need to store the data into a database which has already been created as $ python manage.py syncdb so, that automated data entry is possible in an easier manner, as available in the django shell.


automated tests - Python 2.x: how to automate enforcing unicode instead of string?

How can I automate a test to enforce that a body of Python 2.x code contains no string instances (only unicode instances)? Eg. Can I do it from within the code? Is there a static analysis tool that has this feature? Edit: I wanted this for an application in Python 2.5, but it turns out this is not really possible because: 2.5 doesn't support unico...


Is there an automated way to install python modules (à la rubygems)?

I'm new to python. Are there alternatives to downloading a tarball for a python module &amp; installing it via python setup install ? Anything like rubygems? UPDATE I'm surprised that there are so many solutions for this one problem.


python - Is there any automated tool to generate a script for me to view and edit data based on database table

Currently, I am having Apache running, a PostgreSQL database, and several Python CGI script. I just don't want to write additional script to manipulate the following simple table structure. Machine Name | IP Address ========================= I was wondering, is there any tool, which can help me to auto generate a script/web page for me based on that table, so that I can


vim - How to have automated headers for python files

A proper header format in python is described here . Using either VIM or a shell script, I would like to have the usual metadata (like __author__, __authors__, __contact__, __copyright__, __license__, __deprecated__, __date__ and __version__) added to the file header. SVN keywords would also be nice. Addi...


Automated alignment of "tabular" Python code

Closed. This question does not meet Stack Overflow guid...


python - What controls automated window resizing in Tkinter?

Tkinter top level windows seem to have two "modes": where the size is being determined by the application, and where the user controls the size. Consider this code: from tkinter import * class Test(Frame): def __init__(self,parent): Frame.__init__(self,parent) self.b1 = Button(self, text="Button 1",command=self.b1Press) self.b1.pack() def b1Press(self): print("b1Pre...


python - Non-GUI Automated System Test Framework

I am looking for an automated test framework for testing a system with the following characteristics: 1. Non-GUI System 2. Has backend nodes running on multiple Linux/Unix hosts 3. Has remotely executing processes against which we must test 4. Results in a lot of DB access. The testing performed is system/integration/acceptance testing, not unit testing. Am language agnostic with respect to the fr...






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top