TagUI RPA Challenge submissions

  • TagUI RPA Challenge submissions

     TWS updated 1 year, 1 month ago 7 Members · 85 Posts
  • cheehuat

    August 21, 2021 at 5:58 pm

    My attempt on RPA Challenge by Automation Anywhere (Week 3). Achieved this timing after some guidance from @kensoh !

    • kensoh

      August 23, 2021 at 12:47 pm

      Congrats @cheehuat for being the fastest winner for week 3! 🙂 I’ll private message you on verifying your solution and send you the prize.

      • cheehuat

        August 23, 2021 at 8:32 pm

        Thanking all the experts here for lying low and give chance on this one.


  • kensoh

    August 24, 2021 at 1:11 am

    Congrats to winners for week #3! – cheehuat and Mirza Ahsan Baig. I’ve reached out to send the prize (worth $500 USD, LearnAI premium subscription + Datacamp annual premium subscription).

    A special shout out to Mirza for winning this week but decided to give the prize to somebody else who may benefit more. I’ll recommend him to give the prize to his lecturer who could give it to one of his students working on TagUI RPA.

    My solution and tips for week #3 – https://github.com/kelaberetiv/TagUI/issues/1093#issuecomment-903729704

    Week #4 challenge – https://developer.automationanywhere.com/blog/challenge-page-introaccountspayablechallenge

    Week #4 website – https://developer.automationanywhere.com/challenges/automationanywherelabs-invoiceentry.html

    Cut-off date – by end of Sunday Singapore time (UTC+8)

    Tip 1 – This is a pretty difficult challenge. And there are 3 invoice formats to be taken care of. See this LinkedIn comment thread for a few tips and feel free to ask anything there or in Telegram group chat or at this community thread here if you run into blockers.

    Tip 2 – An updated Tournament Edition of TagUI is created to up the fun factor. This updated version will also speed up the keyboard step. See the link below to see how you can enable and disable turbo mode with 1 line. Instructions included for both TagUI human language and Python versions. https://github.com/kelaberetiv/TagUI/issues/1093

  • kensoh

    August 26, 2021 at 9:49 am

    For week 4 of the RPA challenge, there are 4 main parts to it. I don’t think there is any entry yet using TagUI. If any part is a blocker let me know I can share the solution or tips for the part.

    1. extracting text from image invoices using OCR

    2. looping through the OCR text files to process

    3. extracting useful data from the extracted text

    4. filling up the website and using upload button

    This is the demo of solution I submitted using Python version of TagUI, was the fastest to post solution for the Automation Anywhere Challenge, compared to any other users using any other tools (commercial or open-source) – https://www.linkedin.com/posts/kensoh_botgames-rpaugust-activity-6835498514760499200-DB1M

    • matinciel

      August 26, 2021 at 2:46 pm

      thanks for that, if you could help for the 3, I have seen your comment in Linkedin about using the get_text(), but I not able to find what to search for (eg. for the date as the order date is not extracted on the same line as the actual date.

      • kensoh

        August 26, 2021 at 5:00 pm

        Hi Matinciel, yes sure. Because the solution I chose is on-prem and not relying on any cloud OCR service, I’ll have to use built-in functions like get_text() and del_chars(), along with standard Python functions to extract the data. Below is an example of 1 of the 3 invoice formats.

        There are 3 different formats in the 7 invoice samples. Using below method requires writing the logic for each format and using something unique as the ‘signature’ to decide which format is it. In below case, one format has the ‘Ship to Invoice no.’ string in the extracted OCR.

                if 'Ship to Invoice no.' in r.load(filename):
        ocr_text = r.get_text(r.load(filename), 'Invoice no.', 'Terms: ')
        ocr_line = ocr_text.split('\n')
        invoice_number = ocr_line[0].split(' ')[-1]
        invoice_date = ocr_line[2].split(' ')[-3] + ' ' + ocr_line[2].split(' ')[-2] + ' ' + ocr_line[2].split(' ')[-1]
        invoice_total = r.get_text(ocr_text + '$', 'Invoice Amount', '$')
        r.type('#invoiceNumber', invoice_number)
        r.type('#invoiceDate', invoice_date)
        r.type('#invoiceTotal', invoice_total.replace(',', ''))
        ocr_text = r.get_text(r.del_chars(ocr_text,'{}[]'), 'Tax Amount', 'Subtotal')
        ocr_line = ocr_text.split('\n')
        for item in range(len(ocr_line)):
        if ocr_line[item].count('|') == 1:
        ocr_line[item] = ocr_line[item].replace(' G ', ' | G ')
        quantity = ocr_line[item].split('|')[0].strip()
        item_no = ocr_line[item].split('|')[1].strip().split(' ')[0]
        description = ocr_line[item].split('|')[1].replace(item_no, '').strip()
        unit_price = description.split(' ')[-1]
        description = description.replace(unit_price, '').strip()
        total_price = ocr_line[item].split('|')[-1].split(' ')[-1]
        r.type('#quantity_row_' + str(item + 1), quantity)
        r.type('#description_row_' + str(item + 1), description)
        r.type('#price_row_' + str(item + 1), total_price.replace(',', ''))
        if item != len(ocr_line) - 1:
        • kensoh

          August 26, 2021 at 5:05 pm

          For comparison how using a 3rd party service provider can make things easy, eg Automation Anywhere IQ Bot, or Amazon Textract service below, used by Robocorp for week 4. Amazon Textract comes with 1000 free pages a month for up to 3 months trial.

          And something like below is how to return field details like invoice number, date, total and the different line items. This is of course, base on the premise, that the scanned PDF is of very high-quality, leading to the OCR done being very accurate without further data cleaning.


          def extract_invoice_data(aws_lib, invoice_file):
          model = aws_lib.analyze_document(
          invoice_file, f"{invoice_file}.json", model=True)
          for page in model.pages:
          fields = page.form.fields
          tables = page.tables
          return {"fields": fields, "tables": tables}
  • matinciel

    August 27, 2021 at 8:27 pm

    Tada ! (not claiming the price, before @kensoh remind me 😉 ) but taking first and last rank maybe…

    With a very (very very) ugly code (no function to treat any of the Invoicex.tiff, only copy paste…) but it’s finally working.

    I use PyTessaract as OCR (+ the code provide by Ken to treat the lines, with adjustment to retrieve the first three data (date, total, number).

    Just to add a bit of problem , I encouter a wierd issue (only with the last invoice10.tiff) :

    UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x97 in position 779: invalid start byte

    with open('facture.txt', mode = 'w') as f:


    solved with :

    with open('facture.txt', mode = 'w',encoding='utf-8') as f:


    • kensoh

      August 28, 2021 at 1:21 am

      Oh nice solving this final and toughest challenge! Thanks Matinciel for trying this tough challenge using local on-prem method and sharing back the solution to the roadblock. It looks like probably the OCR engine used gave a non-UTF8 output.

  • TWS

    August 29, 2021 at 8:57 am

    Final problem completed in about 18 seconds using Python TagUI with PyTesseract. The is my fastest timing after few tries.

    My key takeaway is that TagUI, along with other open-source solutions, able to hold up against existing proprietary solution, and performed respectfully well in these challenges. I am particularly excited to see the convenience of using RPA thru Python integration to generate more data for machine learning. @kensoh and your team, thanks for making all this possible!

    • kensoh

      August 29, 2021 at 10:18 am

      Thanks @TWS for sharing your result and thoughts! 😀 I appreciate it, and I’m glad that Automation Anywhere organised these challenges for the broader RPA community to practise and advance together in their skills.

    • matinciel

      August 29, 2021 at 5:47 pm

      Great ! Would you be kind to share your code ?

      • kensoh

        August 29, 2021 at 11:50 pm

        Would like to share a nice solution using tesseract and regex by Daniel from Brazil – https://github.com/DanielCCF/BotGamesAA/blob/master/Week4/Solution-Python.py

      • TWS

        August 30, 2021 at 8:50 am

        Sure, @matinciel! This is my solution for TagUI RPA Challenge Week 4.


        • matinciel

          August 30, 2021 at 4:16 pm

          Thanks for sharing.

          It seems I can’t run ipybn on VS code… (I can, but I got an weird error telling my can’t find pytesseract…but if run the command myself it works.).

          I copy paste the code to make a Python file, and it runs.

          I think you don’t need that (thus, you don’t need visual_automation) @kensoh will confirm or not…



          as r.init() already launch a Chrome instance, then just use r.url().

          It seems we do not need Selenium anywhere ? (I remove that, so maybe that why, I have the issue below)

          from selenium import webdriver
          from selenium.webdriver.common.action_chains import ActionChains

          Running you code always give me the same result (nothing happen on the webpage).

          And the final print indicates :

          Timing= 78 seconds / Accuracy= 100%. Input error. Please try again

          (changing the final if accuracy = 100.00% to 100%, “works” that to say, I get the print as :

          Timing= 78 seconds / Accuracy= 100%

          But the screenshot shows a the webpage as it was opened (not showing the result).

          I have no idea there those values come from :p.

          (thanks for testing the other posted solution).

          • TWS

            August 30, 2021 at 7:51 pm

            One of the rule in the challenge required to use file upload capability. This ‘r.click’ step is to ensure browser stay in the foreground throughout the run so that the bot can type image file path into window dialog.

            We should be able to remove importing of Selenium without any issue. Initially, I was using Selenium action chain for mouse movement but decided not to go ahead with that.

            Since the final print is to extract the timing and accuracy from popup result screen, the accuracy is always in float. I not sure why there is a need to change accuracy format from float to integer in the code. I thought ‘r.read’ should be getting the values from previous chrome browser opened by TagUI. Are you able to run ‘r.close’ line successfully for every run?

            • matinciel

              August 30, 2021 at 7:58 pm

              Thanks, still can’t get it work even with r.close(), the strange part is that I am even not getting anything filled up during the process.

              I don’t think you need the visual automation (nor that the browser to be visible) for the r.click().

              Anyway it seems I having issue on replicating other works (100% due to my newbyness ^^).

            • TWS

              August 30, 2021 at 8:28 pm

              Yeah, ‘r.click()’ itself doesn’t require visual automation but ‘r.click(chrome.png)’ is using visual recognition for application outside of web browser (moving mouse cursor to the chrome icon in the taskbar).

              Thru private message, can share the file you created with my code, I will try to check it out. No problem, everyone is learning.

            • matinciel

              August 30, 2021 at 8:51 pm

              😅 having folder for the invoices works better…

              keyboard and clipboard need visual automation = True…

              Maybe an easier way to handle the file upload (with all the invoice in the same folder as working directory)

              I suppose having a path for the second should work too


            • TWS

              August 30, 2021 at 10:40 pm

              Great! Hope it works for you. 😀

              Yes, ‘r.upload’ can work too. I think it will be faster and more stable. Just that part of the challenge criteria by Automation Anywhere is to open the window dialog for uploading.

  • cheehuat

    August 29, 2021 at 1:33 pm

    Finally solved Week 4 of the RPA challenge with pure TagUI! Due to some Sikulix issue on my local machine, I initially solved this problem with a mix of TagUI and PyAutoGUI. After some guidance from @kensoh I solved it with just TagUI web automation (and hence eligible to post here.. ). I used pytesseract to extract information from the invoice files, and did some processing before launching the RPA flow. Learnt and grow (hopefully) a lot from this week’s challenge!

    • kensoh

      August 29, 2021 at 11:53 pm

      Nice entry Chee Huat! Your solution should be replicable on other computers and OS, and for other invoices. The main limitation of doing on-prem using local OCR is the data cleaning and extraction portion. If that is done for each invoice format, a reliable solution can still be achieve without relying on 3rd party paid OCR services.

  • kensoh

    August 29, 2021 at 11:56 pm

    Would like to post on behalf of a user (since it is too last minute for him to be aware of this challenge here). Daniel, from Brazil, posted great solutions for all 4 challenges and in particular, his week 4 solution should be taking ~ 30 seconds, after moving the OCR part to pre-processing.

    More details – https://www.linkedin.com/feed/update/urn%3Ali%3Aactivity%3A6837140323660025857/

    His week 4 result (solution in the same repo) – https://github.com/DanielCCF/BotGamesAA/blob/master/Week4/Result-Standard.png

    • matinciel

      August 30, 2021 at 5:35 am

      Master piece of code !

      I try and run at 7 seconds (not on the same env. as mine).

      But I am not getting 100% accuracy always 85.71% (both turbo and not turbo).

      If someone can test and report here maybe (?)


    • TWS

      August 30, 2021 at 9:37 am

      Nice solution with good documentation. I’m able to get 100% accuracy with his solution.

  • kensoh

    August 30, 2021 at 12:58 pm

    Congrats to the winner for week #4! – Daniel Correa de Castro Freitas, from Brazil. I’ll be reaching out to send the prize (worth $500 USD, LearnAI premium subscription + Datacamp annual premium subscription).

    My solution and tips for week #4, plus solutions from Daniel and @TWS – https://github.com/kelaberetiv/TagUI/issues/1093#issuecomment-907996687

    Video demo of week #4 solution – https://www.linkedin.com/posts/kensoh_botgames-rpaugust-activity-6835498514760499200-DB1M

    I was the first to post on solving the toughest week 4 challenge, amongst all other users from RPA tools including Automation Anywhere, UiPath, and open-source RPA tools. This shows that free and open-source tools can also solve real world RPA scenarios as well, since these Automation Anywhere challenges are crafted from real customer scenarios.

    During this month of RPA challenges, many users tried out using a temporary turbo mode of TagUI which runs many times faster compared to normal human speed. With sufficient user feedback, am happy to introduce the turbo option permanently into TagUI, which lets you run automation 10X faster than normal human speed –


  • kensoh

    August 30, 2021 at 1:03 pm
Viewing 16 - 24 of 24 replies