There's a chrome extension that records all your steps and then generates a puppeteer/playwright script for you. Then you can just edit bits of that script and run it. This is my go-to method presently.
I feel like you could incorporate some of those features into this and eliminate some of the manual entry of css selectors.
UI Vision is unique because it supports visual automation with image recognition and OCR. A bit like Sikuli but inside the browser. This avoids dealing with CSS selectors and XPaths.
We are using Automa chrome extension for this kind of use cases. We really like it. It is free and open source. If you are thinking of building a product, you can make use of it.
You can do it in the chrome browser tools! There's a record section. Note if you're doing this for automation testing it's going to be pretty flakey and you're better off following best practices and using custom testing selectors instead :)
Why is it flakey for automation testing though? In my experience that's basically targeting page elements and interacting with them. Doesn't this do the same thing?
It uses a combination of xpaths that are like body[1]div[2]div[1]p[2]button[1] so if you edit the UI it'll break, or worst case they do X,Y coordinates of clicks on a page so if the UI changes it'll break. Sometimes it'll get your ID or Name or whatever but some frameworks randomize these on compilation and some devs change them as part of refactoring.
Best practice is to add test attributes to the HTML elements (such as data-testid="SubmitButton") so if the page is moved around or the button is edited the tests remain stable. There's a lot of good online sources on making UI testing somewhat more reliable :)
That’s good to know, I had bad luck with it then. But even so, you also lost the ability to use testing patterns like the page object model where you can update changing IDs or flows more easily - if something changes drastically early in the recording, wouldn’t it need to be re-recorded for all test cases?
Selenium is more widespread but I recommend Playwright as the most modern automation library. Comes with plugins for making your automations more... nimble.
Selenium is the most popular tool in the industry but mostly for legacy reasons, its real competitor is Cypress. If you've just started a project I'd recommend checking it out, a huge time saver is that you install it as an npm package with a headless browser out the box without any webdriver/infra stuff slowing you down.
Really interesting tool. I've always found the task of finding the right CSS selectors tiresome, so I created tool which automatically extracts the right selectors for you (and builds a scraper as well). Open Source on GitHub: https://github.com/lorey/mlscraper
I'll check this repo, thx. It all started as an experiment with Playwright. And I also wanted to try whether I can build a full app, not just some simple scripts.
I remember a Firefox extension that notified me with the diffs of changed pages. I don't remember exactly what I was using it for. I think something related to CVEs for some keywords. Then the service I was using shut down and I lost tracks of that extension.
It could be this one. I remember the yellow highlights.
It was many years ago and definitely not a cloud service so not Distill Web Monitor. It didn't let me define fields to monitor, so not PageProbe.
I wouldn't have used a cloud service. What's the point? My browser already have everything it takes to download pages and diff them. I understand that if the computer is off the browser won't monitor the page but it was ok with my use case: if it's off I won't be able to react to the CVE.
And for other use cases, it's no business of anyone else but me to know what I'm monitoring.
I remember using the addon too. The reason for the addon had words to the effect that it enabled you to keep track of sites that did not have an RSS feed which was the more standard way getting new web content.
I'd love to use this as a webpage interaction/scraping component in chains of webhooks (building on e.g. low-code step pipelines built with Pipedream, could imagine fitting with something like Zapier, too).
Would require the ability to:
(1) make a web request to start a run
(2) collect the scraped info and return it as part of (a) a webhook call or (b) in the response of a synchronous "start run" blocked long http request
It sounds like #1 is already possible. 2a might be possible if there was a call webhook step or notification option? But 2b would be the simplest (if I could make a single GET or POST request, and the response gives me some outputs from an automation session).
Zapier/Pipedream integrations could be nice platform integration marketing for the tool as well.
Two things that have tripped me up
1. "and save the value of this attribute", it wasn't clear that this could only be certain values, text, id.
2. API I can list tasks, get the status of a task, start tasks via API, but I don't se how I can read the results / stored data
Curious what your pricing strategy is here? The free offering is cool for sure. Guess you'll learn how serious users may want to actually use the service, what that means for your unit costs, and how it all scales in your favor? or sort of already figured that out?
more about how the author came up with the pricing and if/how they developed projections around it. There also wasnt a custom offering, so was wondering why not and if the author felt it wouldnt be financially effective.
Alternatively, have a batch script that just runs (Assuming windows) taskkill /f /im firefox.exe which would kill it. Then have firefox startup with no tabs loaded.
I feel like you could incorporate some of those features into this and eliminate some of the manual entry of css selectors.
iMacros? https://chrome.google.com/webstore/detail/imacros-for-chrome...
UI Vision is unique because it supports visual automation with image recognition and OCR. A bit like Sikuli but inside the browser. This avoids dealing with CSS selectors and XPaths.
https://www.automa.site/ https://github.com/AutomaApp/automa
Best practice is to add test attributes to the HTML elements (such as data-testid="SubmitButton") so if the page is moved around or the button is edited the tests remain stable. There's a lot of good online sources on making UI testing somewhat more reliable :)
I remember a Firefox extension that notified me with the diffs of changed pages. I don't remember exactly what I was using it for. I think something related to CVEs for some keywords. Then the service I was using shut down and I lost tracks of that extension.
https://addons.mozilla.org/en-US/firefox/addon/distill-web-m...
https://addons.mozilla.org/en-US/firefox/addon/pageprobe/
It was many years ago and definitely not a cloud service so not Distill Web Monitor. It didn't let me define fields to monitor, so not PageProbe.
I wouldn't have used a cloud service. What's the point? My browser already have everything it takes to download pages and diff them. I understand that if the computer is off the browser won't monitor the page but it was ok with my use case: if it's off I won't be able to react to the CVE.
And for other use cases, it's no business of anyone else but me to know what I'm monitoring.
Would require the ability to:
(1) make a web request to start a run
(2) collect the scraped info and return it as part of (a) a webhook call or (b) in the response of a synchronous "start run" blocked long http request
It sounds like #1 is already possible. 2a might be possible if there was a call webhook step or notification option? But 2b would be the simplest (if I could make a single GET or POST request, and the response gives me some outputs from an automation session).
Zapier/Pipedream integrations could be nice platform integration marketing for the tool as well.
Also the logo in the header looks low definition. Upping its resolution will make the website way better.
I've invested so much time in similar things just to not be able to afford it...
Two things that have tripped me up 1. "and save the value of this attribute", it wasn't clear that this could only be certain values, text, id. 2. API I can list tasks, get the status of a task, start tasks via API, but I don't se how I can read the results / stored data
I’m not seeing any videos on the homepage (I’m using safari on iOS).
I use Firefox. Anyone know of a way I can have it close all tabs once a day? Either via an external script or an extension?
Alternatively, have a batch script that just runs (Assuming windows) taskkill /f /im firefox.exe which would kill it. Then have firefox startup with no tabs loaded.