What you can suggest me to improve in this code and where should i use thisHardeepex/scrapegost#13
![Logo of Sweep](/_next/image?url=%2Flogo.png&w=64&q=75)
What you can suggest me to improve in this code and where should i use this
Hardeepex/scrapegost#13
> > >
✓ Completed in 8 minutes, 7 months ago using GPT-4  •  Book a call  •  Report a bug
Progress
  ModifyChanged
docs/tutorial.md:15-234Â
Changed docs/tutorial.md
in d07c78f
   Â
83 | 83 | ||
84 | All for less than a penny! | 84 | All for less than a penny! |
85 | 85 | ||
86 | ### Best Practices for Efficient Scraping | ||
87 | |||
88 | When constructing a web scraper, it is essential to follow best practices to ensure efficiency and robustness: | ||
89 | |||
90 | - **Use Specific Selectors**: Choose selectors that target the specific content you need. This minimizes the amount of unnecessary HTML sent to the scraper and reduces the likelihood of it breaking when the page structure changes. | ||
91 | |||
92 | - **Preprocess HTML**: Use preprocessors like `CSS` to extract relevant content before passing it to the scraper. Not only does this lower token count, but it also ensures that irrelevant content, such as comments and script tags, do not interfere. | ||
93 | |||
94 | - **Error Handling**: Implement robust error handling to manage exceptions, such as `TooManyTokens`. When you encounter this error, consider using CSS selectors to reduce the HTML content size or splitting the content into smaller chunks that are within token limits for individual scraping operations. | ||
95 | |||
96 | #### Example: Efficient Preprocessing and Error Handling | ||
97 | |||
98 | In the case of the `TooManyTokens` error with our episode details scraper, we can add the `CSS` preprocessor step to avoid the error and streamline the scraping process. | ||
99 | |||
100 | ```python hl_lines="13 14" | ||
101 | # Incorporate CSS preprocessor to focus on relevant content | ||
102 | CSS("div.page-content").preprocess | ||
103 | |||
104 | # Implementation of error handling | ||
105 | try: | ||
106 | # Call to the scraper | ||
107 | data = episode_scraper(scrape_url).data | ||
108 | except scrapeghost.scrapers.TooManyTokens as e: | ||
109 | # Handling the exception | ||
110 | print("Encountered error: ", e) | ||
111 | # Implement a strategy to reduce tokens, like preprocessing or splitting | ||
112 | |||
113 | ``` | ||
114 | |||
115 | By applying these techniques, we adhere to best practices for efficient and reliable scraping. | ||
116 | |||
86 | !!! tip | 117 | !!! tip |
87 | 118 | ||
88 | Even when the page fits under the token limit, it is still a good idea to pass a selector to limit the amount of content that OpenAI has to process. | 119 | Even when the page fits under the token limit, it is still a good idea to pass a selector to limit the amount of content that OpenAI has to process. |
... | |||
202 | 233 | ||
203 | As a safeguard, the maximum cost for a single scrape is configured to $1 by default. If you want to change this, you can set the `max_cost` parameter. | 234 | As a safeguard, the maximum cost for a single scrape is configured to $1 by default. If you want to change this, you can set the `max_cost` parameter. |
204 | 235 | ||
205 | One option is to lower the `auto_split_length` a bit further. Many more requests means it takes even longer, but if you can stick to GPT-3.5-Turbo it was possible to get a scrape to complete for $0.13. | 236 | One option is to lower the `auto_split_length` a bit further. This can help manage large pages and ensure each chunk stays within the token limits. |
206 | 237 | ||
238 | ### Use Cases for the Scraping Tool | ||
239 | |||
240 | `scrapeghost` offers versatility for various scraping scenarios: | ||
241 | |||
242 | - **Scraping Unstructured Text**: When dealing with unstructured data on web pages, the tool can help standardize and extract valuable information. | ||
243 | |||
244 | - **Frequent Page Changes**: Pages that change regularly are challenging to scrape with static selectors. `scrapeghost`'s ability to understand context can be particularly useful here. | ||
245 | |||
246 | - **Quick Prototyping**: When you need to create a proof-of-concept quickly, `scrapeghost` can scrape and structure data without the need for developing complex scraping logic specific to each site. | ||
247 | |||
248 | Incorporating `scrapeghost` into these use cases can significantly streamline the data extraction process. Many more requests means it takes even longer, but if you can stick to GPT-3.5-Turbo it was possible to get a scrape to complete for $0.13. | ||
249 | |||
207 | But as promised, this is something that `scrapeghost` isn't currently very good at. | 250 | But as promised, this is something that `scrapeghost` isn't currently very good at. |
208 | 251 | ||
209 | If you do want to see the pieces put together, jump down to the [Putting it all Together](#putting-it-all-together) section. | 252 | If you do want to see the pieces put together, jump down to the [Putting it all Together](#putting-it-all-together) section. |
- Add a section on best practices for writing efficient and robust scraping code. This could include advice on choosing appropriate selectors, preprocessing HTML to reduce token count, and handling potential errors or exceptions.
- Include examples of these best practices in the tutorial code. For instance, demonstrate how to use the
CSS
preprocessor to extract relevant content, or how to handle theTooManyTokens
error. - Add a section on potential use cases for the scraping tool. This could include examples of scraping unstructured text, dealing with pages that change frequently, or quick prototyping.
Modified file with Assistant API
  Run GitHub Actions for
docs/tutorial.mdÂ
Ran GitHub Actions for d07c78f25e10a729c4736e4ef573801a282dea42:
  ModifyChanged
docs/faq.md:14-86Â
Changed docs/faq.md
in 08fcdaa
   Â
12 | 12 | ||
13 | ## Why would I use this instead of a traditional scraper? | 13 | ## Why would I use this instead of a traditional scraper? |
14 | 14 | ||
15 | It is definitely great for quick prototypes. With the CLI tool, you can try a scrape in a *single command* without writing a line of code. | 15 | It is definitely great for quick prototypes and ad-hoc data extraction. The CLI tool allows you to initiate a scraping session with a *single command* without writing any code, making it ideal for rapid testing and development. |
16 | |||
17 | Advantages over traditional scrapers are several-fold: | ||
18 | |||
19 | - **Flexibility in Unstructured Data Handling**: Traditional scrapers rely on fixed patterns or selectors, which may fail when a website changes. `scrapeghost`'s model-based approach is adept at interpreting unstructured data and adapts more gracefully to changes in page structure. | ||
20 | |||
21 | - **Ease of Use for Non-Developers**: The ability to use natural language instructions makes `scrapeghost` more accessible to those without extensive programming or web scraping experience. | ||
22 | |||
23 | - **Speed of Deployment**: Setting up `scrapeghost` is faster compared to writing a full-fledged scraper, saving valuable time especially when dealing with simple or one-off scraping tasks. | ||
24 | |||
25 | However, there are also challenges and limitations to consider: | ||
26 | |||
27 | - **Costs of API Usage**: While it can be efficient in terms of development time, costs can accumulate with extensive use of the API, especially for larger or more complex scraping tasks. | ||
28 | |||
29 | - **Opaque Errors**: Troubleshooting is made harder by less transparent error messages, which could hinder understanding of why certain extractions fail. | ||
30 | |||
31 | - **Dependence on Upstream Provider**: The reliance on OpenAI's models means any changes in their API, pricing, or availability can directly impact your scraping capabilities. | ||
16 | This means you don't need to sink a bunch of time into deciding if it's worth it or not. | 32 | This means you don't need to sink a bunch of time into deciding if it's worth it or not. |
17 | 33 | ||
18 | Or, imagine a scraper that needs to run infrequently on a page that is likely to break in subtle ways between scrapes. | 34 | Or, imagine a scraper that needs to run infrequently on a page that is likely to break in subtle ways between scrapes. |
... | |||
76 | 92 | ||
77 | ## How much did you spend developing this? | 93 | ## How much did you spend developing this? |
78 | 94 | ||
79 | So far, about $40 on API calls, switching to GPT-3.5 as the default made a big difference. | 95 | So far, the expenditure on API calls is approximately $40, which reflects careful management of the tool's functionality to minimize costs. |
80 | 96 | ||
81 | My most expensive call was a paginated GPT-4 call that cost $2.20. I decided to add the cost-limiting features after that. | 97 | Cost-Control Strategies: |
82 | 98 | ||
99 | - **Max Cost Limiting**: It's possible to set a maximum cost at which the scraping tool will stop processing, ensuring that you never exceed your budget. For instance, a GPT-4 call that would normally cost $2.20 can be limited to a lower threshold. | ||
100 | |||
101 | - **Selective Scraping**: Preprocess the HTML to target only the essential content or use split strategies to efficiently distribute API calls across sections of a page. | ||
102 | |||
103 | - **Efficiency Adjustments**: Switching to models like GPT-3.5 can significantly reduce costs, providing a balance between performance and affordability. Optimization of prompts and careful configuration of tool parameters can also help keep the costs in check. | ||
104 | |||
105 | - **Cost Tracking**: Keeping a close eye on the expenditure and adjusting your scraping strategy allows for incremental improvement in both cost-efficiency and the quality of results. | ||
106 | |||
107 | These examples illustrate how integral cost management is to the sustainable use of scraping tools and highlight the importance of understanding and utilizing cost-limiting features. | ||
108 | |||
83 | ## What's with the license? | 109 | ## What's with the license? |
84 | 110 | ||
85 | I'm still working on figuring this out. | 111 | I'm still working on figuring this out. |
- Expand on the advantages and disadvantages of using the scraping tool. This could include more detailed explanations of when and why the tool might be preferable to traditional scraping methods, as well as potential limitations or challenges.
- Include more detailed information on the cost of using the tool, as this is likely to be a key consideration for users. This could include examples of how to use the cost-limiting features.
Modified file with Assistant API
  Run GitHub Actions for
docs/faq.mdÂ
Ran GitHub Actions for 08fcdaa4faf5a846980253112297d6c2ea1235d3:
  ModifyChanged
docs/contributing.md:12-19Â
Changed docs/contributing.md
in 2b1b210
   Â
8 | 2. Clone your forked repository to your local machine. | 8 | 2. Clone your forked repository to your local machine. |
9 | 3. Install the project's dependencies. You can do this by running `pip install -r requirements.txt`. | 9 | 3. Install the project's dependencies. You can do this by running `pip install -r requirements.txt`. |
10 | 10 | ||
11 | ## Running Tests | 11 | ## Testing Your Changes |
12 | 12 | ||
13 | To ensure that your changes do not break existing functionality, please run the project's test suite before submitting a pull request. You can run the tests by executing `pytest` in the project's root directory. | 13 | It is crucial to test your changes to ensure they do not negatively impact existing functionality. Follow these steps to test your code: |
14 | 14 | ||
15 | 1. Write new test cases that cover the new features or improvements you are making. | ||
16 | 2. Run the existing project's test suite by executing `pytest` in the project's root directory. Ensure existing tests pass along with your new test cases. | ||
17 | 3. If tests fail, review your code, make the necessary corrections, and repeat the testing process. | ||
18 | |||
19 | By thoroughly testing, you help maintain the robustness of Scrapegost. | ||
20 | |||
15 | ## Submitting a Pull Request | 21 | ## Submitting a Pull Request |
16 | 22 | ||
17 | 1. Create a new branch for your changes. The branch name should be descriptive of the changes you are making. | 23 | 1. Create a new branch for your changes. The branch name should be descriptive of the changes you are making. |
- Add a section on how to contribute improvements to the scraping code. This could include guidelines for submitting pull requests, as well as advice on testing changes to ensure they do not break existing functionality.
Modified file with Assistant API
  Run GitHub Actions for
docs/contributing.mdÂ
Ran GitHub Actions for 2b1b210eb7eb0a1b6463b53ab79f63054cc1e5ab:
Plan
This is based on the results of the Planning step. The plan may expand from failed GitHub Actions runs.
  Modify
docs/tutorial.md:15-234Â
  Run GitHub Actions for
docs/tutorial.mdÂ
  Modify
docs/faq.md:14-86Â
  Run GitHub Actions for
docs/faq.mdÂ
  Modify
docs/contributing.md:12-19Â
  Run GitHub Actions for
docs/contributing.mdÂ
Code Snippets Found
This is based on the results of the Searching step.
docs/faq.md:14-86
 14
15It is definitely great for quick prototypes. With the CLI tool, you can try a scrape in a *single command* without writing a line of code.
16This means you don't need to sink a bunch of time into deciding if it's worth it or not.
17
18Or, imagine a scraper that needs to run infrequently on a page that is likely to break in subtle ways between scrapes.
19A CSS/XPath-based scraper will often be broken in small ways between the first run and another run months later, there's a decent chance that those changes won't break a GPT-based scraper.
20
21It is also quite good at dealing with unstructured text. A list of items in a sentence can be hard to handle with a traditional scraper, but GPT handles many of these cases without much fuss.
22
23## What are the disadvantages?
24
25* It is terrible at pages that are large lists (like a directory), they need to be broken into multiple chunks and the API calls can be expensive in terms of time and money.
26* It is opaque. When it fails, it can be hard to tell why.
27* If the page is dynamic, this approach won't work at all. It requires all of the content to be available in the HTML.
28* It is *slow*. A single request can take over a minute if OpenAI is slow to respond.
29* Right now, it only works with OpenAI, that means you'll be dependent on their pricing and availability. It also means
30you need to be comfortable sending your data to a third party.
31
32
33## Why not use a different model?
34
35See <https://github.com/jamesturk/scrapeghost/issues/18>.
36
37## Can I use `httpx`? Or `selenium`/`playwright`? Can I customize the headers, etc.?
38
39This library is focused on handling the HTML that's already been retrieved. There's no reason you can't use any of these libraries to retrieve the HTML. The `scrape` method accepts either a URL or a string of already fetched HTML.
40
41If you'd like to use another library, do it as you usually would, but instead of passing the HTML to `lxml.html` or `BeautifulSoup`, pass it to `scrapeghost`.
42
43## What can I do if a page is too big?
44
45Dealing with large pages requires a strategy that includes scoping and preprocessing. Here are some steps and examples to help you effectively handle large pages:
46
471. Use CSS or XPath selectors to narrow the focus of the page to significant areas. For example:
48- CSS: Use `.main-content` to target the main content area.
49- XPath: Use `//div[@class='product-list']/div` to select only the product list items.
50
512. Pre-process the HTML by removing unnecessary sections, tags, or irrelevant data to streamline the scraping process. This could involve:
52- Stripping out `<script>` and `<style>` tags.
53- Removing comments or non-essential metadata.
54- Simplifying the DOM structure by eliminating redundant wrappers.
55Utilize the library's preprocessing features to automate such tasks wherever possible.
56
573. Finally, you can use the `auto_split_length` parameter to split the page into smaller chunks. This only works for list-type pages, and requires a good choice of selector to split the page up.
58
59## Why not ask the scraper to write CSS / XPath selectors?
60
61While it'd seem like this would perform better, there are a few practical challenges standing in the way right now.
62
63* Writing a robust CSS/XPath selector that'd run against a whole set of pages would require passing a lot of context to the model. The token limit is already the major limitation.
64* The current solution does not require any changes when a page changes. A selector-based model would require retraining every time a page changes as well as a means to detect such changes.
65* For some data, selectors alone are not enough. The current model can easily extract all of the addresses from a page and break them into city/state/etc. A selector-based model would not be able to do this.
66
67I do think there is room for hybrid approaches, and I plan to continue to explore them.
68
69## Does the model "hallucinate" data?
70
71It is possible, but in practice hasn't been observed as a major problem yet.
72
73Because the [*temperature*](https://platform.openai.com/docs/api-reference/completions) is zero, the output is fully deterministic and seems less likely to hallucinate data.
74
75The `HallucinationChecker` class can be used to detect data that appears in the response that doesn't appear on the page. This approach could be improved, but I haven't seen hallucination as a major problem yet. (If you have examples, please open an issue!)
76
77## How much did you spend developing this?
78
79So far, about $40 on API calls, switching to GPT-3.5 as the default made a big difference.
80
81My most expensive call was a paginated GPT-4 call that cost $2.20. I decided to add the cost-limiting features after that.
82
83## What's with the license?
84
85I'm still working on figuring this out.
86
docs/tutorial.md:12-77
 12
13## Writing a Scraper
14
15The goal of our scraper is going to be to get a list of all of the episodes of the podcast [Comedy Bang Bang](https://comedybangbang.fandom.com/wiki/Comedy_Bang_Bang_Wiki).
16
17To do this, we'll need two kinds of scrapers: one to get a list of all of the episodes, and one to get the details of each episode.
18
19### Getting Episode Details
20
21At the time of writing, the most recent episode of Comedy Bang Bang is Episode 800, Operation Golden Orb.
22
23The URL for this episode is <https://comedybangbang.fandom.com/wiki/Operation_Golden_Orb>.
24
25Let's say we want to build a scraper that finds out each episode's title, episode number, and release date.
26
27We can do this by creating a `SchemaScraper` object and passing it a schema.
28
29```python
30--8<-- "src/docs/examples/tutorial/episode_scraper_1.py"
31```
32
33There is no predefined way to define a schema, but a dictionary resembling the data you want to scrape where the keys are the names of the fields you want to scrape and the values are the types of the fields is a good place to start.
34
35Once you have an instance of `SchemaScraper` you can use it to scrape a specific page by passing it a URL (or HTML if you prefer/need to fetch the data another way).
36
37Running our code gives an error though:
38
39```
40scrapeghost.scrapers.TooManyTokens: HTML is 9710 tokens, max for gpt-3.5-turbo is 4096
41```
42
43This means that the content length is too long, we'll need to reduce our token count in order to make this work.
44
45### What Are Tokens?
46
47If you haven't used OpenAI's APIs before, you may not be aware of the token limits. Every request has a limit on the number of tokens it can use. For GPT-4 this is 8,192 tokens. For GPT-3.5-Turbo it is 4,096. (A token is about three characters.)
48
49You are also billed per token, so even if you're under the limit, fewer tokens means cheaper API calls.
50
51--8<-- "docs/snippets/_cost.md"
52
53Ideally, we'd only pass the relevant parts of the page to OpenAI. It shouldn't need anything outside of the HTML `<body>`, anything in comments, script tags, etc.
54
55(For more details on how this library interacts with OpenAI's API, see the [OpenAI API](openai.md) page.)
56
57### Preprocessors
58
59To help with all this, `scrapeghost` provides a way to preprocess the HTML before it is sent to OpenAI. This is done by passing a list of preprocessor callables to the `SchemaScraper` constructor.
60
61!!! info
62
63 A `CleanHTML` preprocessor is included by default. This removes HTML comments, script tags, and style tags.
64
65
66If you visit the page <https://comedybangbang.fandom.com/wiki/Operation_Golden_Orb> viewing the source will reveal that all of the interesting content is in an element `<div id="content" class="page-content">`.
67
68Just as we might if we were writing a real scraper, we'll write a CSS selector to grab this element, `div.page-content` will do.
69The `CSS` preprocessor will use this selector to extract the content of the element.
70
71
72```python hl_lines="1 13 14"
73--8<-- "src/docs/examples/tutorial/episode_scraper_2.py"
74```
75
76Now, a call to our scraper will only pass the content of the `<div>` to OpenAI. We get the following output:
77
docs/tutorial.md:210-234
 210
211## Using `src/main.py` Script
212
213The `src/main.py` script is a new addition to the suite of tools provided. This script utilizes `selectolax` for the initial HTML parsing to efficiently extract relevant content from a webpage. After the initial parse, the content is passed to `scrapeghost` for further processing and filtering. Here is how you might utilize it:
214
2151. Execute the provided Python script `src/main.py`.
2162. The script takes HTML content and uses `selectolax` to parse the main data.
2173. Once the main data is extracted, it is handed off to `scrapeghost` which filters and processes it according to predefined schemas.
218
219You may consider wrapping this process in a function or integrate it into a larger automation workflow depending on your use case.
220
221## Next Steps
222
223If you're planning to use this library, please be aware that while core functionalities like the main scraping mechanisms are stable, certain auxiliary features and interfaces are subject to change. We are continuously working to improve the API based on user feedback and technological advances.
224
225To facilitate smooth transitions, all significant changes will be communicated in advance through our release notes, changelog, and direct notifications if necessary. We encourage you to keep an eye on the repository's 'Releases' section on GitHub, subscribe to our mailing list, or join our community forum to stay updated on the latest developments.
226
227Please rely on the documented interfaces for stable use, and treat undocument features as experimental and subject to change.
228
229If you are going to try to scrape using GPT, it'd probably be good to read the [OpenAI API](openai.md) page to understand a little more about how the underlying API works.
230
231To see what other features are currently available, check out the [Usage](usage.md) guide.
232
233You can also explore the [command line interface](cli.md) to see how you can use this library without writing any Python.
234
docs/contributing.md:12-19
 12
13To ensure that your changes do not break existing functionality, please run the project's test suite before submitting a pull request. You can run the tests by executing `pytest` in the project's root directory.
14
15## Submitting a Pull Request
16
171. Create a new branch for your changes. The branch name should be descriptive of the changes you are making.
182. Make your changes in this branch.
193. Push your changes to your forked repository.