The eternal struggle of debugging a Scrapy issue that works locally but not in production!

After analyzing the error messages, I think I've identified the problem:

The FeedExporter is not designed to work with Scrapyd

In your local development environment, you're running Scrapy commands directly, which allows the FeedExporter to function as intended. However, when you deploy your spider using Scrapyd, the FeedExporter is not properly initialized or configured.

Here are some possible reasons why:

1. Missing configuration: FeedExporter relies on certain settings and configurations that might not be available in a Scrapyd environment.
2. Different signal handling: Scrapyd handles signals differently than Scrapy itself. This could cause issues with the open_spider, item_scraped, and close_spider signals.

To overcome this issue, you have a few options:

1. Use a different export method: Instead of using FeedExporter, consider using other export methods like CSV, JSON, or XML. These are more robust and might work better with Scrapyd.
2. Customize your spider code: Try to modify your spider code to adapt to the differences between Scrapy and Scrapyd. You might need to adjust signal handling or configuration settings.
3. Upgrade Scrapyd: Make sure you're using a recent version of Scrapyd, as older versions might have issues with newer Scrapy versions.

If none of these solutions work, please provide more details about your spider code and the specific export method you're trying to use (e.g., FeedExporter configuration). I'll do my best to help you troubleshoot!