Cover image of a blog post about automating Screaming Frog SEO crawls with command line and batch files

Automated Screaming Frog Reports With Command Line & Batch Files

Did you know that you can automate Screaming Frog crawl reports using Command Line and Windows batch files?

First, the credit goes to Tom Gregan and Open Source SEO for cutting the trail. 🙂

Starting with version 10, Screaming Frog SEO Spider has offered command line functionality– how cool is that? What’s more, the command line functionality is quite customizable, capable of both pulling in configuration files and exporting crawl data with flexibility.

Here’s some sample code for Windows Command Line you can run. Again, credit to Tom and Open Source SEO here. You can drop this into Notepad, Sublime Text, etc. and save as a batch file.

set crawlResults=C:\Users\you\Documents\client\Weekly-SFSEOS-Results
:: Creates variable %results% for where crawl will be saved

set sf=C:\Program Files (x86)\Screaming Frog SEO Spider\
:: Creates another variable telling Command Line where it can find SF

set configFile=C:\Users\you\Documents\client\Weekly-SFSEOS-Results\sample-config.seospiderconfig
:: Creates another variable telling CLI and SF where to find crawl configuration instructions - may be needed for more complex crawls or API settings

set domain=https://www.nike.com/
:: Sets a variable telling CLI and SF which domain or URL to crawl

chdir /d "%sf%"
:: Directs CLI to the directory you specified earlier

ScreamingFrogSEOSpiderCli.exe --config "%configFile%" --crawl "%domain%"  --save-crawl --headless --output-folder "%crawlResults%" --export-format "xlsx" --export-tabs "Internal:All" --timestamped-output
:: Runs the SF CLI exe file, which performs the crawl with the variables specified earlier 

The fun part (as if this wasn’t cool enough!) is that you can make a ton of these batch files, slave them from a separate “master” batch file, and schedule them to run on a recurring basis.

start cmd /k Call website-a.bat
start cmd /k Call website-b.bat

Important to note- the above code sample assumes that you place the master and the “child” Screaming Frog crawl batch files in the same directory on your computer or server.

How nice is that? From here, you could conceivably have lots of options to pull into a business intelligence tool like Tableau, Power BI, Data Studio or pull into data frames in R or Python.

This could mean the end of blind spots for your SEO efforts! Think about monthly, weekly, or even daily crawls of your sites, competitors, publishers, etc. Happy coding!

Photo taken by Texas SEO consultant at Clearwater Falls in Oregon

What Marketing Could Be, Part 1

The rambling preamble

For those imaginary regular readers out there, I find myself again starting a post with, “…I know it’s been awhile since I last posted…” Yet it’s truly been, say, 2 years since my last post?

Anyway, this year marked my 7th year in industry after college. I’m looking to reflect and document some thoughts for future reference, out in the public domain for accountability to myself.

I wish I could say I’ve seen it all in marketing and advertising. But I can only say that I’ve seen a few good and bad things out there in my few years of experience. But put plainly, I now feel like the marketing industry can do better. In every way.

That last sentence conveys a wide range of conversation, but this post more pertains to the operations and setup of agency life as we know it today.

Why is marketing so hard?

For the initiated, the marketing and advertising industry may often feel awash in chaos, churn and turmoil. If you don’t feel that way, I’d love to talk to you. But assuming that chaos reigns, a first glance could say it starts at the top:

Ouch, churn seems to flow down the chain, no?

Do clients (brands) cause marketing churn?

A frustration I’ve long held, but rarely shared, is that many brands and clients have large knowledge gaps. Let me explain:

eMarketer tells us that today, digital takes 50% of marketing spend worldwide. By 2023, we expect that almost 2 out of every 3 marketing dollars will be digital.

Why is this significant?

Digital marketing is predominantly technical marketing, rendering many traditional creative disciplines as the price of admission.

Technical and analytical chops now often set the bar for excellence. So much so that millions of dollars and business returns depend on it. Or risk failure, in some instances, such as Accenture’s shortcomings with Hertz, to the tune of an ugly lawsuit.

To put it more plainly, many clients and brands lack the expertise to execute technical marketing. I would further assert that many clients and brands lack the expertise to manage technical marketing.

But what does that mean? That means you have digital marketing and SEO managers at S&P 500 retailers that don’t understand the basics of optimizing a web page or how search engines crawl sites. That means the head of media for a top 50 Internet Retailer doesn’t know what CPC stands for.

The technical marketing knowledge gaps at brands often prohibit them from achieving marketing performance, much less growth.

The salient question: do clients cause marketing churn?

I would offer a qualified yes, because this post started on, and should end with, a focus on marketing agencies.

Are agencies helping marketing churn?

It’s a loaded question. No, most agencies aren’t helping themselves. Many agencies do have a basic burden to educate their clients. Yet a minimum effective dose of education only extends to ensure the safety of a retainer.

Here’s the fun part:

The technical marketing knowledge gaps at brands often prohibit them from achieving marketing performance, much less growth.

By nature, and the bias of some (almost) funny examples, this could lead us to say that education is the big problem in this equation. Here’s where qualification and nuance mean a lot.

Technical marketing knowledge gaps are easy to pin on uneducated clients. But what about unequipped clients? Therein lies the importance of nuanced views. Do clients need to be proficient tacticians in technical marketing to succeed? Honestly, no.

Do clients need to be competent managers of a technical team to succeed? I would say yes. From this perspective, agencies can thrive when they equip their clients to make smart decisions.

The demise of agency engagements

Stop me if you’ve heard this one before.

  • Client engages agency
  • Agency sends recommendations
  • Client doesn’t implement
  • Nothing gets done
  • Results don’t change
  • Agency gets cut
  • Client engages another agency
  • Agency sends recommendations…

And so on.

What goes on in that ridiculous cycle of a bulleted list? Agency tells client, “You need to fix these render blocking resources to fix your page speed.”

Client asks, “Okay, can you tell me why this is important so I can get it at the top of our dev team’s JIRA queue?”

Agency, “…because a Lighthouse audit told us so!”

And this conversation happens for everything, from hreflang sitemaps to migrating to HTTPS, to putting body copy on product category pages, to….the list goes on.

Most agencies don’t have compelling data or evidence to properly answer client questions – rather, they don’t equip the clients!

Why aren’t agencies better equipped?

When you look at marketing agencies, you can generally place players into one of several buckets:

  • Freelancers or small consultancies
  • Mid-market challengers
  • Big-box holding company conglomerates

Before you @ me, yes, there are exceptions to the above. I could think of 3 agencies offhand that break this mold. But if we look at the aggregate landscape, we could safely say that 80%+ of the market fits these buckets.

Across these buckets, there are a few ironically shared challenges that keep these types of players from delivering outstanding marketing.

  • Lack of strategic acumen
  • Lack of technical expertise
  • Lack of time/staffing

There are no easy answers to lacking strategic acumen. If you’re working with any party lacking this- consider changing/improving your situation as needed. (Ex: your PPC partner doesn’t know the right questions to ask about PPC.)

However, technical expertise and lack of time/staffing may be offset to some degrees- though automation.

Building Smarter Marketing

We (marketers) are un- or under-equipped because of these challenges, which render us unable to provide necessary answers to routine questions as mentioned earlier.

Automation (in varying forms) can be a powerful tool to equip both agencies and brands for smarter marketing.

Think about an SEO engagement. What if:

  • Server logs were routinely and automatically retrieved, stored and analyzed?
  • Market data such as Google Trends were routinely captured on brands, competitors, and industries of interest?
  • Google Search Console queries, crawling activity, links, etc. were captured daily or weekly?
  • Automated crawls of brand and competitor sites took place, for action/alert against new or broken content?
  • Analytics and 3rd party rankings and audit data were routinely, automatically pulled

What if all this data could be stored together, routinely and automatically retrieved, stored, analyzed and actioned upon?

You would certainly be able to answer many questions effectively. Better still, you could learn many more of which to ask.

It’s my hope to answer those questions above and more in the coming days.