awesome_bot

Verify links in awesome projects

awesome_bot checks for valid URLs in a file, it can be used to verify pull requests updating a README.

Installation

$ gem install awesome_bot

Usage

Command Line

Usage: awesome_bot [file or files]
       awesome_bot [options]
    -f, --files [files]              Comma separated files to check
    -a, --allow [errors]             Status code errors to allow
        --allow-dupe                 Duplicate URLs are allowed
        --allow-ssl                  SSL errors are allowed
        --allow-redirect             Redirected URLs are allowed
        --allow-timeout              URLs that time out are allowed
        --base-url [base url]        Base URL to use for relative links
    -d, --request-delay [seconds]    Set request delay
    -t, --set-timeout [seconds]      Set connection timeout (default: 30)
        --skip-save-results          Skip saving results
    -w, --white-list [urls]          Comma separated URLs to white list

• You can check multiple files (comma separated or * pattern, look below for details).

• By default, duplicate URLs or any status code other than 200 are flagged as failures.

  • Use option --allow-dupe to allow duplicates.
  • Use option --allow-redirect to allow redirects.
  • Use option --allow to allow specific status code errors.
  • Use option --white-list (-w for short) to prevent links from being flagged: -w domain1.com/post/article,domain2.com white lists domain1.com/post/article and all links matching domain2.com.

Examples

$ awesome_bot README.md
> Checking links in README.md
Links found: 56, 37 unique
  01. https://github.com/sindresorhus/awesome
  02. http://i.giphy.com/urvsFBDfR6N32.gif
  03. https://travis-ci.org/dkhamsing/awesome_bot.svg
# ...
  37. https://github.com/dkhamsing
  Checking URLs: ✓✓✓→?✓→✓→→✓✓→✓✓✓→✓✓✓✓✓✓✓✓✓✓✓→✓✓✓✓✓→✓✓

Issues :-(
> Links
  1. [L007] 301 https://travis-ci.org/dkhamsing/awesome_bot.svg → https://api.travis-ci.org/dkhamsing/awesome_bot.svg
  2. [L008] 302 https://badge.fury.io/rb/awesome_bot → http://rubygems.org/gems/awesome_bot
# ...
> Dupes
  1. [L03] https://github.com/sindresorhus/awesome
  2. [L05] http://i.giphy.com/urvsFBDfR6N32.gif
# ...

$ awesome_bot README.md --allow-dupe --allow-redirect -w rubydoc,giphy
# allow redirects, dupes and white list all links matching rubydoc and giphy

$ awesome_bot README.md,README-zh.md
# check links in 2 files

$ awesome_bot docs/*.md
# check all Markdown files in the docs/ directory

$ awesome_bot README.md --allow-timeout -t 5
# speed up validation by setting a timeout of 5 seconds per link request and allowing timeouts

$ awesome_bot README.md --allow 403,429
# allow status code errors 403 and 429
# --allow 301 would be similar to --allow-redirect

$ awesome_bot README.md --base-url https://github.com/IDR/idr-notebooks/blob/master/
# check relative links using the base URL provided

(master) $ git branch
* master
(master) $ git checkout -b new-branch
Switched to a new branch 'new-branch'
(new-branch) $ touch new-readme.md && echo 'https://github.com/dkhamsing' >> new-readme.md
(new-branch) $ git add new-readme.md
(new-branch) $ git commit -m 'Testing'
[new-branch ef47336] Testing
 1 file changed, 1 insertion(+)
 create mode 100644 new-readme.md
(new-branch) $ git diff master.. --name-only | grep '.md' | xargs awesome_bot
> Checking links in new-readme.md
Links to check: 1
  1. https://github.com/dkhamsing
Checking URLs: ✓
No issues :-)

Wrote results to ab-results-new-readme.md.json

Docker Examples

If you do not want to install Ruby or its dependencies you can simply use Docker and Docker image.

Here is an example for checking the links in the Markdown files in your current directory/subdirectories:

docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --white-list "test.com" --allow-dupe --allow-redirect --skip-save-results `find . -name "*.md"`

or just check the links in a single file located at ./templates/ubuntu.md:

docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --allow-dupe --allow-redirect --skip-save-results ./templates/ubuntu.md

You always need to specify the path to the file so you cannot simply use *.md; instead use ls *.md":

docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --white-list "test.com" --allow-dupe --allow-redirect --skip-save-results `ls *.md`

Library

irb(main):001:0> require 'awesome_bot'
=> true
irb(main):002:0> content = File.read 'README.md'
=> "..."
irb(main):003:0> result = AwesomeBot.check content
=> #<AwesomeBot::Result:0x007fdde39f4408 @links=...>
# AwesomeBot Result with success, statuses_issues, dupes and more
irb(main):004:0> puts result.success ? 'No errors' : ':-('
:-(

More information at rubydoc.

Validate Pull Requests

Does your GitHub README contain a lot of links? awesome_bot can help you validate them when a pull request is created (or a commit is pushed). It is used by:

• https://github.com/tiimgreen/github-cheat-sheet
• https://github.com/enaqx/awesome-react
• https://github.com/ziadoz/awesome-php
• https://github.com/vsouza/awesome-ios
• https://github.com/alebcay/awesome-shell
• https://github.com/matteocrippa/awesome-swift

and more.

Tips

• Use the keyword [ci skip] in your commit title/message to skip verification.
• Use Danger.

GitHub Actions

To use awesome_bot with GitHub Actions (workflows), here is an example:

name: Ruby

on:
  push:
    branches: [ '*' ]
  pull_request:
    branches: [ '*' ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby 2.6
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.x
    - name: Checks
      run: |
        ruby .github/osia_convert.rb
        gem install awesome_bot
        ruby .github/osia_get_links.rb
        awesome_bot check-unique.txt --allow-ssl -a 302,429 -w xbmc/xbmc

Travis CI

To use awesome_bot with Travis CI, connect your repo and create a .travis.yml file.

language: ruby
rvm: 2.4.1
before_script: gem install awesome_bot
script: awesome_bot README.md

To turn off email notifications, add the lines below

notifications:
  email: false

In case you want to use the docker image inside Travis CI follow this example which will check broken links in all *.md files in your repository:

sudo: required

services:
  - docker

script:
  # Link Checks
  - docker run -ti --rm -v $PWD:/mnt:ro dkhamsing/awesome_bot --allow-dupe --allow-redirect --skip-save-results `find . -name "*.md"`

More

CircleCI, Codeship, and Semaphore CI support running tests without adding a file to the repo (a public configuration file can however help others contribute).

# Codeship
Setup
rvm use 2.4.1 --install
gem install awesome_bot

Test
awesome_bot README.md

# Semaphore CI
Language: Ruby
Ruby version: 2.4.1
Databases for: don't generate
Setup:
gem install awesome_bot
awesome_bot README.md

Status Badge

To add the Travis CI build status badge above to your project, use the following code

[![Build Status](https://travis-ci.org/<username>/<project>.svg)](https://travis-ci.org/<username>/<project>)

i.e.
[![Build Status](https://travis-ci.org/dkhamsing/awesome_bot.svg?branch=master)](https://travis-ci.org/dkhamsing/awesome_bot)

As it happens, the default code snippet provided contains a redirect so adding a badge could fail your status :sob:.. one way to fix this is to white list travis-ci, i.e.

- awesome_bot README.md --white-list travis-ci

You can also add a badge for other CI tools, check out shields.io.

Danger

Integrate awesome_bot with Danger and have results reported back to the pull request.

Here's the step in your Dangerfile:

# Check links
require 'json'
results = File.read 'ab-results-README.md-markdown-table.json'
j = JSON.parse results
if j['error']==true
  fail j['title']
  markdown j['message']
end

Contact

• github.com/dkhamsing
• twitter.com/dkhamsing

License

This project is available under the MIT license. See the LICENSE file for more info.